Skip if the association was already loaded.

This tries to fix #15212 but needs a test.

With this patch I was unable to reproduce the bug.

cc @tenderlove