Fix regression on eager loading association based on SQL query rather
than existing column.

Fixes #15480.