do not load all child records for inverse case

currently `post.comments.find(Comment.first.id)` would load all
comments for the given post to set the inverse association.

This has a huge performance penalty. Because if post has 100k
records and all these 100k records would be loaded in memory
even though the comment id was supplied.

Fix is to use in-memory records only if loaded? is true. Otherwise
load the records using full sql.

Fixes #10509
Merge pull request #11051 from neerajdotname/10509f

do not load all child records for inverse case