do not load all child records for inverse case

PR https://github.com/rails/rails/pull/10566 was reverted because tests were failing.

After this commit https://github.com/rails/rails/commit/6741a0a18761ca9cbbdaeefe70f4d53d68fdf1b5 all the tests are passing.

/cc @jonleighton