Fixed #23001 -- Fixed mixing defer and annotations

I've tested the above on sqlite and postgres, and all tests are passing. I'm not sure that removing the `fill_cache` check in the if block is the right way to go about things though. Perhaps someone with more knowledge of the ORM can take a look and validate the approach.