[FIX] filtering out of records which shouldn't be fetched/prefetched

A todo would only filter out records selectioned by the same field's caching,
it should filter out on the whole prefetching selection or an other field
could/would just add it back to the set of records to fetch (and lead to Bad
Things).

Note: this probably deserves a test somehow, but I'm not quite sure how the
todos thing works so...