Using only the base table was wrong (only_load contains "tables that are deferred, and for each deferred table which columns to load"). So, of course we have to check other tables than the base table, too.