Fixed #16436 -- ``annotate().select_related().only()`` = ``ValueError: invalid literal for int() with base 10``

Correctly calculate the ``aggregate_start`` offset from loaded fields,
if any are deferred, instead of ``self.query.select`` which includes all
fields on the model.