Fields are cached when withRelation, so they should only be accessed when withRelation.

The result of .Fields() is quite different with and without withRelation.
I assume that if you call it without withRelation, you don't want the
fields that were cached when withRelation was true.