Fixed #23001 -- Fixed mixing defer and annotations
Avoided creation of deferred model from another deferred model

Also never create deferred model when no attrs are deferred.