Don't create grouping element for to-one relations.

Fixes 79953026, @afroisalreadyinu can you take a look?
This is also kind-of breaks the recursive nature of this; here we are processing the child while we are processing the parent, before we have descended to the child's level.