Only apply Inverse=true if child has a reference to parent

The bag convention can only apply inverse=true when the child element has a reference back to the parent

@simoncropp @johannesg can you please review ?