Set the inverse association before assigning attributes

Fixes #9157