Fix associated object disappearing

The associated object gets cleared out in invalidate layout, so it
makes sense to add it in prepare layout. Otherwise, once the layout has
been invalidated once, the associated object is gone forever and won't
ever be created again.
Merge pull request #326 from mattjgalloway/missing-assoc-object

Fix associated object disappearing