If the layout isn't prepared, then prepare it when it's about to be used
Fix associated object disappearing

The associated object gets cleared out in invalidate layout, so it
makes sense to add it in prepare layout. Otherwise, once the layout has
been invalidated once, the associated object is gone forever and won't
ever be created again.