Fix crash deallocating a drawer view controller that was never loaded.

Remove key value observation only if the view is loaded, because view loading is where we establish frame observing.
This fix is important for unit tests where the application UI may partially load: instantiating classes and relationships but not view hierarchies.