Make layout caching and removal more robust