Change how cache notification works

Prevent crashes due to destruction/notification races for caches