Remove stale caches on activation