Properly remove the cache entry