Remove the cache directory itself rather than its only content