Be specific about which cache we are invalidating

We may want to invalidate other caches in future (sssd for example).
As suggested by @stdweird.