Fix clearing files in multiple groups.

When using multiple groups on a cache config, clearing was not working
properly. Clearing un-used cache configs also failed when the
directories did not exist.

Fixes #3930