Only look for directory entries before cleaning up old cache dirs

Change-Id: I9af7223656a1091c1566220c1a4ec93c2505efc0