Try and cache the subdirectories of a directory when we gather files.