This could have worse performance than not; it would cause multiple `stat` calls to each directory rather than the ones we are interested in. The ideal scenario would be to use a remote `fs` that has smart caching logic in it.

I think it is best to keep this off the table until we add performance tests (#32).