Several optimizations to reduce peak memory usage when a directory has a large number of files