Changed limit to 1000 files in memory before writing the data to disk