updated to only open 4 files at a time, rather than trying to open all files at once in the hope to prevent running out of memory issues