- Added a mem-cache for files being processed
- After 500 files it will write them do disk and mark a checkpoint
- If processing is aborted or something goes wrong, resume is now supported
- Resume process is slow, should use random position access rather than line iteration in the future