Add a tokenizer, apply a memory limit for files and then merge files.