the tokenization now handle its own output files, which reduces memory usage on tokenizing greatly.