handling special case of a bookworm with no unique metadata
writing tokenizer out to files every 10000 documents to save on RAM