tokens: Having said that , I believe this approach is wrong for your use case - Lucene is optimized to get a few similar documents , not a score for every one , so I predict the runtime will be prohibitive - Hope I am wrong , but nevertheless I suggest you read Mining of Massive Datasets for a better approach - min hashes and shingling .
ner_tags: O O O O O O O O O O O O O O O B-Library O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Algorithm I-Algorithm O B-Algorithm O