tokens: The idea being that when a specific set of fields have the same data in , the checksum generated would be the same and I can deal with duplicate entries by checking the checksum value , instead of lengthy checks of the field to see if they contain matching data .
ner_tags: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O