tokens: Are your needs at all more advanced than simply sequential reading ( e.g . , random access ) , and if so , can you do a first " indexing " pass to get all the offsets from start of file to start of record into a more usable , compact , handily-formatted auxiliary file ?
ner_tags: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-File_Type O O