tokens: You have a very large scale problem ( and no doubt very large scale hardware to support it , since you mention that you can easily read all of the file into memory that means a 64bit box with many tens of GB of RAM - - wow !
ner_tags: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Device O O O O