tokens: Maybe a little specialized Cython-coded helper can offer all the extra performance you need , if no suitable and accessible library ( in C , C++ , Fortran , .. . ) already exists that can be interfaced for the purpose of handling this humongous file as you need to .
ner_tags: O O O O B-Language O O O O O O O O O O O O O O O O O O B-Language O B-Language O B-Language O O O O O O O O O O O O O O O O O O O O O O O