Package: tner
T-NER: An All-Round Python Library for Transformer-based Named Entity RecognitionT-NERis a python tool for language model finetuning on named-entity-recognition (NER) implemented in pytorch, available viapip.
It has an easy interface to finetune models and test on cross-domain and multilingual datasets.
T-NER currently integrates high coverage of publicly available NER datasets and enables an easy integration of custom datasets.
All models finetuned with T-NER can be deployed on our web app for visualization.Our paper demonstrating T-NERhas been accepted to EACL 2021.
All the models and datasets are shared viaT-NER HuggingFace group.NEW (September 2022):We released new dataset of NER on Twittertweetner7and the paper got accepted by AACL 2022 main conference! We release the dataset along with fine-tuned models, and more detail can be foundrepositoryanddataset page.Resources:MODEL_CARD,DATASET_CARD,Gradio Online DEMOHuggingFace:https://huggingface.co/tnerGitHub:https://github.com/asahi417/tnerPapersT-NER (EACL2021):acl anthology,arxivTweetNER7 (AACL 2022):arxivInstalltnervia pip to get started!pipinstalltnerTable of ContentsDataset1.1Preset Dataset1.2Custom DatasetModelFine-Tuning Language Model on NEREvaluating NER ModelWeb APIColab ExamplesReferenceGoogle Colab ExamplesDescriptionLinkModel Finetuning & EvaluationModel PredictionMultilingual NER WorkflowDatasetAn NER dataset contains a sequence of tokens and tags for each split (usuallytrain/validation/test),{'train':{'tokens':[['@paulwalk','It',"'s",'the','view','from','where','I',"'m",'living','for','two','weeks','.','Empire','State','Building','=','ESB','.','Pretty','bad','storm','here','last','evening','.'],['From','Green','Newsfeed',':','AHFA','extends','deadline','for','Sage','Award','to','Nov','.','5','http://tinyurl.com/24agj38'],...],'tags':[[0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,2,0,1,0,0,0,0,0,0,0,0],[0,0,0,0,3,0,0,0,0,0,0,0,0,0,0],...]},'validation':...,'test':...,}with a dictionary to map a label to its index (label2id) as below.{"O":0,"B-ORG":1,"B-MISC":2,"B-PER":3,"I-PER":4,"B-LOC":5,"I-ORG":6,"I-MISC":7,"I-LOC":8}Preset DatasetA variety of public NER datasets are available on ourHuggingFace group, which can be used as below
(seeDATASET CARDfor full dataset lists).fromtnerimportget_datasetdata,label2id=get_dataset(dataset="tner/wnut2017")User can specify multiple datasets to get a concatenated dataset.data,label2id=get_dataset(dataset=["tner/conll2003","tner/ontonotes5"])In concatenated datasets, we use theunified label setto unify the entity label.
The idea is to share all the available NER datasets on the HuggingFace in a unified format, so let us know if you want any NER datasets to be added there!Custom DatasetTo go beyond the public datasets, users can use their own datasets by formatting them into
the IOB format described inCoNLL 2003 NER shared task paper,
where all data files contain one word per line with empty lines representing sentence boundaries.
At the end of each line there is a tag which states whether the current word is inside a named entity or not.
The tag also encodes the type of named entity. Here is an example sentence:EU B-ORG
rejects O
German B-MISC
call O
to O
boycott O
British B-MISC
lamb O
. OWords tagged with O are outside of named entities and the I-XXX tag is used for words inside a
named entity of type XXX. Whenever two entities of type XXX are immediately next to each other, the
first word of the second entity will be tagged B-XXX in order to show that it starts another entity.
Please take a looksample custom data.
Those custom files can be loaded in a same way as HuggingFace dataset as below.fromtnerimportget_datasetdata,label2id=get_dataset(local_dataset={"train":"examples/local_dataset_sample/train.txt","valid":"examples/local_dataset_sample/train.txt","test":"examples/local_dataset_sample/test.txt"})Same as the HuggingFace dataset, one can concatenate dataset.data,label2id=get_dataset(local_dataset=[{"train":"...","valid":"...","test":"..."},{"train":"...","valid":"...","test":"..."}])ModelT-NER currently has shared more than 100 NER models onHuggingFace group, as shown in the above table, which reports the major models only and seeMODEL_CARDfor full model lists.
All the models can be used withtneras below.fromtnerimportTransformersNERmodel=TransformersNER("tner/roberta-large-wnut2017")# provide model alias on huggingfaceoutput=model.predict(["Jacob Collier is a Grammy awarded English artist from London"])# give a list of sentences (or tokenized sentence)print(output){'prediction':[['B-person','I-person','O','O','O','O','O','O','O','B-location']],'probability':[[0.9967652559280396,0.9994561076164246,0.9986955523490906,0.9947081804275513,0.6129112243652344,0.9984312653541565,0.9868122935295105,0.9983410835266113,0.9995284080505371,0.9838910698890686]],'input':[['Jacob','Collier','is','a','Grammy','awarded','English','artist','from','London']],'entity_prediction':[[{'type':'person','entity':['Jacob','Collier'],'position':[0,1],'probability':[0.9967652559280396,0.9994561076164246]},{'type':'location','entity':['London'],'position':[9],'probability':[0.9838910698890686]}]]}Themodel.predicttakes a list of sentences and batch sizebatch_sizeoptionally, and tokenizes the sentence by a half-space or
the symbol specified byseparator, which is returned asinputin its output object. Optionally, user can tokenize the
inputs beforehand with any tokenizer (spacy, nltk, etc) and the prediction will follow the tokenization.output=model.predict([["Jacob Collier","is","a","Grammy awarded","English artist","from","London"]])print(output){'prediction':[['B-person','O','O','O','O','O','B-location']],'probability':[[0.9967652559280396,0.9986955523490906,0.9947081804275513,0.6129112243652344,0.9868122935295105,0.9995284080505371,0.9838910698890686]],'input':[['Jacob Collier','is','a','Grammy awarded','English artist','from','London']],'entity_prediction':[[{'type':'person','entity':['Jacob Collier'],'position':[0],'probability':[0.9967652559280396]},{'type':'location','entity':['London'],'position':[6],'probability':[0.9838910698890686]}]]}A local model checkpoint can be specified instead of model aliasTransformersNER("path-to-checkpoint").
Script to re-produce those released models ishere.command-line toolFollowing command-line tool is available for model prediction.tner-predict[-h]-mMODELcommandlinetooltotestfinetunedNERmodel

optionalarguments:-h,--helpshowthishelpmessageandexit-mMODEL,--modelMODELmodelaliasofhuggingfaceorlocalcheckpointExampletner-predict-m"tner/roberta-large-wnut2017"Web AppTo install dependencies to run the web app, add option at installation.pipinstalltner[app]Then, clone the repositorygitclonehttps://github.com/asahi417/tnercdtnerand launch the server.uvicornapp:app--reload--log-leveldebug--host0.0.0.0--port8000Open your browserhttp://0.0.0.0:8000once ready.
You can specify model to deploy by an environment variableNER_MODEL, which is set astner/roberta-large-wnut2017as a default.NER_MODELcan be either path to your local model checkpoint directory or model name on transformers model hub.AcknowledgementThe App interface is heavily inspired bythis repository.Fine-Tuning Language Model on NERT-NER provides an easy API to run language model fine-tuning on NER with an efficient parameter-search as described above.
It consists of 2 stages: (i) fine-tuning with every possible configurations for a small epoch and
compute evaluation metric (micro F1 as default) on the validation set for all the models, and (ii) pick up top-Kmodels to continue fine-tuning tillLepoch.
The best model in the second stage will continue fine-tuning till the validation metric get decreased.This fine-tuning with two-stage parameter search can be achieved in a few lines withtner.fromtnerimportGridSearchersearcher=GridSearcher(checkpoint_dir='./ckpt_tner',dataset="tner/wnut2017",# either of `dataset` (huggingface dataset) or `local_dataset` (custom dataset) should be givenmodel="roberta-large",# language model to fine-tuneepoch=10,# the total epoch (`L` in the figure)epoch_partial=5,# the number of epoch at 1st stage (`M` in the figure)n_max_config=3,# the number of models to pass to 2nd stage (`K` in the figure)batch_size=16,gradient_accumulation_steps=[4,8],crf=[True,False],lr=[1e-4,1e-5],weight_decay=[1e-7],random_seed=[42],lr_warmup_step_ratio=[0.1],max_grad_norm=[10])searcher.train()Following parameters are tunable at the moment.gradient_accumulation_steps: the number of gradient accumulationcrf: use CRF on top of output embeddinglr: learning rateweight_decay: coefficient for weight decayrandom_seed: random seedlr_warmup_step_ratio: linear warmup ratio of learning rate, eg) if it's 0.3, the learning rate will warmup linearly till 30% of the total step (no decay after all)max_grad_norm: norm for gradient clippingSeesourcefor more information about each argument.command-line toolFollowing command-line tool is available for fine-tuning.tner-train-search[-h]-cCHECKPOINT_DIR[-dDATASET[DATASET...]][-lLOCAL_DATASET[LOCAL_DATASET...]][--dataset-nameDATASET_NAME[DATASET_NAME...]][-mMODEL][-bBATCH_SIZE][-eEPOCH][--max-lengthMAX_LENGTH][--use-auth-token][--dataset-split-trainDATASET_SPLIT_TRAIN][--dataset-split-validDATASET_SPLIT_VALID][--lrLR[LR...]][--random-seedRANDOM_SEED[RANDOM_SEED...]][-gGRADIENT_ACCUMULATION_STEPS[GRADIENT_ACCUMULATION_STEPS...]][--weight-decayWEIGHT_DECAY[WEIGHT_DECAY...]][--lr-warmup-step-ratioLR_WARMUP_STEP_RATIO[LR_WARMUP_STEP_RATIO...]][--max-grad-normMAX_GRAD_NORM[MAX_GRAD_NORM...]][--crfCRF[CRF...]][--optimizer-on-cpu][--n-max-configN_MAX_CONFIG][--epoch-partialEPOCH_PARTIAL][--max-length-evalMAX_LENGTH_EVAL]Fine-tunetransformersonNERdatasetwithRobustParameterSearch

optionalarguments:-h,--helpshowthishelpmessageandexit-cCHECKPOINT_DIR,--checkpoint-dirCHECKPOINT_DIRcheckpointdirectory-dDATASET[DATASET...],--datasetDATASET[DATASET...]datasetname(oralistofit)onhuggingfacetnerorganizationeg.'tner/conll2003'['tner/conll2003','tner/ontonotes5']]seehttps://huggingface.co/datasets?search=tnerforfulldatasetlist-lLOCAL_DATASET[LOCAL_DATASET...],--local-datasetLOCAL_DATASET[LOCAL_DATASET...]adictionary(oralist)ofpathstolocalBIOfileseg.{"train":"examples/local_dataset_sample/train.txt","test":"examples/local_dataset_sample/test.txt"}--dataset-nameDATASET_NAME[DATASET_NAME...][optional]datanameofhuggingfacedataset(shouldbesamelengthasthe`dataset`)-mMODEL,--modelMODELmodelnameofunderlyinglanguagemodel(huggingfacemodel)-bBATCH_SIZE,--batch-sizeBATCH_SIZEbatchsize-eEPOCH,--epochEPOCHthenumberofepoch--max-lengthMAX_LENGTHmaxlengthoflanguagemodel--use-auth-tokenHuggingfacetransformersargumentof`use_auth_token`--dataset-split-trainDATASET_SPLIT_TRAINdatasetsplittobeusedfortraining('train'asdefault)--dataset-split-validDATASET_SPLIT_VALIDdatasetsplittobeusedforvalidation('validation'asdefault)--lrLR[LR...]learningrate--random-seedRANDOM_SEED[RANDOM_SEED...]randomseed-gGRADIENT_ACCUMULATION_STEPS[GRADIENT_ACCUMULATION_STEPS...],--gradient-accumulation-stepsGRADIENT_ACCUMULATION_STEPS[GRADIENT_ACCUMULATION_STEPS...]thenumberofgradientaccumulation--weight-decayWEIGHT_DECAY[WEIGHT_DECAY...]coefficientofweightdecay(set0forNone)--lr-warmup-step-ratioLR_WARMUP_STEP_RATIO[LR_WARMUP_STEP_RATIO...]linearwarmupratiooflearningrate(nodecay).eg)ifit's0.3,thelearningratewillwarmuplinearlytill30%ofthetotalstep(set0forNone)--max-grad-normMAX_GRAD_NORM[MAX_GRAD_NORM...]normforgradientclipping(set0forNone)--crfCRF[CRF...]useCRFontopofoutputembedding(0or1)--optimizer-on-cpuputoptimizeronCPUtosavememoryofGPU--n-max-configN_MAX_CONFIGthenumberofconfigstorun2ndphasesearch--epoch-partialEPOCH_PARTIALthenumberofepochfor1stphasesearch--max-length-evalMAX_LENGTH_EVALmaxlengthoflanguagemodelatevaluationExampletner-train-search-m"roberta-large"-c"ckpt"-d"tner/wnut2017"-e15--epoch-partial5--n-max-config3-b64-g12--lr1e-61e-5--crf01--max-grad-norm010--weight-decay01e-7Evaluating NER ModelEvaluation of NER models is done bymodel.evaluatefunction that takesdatasetorlocal_datasetas the dataset to evaluate on.fromtnerimportTransformersNERmodel=TransformersNER("tner/roberta-large-wnut2017")# provide model alias on huggingface# huggingface datasetmetric=model.evaluate('tner/wnut2017',dataset_split='test')# local datasetmetric=model.evaluate(local_dataset={"test":"examples/local_dataset_sample/test.txt"},dataset_split='test')An example of the output objectmetriccan be foundhere.entity span predictionFor better understanding of out-of-domain accuracy, we provide the entity span prediction
pipeline, which ignores the entity type and compute metrics only on the IOB entity position (binary sequence labeling).metric=model.evaluate(datasets='tner/wnut2017',dataset_split='test',span_detection_mode=True)Command line toolFollowing command-line tool is available for model prediction.tner-evaluate[-h]-mMODEL-eEXPORT[-dDATASET[DATASET...]][-lLOCAL_DATASET[LOCAL_DATASET...]][--dataset-nameDATASET_NAME[DATASET_NAME...]][--dataset-splitDATASET_SPLIT][--span-detection-mode][--return-ci][-bBATCH_SIZE]EvaluateNERmodel

optionalarguments:-h,--helpshowthishelpmessageandexit-mMODEL,--modelMODELmodelaliasofhuggingfaceorlocalcheckpoint-eEXPORT,--exportEXPORTfiletoexporttheresult-dDATASET[DATASET...],--datasetDATASET[DATASET...]datasetname(oralistofit)onhuggingfacetnerorganizationeg.'tner/conll2003'['tner/conll2003','tner/ontonotes5']]seehttps://huggingface.co/datasets?search=tnerforfulldatasetlist-lLOCAL_DATASET[LOCAL_DATASET...],--local-datasetLOCAL_DATASET[LOCAL_DATASET...]adictionary(oralist)ofpathstolocalBIOfileseg.{"train":"examples/local_dataset_sample/train.txt","test":"examples/local_dataset_sample/test.txt"}--dataset-nameDATASET_NAME[DATASET_NAME...][optional]datanameofhuggingfacedataset(shouldbesamelengthasthe`dataset`)--dataset-splitDATASET_SPLITdatasetsplittobeusedfortest('test'asdefault)--span-detection-modereturnF1ofentityspandetection(ignoringentitytypeerrorandcastasbinarysequenceclassificationasbelow)-NER:["O","B-PER","I-PER","O","B-LOC","O","B-ORG"]-Entity-spandetection:["O","B-ENT","I-ENT","O","B-ENT","O","B-ENT"]--return-cireturnconfidenceintervalbybootstrap-bBATCH_SIZE,--batch-sizeBATCH_SIZEbatchsizeExampletner-evaluate-m"tner/roberta-large-wnut2017"-e"metric.json"-d"tner/conll2003"-b"32"ReferenceIf you use any of these resources, please cite the followingpaper:@inproceedings{ushio-camacho-collados-2021-ner,
    title = "{T}-{NER}: An All-Round Python Library for Transformer-based Named Entity Recognition",
    author = "Ushio, Asahi  and
      Camacho-Collados, Jose",
    booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations",
    month = apr,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2021.eacl-demos.7",
    pages = "53--62",
    abstract = "Language model (LM) pretraining has led to consistent improvements in many NLP downstream tasks, including named entity recognition (NER). In this paper, we present T-NER (Transformer-based Named Entity Recognition), a Python library for NER LM finetuning. In addition to its practical utility, T-NER facilitates the study and investigation of the cross-domain and cross-lingual generalization ability of LMs finetuned on NER. Our library also provides a web app where users can get model predictions interactively for arbitrary text, which facilitates qualitative model evaluation for non-expert programmers. We show the potential of the library by compiling nine public NER datasets into a unified format and evaluating the cross-domain and cross- lingual performance across the datasets. The results from our initial experiments show that in-domain performance is generally competitive across datasets. However, cross-domain generalization is challenging even with a large pretrained LM, which has nevertheless capacity to learn domain-specific features if fine- tuned on a combined dataset. To facilitate future research, we also release all our LM checkpoints via the Hugging Face model hub.",
}