Package: eval4ner
eval4ner: An All-Round Evaluation for Named Entity RecognitionTable of ContentsTL;DRPreliminaries for NER EvaluationUser GuideInstallationUsageCitationReferencesThis is a Python toolkit of MUC-5 evaluation metrics for evaluating Named Entity Recognition (NER) results.TL;DRIt considers not only the mode of strict matching,i.e., extracted entities are correct w.r.t both boundaries and types, but that of partial match, summarizing as following four modes:Strict：exact match (Both entity boundary and type are correct)Exact boundary matching：predicted entity boundary is correct, regardless of entity boundaryPartial boundary matching：entity boundaries overlap, regardless of entity boundaryType matching：some overlap between the system tagged entity and the gold annotation is required;Refer to the blogEvaluation Metrics of Name Entity Recognitionfor explanations of MUC metric.Preliminaries for NER EvaluationIn research and production, following scenarios of NER systems can occur frequently:ScenarioGolden StandardNER system predictionMeasureEntity TypeEntity Boundary (Surface String)Entity TypeEntity Boundary (Surface String)TypePartialExactStrictIIIMUSIC_NAME告白气球MISMISMISMISIIMUSIC_NAME年轮SPUSPUSPUSPUVMUSIC_NAME告白气球MUSIC_NAME一首告白气球CORPARINCINCIVMUSIC_NAME告白气球SINGER告白气球INCCORCORINCIMUSIC_NAME告白气球MUSIC_NAME告白气球CORCORCORCORVIMUSIC_NAME告白气球SINGER一首告白气球INCPARINCINCThus, MUC-5 takes into account all these scenarios for all-sided evaluation.Then we can compute:Number of golden standard:Number of predictee:The evaluation type of exact match and partial match are as follows:Exact match(i.e. Strict, Exact)Partial match (i.e. Partial, Type)F-MeasureTherefore, we can get the results:MeasureTypePartialExactStrictCorrect2221Incorrect2023Partial0200Missed1111Spurius1111Precision0.40.60.40.2Recall0.40.60.40.2F1 score0.40.60.40.2User GuideInstallationpipinstall[-U]eval4nerUsage1. Evaluate single predictionimporteval4ner.mucasmucimportpprintgrount_truth=[('PER','John Jones'),('PER','Peter Peters'),('LOC','York')]prediction=[('PER','John Jones and Peter Peters came to York')]text='John Jones and Peter Peters came to York'one_result=muc.evaluate_one(prediction,grount_truth,text)pprint.pprint(one_result)Output:{'exact':{'actual':1,'correct':0,'f1_score':0,'incorrect':1,'missed':2,'partial':0,'possible':3,'precision':0.0,'recall':0.0,'spurius':0},'partial':{'actual':1,'correct':0,'f1_score':0.25,'incorrect':0,'missed':2,'partial':1,'possible':3,'precision':0.5,'recall':0.16666666666666666,'spurius':0},'strict':{'actual':1,'correct':0,'f1_score':0,'incorrect':1,'missed':2,'partial':0,'possible':3,'precision':0.0,'recall':0.0,'spurius':0},'type':{'actual':1,'correct':1,'f1_score':0.5,'incorrect':0,'missed':2,'partial':0,'possible':3,'precision':1.0,'recall':0.3333333333333333,'spurius':0}}2. Evaluate all predictionsimporteval4ner.mucasmuc# ground truthgrount_truths=[[('PER','John Jones'),('PER','Peter Peters'),('LOC','York')],[('PER','John Jones'),('PER','Peter Peters'),('LOC','York')],[('PER','John Jones'),('PER','Peter Peters'),('LOC','York')]]# NER model predictionpredictions=[[('PER','John Jones and Peter Peters came to York')],[('LOC','John Jones'),('PER','Peters'),('LOC','York')],[('PER','John Jones'),('PER','Peter Peters'),('LOC','York')]]# input textstexts=['John Jones and Peter Peters came to York','John Jones and Peter Peters came to York','John Jones and Peter Peters came to York']muc.evaluate_all(predictions,grount_truths*1,texts,verbose=True)Output:NERevaluationscores:strictmode,Precision=0.4444,Recall=0.4444,F1:0.4444exactmode,Precision=0.5556,Recall=0.5556,F1:0.5556partialmode,Precision=0.7778,Recall=0.6667,F1:0.6944typemode,Precision=0.8889,Recall=0.6667,F1:0.7222This repo will be long-term supported. Welcome to contribute and PR.CitationFor attribution in academic contexts, please cite this work as:@misc{eval4ner,
  title={Evaluation Metrics of Named Entity Recognition},
  author={Chai, Yekun},
  year={2018},
  howpublished={\url{https://cyk1337.github.io/notes/2018/11/21/NLP/NER/NER-Evaluation-Metrics/}},
}

@misc{chai2018-ner-eval,
  author = {Chai, Yekun},
  title = {eval4ner: An All-Round Evaluation for Named Entity Recognition},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/cyk1337/eval4ner}}
}ReferencesEvaluation of the SemEval-2013 Task 9.1: Recognition and Classification of pharmacological substancesMUC-5 Evaluation Metrics