Package: ner-error-analysis
NER Error AnalyzerQuick Startfromnlu.errorimport*fromnlu.parserimport*cols_format=[{'type':'predict','col_num':1,'tagger':'ner'},{'type':'gold','col_num':2,'tagger':'ner'}]parser=ConllParser('testb.pred.gold',cols_format)parser.obtain_statistics(entity_stat=True,source='predict')parser.obtain_statistics(entity_stat=True,source='gold')parser.set_entity_mentions()NERErrorAnnotator.annotate(parser)parser.print_corrects()parser.print_all_errors()parser.error_overall_stats()see the section Input Format below to know what the input format isUsageimportfromnlu.errorimport*fromnlu.parserimport*Create aConllParserinstance first with the input of the file path with specifying the column number incols_formatfieldConllParser(filepath)cols_format=[{'type':'predict','col_num':1,'tagger':'ner'},{'type':'gold','col_num':2,'tagger':'ner'}]parser=ConllParser('testb.pred.gold',cols_format)obtain the basic statistics byobtain_statistics()methodparser.obtain_statistics(entity_stat=True,source='predict')parser.obtain_statistics(entity_stat=True,source='gold')To "Annotate" NER Errors in the documents inside ConllParserNERErrorAnnotator.annotate(parser)To print out all corrects/errors, useparser.print_corrects()orparser.print_all_errors()or use the functionerror_overall_stats()method to get the statsInput File FormatThe input file format ofConllParseris following the column format used by Conll03.For example,Natural I-ORG O
Language I-ORG O
Laboratory I-ORG I-ORG
...where the first column is the text, the second and the third are the predicted and the ground truth tag respectively, where the order can be specified in the keywordcols_formatinConllParserin instantialization:cols_format = [{'type': 'predict', 'col_num': 1, 'tagger': 'ner'},
               {'type': 'gold', 'col_num': 2, 'tagger': 'ner'}]  # col_num starts from 0I recommend to use shell commandawk '{print $x}' filepathto obtain the x-th column, likeawk '{print $4} filepath'to obtain the 4-th column.And usepaste file1.txt file2.txtto concatenate two files.For example,awk'{print $4}'eng.train>ner_tags_file# $num starts from 1pastener_pred_tags_filener_tags_fileTypes of Span ErrorsTypesNumber of Mentions (Predicted and Gold)SubtypesExamplesNotesMissing Mention(False Negative)1TYPES→O[] → None # todoExtra Mention(False Positive)1O→TYPESNone → [...]  # todoMention with Wrong Type(Type Errors)≥ 2TYPES-> TYPES - self( {(p, g) | p ∈ T, g ∈ T - p } )[PER...] → [ORG...] # todoBut the spans are the sameMissing Tokens2L/ R/ LR Diminished[MISC1991 World Cup] → [MISC1991] [MISCWorld Cup]also possible with type errorsExtra Tokens2L/R/LR Expanded[...] → [......]  # todoalso possible with type errorsMissing + Extra Tokens2L/R Crossed..[...].. → .[..]...also possible with type errorsConflated Mention≥ 3[][][] → []  # todoalso possible with type errorsDivided Mention≥ 3[MISC1991 World Cup] → [MISC1991] [MISCWorld Cup][PERBarack Hussein Obama] → [PERBarack][PERHussein][PERObama]also possible with type errorsComplicated Case≥ 3[][][] → [][]  # todoalso possible with type errorsEx -Mention with Wrong Segmentation(Same overall range but wrong segmentation)≥ 4[...][......][.] → [......][.....]also possible with type errors