Package: nerval
nervalEntity-level confusion matrix and classification report to evaluate Named Entity Recognition (NER) models.Labelling schemes supported:IOBIO1 (IOB1)BIO2 (IOB2)IOE1IOE2IOBESBILOUBMEWOOptions for the 'scheme' argument:BIOfor the following schemes: IO / BIO1 (IOB1) / BIO2 (IOB2) / IOBES / BILOU / BMEWOIOEfor the following schemes: IOE1 / IOE2BIOis the default scheme.Output:Classification reportConfusion matrixLabels for the confusion matrixConfusion matrix plotHow to use it:>>>fromnervalimportcrm>>>y_true=[['O','B-PER','I-PER','O','O','O','O','B-LOC','I-LOC']]>>>y_pred=[['O','B-PER','O','O','O','O','O','O','B-LOC']]>>>cr,cm,cm_labels=crm(y_true,y_pred,scheme='BIO')TrueEntities:2PredEntities:2TrueEntitieswith3ormoretags:0PredEntitieswith3ormoretags:0Truepositives:0Falsepositives(true='O'):1Falsepositives(true<>pred):1ToTFalsepositives:2Falsenegatives:1>>>print(cr)precisionrecallf1_scoretrue_entitiespred_entitiesPER0.000.000.001.000.00LOC0.000.000.001.001.00PER__0.000.000.000.001.00micro_avg0.000.000.002.002.00macro_avg0.000.000.002.002.00weighted_avg0.000.000.002.002.00>>>print(cm)[[0100][1000][0001][0000]]>>>print(cm_labels)['LOC','O','PER','PER__']>>>fromnervalimportplot_confusion_matrix>>>y_true=[['O','B-PER','I-PER','O','O','O','O','B-LOC','I-LOC']]>>>y_pred=[['O','B-PER','O','O','O','O','O','O','B-LOC']]>>>plot_confusion_matrix(cm,cm_labels,show=True,save=False,img_path=None,normalize=None,decimal_places=2,figsize=(15,15),SMALL_SIZE=8,MEDIUM_SIZE=12,BIGGER_SIZE=14,cmap='OrRd',xticks_rotation='vertical',title='Confusion Matrix')>>>plot_confusion_matrix(cm,cm_labels,show=True,save=True,img_path=None)>>>plot_confusion_matrix(cm,cm_labels,show=True,save=True,img_path=r'C:\Users\...\my_conf_matrix.png')>>>plot_confusion_matrix(cm,cm_labels,show=False,save=True,img_path=None)>>>plot_confusion_matrix(cm,cm_labels,show=False,save=True,img_path=r'C:\Users\...\my_conf_matrix.png')Note 1:y_trueandy_predcould be:flat listslists of flat listslists of nested lists.Flat lists and lists of nested lists will be converted to lists of lists.Note 2:The __ at the end of some entities means that true and pred have the same entity name for the first token but the prediction is somewhat different from the true label.
Examples:>>>y_true=['B-ORG','I-ORG','I-ORG']>>>y_pred=['B-ORG']>>>y_true=['B-ORG','I-ORG','I-ORG']>>>y_pred=['B-ORG','I-ORG','I-ORG','I-ORG','I-ORG']>>>y_true=['B-ORG','I-ORG','I-ORG']>>>y_pred=['B-ORG','I-PER']>>>y_true=['B-ORG','I-ORG','I-ORG']>>>y_pred=['I-ORG','I-PER']Note 3:The normalize argument could be:None'true''pred''all'Default is None.Note 4:In case of division by zero, the result will default to zero.Note 5:Parameters in function plot_confusion_matrix():show: show the plot (default: True)save: save the plot (default: False)img_path: where to save the plot - e.g. r'C:\Users\User...\my_conf_matrix.png' (default: None - this means save the plot in current dir)Installationpipinstallnerval
condainstall-cconda-forgenervalLicenseMITCitation@misc{nerval,
  title={{nerval}: Entity-level confusion matrix and classification report to evaluate Named Entity Recognition (NER) models.},
  url={https://github.com/mdadda/nerval},
  note={Software available from https://github.com/mdadda/nerval},
  author={Mariangela D'Addato},
  year={2022},
}