Entity Level Evaluation for NER Task | by Xu LIANG | Towards Data Science

When we evaluate the NER (Named Entity Recognition) task, there are two kinds of methods, the token-level method, and the entity-level method. For example, we have this sentence predicted below: “Foreign Ministry spokesman Shen Guofang told Reuters”. If we use the token-level evaluation, the token “Shen” is correct, and the token “Guofang” is wrong. But if we use the entity-level evaluation, “Shen Guofang” is a complete named entity, so the predictions for “Shen” and “Guofang” have to be “PER” and “PER”. Otherwise, this is the wrong predicted entity, even token “Shen” is predicted as “PER”.
So which method we should use?
The answer is the entity-level evaluation. As the task name “Named Entity” indicates, what we really care about is how our model predicts the whole entity, instead of separate tokens.
I usually use the sklearn-crfsuite to implement the CRF model, which is a great library. But one shortcoming of it is the evaluation method used is the token-level evaluation.
precision    recall  f1-score   support      B-LOC      0.775     0.757     0.766      1084      I-LOC      0.601     0.631     0.616       325     B-MISC      0.698     0.499     0.582       339     I-MISC      0.644     0.567     0.603       557      B-ORG      0.795     0.801     0.798      1400      I-ORG      0.831     0.773     0.801      1104      B-PER      0.812     0.876     0.843       735      I-PER      0.873     0.931     0.901       634avg / total      0.779     0.764     0.770      6178
We want to change these to result in entity-level evaluation like below:
precision    recall  f1-score   support      LOC      0.775     0.757     0.766      1084     MISC      0.698     0.499     0.582       339      ORG      0.795     0.801     0.798      1400      PER      0.812     0.876     0.843       735avg/total      0.779     0.764     0.770      6178
Instead of using the official evaluation method, I recommend using this tool, seqeval. This library could run the evaluation at entity-level.
>>> from seqeval.metrics import accuracy_score>>> from seqeval.metrics import classification_report>>> from seqeval.metrics import f1_score>>> >>> y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]>>> y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]>>>>>> f1_score(y_true, y_pred)0.50>>> accuracy_score(y_true, y_pred)0.80>>> classification_report(y_true, y_pred)             precision    recall  f1-score   support       MISC       0.00      0.00      0.00         1        PER       1.00      1.00      1.00         1  micro avg       0.50      0.50      0.50         2  macro avg       0.50      0.50      0.50         2
Ok, we already have a great tool to solve the metric calculation, why do we have to care about the calculations behind the scenes?
Give a man a fish, and you feed him for a day; teach a man to fish, and you feed him for a lifetime.
The confusion matrix is an important topic in the machine learning field, but there are few posts about how to calculate it for the NER task, so I hope this post can clear the uncertainty.
First, we write the confusion matrix table:
And then calculate precision, recall, and F1:
We use the above example to fill the confusion matrix table:
The “Foreign Ministry” is counted as an FN, and “Reuters” is counted as a TP. These two are easy to distinguish. The tricky one is when the prediction makes the boundary error. If the model makes the boundary error, we count it as two errors. For example, we count the entity “Shen Guofang” as two errors, one is the FN, and one is FP.
We ignore the entity type and fill the confusion matrix table like this:
And then calculate the matrices:
We ignore the entity type and only calculate the confusion matrix for one sentence in the above example. For better generalization, the code implementation has to consider the entity type and calculate the confusion matrix for all sentences.
Check out my other posts on Medium with a categorized view!GitHub: BrambleXuLinkedIn: Xu LiangBlog: BrambleXu.com