A novel evaluation technique for Named Entity Recognition (NER)
By: Charudatta Manwatkar
Tags: Naturallanguageprocessing, Named Entity Recognition, Ner, Machine Learning, Artificial Intelligence

New research explained

A novel evaluation technique for Named Entity Recognition (NER)

Interpretable and fine-grained evaluation techniques are crucial for making well-grounded improvements in any ML model. Photo by Kent Pilcher on Unsplash

Prologue

NER is typically approached as a sequence labelling problem and the models for it are usually evaluated through traditional classification metrics such as precision, recall, F-score, etc. You can read a primer on the topic in this post.

Example of Named Entity Recognition in action. Image by author.

The problem with traditional evaluation metrics

These simple metrics are fine if you just want a quick comparison of different models, but they are really opaque and don’t tell you a lot. They don’t tell you anything about what are the strengths and weaknesses of the model or what characteristics of the data are most influential on the model performance.

You are left guessing whether the model is suffering due to long sentence length, or is it because there are too many (or too few) entities, the entities are spread out too far (or too close), or the entities are highly ambiguous, etc.

New Development

To tackle this problem, researchers from Carnegie Mellon University and Fudan University, China have come up with a novel evaluation technique which tackles some of these problems. You can read the original paper here.

The authors have also released an open source repository that has the implementation of the concepts in this paper.

This article is my attempt to explain the findings in this paper to a broader audience in a simplified way. It is in no way a complete and exhaustive reproduction of the paper. The original paper is rich in details and nuances, so do give it a read to know more.

This recent paper by Jinlan Fu (Fudan University), Pengfei Liu and Graham Neubig (Carnegie Mellon University) presents new evaluation techniques for NER that are interpretable and fine-grained.

Core idea

The main idea in the paper is to divide the data into buckets of entities based on attributes such as entity length, label consistency, entity density, sentence length, etc. and then evaluate the model on each of these buckets separately.

This makes it easy to locate the factors where the model is performing (or not performing) well.

The core idea of the paper is to divide the dataset into distinct buckets to understand what factors affect the model performance. Photo by Pascal Brokmeier on Unsplash

Let’s go into some of these details.

Attributes

Attributes are different characteristics of the training examples, which may affect the performance of the model. These are:

1) Local Attributes

Entity Length : String length of the entity

: String length of the entity Sentence Length

OOV Density : Fraction of ‘Out of Vocabulary’ words in the sentence

: Fraction of ‘Out of Vocabulary’ words in the sentence Entity Density: Number of entities in a sentence

2) Aggregate Attributes

Token Label Consistency: This is a measure of how versatile an entity is. E.g. in

“I live in New York”

‘New York’ is labelled as ‘LOC’, but in

“I study in New York University”

‘New York’ is a part of entity labelled ‘ORG’.

Token frequency : How many times the token appears in the dataset

: How many times the token appears in the dataset Entity Label Consistency : Same as Token Label Consistency, but calculated for entity text instead of tokens.

: Same as Token Label Consistency, but calculated for entity text instead of tokens. Entity Frequency : Same as Token Frequency, but calculated for entity text instead of tokens.

Bucketing

For any of the above-mentioned attributes, we may divide the dataset into as many buckets as we want (authors prefer 4 buckets). On each of these buckets, the model performance is measured by its micro-average F-1 score.

After gathering this diverse performance data, it’s time to find out the correlation between attributes and performance.

Correlation between attributes and performance

A typical reader would infer correlation between the model performance and the attribute values using Pearson Correlation Coefficient.

However the authors infer correlations not directly between F-1 scores and the attribute values, but between the ranks of the F-1 scores and the ranks of the attribute values.

My guess is that this allows the authors to create more convenient buckets which may not be strictly in the increasing order of the attribute values.

For this purpose, the authors use Spearman’s rank correlation coefficient. To test the significance of these correlations, they use the Friedman Test and Wilcoxon signed-rank test as appropriate.

To quote a wise man, “Don’t worry about it if you don’t understand it right away”.

To Spearman Rank Correlation Coefficient, rank is all that matters, not the actual value. Photo by Joshua Golde on Unsplash

Empirical Results

With this framework, the authors carry out an extensive analysis of 12 models, 8 attributes, and 6 datasets. The paper describes the results in great details. The analysis is roughly divided in 3 types:

1] Holistic Analysis

This analysis is similar to how traditional multi-dataset analysis are performed (without fine-graining into attributes or buckets). The authors argue that it does not create a very clear picture of what are the crucial factors for the performance.

2] Fine-grained analysis

This is the analysis which takes full advantage of the new ideas presented in the paper. After measuring the performance of all the models on all the buckets of all the datasets, the authors present 2 methods to make sense of all the results.

2.1] Model-wise analysis

Here, the authors talk about the Spearman’s Rank Correlation between model performance (F- score) and the attribute values. The authors make the following observations:

Observation 1: The performance of character-unaware models is more sensitive to the label consistency.

Observation 2: The influence of entity length varies greatly between different decoders.

The table from the paper presents Spearman Rank Correlation Coefficient of model performance with attribute values. eDen, oDen, etc. are shorthand for attribute names like Entity Density, OOV Density, etc. Green/red values indicate correlation with Observation 1/Observation 2. Gray values indicate correlation which did not pass the Friedman Significance test. [Minor modifications to the original table by me for greater clarity]

2.2] Attribute-wise analysis

Here the authors discuss how an attribute affects the NER (independent of the model). They make the following observations:

Observation 1: Label consistency and entity length have a more consistent influence on NER performance.

Observation 2: Frequency and sentence length matter but are minor factors.

The distance from the center is proportional to the average of Spearman’s Rank Correlation Coefficient across all models. Each color corresponds to a particular dataset as specified by the legend. Source

The authors also characterize the datasets themselves in terms of average values of attribute values (without referring to any NER model at all). This gives us a great way to assess the character of a new dataset (which may be specific to a particular application). This kind of characterization may enable us to choose the model in an informed way (e.g. if your dataset has high OOV Density, then you might choose a CNN based model rather than BERT). The following graph maps this characterization:

Distance from origin denotes the average value of the attribute in a dataset. Colors denote the dataset similar to the image above. Source

3] Bucket-wise analysis

Bucket-wise analysis aims to:

Self Diagnosis: Identify buckets where a specific model shows good/bad performance. E.g. BERT performs best/worst on buckets with most/least label consistency

Identify buckets where a specific model shows good/bad performance. E.g. BERT performs best/worst on buckets with most/least label consistency Comparative Diagnosis: Identify buckets where 2 models show the most/least difference in performance.

From the paper: “M1 and M2 denote two models. We classify the attribute values into four categories: extra-small (XS), small (S), large (L) and extra-large (XL). In the self-diagnosis histogram, green (red) x-tick labels represents the bucket value of a specific attribute on which system achieved best (worst) performance. Gray bins represent worst performance while blue bins denote the gap between best and worst performance. In the comparative diagnosis histogram, green (red) x-tick labels represents the bucket value of a specific attribute on which system M1 surpasses (under-performs) M2 by the largest margin that is illustrated by a green (red) bin”. Source

Application: Well-grounded Model Improvement

With the use of these fine-grained and interpretable evaluation metrics, we can take informed decisions about how to improve model performance.

Example: Improving BERT performance by over 10%

The authors present the case that since BERT and Flair are most sensitive to Label Consistency, adding more context to each training example would enable the model to make substantial improvement.

To test this hypothesis, they train the models again with multiple sentences in each training example. As a result, the performance of the model improves substantially.

The improvement of performance on different datasets as a result of increase in context. Y-axis is the increase in F-score (%). X- axis is the number of sentences in each training example. More than 10% improvement is seen in “WB”, which is a domain from OntoNotes dataset when X=6. Source

TL;DR

Named Entity Recognition (NER) is a sequence labelling task in ML. The models performing the task are traditionally evaluated with classification metrics like precision, recall, F-1 score, etc. These traditional NER evaluation metrics don’t tell you where the model is failing or what actions you can take to improve performance. A new research paper has presented the following attempt to solve this problem:

Define explainable attributes of data like Entity Density, Label Consistency, token frequency, etc.

Divide the dataset into different buckets based on these attributes.

Evaluate the models on each of those buckets separately.

5. This allows us to:

— Find out which attributes of the data affect the performance the most.

— Choose the best suited model for a particular dataset in an informed way.

— Come up with well-grounded ideas for model improvement.