SlowGuess/NAACL_2022_OCR huggingface dataset:
Xiaochen Wang
University of North Carolina at
Chapel Hill, NC, USA
xcwang@email.unc.edu
wangyue@unc.edu
As a fundamental task in natural language processing, named entity recognition (NER) aims to locate and classify named entities in unstructured text. However, named entities are always the minority among all tokens in the text. This data imbalance problem presents a challenge to machine learning models as their learning objective is usually dominated by the majority of non-entity tokens. To alleviate data imbalance, we propose a set of sentence-level resampling methods where the importance of each training sentence is computed based on its tokens and entities. We study the generalizability of these resampling methods on a wide variety of NER models (CRF, Bi-LSTM, and BERT) across corpora from diverse domains (general, social, and medical texts). Extensive experiments show that the proposed methods improve performance of the evaluated NER models especially on small corpora, frequently outperforming sub-sentence-level resampling, data augmentation, and special loss functions such as focal and Dice loss.
In natural language processing, named entity recognition (NER) is an important task both on its own and for numerous downstream tasks such as entity linking and question answering. NER has an inherent data imbalance problem: named entities of interest are almost always the minority among irrelevant (Other type) tokens in a text corpus. Table 1 shows the prevalent imbalanced nature of NER corpora from multiple domains. As shown in Table 1, entity tokens (tokens associated with any named entity) account for  $3.9 - 16.6\%$  of all tokens in any of these corpora. Within entity tokens, the most frequent entity type may cover 2-200 times more tokens than the least frequent entity type. At the sentence level,  $23 - 85\%$  sentences contain at least
one entity, suggesting that  $15 - 77\%$  sentences contain no entity at all.
Data imbalance is even more severe in real-world bespoke NER tasks, which directly motivated this work. For example, given full-text articles from a medical subfield, domain experts may wish to recognize only those concepts related to specific aspects of the subfield (e.g., symptoms and medicine related to a specific disease). Compared to all tokens in the full text, extremely few tokens are annotated with any entity type. Because domain experts have limited availability, annotated corpus are usually small in such tasks. As a result, some rare entity types may have less than 10 tokens across the corpus. Such severe data imbalance and scarcity makes many NER models suffer.
Data imbalance in NER challenges machine learning-based models because their learning objective is dominated by entities of the majority type (Other), causing the model to be reluctant to predict the types of interest. Various techniques have been studied to tackle this challenge. Active learning was applied to collect a more balanced dataset at annotation time (Tomanek and Hahn, 2009). Special loss functions including focal loss (Lin et al., 2017) and Dice loss (Li et al., 2019) are proposed to deal with data imbalance. Data augmentation was shown to be effective by enriching entity-bearing sentences through methods like segment shuffling and mention replacement (Dai and Adel, 2020; Issifu and Ganiz, 2021; Wang and Henao, 2021).
The classical method for alleviating data imbalance is resampling (upsampling the minority class or downsampling the majority class) and its close relative, cost-sensitive learning (assigning larger weight to the minority class or smaller weight to the majority class in the learning objective) (He and Garcia, 2009). A natural question is: Can we apply resampling to address the data imbalance problem in NER? It turns out that unlike classification tasks, applying resampling to sequence
The source code is available at https://github. com/XiaoChen-W/NER_Adaptive_Resampling.
Human Language Technologies, pages 2151 - 2165
Domain	NER Corpus	# of Tokens	# of Entity Types	% of Least vs. Most Freq. Type Entity Tokens	% of All Entity Tokens	# of Sent.	% of Sent. w/ Entities
Social	WNUT	59,570	6	0.43 vs. 1.59	5.03	3,394	36.18
General	GMB subset	66,161	8	0.03 vs. 4.00	15.03	2,999	85.40
Medical	AnEM	71,697	11	0.03 vs. 1.08	3.91	2,815	35.38
Medical	CADEC	121,307	5	0.21 vs. 6.65	15.76	5,719	58.86
General	CoNLL	204,567	4	2.26 vs. 5.46	16.64	14,986	74.28
Medical	n2c2 ADE	813,277	9	0.19 vs. 2.34	10.89	65,293	22.73
General	OntoNotes	2,200,865	18	0.01 vs. 2.59	10.89	115,812	50.11
Table 1: Imbalance ratio statistics in NER corpora from different domains. Corpora references: WNUT (Derczynski et al., 2017); GMB subset (Bos et al., 2017); AnEM (Ohta et al., 2012); CADEC (Karimi et al., 2015); CoNLL (Sang and De Meulder, 2003); n2c2 ADE (Henry et al., 2020); OntoNotes (Ralph et al., 2013). 'Sent.' = Sentences; 'Freq.' = Frequent.
tagging tasks like NER is not straightforward. Recent work attempted sub-sentence-level resampling – dropping tokens from the majority class either at random (Akkasi, 2018) or using heuristic rules (Akkasi et al., 2018; Akkasi and Varoglu, 2019; Grancharova et al., 2020). These methods were shown to perform well with shallow NER models – conditional random fields with local  $n$ -gram and word shape features. However, sub-sentence-level resampling inevitably destroys the structure of complete sentences and distort the contextual information around entities of interest. Complete sentences are essential for state-of-the-art NER models based on contextual word representations, e.g., deep Transformers (Devlin et al., 2018). As shown in our experiments, incomplete sentences generated by sub-sentence-level resampling often hurt the performance of deep NER models.
In this paper, we propose sentence-level resampling methods for NER, an underexplored problem in this area. As sentences are the natural units of data in NER, sentence-level resampling leaves the contextual information intact in a natural sentence needed by deep models like Transformers. Since a sentence may contain a mixture of entities whose types have different levels of rareness, traditional resampling method for imbalanced classification (e.g., inverse probability resampling) cannot be applied. Instead, we develop a set of methods for computing integer-valued importance score for a sentence based on its entity composition, and resample the sentence accordingly. Experiments show that our methods can improve performance of a variety of NER models and are especially effective on tasks with small annotated corpora, which
is often seen in real-world bespoke NER tasks.
2.1 Learning from Imbalanced Data
Class imbalance is a long-standing problem in machine learning tasks, posing challenges to researchers and practitioners in many domains (King and Zeng, 2001; Lu and Jain, 2003; He and Garcia, 2009; Moreno et al., 2016). Classes in real-world data often have highly skewed distribution, leading to substantial gaps between majority and minority classes. While the positive (minority) class is often of interest, the lack of positive examples makes classifiers conservative, i.e., they incline to predict all example as the negative (majority) class. This often results in a low recall of the positive class. Because only a small number of examples are predicted as positive, precision of the positive class tends to be high or unstable. Such a low-recall, high-precision pattern often hurts the F1-score, the standard metric that emphasizes a balanced precision and recall (Juba and Le, 2019). This performance pattern is observed not only in classification tasks, but also in NER tasks where named entity tokens are the minority compared to non-entity tokens (Mao et al., 2007; Kuperus et al., 2013).
Researchers have proposed various techniques for imbalanced learning, including resampling and cost-sensitive learning (He and Garcia, 2009). Both aim to re-balance the representation of different classes in the loss function, such that the classifier is less conservative in making positive predictions. In principle, by equating per-instance resampling frequency with per-instance cost, resampling can be implemented as cost-sensitive learning. How-
ever, resampling can be applied to models that do not support cost-sensitive learning, making it conveniently applicable to all models.
2.2 Resampling in Sequence Tagging Tasks
Resampling (and cost-sensitive learning) can be conveniently used in classification and regression tasks where a model makes pointwise predictions (a single categorical or scalar value). Each example has a clearly defined sampling rate (or cost) according to its class label. However, in sequence tagging tasks like NER (more broadly, structured prediction tasks (BakIr et al., 2007; Smith, 2011)), a model predicts multiple values for a sequence (or structured output). For sequence learning algorithms such as linear-chain conditional random fields, while the learning objective is formulated at the sequence level, the evaluation metrics are defined at the entity span level. This makes it nontrivial to determine the sampling rate (or cost) for a sequence that contains tokens from both majority and minority entity types. Simply resampling entities by stripping surrounding context is problematic as sequence tagging algorithms depend on context to make predictions. Recent works proposed to randomly or heuristically drop tokens from sentences to re-balance NER data, which had success using conditional random fields and shallow  $n$ -gram features (Akkasi, 2018; Akkasi and Varoglu, 2019; Grancharova et al., 2020). However, these methods distort the syntactic and semantic structure of complete sentences, which may generate low-quality data for models that are capable of capturing long-distance linguistic dependencies (e.g. BERT) and hurt performance of those models. In this work, we focus on resampling strategies that leaves sentences intact.
2.3 Loss Functions for Imbalanced Data
Recent literature proposed special loss functions for tackling data imbalance, including focal loss (Lin et al., 2017) and Dice loss (Li et al., 2019). They increase the cost of 'hard positives' where the correct label has low predicted probability and decrease the cost of 'easy negatives' where the correct label has high predicted probability. However, these loss functions do not fully address data imbalance in NER. First, the formulation does not always emphasize the loss of minority-class tokens – majority-class tokens can also be hard to classify, and minority-class tokens can also be easy to classify. Second, these loss functions only work on
token-wise prediction outputs. They cannot work on sequence-level outputs generated by conditional random fields, which is commonly used in NER. Our resampling methods can be seen as estimating sentence-level losses with explicit emphasis on sentences containing minority-class tokens.
3 Resampling Strategy Design
For a sequence tagging task like NER, resampling cannot be as simple as what it is in classification and regression tasks, in which data points can be individually replicated, discarded, or synthesized. In NER, named entities cannot be resampled out of context. The surrounding context of named entities – albeit tokens from the irrelevant Other type – should be considered as well. Resampling named entities with context is a double-edged sword: preserving context will help NER models, but too much context increases the amount of non-entity tokens and aggravates the data imbalance problem. The goal of sentence-level resampling is to find the balance between too little and too much context accompanying named entities in complete sentences.
3.1 Sentence Importance Factors in NER
Intuitively, sentences that are worth resampling are those that are more important towards constructing a balanced NER dataset. We start by proposing factors that influence the importance of a sentence in resampling. These factors share the theoretical foundation of retrieval functions in information retrieval (Fang et al., 2004). A retrieval function evaluates the utility of a document with respect to the query terms it contains. By direct analogy, sentence importance score measures the utility of a sentence with respect to the entities it contains.
Count of entity tokens. Regardless of entity types, a sentence containing more entity tokens is more important than a sentence filled with nonentity tokens. This factor mirrors term frequency in retrieval functions (Salton and Buckley, 1988).
Rareness of entity type. The general idea of resampling for minority classes says that the rarer an entity type is, the more times we should resample sentences containing this type of entity. This factor mirrors inverse document frequency in retrieval functions (Salton and Buckley, 1988).
Density of tokens labeled as any entity. Including too much context can aggravate the imbalance problem. While the absolute count of entity tokens matters, the density of entity tokens in a sentence
(number of entity tokens compared to the length of a sentence) should also be concerned. The higher the density, the more important a sentence. This factor mirrors document length normalization in retrieval functions (Singhal et al., 1996).
Diminishing marginal utility. If one sentence contains twice as many tokens with a specific entity type as the other sentence with the same length, does that mean the first sentence is twice as important as the second? In reality, an entity may contain numerous tokens, or a sentence may include multiple entities of the same type. Twice as many tokens from the same entity type may not offer twice as much information (for the same reason why too many tokens from the Other type is not helpful). Therefore, as the number of tokens from the same entity type increases, they generate diminishing marginal utility to a sentence. This factor mirrors diminishing marginal gain of repeated query terms in retrieval functions (Fang et al., 2004).
3.2 Resampling Functions
Based on the above importance factors, we design a suite of functions  $f_{s} \in \mathbb{Z}^{+}$  to determine the number of times a sentence  $s$  should be resampled in a NER training set. These functions incorporate progressively more factors discussed previously.
In a given corpus, let us denote the set of all entity types except for the majority type Other as  $T$ . Let  $c(t,s)$  be the count of tokens with entity type  $t \in T$  in sentence  $s$ . We define the resampling function with respect to the smoothed count (sC) of all entity tokens as
$$
f _ {s} ^ {\mathrm {s C}} = 1 + \sum_ {t \in T} c (t, s). \tag {1}
$$
Here,  $\sum_{t\in T}c(t,s)$  is the total number of entity tokens in sentence  $s$ . ‘+1’ is to avoid removing entity-less sentences from the training set, in reminiscence of add-one smoothing in empirical probability estimates. It guarantees that all training sentences are resampled as least once. This smoothing-like process maintains consistency between training and test sets. If the training set contains entity-less sentences, it is highly likely that the test set will contain entity-less sentences as well.
The next function incorporates entity rareness factor. The rareness  $r_t$  of an entity type  $t \in T$  is measured as the self-information of the event that any token carries this type:
$$
r _ {t} = - \log_ {2} \frac {\sum_ {s \in S} c (t , s)}{N},
$$
where  $S$  is the set of all sentences in the training set, and therefore  $\sum_{s\in S}c(t,s)$  is the total number of tokens with entity type  $t$ .  $N$  is number of all tokens (including Other tokens) in the training set. By introducing the rareness of an entity type we propose another function called the smoothed resampling incorporating count and rareness (sCR):
$$
f _ {s} ^ {\mathrm {s C R}} = 1 + \left\lceil \sqrt {\sum_ {t \in T} r _ {t} \cdot c (t , s)} \right\rceil . \tag {2}
$$
Ceiling function  $\lceil \cdot \rceil$  ensures  $f_{s}^{\mathrm{sCR}} \in \mathbb{Z}^{+}$ . Square root is to slow down the increase of  $f_{s}^{\mathrm{sCR}}$  when an entity type  $t$  is extremely rare (when  $r_t$  is large).
According to the density factor in the previous section, the length of sentence  $s$  plays a role in determining the density of entity tokens within a sentence. Let  $l_{s}$  be the length of sentence  $s$  measured in number of tokens. We define the following function called the smoothed resampling incorporating count, rareness, and density (sCRD):
$$
f _ {s} ^ {\mathrm {s C R D}} = 1 + \left\lceil \frac {\sum_ {t \in T} r _ {t} \cdot c (t , s)}{\sqrt {l _ {s}}} \right\rceil . \tag {3}
$$
We use  $\sqrt{l_s}$  instead of  $l_{s}$  to slow down the decrease of  $f_{s}^{\mathrm{sCRD}}$  when a sentence is too long.
Lastly, we incorporate the diminishing marginal utility factor and propose a function called the normalized and smoothed resampling incorporating count, rareness, and density (nsCRD):
$$
f _ {s} ^ {\mathrm {n s C R D}} = 1 + \left\lceil \frac {\sum_ {t \in T} r _ {t} \cdot \sqrt {c (t , s)}}{\sqrt {l _ {s}}} \right\rceil . \tag {4}
$$
Here,  $\sqrt{c(t,s)}$  applies a sublinear increasing function on  $c(t,s)$  to implement the diminishing marginal utility when a sentence contains many tokens with the same type.
In summary, we proposed four functions for determining resampling frequencies for each sentence, representing four resampling methods.
Resampling should be a domain- and model-agnostic strategy in tackling data imbalance. Therefore, the goal of our experiments is to evaluate if the proposed resampling methods are effective in an extensive array of NER corpora and base models. Towards this goal, we apply the four resampling methods (together with baseline methods) on three representative NER models (each has two variants),
and evaluate the resulting models on four corpora from diverse domains.
4.1 Evaluation Metric
We use span-level strict-match macro-averaged F1 score as our main evaluation metric. Other is not viewed as an entity type. Macro-averaged metrics emphasize a balanced treatment of all entity types, which align with our main goal. See Appendix C for micro-averaged and per-entity-type results.
We compare the following baseline methods for dealing with data imbalance in NER.
Original corpus: training data untreated.
Balanced undersampling: We implement the algorithm proposed in (Akkasi et al., 2018) as a representative of sub-sentence-level resampling.
Data augmentation: The data augmentation techniques that includes all transformations as proposed in (Dai and Adel, 2020).
Focal loss(Lin et al., 2017), Dice loss (Li et al., 2019): We apply these loss functions on tokenwise predictions made by a softmax output layer. Note that they are not applicable to sequence-level predictions made by a CRF output layer.
sC, sCR, sCRD, nsCRD: the four resampling methods proposed in this work.
4.3 NER Corpora
We select four corpora from different domains. The first three are of small scale, representing bespoke NER tasks in practice where entity types are task-specific and annotation efforts are limited.
AnEM (Ohta et al., 2012): The Anatomical Entity Mention (AnEM) corpus consists of 500 documents selected randomly from citation abstracts and full-text papers concerning both health and pathological anatomy. With only  $3.91\%$  entity tokens and  $35.38\%$  sentences having any entity, this is a very imbalanced corpus in Table 1.
WNUT (Derczynski et al., 2017): This is a social domain corpus released in the 2017 Workshop on Noisy User-generated Text (W-NUT). It contains noisy user-generated texts found in social media, online review, crowdsourcing, web forums, clinical records, and language learner essays. This is another very imbalanced corpus in Table 1.
GMB subset (Bos et al., 2017; Kaggle, 2018): The Groningen Meaning Bank (GMB) corpus consists of public domain English texts with corresponding syntactic and semantic representations.
The GMB subset is extracted from the larger GMB 2.0.0 corpus which is built specially for NER.
To test the generalizability of our methods, we also include a standard NER benchmark dataset.
CoNLL (Sang and De Meulder, 2003): The CoNLL-2003 English news NER corpus.
For AnEM, WNUT, and CoNLL, we use their pre-existing training/test split. For GMB subset, we use 3:1 training/test split.
4.4 Base NER Models and Variants
To comprehensively evaluate the combinations of our upstream resampling strategies with many downstream sequence tagging models, we select the following models:
Shallow Model. We construct shallow NER models that use pretrained word embeddings as per-word feature vectors. We consider two variants: one using a softmax output layer making tokenwise predictions; the other using a CRF (conditional random fields) output layer making sequence-level predictions. Considering domains of the corpora, we select embeddings trained on biomedical literature (Huang et al., 2016), tweets (Glove-27B-twitter-27B), and Wikipedia+news (Glove-6B), for AnEM, WNUT, and datasets from general domain (GMB subset and CoNLL), respectively. All are 50-dimensional embeddings. CrfSuite is applied with default hyperparameters.
Bi-LSTM (Bidirectional Long Short-Term Memory). LSTM is a special recurrent neural network architecture in which the vanishing gradient problem can be effectively mitigated. Bi-LSTM consists of two LSTMs taking inputs in both forward and backward directions. Even though more recent models (e.g., GPT-2, BERT) are shown to outperform Bi-LSTM, it is still regarded as one of the most prevalent tools for solving sequence tagging problems. We implement two variants of Bi-LSTM: one with a softmax output layer making token-wise predictions; the other with a CRF decoding layer<sup>5</sup>, to ensure the local consistency of output tags. Different from the default hyperparameters, batch size and number of epochs are set to 32 and 20, respectively. Embeddings are used in the same way as in the shallow models above.
BERT (Bidirectional Encoder Representations from Transformers). BERT is widely re
<sup>2</sup>http://nlp.stanford.edu/data/glove.twitter.27B.zip
<sup>3</sup>http://nlp.stanford.edu/data/glove.6B.zip
<sup>4</sup>https://github.com/scrapinghub/python-crfsuite
$^{5}$ https://github.com/guillaumeengenthal/sequence_tagging
Shallow model	Bi-LSTM	BERT
Softmax	CRF	Softmax	CRF	Softmax	CRF
Original corpus	27.36	52.39	41.76	45.21	58.60	57.68
Balanced undersampling	24.17	52.06	21.22	26.19	62.59	63.03
Data augmentation	22.78	53.20	40.73	41.05	61.34	63.38
Focal loss	27.65	-	40.73	-	58.39	-
Dice loss	25.87	-	2.31	-	47.36	-
sC	30.39	52.13	44.69	47.55	62.86	65.47
sCR	30.94	53.38	48.37	48.90	65.22	62.41
sCRD	29.98	50.67	45.69	44.31	61.70	60.50
nsCRD	30.57	53.41	39.54	46.10	64.63	62.59
Table 2: Macro F1 scores on AnEM. The three NER models using either softmax or CRF output layer are reported. ' -' means that focal loss and Dice loss do not apply to CRF outputs. In each column, the highest F1-score is shown in boldface and the second highest is shown in underline.
Shallow model	Bi-LSTM	BERT
Softmax	CRF	Softmax	CRF	Softmax	CRF
Original corpus	14.56	4.31	18.27	18.46	37.67	39.98
Balanced undersampling	16.94	4.45	15.81	16.68	33.73	30.86
Data augmentation	11.05	4.49	20.10	10.06	35.03	36.08
Focal loss	14.25	-	17.73	-	39.20	-
Dice loss	15.74	-	15.26	-	31.62	-
sC	16.20	4.58	16.90	19.21	37.16	49.44
sCR	15.94	4.39	21.52	21.40	44.60	45.06
sCRD	15.95	4.94	17.15	19.80	43.82	42.34
nsCRD	16.06	4.62	18.31	23.71	41.65	41.71
Table 3: Macro-averaged F1-scores on the WNUT corpus. See the caption of Table 2 above for details.
Shallow model	Bi-LSTM	BERT
Softmax	CRF	Softmax	CRF	Softmax	CRF
Original corpus	28.71	41.42	45.73	44.91	50.20	54.32
Balanced undersampling	29.56	40.52	41.73	42.70	57.01	56.89
Data augmentation	27.53	41.24	47.63	49.76	54.78	55.25
Focal loss	28.47	-	41.25	-	53.66	-
Dice loss	33.42	-	40.16	-	52.10	-
sC	29.17	41.34	44.52	48.39	54.52	54.58
sCR	29.85	40.39	45.85	46.91	52.96	54.33
sCRD	30.99	41.54	44.60	48.07	52.72	55.12
nsCRD	29.37	41.76	46.16	45.69	55.14	54.60