SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 4: Macro-averaged F1-scores on the GMB subset corpus. See the caption of Table 2 above for details.
Shallow model	Bi-LSTM	BERT
Softmax	CRF	Softmax	CRF	Softmax	CRF
Original corpus	42.66	67.71	75.78	78.63	88.20	88.45
Balanced undersampling	42.55	66.77	66.36	76.54	86.91	86.75
Data augmentation	42.73	67.06	77.70	77.31	87.98	88.37
Focal loss	43.47	-	77.24	-	88.44	-
Dice loss	48.58	-	72.80	-	88.82	-
sC	42.20	66.84	76.82	75.03	88.94	88.81
sCR	42.22	66.63	77.72	74.13	88.36	89.11
sCRD	42.98	66.97	78.06	77.63	88.30	88.61
nsCRD	41.98	67.04	78.80	78.87	87.58	88.03
Table 5: Macro-averaged F1-scores on the CoNLL corpus. See the caption of Table 2 above for details.
garded as the most significant improvement in natural language processing. Its outstanding capability of learning contextualized word representations makes it the representative of advanced NER model in this work. Again, we implement two variants of BERT: one with a softmax output layer making token-wise predictions; the other with a CRF decoding layer<sup>6</sup>. Default hyperparameters are used. More implementation details are in Appendix A.
Macro-averaged F1-scores of different methods applied to four corpora and three base NER models are reported in Tables 2-5.
Our goal is not to compete with state-of-the-art methods on these corpora. Instead, we aim to present an interesting and underexplored problem (sentence-level resampling for NER) and a set of simple yet promising methods. In principle, our proposed resampling methods are model-agnostic and can provide an additional performance boost for a variety of NER models. We observe the following trends in Tables 2-5.
Overall performance of our resampling methods: Across Tables 2-5, our methods (sC, sCR, sCRD, nsCRD) generally performed well, achieving the highest or second highest F1-scores in almost every column (except for the condition 'Shallow model, Softmax' on CoNLL). Although no specific method consistently outperforms others in every condition, it is clear that sentence-level resampling is overall a promising approach to tackling the data imbalance problem in NER. The best resampling method depends on the specific base model, output layer, and corpus used. Just as the best hyperparameter values have to be empirically determined, so could be the most suitable resampling method. Fortunately, all our resampling methods are simple and straightforward, which allows for convenient experimentation.
Shallow vs. deep models: We observe a clear trend that shallow models using word embedding as features and softmax/CRF as the output layer underperform deep models such as Bi-LSTM and BERT. We view this as a sanity check. Bi-LSTMs and BERT can learn word representations that account for long-distance dependencies, and BERT should be even more powerful with contextual word representations pretrained on massive texts.
Softmax vs. CRF output layer: Using the
same base model, CRF output layer often (but not always) outperforms softmax output layer. The performance gap is larger on shallow models and small corpora (AnEM, WNUT, GMB) than on deep models and large corpus (CoNLL). Indeed, BiLSTM and BERT are capable of learning word representations that account for long-distance word dependencies, reducing the benefit of tag dependencies offered by a CRF layer. Similar observation was made by previous work (Devlin et al., 2018). An exception is the combination (WNUT, Shallow model), where the CRF layer suffered from severe overfitting caused by noisy text and extremely imbalanced data distribution in WNUT corpus.
Small vs. large corpus: On small corpora (AnEM, WNUT, GMB subset), our resampling methods usually outperform the original corpus baseline by a big margin. These benefits become less salient on large corpus (CoNLL). This implies that our methods are especially effective when the corpus is small and annotations are few. As corpus size gets large, even rare entity types are covered by many examples and therefore sufficiently trained.
Sub-sentence resampling and data augmentation: Sub-sentence resampling (balanced undersampling) has large variance in its performance. In some cases it gives the highest gain (GMB subset, BERT model), and in other cases it performs worse than just using the original corpus (all corpora, Bi-LSTM models). It suggests that sub-sentence resampling is highly sensitive to the corpus and model choice. Data augmentation also shows high variance in its performance. It gives the highest gain on (GMB subset, Bi-LSTM model), and performs worse than the original corpus on (WNUT, BERT model). Sentences generated by data augmentation generally have correct syntax but garbled semantics (e.g., one entity is replaced by another same-type, out-of-context entity). The nonsensical sentences may confuse NER models. In contrast, whole-sentence resampling methods give more stable improvements over the original corpus baseline largely because they preserve the naturalness of resampled sentences.
Focal loss and Dice loss: These loss functions are applicable only on pointwise predictions made by the softmax output layer. A major trend is that their performance tend to be unreliable across scenarios. We attribute this behavior to the difficulty in optimizing these losses. For shallow models, we optimize them by feeding gradients of either
$^{6}$ https://github.com/kyzhouhzau/BERT-NER
loss function (see Appendix B for their derivation) into a L-BFGS optimizer (Liu and Nocedal, 1989) in Scikit-Learn. As shown in the (Shallow model, Softmax) column of GMB subset and CoNLL corpora, the two loss functions (especially the Dice loss) performed well. For deep models (Bi-LSTM and BERT), we rely on TensorFlow's automatic differentiation and Adam gradient descent optimizer (Kingma and Ba, 2014) because manually deriving gradients for deep models is infeasible. The two loss functions sometimes give poor performance. The Bi-LSTM model with Dice loss failed completely on AnEM (F1-score: 2.31). A possible explanation is that Dice loss is non-convex and it may be difficult for first-order optimizers in current deep learning toolkits (e.g. Adam in TensorFlow) to find high-quality local minima than second-order methods like L-BFGS.
Precision and recall: To illustrate performance changes in terms of precision and recall, Figure 1 visualizes the changes before and after resampling as displacement vectors in precision-recall plots with F1-score contour lines. Some arrows are pointing to the upper right corner of the plots, indicating the associated methods improve F1-score by improving both precision and recall. Other arrows point to the upper left, indicating the associated methods increase recall at the sacrifice of precision. In this case, most of our methods improve macroaveraged precision and recall of the BERT model on WNUT. See Appendix C.2 for more details.
4.6 Effect on Training Corpus Size
Table 6 shows the effect of training corpus size as a result of resampling or data augmentation. These factors are the average across four corpora.
The balanced undersampling method drops tokens from sentences, and therefore reduces training corpus size. Data augmentation method doubles the corpus size as many sentences are paraphrased into multiple versions. Our proposed methods increase corpus size by a slightly larger factor because sentences that contain rare entity types are resampled multiple times. Although increased training corpus size leads to increased training time, note that our methods are especially suitable for scenarios where the annotated corpus is small and hence the training time is still relatively short.
Figure 1: Displacement vectors of F1-scores in precision-recall plots for WNUT corpus. 'BUS': balanced undersampling. 'DA': data augmentation. All models use softmax output layer. The downward curves are contours of F1-scores in the precision-recall space. Each NER model (Shallow, Bi-LSTM, BERT) is associated with a cluster of vectors sharing the same starting point in the space, which represents the performance on the original corpus.
Methods	Size increase factor
Original corpus	1.00
Balanced undersampling	0.32
Data augmentation	2.00
sC	3.80
sCR	4.60
sCRD	3.91
nsCRD	2.82
Table 6: Effect of data resampling/augmentation methods on training corpus size. The factors are averaged across four evaluated corpora.
Our proposed sentence resampling methods generalize well across diverse NER corpora and models. They enjoy the following advantages:
Model-agnostic: Since resampling only manipulates datasets and not models, the proposed methods can be directly applied to any NER model, requiring no knowledge of its functioning or any change to it. Resampling is also more convenient than cost-sensitive learning as the latter still requires changing the model training process.
Domain-agnostic: Compared with data preprocessing methods such as data augmentation, sentence-level resampling methods are simple and do not require domain- or language-specific manipulations such as synonym replacement, saving practitioners from excessive data engineering.
Note that data augmentation and sentence-level resampling (and resampling methods in general) are complementary methods for improving NER model training. Data augmentation improves the semantic richness of training instances by expanding the coverage of training data in the input feature space, while sentence-level resampling refines the importance weighting of training instances by bridging the gap between the training objective and evaluation metrics. Therefore, they work in orthogonal directions. This points to a promising direction for future work: to explore the two line of methods in combination rather than in competition.
Various other avenues exist for future work. First, further theoretical and empirical research can explore more effective resampling functions that deliver consistently better performance across corpora and base NER models. Second, more corpora and models can be examined under these resampling strategies to evaluate their generalizability. Third, the variance of performance in different scenarios may potentially relate to characteristics of specific corpora. Future research may seek for corpora-level statistics that can assist practitioners in selecting the appropriate resampling methods.
Acknowledgments. The authors thank anonymous reviewers for their useful comments. This work is partially supported by the National Institute of Health under the grant number R01DA053028.
A Akkasi and E Varoglu. 2019. Improvement of chemical named entity recognition through sentence-based random under-sampling and classifier combination. Journal of AI and Data Mining, 7(2):311-319.
Abbas Akkasi. 2018. Sentence-based undersampling for named entity recognition using genetic algorithm. *Iran Journal of Computer Science*, 1(3):165-174.
Abbas Akkasi, Ekrem Varoglu, and Nazife Dimililer. 2018. Balanced undersampling: a novel sentence-based undersampling method to improve recognition of named entities in chemical and biomedical text. Applied Intelligence, 48(8):1965-1978.
Gökhan BakIr, Thomas Hofmann, Bernhard Schölkopf, Alexander J Smola, and Ben Taskar. 2007. Predicting structured data. MIT press.
Johan Bos, Valerio Basile, Kilian Evang, Noortje Venhuizen, and Johannes Bjerva. 2017. The Groningen meaning bank. In Nancy Ide and James Pustejovsky, editors, Handbook of Linguistic Annotation, volume 2, pages 463-496. Springer.
Xiang Dai and Heike Adel. 2020. An analysis of simple data augmentation for named entity recognition. arXiv preprint arXiv:2010.11683.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Hui Fang, Tao Tao, and ChengXiang Zhai. 2004. A formal study of information retrieval heuristics. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 49-56.
Mila Grancharova, Hanna Berg, and Hercules Dalianis. 2020. Improving named entity recognition and classification in class-imbalanced swedish electronic patient records through resampling. In *Eighth Swedish Language Technology Conference (SLTC)*. Förlag Göteborgs Universitet.
Haibo He and Edwardo A Garcia. 2009. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9):1263-1284.
Sam Henry, Kevin Buchan, Michele Filannino, Amber Stubbs, and Ozlem Uzuner. 2020. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. Journal of the American Medical Informatics Association, 27(1):3-12.
Jian Huang, Keyang Xu, and VG Vinod Vydiswaran. 2016. Analyzing multiple medical corpora using word embedding. In 2016 IEEE International Conference on Healthcare Informatics (ICHI), pages 527-533. IEEE.
Abdul Majeed Issifu and Murat Can Ganiz. 2021. A simple data augmentation method to improve the performance of named entity recognition models in medical domain. In 2021 6th International Conference on Computer Science and Engineering (UBMK), pages 763-768. IEEE.
Brendan Juba and Hai S Le. 2019. Precision-recall versus accuracy and the role of large data sets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4039-4048.
Kaggle. 2018. Annotated GMB corpus. https://www.kaggle.com/shoumikgoswami/annotated-gmb-corpus. Accessed: 2022-05-01.
Sarvnaz Karimi, Alejandro Metke-Jimenez, Madonna Kemp, and Chen Wang. 2015. Cadec: A corpus of adverse drug event annotations. Journal of biomedical informatics, 55:73-81.
Gary King and Langche Zeng. 2001. Logistic regression in rare events data. Political analysis, 9(2):137-163.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Jasper Kuperus, Cor J Veenman, and Maurice van Keulen. 2013. Increasing ner recall with minimal precision loss. In 2013 European Intelligence and Security Informatics Conference, pages 106-111. IEEE.
Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2019. Dice loss for data-imbalanced nlp tasks. arXiv preprint arXiv:1911.02855.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dólar. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
Dong C Liu and Jorge Nocedal. 1989. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):503-528.
Xiaoguang Lu and Anil K Jain. 2003. Resampling for face recognition. In International Conference on Audio-and Video-based Biometric Person Authentication, pages 869-877. Springer.
Xinnian Mao, Wei Xu, Yuan Dong, Saike He, and Haila Wang. 2007. Using non-local features to improve named entity recognition recall. In Proceedings of the 21st Pacific Asia Conference on Language, Information and Computation, pages 303-310.
Alejandro Moreno, Andrea Esuli, and Fabrizio Sebastiani. 2016. Distributional random oversampling for imbalanced text classification. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 805-808.
Tomoko Ohta, Sampo Pyysalo, Jun'ichi Tsujii, and Sophia Ananiadou. 2012. Open-domain anatomical entity mention detection. In Proceedings of the workshop on detecting structure in scholarly discourse, pages 27-36.
Weischedel Ralph, Palmer Martha, Marcus Mitchell, Hovy Eduard, Pradhan Sameer, Ramshaw Lance, Xue Nianwen, Taylor Ann, Kaufman Jeff, Franchini Michelle, El-Bachouti Mohammed, Belvin Robert, and Houston Ann. 2013. Ontonotes release 5.0. https://doi.org/10.35111/xmhb-2b84.
Gerard Salton and Christopher Buckley. 1988. Termweighting approaches in automatic text retrieval. Information processing & management, 24(5):513-523.
Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Amit Singhal, Gerard Salton, Mandar Mitra, and Chris Buckley. 1996. Document length normalization. Information Processing & Management, 32(5):619-633.
Noah A Smith. 2011. Linguistic structure prediction. Synthesis lectures on human language technologies, 4(2):1-274.
Katrin Tomanek and Udo Hahn. 2009. Reducing class imbalance during active learning for named entity annotation. In Proceedings of the fifth international conference on Knowledge capture, pages 105-112.
Rui Wang and Ricardo Henao. 2021. Unsupervised paraphrasing consistency training for low resource named entity recognition. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5303-5308.
A.1 Software and Hardware Environment
All the deep learning models are implemented in Tensorflow 1.12.0 environment.
Softmax regression (or multinomial logistic regression) model is from Scikit-Learn package in version 0.23.2. The CRF model is implemented by the package sklearn-crfsuite in version of 0.3.6.
Data resampling and CRF training/evaluation were performed on 2.60 GHz Intel CPUs and 8GB RAM. Bi-LSTM and BERT training/evaluation were performed on GPUs (GeForce GTX1080 8GB and Tesla V100 16GB).
A.2 Hyperparameters for Machine Learning Models
For shallow models and BERT, all hyperparameters are set by default. For details of them, please see documents of sklearn, crfsuite and BERT-NER.
For Bi-LSTM, we adjust a few of parameters as there are some drawbacks of the default settings: 20 is not a commonly used number for batch size, and loss of Bi-LSTM model fails to converge under some circumstances. So we set them to 32 and 20, instead of default values 20 and 15. Other hyperparameters are applied according to default settings.
For the fairness in the comparison, we do not alter any hyperparameters while switching resampling methods and loss functions without changing dataset and models. We believe that it is totally appropriate in the process of comparing, despite that better performance of specific methods may be obtained after tuning hyperparameters, which beyond the scope of this exploring research.
A.3 Hyperparameters for Loss Functions
There are two hyperparameters in the focal loss and Dice Loss, determining converging speed and smooth degree. For focal loss, we set  $\gamma$  to 2, as what authors of (Lin et al., 2017) recommend. According to (Li et al., 2019), it is appropriate to set  $\gamma$  of Dice loss to 1 for the purpose of smoothing. In our implementation of loss function in shallow model, this setting is found effective. However, while using it in deep learning model, its effectiveness cannot be ensured. Hence, we adopt another setting of  $\gamma = 10^{-5}$  in the tensor computing and obtain better results compared with those obtained with a larger  $\gamma$ .
B Derivation of Loss Function Gradients for Softmax Regression
When using the shallow model with softmax output layer and focal/Dice loss functions, we optimize the model parameters by the quasi-Newton method L-BFGS provided by Python Scikit-Learn. This approach requires us to provide the gradients of current model parameters. Below we show our derivation of these gradients.
Notations and Preliminaries. Scalar values are denoted by non-  $\mathbf{bold}$  , lowercase letters such as  $x$  Row vectors are denoted by bold, lowercase letters such as  $\mathbf{x}$  .Matrices are denoted by bold, uppercase letters such as  $\mathbf{X}$
Softmax regression has the following components:
Feature vector:  $\mathbf{x} \in \mathbb{R}^m, \mathbf{x} = [x_1, \dots, x_i, \dots, x_m]$ .
Label vector:  $\mathbf{y} \in \{0,1\}^k, \mathbf{y} = [y_1, \dots, y_j, \dots, y_k]$ . If the ground truth is the  $c$ -th class,  $1 \leq c \leq k$ , then  $y_c = 1$ , and  $y_j = 0$  if  $j \neq c$ .
- Weight vector for the  $j$ -th class:  $\mathbf{w}_j \in \mathbb{R}^m, \mathbf{w}_j = [w_{j1}, \dots, w_{ji}, \dots, w_{jm}]$ .
Weight matrix  $\mathbf{W} \in \mathbb{R}^{m \times k}$ ,  $\mathbf{W} = [\mathbf{w}_1^\top, \dots, \mathbf{w}_j^\top, \dots, \mathbf{w}_k^\top]$ . “ $\top$ ” is the transpose operation.  $\mathbf{w}^\top$  is the transpose of  $\mathbf{w}$ , which is a column vector.
- Bias for the  $j$ -th class:  $b_{j} \in \mathbb{R}$ .
- Predicted probability vector:  $\mathbf{p} \in [0,1]^k$ ,  $\mathbf{p} = [p_1, \dots, p_j, \dots, p_k]$ .
$$
\begin{array}{l} p _ {j} = \Pr (y _ {j} = 1 | \mathbf {x}) (5) \\ = \frac {\exp \left(\left\langle \mathbf {w} _ {j} , \mathbf {x} \right\rangle + b _ {j}\right)}{\sum_ {j ^ {\prime} = 1} ^ {k} \exp \left(\left\langle \mathbf {w} _ {j ^ {\prime}} , \mathbf {x} \right\rangle + b _ {j ^ {\prime}}\right)} (6) \\ \end{array}
$$
$\langle \mathbf{w},\mathbf{x}\rangle$  is the inner product of vector w and vector x.
One can verify that the partial derivative of  $p_c$  with respect to  $w_{ji}$ , the weight of the  $j$ -th class, the  $i$ -th dimension, is the following:
$$
\frac {\partial p _ {c}}{\partial w _ {j i}} = \left[ \mathbf {1} \{j = c \} - p _ {j} \right] p _ {c} x _ {i} \tag {7}
$$
B.1 Focal Loss Gradient
Suppose the ground truth is the  $c$ -th class for a given example  $\mathbf{x}$ .
$$
L _ {F L} (\mathbf {x}, \mathbf {y}) = - (1 - p _ {c}) ^ {\gamma} \log p _ {c} \tag {8}
$$
$$
\begin{array}{l} \frac {\partial L _ {F L} (\mathbf {x} , \mathbf {y})}{\partial w _ {j i}} (9) \\ = - \frac {\partial}{\partial p _ {c}} \left[ (1 - p _ {c}) ^ {\gamma} \log p _ {c} \right] \cdot \frac {\partial p _ {c}}{\partial w _ {j i}} (10) \\ = - \left[ - \gamma (1 - p _ {c}) ^ {\gamma - 1} \log p _ {c} + \frac {(1 - p _ {c}) ^ {\gamma}}{p _ {c}} \right] \cdot \frac {\partial p _ {c}}{\partial w _ {j i}} (11) \\ = - \left[ - \gamma (1 - p _ {c}) ^ {\gamma - 1} \log p _ {c} + \frac {(1 - p _ {c}) ^ {\gamma}}{p _ {c}} \right] \\ \cdot \left[ \mathbf {1} \{j = c \} - p _ {j} \right] p _ {c} x _ {i} (12) \\ = - \left[ - \gamma p _ {c} \left(1 - p _ {c}\right) ^ {\gamma - 1} \log p _ {c} + \left(1 - p _ {c}\right) ^ {\gamma} \right] \\ \cdot \left[ \mathbf {1} \{j = c \} - p _ {j} \right] x _ {i} (13) \\ = a _ {c} \left[ p _ {j} - \mathbf {1} \{j = c \} \right] x _ {i} (14) \\ \end{array}
$$
Here we set
$$
a _ {c} = - \gamma p _ {c} (1 - p _ {c}) ^ {\gamma - 1} \log p _ {c} + (1 - p _ {c}) ^ {\gamma} \tag {15}
$$
to reduce notational clutter.  $a_{c}$  has nothing to do with  $i$  or  $j$ ; it only has to do with  $c$ , the index of the ground truth label for the training example  $\mathbf{x}$ . When  $\gamma = 0$ ,  $a_{c} = 1$ . When  $\gamma > 0$ ,  $a_{c}$  decreases when  $p_{c}$  increases from 0 to 1. This means the gradient for an easy example (when  $p_{c}$  is close to 1) have a smaller magnitude than the gradient for a hard example (when  $p_{c}$  is close to 0).
Generalizing the scalar gradient in Equation (14) to matrix gradient, we have
$$
\frac {\partial L _ {F L} (\mathbf {x} , \mathbf {y})}{\partial \mathbf {W}} = a _ {c} \cdot \mathbf {x} ^ {\top} (\mathbf {p} - \mathbf {y}). \tag {16}
$$
The shape of  $a_{c}\cdot \mathbf{x}^{\top}(\mathbf{p} - \mathbf{y})$  is  $m\times k$ , the same shape as  $\mathbf{W}$ .
An important note is that here  $a_{c}$  is specific to that single example  $\mathbf{x}$ , which has ground truth label  $y_{c} = 1$ . If we have  $n$  different training examples  $\mathbf{x}^{(1)}, \dots, \mathbf{x}^{(n)}$ , then every example will have a different  $a_{c}$  value:  $a_{c}^{(1)}, \dots, a_{c}^{(n)}$ . Let's create a diagonal matrix  $\mathbf{A}_{c} \in \mathbb{R}^{n \times n}$ ,  $\mathbf{A}_{c} = \text{diag}(a_{c}^{(1)}, \dots, a_{c}^{(n)})$ .
If we have  $n$  training examples, then the feature matrix  $\mathbf{X} \in \mathbb{R}^{n \times m}$ , the label matrix  $\mathbf{Y} \in \{0,1\}^{n \times k}$ , and the predicted probability matrix  $\mathbf{P} \in [0,1]^{n \times k}$ . We have:
$$
\frac {\partial L _ {F L} (\mathbf {X} , \mathbf {Y})}{\partial \mathbf {W}} = \mathbf {X} ^ {\top} \mathbf {A} _ {c} (\mathbf {P} - \mathbf {Y}). \tag {17}
$$