SlowGuess/NAACL_2022_OCR huggingface dataset:
As mentioned in Section 2.2, we defined two core hyper-parameters for our self-augmentation methods, one for TS (i.e.,  $\gamma$ ) and the other for mixup (i.e.,  $\lambda$ ). Specifically, we set  $\gamma = 20\%$  and  $\lambda$  by sampled from the  $\mathrm{Beta}(\alpha, \alpha)$  distribution with  $\alpha = 7$ , where the details will be shown in the analysis section. Meanwhile, we conduct the augmentation up to 5 times corresponding to the original training data.
Evaluation We conduct each experiment by 5 times and report the average F1 score. The best-performing model on the development set is then used to evaluate on the test set.
$^{4}$ https://github.com/facebookresearch/higher
<sup>5</sup>https://dumps.wikimedia.org/
The main results are presented in Table 1, 2 and 3, verifying the effectiveness of our method under the low-resource setting and the standard full-scale setting, respectively. Since Weibo is a small-scale dataset, we do not consider its partial training set for the low-resource setting.
Low-Resource Setting We randomly sample  $5\%$ ,  $10\%$ , and  $30\%$  of the original training data from OntoNotes and CoNLL03 for simulation studies. Table 1 shows the results where F1 scores of the baseline, +TS, +Mixup, and +Both are reported. We can observe that: (1) the baseline performance will drop significantly as the size of training data get reduced gradually, which demonstrates the performance of the supervised NER model relies heavily on the scale of the labeled training data. (2) although the number of training examples has increased, vanilla self-augmentation (without meta reweighting) might degrade the model performance due to potentially unreliable pseudo-labeled examples. The meta reweighting strategy helps to adaptively weight the augmented examples during training, which combats the negative impact and leads to a stable and positive performance boost.
In addition, as the scale of the training data decreases, the effectiveness of the augmentation methods can be more significant, indicating that our self-augmentation methods are highly beneficial for the low-resource settings, and the two-stage combination of the two heterogeneous methods can yield better performance consistently.
Full-Scale Setting Table 2 and 3 show the results using full-scale training data. The results demonstrate that our baseline model is already strong. The model after vanilla augmentation could perform slightly worse since each training example is treated equally even if it is noisy. This also implies our meta reweighting makes great sense. Furthermore, our final model (+Both) can further achieve performance gains by integrating these self-augmentation methods with the meta reweighting mechanism. The overall trend is similar to the low-resource setting, but the gains are relatively smaller when the training data is sufficient. That may be attributed that the size of training data is large enough to narrow the performance gap between the baseline and augmented models. It also suggests that our method does not hurt the model performance even when using enough training data.
Comparison with Previous Work We also compare our method with previous representative SOTA work, where all referred systems exploit the pre-trained BERT model. As shown, compared to Dai and Adel (2020) and Chen et al. (2020), our method either outperforms or performs on par with theirs when using limited training data. For Chen et al. (2020), the pure mixup performs slightly better due to the well-designed example sampling strategy, but our overall framework outperforms theirs. Moreover, our method can match the performance of the semi-supervised setting that uses additional 10K unlabeled training data. Besides, our final model, without utilizing much external knowledge, can achieve very competitive results on the full training set in comparison to most previous systems.
In this subsection, we further conduct detailed experimental analyses on the CoNLL03 dataset for a better understanding of our method. Our main concern is on the low-resource setting, therefore the models based on  $5\%$ ,  $10\%$  and  $30\%$  of the original training data are our main focus.
Augmentation Times The size of augmented examples is an essential factor in final model performance. Typically, we examine the  $5\%$  CoNLL03 training data. As illustrated in Figure 3, the larger pseudo examples can obtain better performance in a certain range. However, as the times of augmentation increases, the uptrend of performance slows down. The improvement tends to be stable when the pseudo samples are increased to about 5 times the original training data. Excessively increasing the augmentation times does not necessarily bring consistent performance improvement. And we select an appropriate value for training data of different sizes from a range [1, 8].
Influence of  $\gamma$  for Token Substitution Regarding our TS strategy, we take both NWS and EMS into account simultaneously. The two parts are blended by a percentage parameter  $\gamma$ , namely  $\gamma$  for EMS and  $1 - \gamma$  for NWS. Here we examine the influence of  $\gamma$  in the sole self-augmentation model by TS. Figure 4 shows the results, where  $\gamma = 0$  and  $\gamma = 100\%$  denote the model with only NWS and EMS, respectively. As shown, our model can achieve the overall better performance when  $\gamma = 20\%$ , indicating that both of them are helpful
Figure 3: Influence of augmentation times for  $5\%$  training data of CoNLL03. Times=0 means original training data without any augmentation.
Figure 4: Performance against different EMS rate  $\gamma$ .
for the TS strategy, and NWS can be slightly better. One possible reason is that the entity words in original training examples are relatively sparse (i.e., the 'O' label is dominant), allowing the NWS to produce more diverse pseudo examples.
Mixup Parameter We further inspect the model with the mixup strategy alone so as to understand the important factors of the mixup model. First, we analyze the influence of the mixing parameter  $\alpha$ . As depicted in Figure 5, we can see that  $\alpha$  indeed affects the effectiveness of the mixup method greatly. Considering the feature of Beta distribution, the sampled  $\lambda$  will be more concentrated around 0.5 as the  $\alpha$  value becomes large, resulting in a relatively balanced weight between the mixed example pairs. The model performance remains stable when  $\alpha$  is around 7. Second, we study where to conduct the mixup operation since there are two main options in our framework, i.e., the hidden representations of either the BERT or BiLSTM for linear interpolation. Table 5 reports the comparison results, demonstrating the former is a better choice.
Case Study To further understand the effectiveness of the meta-reweighting mechanism, we present several high-quality and low-quality examples in Table 6. As shown, the difference between the positive and negative examples for TS could be
Figure 5: Performance against different mixing parameter by  $\mathrm{Beta}(\alpha, \alpha)$  distribution.
Method	5%	10%	30%
Baseline	85.12	87.11	89.24
Mixing on BiLSTM	86.15	87.66	89.41
Mixing on BERT	86.33	88.03	89.75
Table 5: Performance comparison of the mixup strategy on BERT or BiLSTM layer.
reflected in the syntactic and semantic validity of the augmented examples. Similarly, for the mixup, it seems that the valid example pairs are more likely to generate positive augmented examples.
In recent years, research on NER has concentrated on either enriching input text representations (Zhang and Yang, 2018; Nie et al., 2020b; Ma et al., 2020) or refining model architectures with various external knowledge (Zhang and Yang, 2018; Ye and Ling, 2018; Li et al., 2020a; Xuan et al., 2020; Li et al., 2020b; Yu et al., 2020; Shen et al., 2021). Particularly, NER model, with the aid of large pretrained language models (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019), has achieved impressive performance gains. However, these models mostly depend on rich manual annotations, making it hard to cope with the low-resource challenges in real-world applications. Instead of pursuing a sophisticated model architecture, in this work, we exploit the BiLSTM-CRF model coupled with the pre-trained BERT as our basic model structure.
Self-augmentation methods have been widely investigated in various NLP tasks (Zhang et al., 2018; Wei and Zou, 2019; Dai and Adel, 2020; Zeng et al., 2020; Ding et al., 2020). The mainstream methods can be broadly categorized into three types: (1) token substitution (Kobayashi, 2018; Wei and Zou, 2019; Dai and Adel, 2020; Zeng et al., 2020), which performs local substitution for a given sentence, (2) paraphrasing (Kumar
Augmentation	Examples
Original	[Diana]PER met [Will Carling]PER at an exclusive gymnasium in [London]LOC.
Positive TS	[Freddy Pinas]PER invited [John Marzano]PER at an available gym in [UK]LOC.
Negative TS	[Tim Henman]PER visit [Simpson]PER at an avail-able room in [NICE]LOC.
Positive Mixup	French 1997 budget due around September 10 - Juppe.
Jewish 1999 deficit due about October 20 M.Atherton.
Negative Mixup	Olympic champion Agassi meets Karim Alami of Morocco in the first round.
Olympic champion Nathalie Lancien of France also missed the winning attack.
Table 6: Case study on positive and negative augmentation with respect to the TS and mixup.
et al., 2019; Xie et al., 2020; Zhang et al., 2020b), which involves sentence-level rewriting without significantly changing the semantics, and (3) mixup (Zhang et al., 2018; Chen et al., 2020; Sun et al., 2020), which carries out the feature-level augmentation. As a data-agnostic augmentation technique, mixup can help improve the generalization and robustness of our neural model acting as an useful regularizer (Verma et al., 2019). For NER, token substitution and mixup are very suitable and have been exploited successfully with specialized efforts (Dai and Adel, 2020; Chen et al., 2020; Zeng et al., 2020), while the paraphrasing strategy may result in structure incompleteness and token-label inconsistency, thus there has not been widely concerned yet. In this work, we mainly investigate the token substitution and mixup techniques for NER, as well as their integration. Despite the success of various self-augmentation methods, quality control may be an issue easily overlooked by most methods.
Many previous studies have explored the example weighting mechanism in domain adaption (Jiang and Zhai, 2007; Wang et al., 2017; Osumi
et al., 2019). Xia et al. (2018) and Wang et al. (2019) looked into the example weighting methods for cross-domain tasks. Ren et al. (2018) adapted the MAML algorithm (Finn et al., 2017) and proposed a meta-learning algorithm to automatically weight training examples of the noisy label using a small unbiased validation set. Inspired by their work, we extend the meta example reweighting mechanism to the NER task, which is exploited to adaptively reweight mini-batch augmented examples during training. The main purpose is to mitigate the potential noise effects brought by the self-augmentation techniques, advancing a noiserobust model, especially in low-resource scenarios.
In this paper, we re-examine two heterogeneous self-augmentation methods (i.e., TS and mixup) for NER, extending them into more unrestricted augmentations without heuristic constraints. We further exploit a meta reweighting strategy to alleviate the potential negative impact of noisy augmented examples introduced by the aforementioned relaxation. Experiments conducted on several benchmarks show that our self-augmentation methods along with the meta reweighting mechanism are very effective in low-resource settings, and still work when enough training data is used. The combination of the two methods can lead to consistent performance improvement across all datasets. Since our framework is general and does not rely on a specific model backbone, we will further investigate other feasible model structures.
We thank the valuable comments of all anonymous reviewers. This work is supported by grants from the National Natural Science Foundation of China (No. 62176180).
Darina Benikova, Chris Biemann, Max Kisselew, and Sebastian Padó. 2014. Germeval 2014 named entity recognition shared task: Companion paper. the KONVENS GermEval Shared Task on Named Entity Recognition.
Jiaao Chen, Zhenghui Wang, Ran Tian, Zichao Yang, and Diyi Yang. 2020. Local additivity based data augmentation for semi-supervised NER. In Proceedings of EMNLP, November 16-20, 2020, pages 1241-1251.
Jason P. C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional lstm-cnns. TACL, pages 357-370.
Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc V. Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of EMNLP, October 31 - November 4, 2018, pages 1914-1925.
Xiang Dai and Heike Adel. 2020. An analysis of simple data augmentation for named entity recognition. In Proceedings of COLING, December 8-13, 2020, pages 3861-3867.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, June 2-7, 2019, pages 4171-4186.
Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kru-engkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si, and Chunyan Miao. 2020. DAGA: Data augmentation with a generation approach for low-resource tagging tasks. In Proceedings of EMNLP, pages 6045-6057.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of ICML, 6-11 August 2017, volume 70, pages 1126-1135.
Joseph Fisher and Andreas Vlachos. 2019. Merge and label: A novel neural network architecture for nested NER. In Proceedings of ACL, July 28- August 2, 2019, pages 5840-5850.
Hongyu Guo, Yongyi Mao, and Richong Zhang. 2019. Augmenting data with mixup for sentence classification: An empirical study. CoRR, abs/1905.08941.
James Hammerton. 2003. Named entity recognition with long short-term memory. In Proceedings of HLT-NAACL, May 31 - June 1, 2003, pages 172-175.
Dou Hu and Lingwei Wei. 2020. SLK-NER: exploiting second-order lexicon knowledge for chinese NER. In SEKE, July 9-19, 2020, pages 413-417.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. CoRR, abs/1508.01991.
Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in NLP. In Proceedings of ACL, pages 264-271.
Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021. AEDA: an easier data augmentation technique for text classification. In *Findings of EMNLP*, 16-20 November, 2021, pages 2748-2754.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of NAACL-HLT, June 1-6, 2018, pages 452-457.
Ashutosh Kumar, Satwik Bhattachamishra, Manik Bhandari, and Partha P. Talukdar. 2019. Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation. In Proceedings of NAACL-HLT, June 2-7, 2019, pages 3609-3619.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-HLT, June 12-17, 2016, pages 260-270.
Xiaonan Li, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020a. FLAT: chinese NER using flat-lattice transformer. In Proceedings of ACL, July 5-10, 2020, pages 6836-6842.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020b. A unified MRC framework for named entity recognition. In Proceedings of ACL, July 5-10, 2020, pages 5849-5859.
Wei Liu, Xiyan Fu, Yue Zhang, and Wenming Xiao. 2021. Lexicon enhanced chinese sequence labeling using BERT adapter. In Proceedings of ACL/IJCNLP, August 1-6, 2021, pages 5847-5858.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of ICLR 2019, May 6-9, 2019.
Ruotian Ma, Minlong Peng, Qi Zhang, Zhongyu Wei, and Xuanjing Huang. 2020. Simplify the usage of lexicon in chinese NER. In Proceedings of ACL, July 5-10, 2020, pages 5951-5960.
Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In Proceedings of ACL, August 7-12, 2016, pages 1064-1074.
Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for chinese character representations. In Proceedings of NeurIPS, December 8-14, 2019, pages 2742-2753.
Xue Mengge, Bowen Yu, Tingwen Liu, Yue Zhang, Erli Meng, and Bin Wang. 2020. Porous lattice transformer encoder for Chinese NER. In Proceedings of COLING, pages 3831-3841.
Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NeurIPS, pages 3111-3119.
Yuyang Nie, Yuanhe Tian, Yan Song, Xiang Ao, and Xiang Wan. 2020a. Improving named entity recognition with attentive ensemble of syntactic information. In Findings of EMNLP 2020, 16-20 November 2020, pages 4231-4245.
Yuyang Nie, Yuanhe Tian, Xiang Wan, Yan Song, and Bo Dai. 2020b. Named entity recognition for social media texts with semantic augmentation. In Proceedings of EMNLP, November 16-20, 2020, pages 1383-1391.
Kosuke Osumi, Takayoshi Yamashita, and Hironobu Fujiyoshi. 2019. Domain adaptation using a gradient reversal layer with instance weighting. In 16th International Conference on Machine Vision Applications, May 27-31, 2019, pages 1-5.
Nanyun Peng and Mark Dredze. 2015. Named entity recognition for chinese social media with jointly trained embeddings. In Proceedings of EMNLP, September 17-21, 2015, pages 548-554.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, October 25-29, 2014, pages 1532-1543.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, June 1-6, 2018, pages 2227-2237.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In Proceedings of CoNLL, August 8-9, 2013, pages 143-152.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Proceedings of EMNLP-ConLL, July 13, 2012, pages 1-40.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2018. Learning to reweight examples for robust deep learning. In Proceedings of ICML, July 10-15, 2018, volume 80, pages 4331-4340.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of CoNLL, May 31 - June 1, 2003, pages 142-147.
Yongliang Shen, Xinyin Ma, Zeqi Tan, Shuai Zhang, Wen Wang, and Weiming Lu. 2021. Locate and label: A two-stage identifier for nested named entity recognition. In Proceedings of ACL/IJCNLP, August 1-6, 2021, pages 2782-2794.
Lichao Sun, Congying Xia, Wenpeng Yin, Tingting Liang, Philip S. Yu, and Lifang He. 2020. Mixup-transformer: Dynamic data augmentation for NLP
tasks. In Proceedings of COLING, December 8-13, 2020, pages 3436-3440.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. 2019. Manifold mixup: Better representations by interpolating hidden states. In Proceedings of ICML, 9-15 June 2019, pages 6438-6447.
Andrew J. Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Trans. Inf. Theory, 13(2):260-269.
Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2017. Instance weighting for neural machine translation domain adaptation. In Proceedings of EMNLP, September 9-11, 2017, pages 1482-1488.
Zhi Wang, Wei Bi, Yan Wang, and Xiaojiang Liu. 2019. Better fine-tuning via instance weighting for text classification. In Proceedings of AAAI, January 27 - February 1, 2019, pages 7241-7248.
Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of EMNLP-IJCNLP, November 3-7, 2019, pages 6381-6387.
Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Martha Palmer, Nianwen Xue, Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, Robert Belvin, et al. 2011. Ontonotes release 4.0. LDC2011T03, Penn.: Linguistic Data Consortium.
Fangzhao Wu, Junxin Liu, Chuhan Wu, Yongfeng Huang, and Xing Xie. 2019. Neural chinese named entity recognition via CNN-LSTM-CRF and joint training with word segmentation. In Proceedings of WWW, May 13-17, 2019, pages 3342-3348.
Rui Xia, Zhenchun Pan, and Feng Xu. 2018. Instance weighting for domain adaptation via trading off sample selection bias and variance. In Proceedings of IJCAI, page 4489-4495.
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. In Proceedings of NeurIPS, December 6-12, 2020.
Lu Xu, Zhanming Jie, Wei Lu, and Lidong Bing. 2021. Better feature integration for named entity recognition. In Proceedings of NAACL-HLT, June 6-11, 2021, pages 3457-3469.
Zhenyu Xuan, Rui Bao, and Shengyi Jiang. 2020. FGN: fusion glyph network for chinese named entity recognition. In CCKS, November 12-15, 2020, volume 1356, pages 28-40.
Ikuya Yamada, Akari Asai, Jin Sakuma, Hiroyuki Shindo, Hideaki Takeda, Yoshiyasu Takefuji, and Yuji Matsumoto. 2020. Wikipedia2vec: An efficient toolkit for learning and visualizing the embeddings of words and entities from wikipedia. In Proceedings of
EMNLP: System Demonstrations, November 16-20, 2020, pages 23-30.
Zhi-Xiu Ye and Zhen-Hua Ling. 2018. Hybrid semimarkov CRF for neural sequence labeling. In Proceedings of ACL, July 15-20, 2018, pages 235-240.
Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020. Named entity recognition as dependency parsing. In Proceedings of ACL, July 5-10, 2020, pages 6470-6476.
Xiangji Zeng, Yunliang Li, Yuchen Zhai, and Yin Zhang. 2020. Counterfactual generator: A weakly-supervised method for named entity recognition. In Proceedings of EMNLP, November 16-20, 2020, pages 7270-7280.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In Proceedings of ICLR, April 30 - May 3, 2018.
Rongzhi Zhang, Yue Yu, and Chao Zhang. 2020a. Seqmix: Augmenting active sequence labeling via sequence mixup. In Proceedings of EMNLP, November 16-20, 2020, pages 8566-8579.
Yi Zhang, Tao Ge, and Xu Sun. 2020b. Parallel data augmentation for formality style transfer. In Proceedings of ACL, pages 3221-3228.
Yue Zhang and Jie Yang. 2018. Chinese NER using lattice LSTM. In Proceedings of ACL, July 15-20, 2018, pages 1554-1564.