SlowGuess/NAACL_2021_OCR huggingface dataset:
To investigate where the gains of the larger-context training come, we conduct a fine-grained evaluation with the evaluation attributes defined in Sec. 5.1. We use the cPre-seq larger-context aggregation method as the base model. Fig. 3 shows the relative improvement of the cPre-seq larger-context aggregation method in NER (7 datasets) and CWS tasks (4 datasets). The relative improvement is the performance of cPre-seq larger-context tagging minus sentence-level tagging.
Results Our findings from Fig. 3 are:
1) Test spans with lower label consistency can benefit much more from the larger-context training. As shown in Fig. 3 (a,b,i,j), test spans with lower label consistency (NER:eCon, tCon = S/XS, CWS: wCon, cCon = S/XS) can achieve higher relative improvement using the larger-context training, which holds for both NER and CWS tasks.
2) NER task has achieved more gains on lower and higher-frequency test spans, while CWS task obtains more gains on lower-frequency test spans. As shown in Fig. 2 (c,d,k,l), in NER task, test spans with higher or lower frequency (NER:eFre=XS/XL; tFre=XS/XL) will achieve larger improvements with the help of more contextual sentences; while for the CWS task, only the test spans with lower frequency will achieve more gains.
3) Test spans of NER task with lower entity density have obtained larger improvement with the help of a larger-context training. In terms of entity density shown in Fig. 3 (e), an evaluation attribute specific to the NER task, the larger-context training is not good at dealing with the test spans with high entity density (NER:eDen=XL/L), while doing well in test spans with low entity density (NER:eDen=XS/S).
4) Larger-context training can achieve more gains on short entities in NER task while long words in CWS task. As shown in Fig. 3 (f,m), the dark blue boxes can be seen in the short entities  $(\mathrm{eLen} = \mathrm{XS} / \mathrm{S})$  of NER task, and long words  $(\mathrm{wLen} = \mathrm{XL} / \mathrm{L})$  of CWS task.
5) Both NER and CWS tasks will achieve more gains on spans with higher OOV density. For the OOV density shown in Fig. 2 (h,o), the test spans with higher OOV density (NER,CWS:dOov=L/XL) will achieve more gains
Figure 3: The relative increase  $(\in [0,1])$  of the cPre-seq larger-context training on NER (a-h) and CWS (i-o) tasks based on their evaluation attributes. "co" denotes the CoNLL-2003 dataset. In order to facilitate observation, we divide the attribute value range into four categories: extra-small (XS), small (S), large (L), and extra-large (XL). The darker blue implies more significant improvement while the darker red suggests larger-context leads to worse performance. For the attribute name, "e", "t", "w", and "c" refers to "entity", "token", "word", and "character", respectively.
from the larger-context training, which holds for both NER and CWS tasks.
5.4 Exp-II: Quantifying and Understanding Dataset Bias
Different datasets (e.g. CN03) may match different information aggregators (e.g. cPre-seq). Figuring out how different datasets influence the choices of aggregators is a challenging task. We try to approach this goal by (i) designing diverse measures that can characterize a given dataset from different perspectives, (ii) analyzing the correlation between different dataset properties and improvements brought by different aggregators.
Dataset-level Measure Given a dataset  $\mathcal{E}$  and an attribute  $p$  as defined in Sec. 5.1, the data-level measure can be defined as:
$$
\zeta_ {p} (\mathcal {E}) = \frac {1}{| \mathcal {E} ^ {t e} |} \sum_ {\varepsilon \in \mathcal {E} ^ {t e}} \phi_ {p} (\varepsilon), \tag {12}
$$
where  $\mathcal{E}^{te}\in \mathcal{E}$  is a test set that contains entities/tokens in the NER task or word/character in the CWS task.  $\phi_p(\cdot)$  is a function (as defined in Sec. 5.1) that computes the attribute value for a given span. For example,  $\zeta_{sLen}(\mathrm{CN03})$  represents the average sentence length of CN03's test set.
Correlation Measure Statistically, we define a variable of  $\rho$  to quantify the correlation between a dataset-level attribute and the relative improvement of an aggregator:  $\rho = \operatorname{Spearman}(\zeta_p, f_y)$ ,
where Spearman denotes the Spearman's rank correlation coefficient (Mukaka, 2012).  $\zeta_p$  represents dataset-level attribute values on all datasets with respect to attribute  $p$  (e.g., eLen) while  $f_y$  denotes the relative improvements of larger-context training on corresponding datasets with respect to a given aggregator  $y$  (e.g., cPre-seq).
Results Tab. 2 displays (using spider charts) measure  $\zeta_{p}^{5}$  of seven datasets with respect to diverse attributes, and correlation measure  $\rho$  in the NER task. Based on these correlations, which passed significantly test  $(p < 0.05)$ , between dataset-level measure (w.r.t a certain attribute, e.g. eCon) and gains from larger-context training (w.r.t an aggregator, e.g. seq), we can obtain that:
(1) Regarding the  $cPre$ -seq aggregator, it negatively correlated with  $\zeta_{\mathrm{eCon}}$ ,  $\zeta_{\mathrm{tCon}}$ ,  $\zeta_{\mathrm{eFre}}$ , and  $\zeta_{\mathrm{eDen}}$  with larger correlation values. Therefore, the  $cPre$ -seq aggregator is more appropriate to deal with WB, TC, BC and NW datasets, since these four datasets have a lower value of  $\zeta_p$  with respect to the attribute eCon (TC, WB), tCon (TC, WB), eFre (NW, TC), and eDen (BC, WB, TC). Additionally, since the  $cPre$ -seq aggregator obtains the highest positive correlation with  $\zeta_{\mathrm{doov}}$ , and  $\zeta_{\mathrm{doov}}$  (CN03), as well as  $\zeta_{\mathrm{doov}}$  (BC), achieve the highest value,  $cPre$ -seq aggregator is suitable for CN03 and BC.
5The specific value of  $\zeta_{p}$  in NER and CWS task can be found in the appendix.
Analysis of other tasks can be found in our appendix section.
Attr.	eCon	tCon	eFre	tFre	eLen	dOov	sLen	eDen
ζp	BN MZ CW NB CN03 TC	BN MZ CW NB CN03 TC	BN MZ CW NB CN03 TC	BN MZ CW NB CN03 TC	BN MZ CW NB CN03 TC	BN MZ CW NB CN03 TC	BN MZ CW NB CN03 TC	BN MZ CW NB CN03 TC
ρ	bow	-0.179	-0.607	0.143	0.396	0.643	-0.036	0.468	-0.571
graph	-0.571	-0.143	-0.393	-0.919	-0.643	0.286	-0.288	-0.107
seq	-0.643	-0.857	-0.429	0.162	0.143	0.071	0.180	-0.714
cPre	-0.714	-0.750	-0.571	-0.306	0.000	0.357	-0.180	-0.643
Table 2: Illustration of measures  $\zeta_{p}$  in seven datasets (CN03, TC, NW, WB, MZ, BN, BC) with respect to eight attributes (e.g., eCon) and correlation measure  $\rho$  in NER task. A higher absolute value  $\rho$  (e.g  $| - 0.714|$ ) represents the improvement of the corresponding aggregator (e.g., seq) heavily correlates with corresponding attribute (e.g. eCon). The number with the highest absolute value of each column is colored by green. "cPre" represents "cPre-seq" and the values in grey denote correlation values do not pass a significance test ( $p = 0.05$ ). "Attr." denotes attributes.
(2) Regarding the seq aggregator, it negatively correlated with  $\zeta_{\mathrm{eCon}}$ ,  $\zeta_{\mathrm{tCon}}$ , and  $\zeta_{\mathrm{eDen}}$ . Therefore, the seq aggregator is better at dealing with datasets WB, TC, and BC, since these datasets are with lower  $\zeta_p$  value on one of the attributes (eCon, tCon, and eDen).
Takeaways: We can conduct a similar analysis for bow and graph aggregators. Due to limited pages, we detail them in our appendix and highlight the suitable NER datasets for each aggregator as follows.
(1) bow: WB, TC, NW, MZ, BC.
(2) graph: WB, TC, BN, CN03.
(3) seq: WB, TC, BC.
(4) cPre-seq: CN03, WB, TC, BC, NW.
6 Adapting to Top-Scoring Systems
Beyond the above quantitative and qualitative analysis of our instantiated typical tagging models (Sec.2.3), we are also curious about how well modern top-scoring tagging systems perform when equipped with larger-context training.
To this end, we choose the NER task as a case study and first re-implement existing top-performing models for different NER datasets separately, and then adapt larger-context approach to them based on the seq or cPre-seq aggregator, which has shown superior performance in our above analysis.
Settings We collect five top-scoring tagging systems (Luo et al., 2020; Lin et al., 2019; Chen et al., 2019; Yan et al., 2019; Akbik et al., 2018) that
are most recently proposed  $^{8}$ . Among these five models, regarding Akbik et al. (2018), we use cPre-seq aggregator for the larger-context training, since this model originally relies on a contextualized pre-trained layer. Besides, from above analysis in Sec. 5.4 we know the suitable datasets for cPre-seq aggregator: CN03, WB, TC, BC, and NW. Regarding the other four models, we use the seq aggregator for the larger-context training and the matched datasets are: WB, TC, and BC.
Results Tab. 3 shows the relative improvement of larger-context training on five modern top-scoring models in the NER task. We observe that the larger-context training has achieved consistent gains on all chosen datasets, which holds for both seq and cPre-seq aggregators. Notably, the larger-cotext training achieves sharp improvement on WB, which holds for all the five top-scoring models. For example, with the help of larger-context training, the performance can be improved significantly using Akbik et al. (2018) and 7.18 F1 score using Luo et al. (2020). This suggests that modern top-scoring NER systems can also benefit from larger-context training.
Our work touches the following research topics for tagging tasks.
Sentence-level Tagging Existing works have achieved impressive performance at sentence-level tagging by extensive structural explorations with different types of neural components. Regarding sentence encoders, recurrent neural nets (Huang et al., 2015; Chiu and Nichols, 2015; Ma and Hovy,
<sup>8</sup>We originally aimed to select more (10 systems) but suffer from reproducibility problems (Pineau et al., 2020), even after contacting the first authors.
7Training all four aggregators for all tagging tasks is much more costly and here we choose these two since they can obtain better performance at a relatively lower cost.
Models	Aggregator	Datasets
norm seq cPre	BC	WB	TC	CN03	NW
Luo et al. (2020)	✓	78.78	62.38	65.56	-	-
✓	+0.54	+7.18	+1.7
Lin et al. (2019)	✓	77.80	63.16	65.19	-	-
✓	+2.94	+5.07	+2.25
Chen et al. (2019)	✓	77.50	66.51	65.49	-	-
✓	+1.96	+3.98	+0.89	-	-
Yan et al. (2019)	✓	81.29	65.05	67.92	92.17	90.37
✓	+0.16	+6.79	+3.03	+0.04	+1.11
Akbik et al. (2018)	✓	81.13	64.79	69.00	93.03	90.76
✓+1.12	+10.78	+2.12	+0.05	+1.03
Table 3: The relative improvement of larger-context training on top-scoring models in the NER task. "cPre" represents "cPre-seq". "norm" denotes the normal setting  $(K = 1)$ . The testing datasets are chosen based on the analysis in Sec. 5.4.
2016; Lample et al., 2016; Li et al., 2019; Lin et al., 2020) and convolutional neural nets (Strubell et al., 2017; Yang et al., 2018; Chen et al., 2019; Fu et al., 2020a) were widely used while transformer were also studied to get sentential representations (Yan et al., 2019; Yu et al., 2020). Some recent works consider the NER as a span classification (Li et al., 2019; Jiang et al., 2019; Mengge et al., 2020; Ouchi et al., 2020) task, unlike most works that view it as a sequence labeling task. To capture morphological information, some previous works introduced a character or subword-aware encoders with unsupervised pre-trained knowledge (Peters et al., 2018; Akbik et al., 2018; Devlin et al., 2018; Akbik et al., 2019; Yang et al., 2019; Lan et al., 2019).
Document-level Tagging Document-level tagging introduced more contextual features to improve the performance of tagging. Some early works introduced non-local information (Finkel et al., 2005; Krishnan and Manning, 2006) to enhance traditional machine learning methods (e.g., CRF (Lafferty et al., 2001)) and achieved impressive results. Qian et al. (2018); Wadden et al. (2019) built graph representation based on the broad dependencies between words and sentences. Luo et al. (2020) proposed to use a memory network to record the document-aware information. Besides, document-level features was introduced by different domains to alleviate label inconsistency problems, such as news NER (Hu et al., 2020, 2019), chemical NER (Luo et al., 2018), disease NER (Xu et al., 2019), and Chinese patent (Li and Xue, 2014, 2016). Compared with these works, instead of proposing a novel model, we focus on investigating when and why the larger-context training, as a general strategy, can work.
Interpretability and Robustness of Sequence Labeling Systems Recently, there is a popular trend that aims to (i) perform a glass-box analysis of sequence labeling systems (Fu et al., 2020b; Agarwal et al., 2020), understanding their generalization ability and quantify robustness (Fu et al., 2020c), (ii) interpretable evaluation of them (Fu et al., 2020a), making it possible to know what a system is good/bad at and where a system outperforms another, (iii) reliable analysis (Ye et al., 2021) for test set with fewer samples. Our work is based on the technique of interpretable evaluation, which provides a convenient way for us to diagnose different systems.
We summarize the main observations from our experiments and try to provide preliminary answers to our proposed research questions:
(i) How do different integration ways of larger-context information influence the system's performance? Overall, introducing larger-context information will bring gains regardless of the ways how to introduce it (e.g., seq, graph). Particularly, larger-context training with seq aggregator can achieve better performance at lower training cost compared with graph and bow aggregators (Sec. 4.1).
(ii) Can the larger-context training easily play to its strengths with the help of contextualized pre-trained models? Yes for all datasets on NER, Chunk, and POS tasks. By contrast, for CWS tasks, the aggregator without BERT (e.g., seq) can achieve better improvement (Sec. 4.2).
(iii) Where does the gain of larger-context training come? And how do different characteristics of datasets affect the amount of gain? The source of gains, though, is dataset- and aggregator-dependent, a relatively consensus observation is that text spans with lower label consistency and higher OOV density can benefit a lot from larger-context training (Sec. 5.3). Regarding different datasets, diverse aggregators are recommended in Sec. 5.4.
We would like to thank the anonymous reviewers for their valuable comments. This work was supported by China National Key R&D Program (No.2018YFC0831105).
Oshin Agarwal, Yinfei Yang, Byron C Wallace, and Ani Nenkova. 2020. Interpretability analysis for named entity recognition to understand system predictions and how they can improve. arXiv preprint arXiv:2004.04564.
Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named entity recognition. pages 724-728.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638-1649.
Hui Chen, Zijia Lin, Guiguang Ding, Jianguang Lou, Yusen Zhang, and Borje Karlsson. 2019. Grn: Gated relation network to enhance convolutional neural network for named entity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6236-6243.
Jason P C Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns. arXiv: Computation and Language.
Jason PC Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional lstm-cnns. Transactions of the Association for Computational Linguistics, 4:357-370.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. Transactions of the association for computational linguistics, 2:477-490.
Bradley Efron and Robert Tibshirani. 1986. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical science, pages 54-75.
Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 363-370. Association for Computational Linguistics.
Jinlan Fu, Pengfei Liu, and Graham Neubig. 2020a. Interpretable multi-dataset evaluation for named entity recognition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6058-6069, Online. Association for Computational Linguistics.
Jinlan Fu, Pengfei Liu, and Qi Zhang. 2020b. Rethinking generalization of neural models: A named
entity recognition case study. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7732-7739.
Jinlan Fu, Pengfei Liu, Qi Zhang, and Xuanjing Huang. 2020c. RethinkCWS: Is Chinese word segmentation a solved task? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5676-5686, Online. Association for Computational Linguistics.
Abbas Ghaddar and Philippe Langlais. 2018. Robust lexical features for improved neural network named-entity recognition. arXiv preprint arXiv:1806.03489.
Anwen Hu, Zhicheng Dou, and Ji-rong Wen. 2019. Document-level named entity recognition by incorporating global and neighbor features. In China Conference on Information Retrieval, pages 79-91. Springer.
Anwen Hu, Zhicheng Dou, Jirong Wen, and Jianyun Nie. 2020. Leveraging multi-token entities in document-level named entity recognition.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-crf models for sequence tagging. arXiv: Computation and Language.
Zhengbao Jiang, Wei Xu, Jun Araki, and Graham Neubig. 2019. Generalizing natural language analysis through span-relation representations. arXiv preprint arXiv:1911.03822.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Vijay Krishnan and Christopher D Manning. 2006. An effective two-stage model for exploiting non-local dependencies in named entity recognition. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 1121-1128. Association for Computational Linguistics.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-HLT, pages 260-270.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Si Li and Nianwen Xue. 2014. Effective document-level features for chinese patent word segmentation. 2:199-205.
Si Li and Nianwen Xue. 2016. Towards accurate word segmentation for chinese patents. arXiv: Computation and Language.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2019. A unified mrc framework for named entity recognition. arXiv preprint arXiv:1910.11476.
Bill Yuchen Lin, Dong-Ho Lee, Ming Shen, Ryan Moreno, Xiao Huang, Prashant Shiralkar, and Xiang Ren. 2020. Triggerner: Learning with entity triggers as explanations for named entity recognition. arXiv preprint arXiv:2004.07493.
Ying Lin, Liyuan Liu, Heng Ji, Dong Yu, and Jiawei Han. 2019. Reliability-aware dynamic feature composition for name tagging. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 165-174.
Ling Luo, Zhihao Yang, Pei Yang, Yin Zhang, Lei Wang, Hongfei Lin, and Jian Wang. 2018. An attention-based bilstm-crf approach to document-level chemical named entity recognition. Bioinformatics, 34(8):1381-1388.
Ying Luo, Fengshun Xiao, and Hai Zhao. 2020. Hierarchical contextualized representation for named entity recognition.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In Proceedings of the 54th Annual Meeting of ACL, volume 1, pages 1064-1074.
Xue Mengge, Bowen Yu, Zhenyu Zhang, Tingwen Liu, Yue Zhang, and Bin Wang. 2020. Coarse-to-fine pretraining for named entity recognition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6345-6354.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Mavuto Mukaka. 2012. Statistics corner: A guide to appropriate use of correlation coefficient in medical research. Malawi medical journal : the journal of Medical Association of Malawi, 24(3):69-71.
Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi, Sho Yokoi, Tatsuki Kuribayashi, Ryuto Konno, and Kentaro Inui. 2020. Instance-based learning of span representations: A case study through named entity recognition. arXiv preprint arXiv:2004.14514.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the EMNLP, 12:1532-1543.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2227-2237.
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d'Alché Buc, Emily Fox, and Hugo Larochelle. 2020. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). arXiv preprint arXiv:2003.12206.
Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, and Regina Barzilay. 2018. Graphie: A graph-based framework for information extraction. arXiv: Computation and Language.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pages 593-607. Springer.
Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017. Fast and accurate entity recognition with iterated dilated convolutions. pages 2670-2680.
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. Entity, relation, and event extraction with contextualized span representations. arXiv preprint arXiv:1909.03546.
Frank Wilcoxon, SK Katti, and Roberta A Wilcox. 1970. Critical values and probability levels for the wilcoxon rank sum test and the wilcoxon signed rank test. Selected tables in mathematical statistics, 1:171-259.
Kai Xu, Zhenguo Yang, Peipei Kang, Qi Wang, and Wenyin Liu. 2019. Document-level attention-based bilstm-crf incorporating disease dictionary for disease named entity recognition. Computers in biology and medicine, 108:122-132.
Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. 2019. Tener: Adapting transformer encoder for name entity recognition. arXiv preprint arXiv:1911.04474.