SlowGuess/NAACL_2021_OCR huggingface dataset:
Jinlan Fu†, Liangjing Feng†, Qi Zhang†, Xuanjing Huang†, Pengfei Liu‡ * † School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University, ‡Carnegie Mellon University {fujl16, qz, xjhuang} @fudan.edu.cn, pliu3@cs.cmu.edu
The development of neural networks and pretraining techniques has spawned many sentence-level tagging systems that achieved superior performance on typical benchmarks. However, a relatively less discussed topic is what if more context information is introduced into current top-scoring tagging systems. Although several existing works have attempted to shift tagging systems from sentence-level to document-level, there is still no consensus conclusion about when and why it works, which limits the applicability of the larger-context approach in tagging tasks. In this paper, instead of pursuing a state-of-the-art tagging system by architectural exploration, we focus on investigating when and why the larger-context training, as a general strategy, can work.
To this end, we conduct a thorough comparative study on four proposed aggregators for context information collecting and present an attribute-aided evaluation method to interpret the improvement brought by larger-context training. Experimentally, we set up a testbed based on four tagging tasks and thirteen datasets. Hopefully, our preliminary observations can deepen the understanding of larger-context training and enlighten more follow-up works on the use of contextual information.
The rapid development of deep neural models has shown impressive performances on sequence tagging tasks that aim to assign labels to each token of an input sequence (Sang and De Meulder, 2003; Lample et al., 2016; Ma and Hovy, 2016). More recently, the use of unsupervised pre-trained models (Akbik et al., 2018, 2019; Peters et al., 2018; Devlin et al., 2018) (especially contextualized version) has driven state-of-the-art performance to a
new level. Among these works, researchers frequently choose the boundary with the granularity of sentences for tagging tasks (i.e., sentence-level tagging) (Huang et al., 2015; Chiu and Nichols, 2015; Ma and Hovy, 2016; Lample et al., 2016). Undoubtedly, as a transient, sentence-level setting enables us to develop numerous successful tagging systems, nevertheless the task itself should have not be defined as sentence-level but for simplifying the learning process for machine learning models. Naturally, it would be interesting to see what if larger-context information (e.g., taking information of neighbor sentences into account) is introduced to modern top-scoring systems, which have shown superior performance under the sentence-level setting. A small number of works have made seminal exploration in this direction, in which part of works show significant improvement of larger-context (Luo et al., 2020; Xu et al., 2019) while others don't (Hu et al., 2020, 2019; Luo et al., 2018). Therefore, it's still unclear when and why larger-context training is beneficial for tagging tasks. In this paper, we try to figure it out by asking the following three research questions:
Q1: How do different integration ways of larger-context information influence the system's performance? The rapid development of neural networks provides us with diverse flavors of neural components to aggregate larger-context information, which, for example, can be structured as a sequential topology by recurrent neural networks (Ma and Hovy, 2016; Lample et al., 2016) (RNNs) or graph topology by graph neural networks (Kipf and Welling, 2016; Schlichtkrull et al., 2018).
Understanding the discrepancies of these aggregators can help us reach a more generalized conclusion about the effectiveness of larger-context training. To this end, we study larger-context aggregators with three different structural priors (defined in Sec. 3.2) and comprehensively evaluate their efficacy.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1463-1475 June 6-11, 2021. ©2021 Association for Computational Linguistics
Q2: Can the larger-context training easily play to its strengths with the help of recently arising contextualized pre-trained models (Akbik et al., 2018, 2019; Peters et al., 2018; Devlin et al., 2018) (e.g. BERT)? The contextual modeling power of these pre-trained methods makes it worth looking at its effect on larger-context training. In this work, we take BERT as a case study and assess its effectiveness quantitatively and qualitatively.
Q3: If improvements could be observed, where does the gain come and how do different characteristics of datasets affect the amount of gain? Instead of simply figuring out whether larger-context training could work, we also try to interpret its gains. Specifically, we propose to use fine-grained evaluation to explain where the improvement comes from and why different datasets exhibit discrepant gains.
Overall, the first two questions aim to explore when larger-context training can work while the third question addresses why. Experimentally, we try to answer these questions by conducting a comprehensive analysis, which involves four tagging tasks and thirteen datasets. Our main observations are summarized in Sec. 8. Furthermore, we show, with the help of these observations, it's easier to adapt larger-context training to modern top-performing tagging systems with significant gains. We brief our contributions below:
1) We try to bridge the gap by asking three research questions, between the increasing top-performing sentence-level tagging systems and insufficient understanding of larger-context training, encouraging future research to explore more larger-context tagging systems. 2) We systematically investigate four aggregators for larger-context and present an attribute-aided evaluation methodology to interpret the relative advantages of them, and why they can work (Sec. 3.2). 3) Based on some of our observations, we adapt larger-context training to five modern top-scoring systems in the NER task and observe that all larger-context enhanced models can achieve significant improvement (Sec. 6). Encouragingly, with the help of larger-context training, the performance of Akbik et al. (2018) on the WB (OntoNotes5.0-WB) dataset can be improved by a 10.78 F1 score.
2 Task, Dataset, and Model
We first explicate the definition of tagging task and then describe several popular datasets as well as typical methods of this task.
Sequence tagging aims to assign one of the predefined labels to each token in a sequence. In this paper, we consider four types of concrete tasks: Named Entity Recognition (NER), Chinese Word Segmentation (CWS), Part-of-Speech (POS) tagging, and Chunking.
The datasets used in our paper are naturally ordered without random shuffling according to the paper that constructed these datasets, except for WNUT-2016 dataset.
Named Entity Recognition (NER) We consider two well-established benchmarks: CoNLL-2003 (CN03) and OntoNotes 5.0. OntoNotes 5.0 is collected from six different genres: broadcast conversation (BC), broadcast news (BN), magazine (MZ), newswire (NW), telephone conversation (TC), and web data (WB). Since each domain of OntoNotes 5.0 has its nature, we follow previous works (Dur-rett and Klein, 2014; Chiu and Nichols, 2016; Ghaddar and Langlais, 2018) that utilize different domains of this dataset, which also paves the way for our fine-grained analysis.
Chinese Word Segmentation (CWS) We use four mainstream datasets from SIGHAN2005 and SIGHAN2008, in which CITYU is traditional Chinese, while PKU, NCC, and SXU are simplified ones.
Chunking (Chunk) CoNLL-2000 (CN00) is a benchmark dataset for text chunking.
Part-of-Speech (POS) We use the Penn Treebank (PTB) III dataset for POS tagging.2
2.3 Neural Tagging Models
Despite the emergence of a bunch of architectural explorations (Ma and Hovy, 2016; Lample et al., 2016; Yang et al., 2018; Peters et al., 2018; Akbik et al., 2018; Devlin et al., 2018) for sequence tagging, two general frameworks can be summarized: (i) cEnc-wEnc-CRF consists of the word-level encoder, sentence-level encoder, and CRF
<sup>1</sup>Putting the conclusion at the end can help the reader understand it better since more contextual information about experiments has been introduced.
<sup>2</sup>It's hard to cover all datasets for all tasks. For Chunk and POS tasks, we adopt the two most popular benchmark datasets.
layer (Lafferty et al., 2001); (ii) ContPre-MLP is composed of a contextualized pre-trained layer, followed by an MLP or CRF layer. In this paper, we take both frameworks as study objects for our three research questions first, and instantiate them as two specific models: CNN-LSTM-CRF (Ma and Hovy, 2016) and BERT-MLP (Devlin et al., 2018).
3 Larger-Context Tagging
3.1 Sentence-level Tagging
Let  $S = s_1, \dots, s_k$  represent a sequence of sentences, where sentence  $s_i$  contains  $n_i$  words:  $s_i = w_{i,1}, \dots, w_{i,n_i}$ . Sentence-level tagging models predict the label for each word  $w_{i,t}$  sentence-wisely (within a given sentence  $s_i$ ). CNN-LSTM-CRF, for example, first converts each word  $w_{i,t} \in s_i$  into a vector by different word-level encoders wEnc( $\cdot$ ):
$$
\mathbf {w} _ {i, t} = \mathrm {w E n c} \left(w _ {i, t}\right) = \operatorname {L o o k u p} \left(w _ {i, t}\right) \oplus \operatorname {C N N} \left(w _ {i, t}\right), \tag {1}
$$
where  $\oplus$  denotes the concatenation operation,  $\mathrm{Lookup}(w_{i,t})$  can be pre-trained by context-free (e.g., GloVe) or context-dependent (e.g., BERT) word representations.
And then the concatenated representation of them will be fed into sentence encoder  $\mathrm{sEnc}(\cdot)$  (e.g., LSTM layer) to derive a contextualized representation for each word.
$$
\mathbf {h} _ {i, t} = \operatorname {s E n c} (\cdot) = \operatorname {L S T M} ^ {(s)} \left(\mathbf {w} _ {i, t}, \mathbf {h} _ {i, t - 1}, \theta\right), \tag {2}
$$
where the lower case “ $s$ ” of  $\mathrm{LSTM}^{(s)}$  represents a sentence-level LSTM. Finally, a CRF layer will be used to predict the label for each word.
3.2 Contextual Information Aggregators
Instead of predicting entity tags sentence-wisely, more contextual information of neighbor sentences can be introduced in diverse ways. Following, we elaborate on how to extend sentence-level tagging to a larger-context setting. The high-level idea is to introduce more contextual information into word- or sentence-level encoder defined in Eq. 1 and Eq. 2. Here, we propose four larger-context aggregators, whose architectures are illustrated in Fig. 1.
Bag-of-Word Aggregator (bow) calculates a fused representation  $\mathbf{r}$  for a sequence of sentences.
$$
\mathbf {r} = \operatorname {B O W} \left(w _ {1, 1}, \dots , w _ {1, n _ {1}}, \dots , w _ {k, n _ {k}}\right), \tag {3}
$$
where  $\mathrm{BOW}(\cdot)$  is a function that computes the average of all word representations of input sentences. Afterward,  $\mathbf{r}$ , as additional information, will be injected into the word encoder.
More precisely, the word-level encoder and sentence-level encoder can be re-written below:
$$
\mathbf {w} _ {i, t} ^ {b o w} = \operatorname {G l o V e} \left(w _ {i, t}\right) \oplus \operatorname {C N N} \left(w _ {i, t}\right) \oplus \mathbf {r}, \tag {4}
$$
$$
\mathbf {h} _ {i, t} ^ {b o w} = \operatorname {L S T M} ^ {(S)} \left(\mathbf {w} _ {i, t} ^ {b o w}, \mathbf {h} _ {i, t - 1} ^ {b o w}, \theta\right), \tag {5}
$$
where the upper case "S" of  $\mathrm{LSTM}^{(S)}$  denotes the larger-context encoder that utilizes an LSTM deal with a sequence of sentences  $(S = s_{1},\dots ,s_{k})$  (instead of solely one sentence).
Sequential Aggregator (seq) first concatenates all sentences  $s_i \in S$  and then encode it with a larger-context encoder  $\mathrm{LSTM}^{(S)}$ . Formally, seq aggregator can be represented as:
$$
\mathbf {h} _ {i, t} ^ {\text {s e q}} = \operatorname {L S T M} ^ {(S)} \left(\mathbf {w} _ {i, t} ^ {\text {s e q}}, \mathbf {h} _ {i, t - 1} ^ {\text {s e q}}, \theta\right), \tag {6}
$$
where  $\mathbf{w}_{i,t}^{seq}$  is defined as Eq. 1, and the  $\mathrm{Lookup}(w_{i,t})$  is GloVe. Then, a CRF decoder is utilized to predict the tags for each word.
Graph Aggregator (graph) incorporates non-local bias into tagging models. Each word  $w_{i}$  is conceptualized as a node. For edge connections, we define the following types of edges between pairs of nodes (i.e.  $w_{i}$  and  $w_{j}$ ) to encode various structural information in the context graph: i) if  $|i - j| = 1$ ; ii) if  $w_{i} = w_{j}$ . In practice, the graph aggregator first collects contextual information over a sequence of sentences, and generate the word representation:
$$
\mathbf {G} = \operatorname {G r a p h N N} (\mathbf {V}, E, \theta), \tag {7}
$$
where  $\mathbf{V} = \{\mathbf{w}_{1,1},\dots ,\mathbf{w}_{1,n_1},\dots ,\mathbf{w}_{k,n_k}\}$  and  $\mathbf{w}_i$  can be obtained as defined in Eq. 1. Additionally,  $\mathbf{G} = \{\mathbf{g}_{1,1},\dots ,\mathbf{g}_{1,n_1},\dots ,\mathbf{g}_{k,n_k}\}$  stores aggregated contextual information for each word. We instantiate GraphNN(·) as graph convolutional neural networks (Kipf and Welling, 2016).
3Notably, in the setting, we don't aim to improve performance over state-of-the-art models.
(a) bow
(b) seq
(c) graph
Figure 1: Illustration of four larger-context aggregators.
(d) cPre-seq
Afterwords, the contextual vector  $\mathbf{g}$  will be introduced into larger-context encoder, i.e.,  $\mathrm{LSTM}^{(S)}$ :
$$
\mathbf {h} _ {i, t} ^ {\text {g r a p h}} = \operatorname {L S T M} ^ {(S)} \left(\mathbf {g} _ {i, t}, \mathbf {h} _ {i, t - 1} ^ {\text {g r a p h}}, \theta\right), \tag {8}
$$
Contextualized Sequential Aggregator (cPre-seq) is an extension of seq aggregator by using contextualized pre-trained models, such as BERT (Devlin et al., 2018), Flair (Akbik et al., 2018), and ELMo (Peters et al., 2018), as a word encoder. Here, cPre-seq is instantiated as BERT to get the word representation, then followed by a larger-context encoder  $\mathrm{LSTM}^{(S)}$ . We make the length of larger-context for the cPre-seq aggregator within 512. cPre-seq can be formalized as:
$$
\mathbf {h} _ {i, t} ^ {c P r e} = \operatorname {L S T M} ^ {(S)} \left(\operatorname {B E R T} \left(w _ {i, t}\right), \mathbf {h} _ {i, t - 1} ^ {c P r e}, \theta\right). \tag {9}
$$
4 Experiment: When Does It Work?
The experiment in this section is designed to answer the first two research questions: Q1 and Q2 (Sec. 1). Specifically, we investigate whether larger-context training can achieve improvement and how different structures of aggregator, contextualized pre-trained models influence it.
Settings and Hyper-parameters We adopt CNN-LSTM-CRF as a prototype and augment it with larger-context information by four categories of aggregators: bow, seq, graph, and cPre-seq. We use Word2Vec (Mikolov et al., 2013) (trained on simplified Chinese Wikipedia dump) as noncontextualized embeddings for CWS task, and GloVe (Pennington et al., 2014) for NER, Chunk, and POS tasks.
The window size (the number of sentence)  $k$  of larger-context aggregators will be explored with a range of  $k = \{1,2,3,4,5,6,10\}$  for seq, bow, and cPre-seq. We chose the best performance that the larger-context aggregator achieved with window
size  $k \neq 1$  as the final performance of a larger-context aggregator. We use the result from the model with the best validation set performance, terminating training when the performance on development is not improved in 20 epochs.
For the POS task, we adopt dataset-level accuracy as evaluated metric while for other tasks, we use a corpus-level  $F1$ -score (Sang and De Meulder, 2003) to evaluate.
4.1 Exp-I: Effect of Structured Typologies
Tab. 1 illustrates the relative improvement results of four larger-context training ( $k > 1$ ) relative to the sentence-level tagging ( $k = 1$ ). To examine whether the larger-context aggregation method has a significant improvement over the sentence-level tagging, we used significant test with Wilcoxon Signed-RankTest (Wilcoxon et al., 1970) at  $p = 0.05$  level. Results are shown in Tab. 1 (the last column). We find that improvements brought by four larger-context aggregators are statistically significant ( $p < 0.05$ ), suggesting that the introduction of larger-context can significantly improve the performance of sentence-level models.
Results We detail main observations in Tab. 1:
1) For most of the datasets, introducing larger-context information will bring gains regardless of the ways how to introduce it (e.g. bow or graph), indicating the efficacy of larger contextual information. Impressively, the performance on dataset WB is significantly improved by 7.26 F1 score with the cPre-seq aggregator  $(p = 5.1 \times 10^{-3} < 0.05)$ .
2) Overall, comparing with bow and graph aggregators, seq aggregator has achieved larger improvement by average, which can be further enhanced by introducing contextualized pre-trained models (e.g. BERT).
3) Incorporating larger-context information with some aggregators also can lead to performance drop on some datasets (e.g, using graph aggrega
4The settings of window size  $k$  are listed in the appendix.
Emb.	Agg.	CWS	NER	Chunk	POS	Avg.	Signi. (×10-2)
CITYU	NCC	SXU	PKU	CN03	BC	BN	MZ	WB	NW	TC	CN00	PTB
Non- Con.	norm	93.70	92.26	94.94	94.35	90.46	75.38	86.89	85.42	62.09	88.38	63.69	93.85	97.25	-	-
bow	+0.17	+0.42	+0.03	+0.04	-0.39	+1.66	+0.32	+1.51	+3.49	+0.92	+0.42	-0.29	-0.14	+0.54	1.74
graph	-0.15	-0.61	-0.02	+0.33	+1.47	+0.17	+0.42	-0.16	+4.84	+0.34	+0.90	-0.15	+0.17	+0.61	2.17
seq	+0.27	+0.34	+0.18	+0.08	-0.14	+0.65	-0.50	+1.49	+5.61	+1.13	+2.39	-0.08	+0.03	+0.77	0.86
Con.	norm	97.09	95.77	97.49	96.47	90.77	80.46	89.67	87.03	68.78	90.04	63.34	96.45	97.62	-	-
cPre	+0.07	+0.07	+0.13	+0.14	+0.72	+1.27	+0.39	+0.19	+7.26	+0.99	+6.00	+0.11	+0.04	+1.15	0.26
Table 1: The relative improvement (the performance difference between a model with larger-context aggregator (e.g. bow) and the one without it) on tasks CWS, NER, Chunk, and POS. "norm" denotes the normal setting ( $K = 1$ ). The values in red are the performance of larger-context tagging ( $k > 1$ ) lower than sentence-level tagging ( $k = 1$ ). "Signi." denotes p-value of "significant test". "Emb.", "Non-Con.", "Con.", and "Agg." are the abbreviations of "Embeddings", "Non-Contextualized", "Contextualized", and "Aggregator" respectively. The values in pink indicate that the value is less than zero.
tor on dataset MZ lead to 0.16 performance drop), which suggests the importance of a better match between datasets and aggregators.
4.2 Exp-II: Effect of BERT
To answer the research question Q2 ( Can the larger-context approach easily play to its strengths with the help of recently arising contextualized pretrained models?), we elaborate on how cPre-seq and seq aggregators influence the performance.
Results Fig. 2 illustrates the relative improvement achieved by two larger-context methods: seq (blue bar) and cPre-seq (red bar) on four different tagging tasks. We observe that:
1) In general, aggregators equipped with BERT can not guarantee a better improvement, which is dataset-dependent. 2) Task-wisely,  $cPre-seq$  can improve performance on all datasets on NER, Chunk, and POS tasks. By contrast, seq is beneficial to all datasets on CWS task. It could be attributed to the difference in language and characteristics of the task. Specifically, for most non-CWS task datasets,  $cPre-seq$  (7 out of 9 datasets) performs better than seq ( $p < 0.05$ ).
5 Experiment: Why Does It Work?
Experiments in this section are designed for the research questions Q3, interpreting where the gains of a larger-context approach come and why different datasets exhibit diverse improvements. To achieve this goal, we use the concept of interpretable evaluation (Fu et al., 2020a) that allows us to perform fine-grained evaluation of one or multiple systems.
5.1 Attribute Definition
The first step of interpretable evaluation is attribute definition. The high-level idea is, given one attribute, the test set of each tagging task will be partitioned into several interpretable buckets based on it. And  $F1$  score (accuracy for POS) will be calculated bucket-wisely. Next, we will explicate the general attributes we defined in this paper.
We first detail some notations to facilitate definitions of our attributes. We define  $x$  as a token and a bold form  $\mathbf{x}$  as a span, which occurs in a test sentence  $X = \mathrm{sent}(\mathbf{x})$ . We additionally define two functions  $\mathrm{oov}(\cdot)$  that counts the number out of training set words, and  $\mathrm{ent}(\cdot)$  that tallies the number of entity words. Based on these notations, we introduce some feature functions that can compute different attributes for each span or token. Following, we will give the attribute definition of the NER.
Training set-independent Attributes
-  $\phi_{\mathrm{elen}}(\mathbf{x}) = |\mathbf{x}|$ : entity span length
-  $\phi_{\mathrm{SLen}}(\mathbf{x}) = |\mathrm{sent}(\mathbf{x})|$ : sentence length
-  $\phi_{\mathrm{eDen}}(\mathbf{x}) = |\mathrm{ent}(sent(\mathbf{x}))| / \phi_{\mathrm{sLen}}(\mathbf{x})$  : entity density
$\phi_{\mathrm{d0ov}}(\mathbf{x}) = |\mathrm{oov}(\mathrm{sent}(\mathbf{x}))| / \phi_{\mathrm{sLen}}(\mathbf{x})$  .  $OOV$  density
Training set-dependent Attributes
-  $\phi_{\mathrm{eFre}}(\mathbf{x}) = \operatorname{Fre}(\mathbf{x})$ : entity frequency
-  $\phi_{\mathrm{eCon}}(\mathbf{x}) = \operatorname{Con}(\mathbf{x})$ : label consistency of entity
where  $\operatorname{Fre}(\mathbf{x})$  calculates the frequency of input  $\mathbf{x}$  in the training set.  $\operatorname{Con}(\mathbf{x})$  quantifies how consistently a given span is labeled with a particular label, and  $\operatorname{Con}(\mathbf{x})$  can be formulated as:
(a) CWS
(b) NER
Figure 2: Illustration of the relative improvement  $(\%)$  achieved by two larger-context methods (i.e., seq and cPre-seq) on four different tagging tasks. The red and blue bars represent the improvements from seq and cPre-seq, respectively. The error bars represent  $95\%$  confidence intervals of the relative improvement that are computed based on Bootstrap method (Efron and Tibshirani, 1986).
(c) Chunk & POS
$$
\operatorname {C o n} (\mathbf {x}) = \frac {\left| \left\{\varepsilon \mid \operatorname {l a b} (\varepsilon) (\mathbf {x}) , \forall \varepsilon \in \mathcal {E} ^ {t r} \right\} \right|}{\left| \operatorname {s t r} (\varepsilon) = \operatorname {s t r} (\mathbf {x}) , \forall \varepsilon \in \mathcal {E} ^ {t r} \right\} |}, \tag {10}
$$
$$
\operatorname {l a b} (\varepsilon) = \operatorname {l a b} (\mathbf {x}) \cap \operatorname {s t r} (\varepsilon) = \operatorname {s t r} (\mathbf {x}), \tag {11}
$$
where  $\mathcal{E}^{tr}$  denotes entities in the training set,  $\mathrm{lab}(\cdot)$  denotes the label of input span while  $\mathrm{str}(\cdot)$  represents the surface string of input span. Similarly, we can extend the above two attributes to token-level, therefore obtaining  $\phi_{\mathrm{tFre}}(x)$  and  $\phi_{\mathrm{tCon}}(x)$ .
Attributes for CWS task can be defined in a similar way. Specifically, the entity (or token) in NER task corresponds to the word (or character) in CWS task. Note that we omit word density for CWS task since it equals to one for any sentence.
5.2 Attribute Buckets
We breakdown all test examples into different attribute buckets according to the given attribute. Take entity length (eLen) attribute of NER task as an example, first, we calculate each test sample's entity length attribute value. Then, divide the test entities into  $N$  attribute buckets ( $N = 4$  by default) where the numbers of the test samples in all attribute intervals (buckets) are equal, and calculate the performance for those entities falling into the same bucket.
5.3 Exp-I: Breakdown over Attributes