SlowGuess/ACL_2020_OCR huggingface dataset:
Ouyu Lan\* Xiao Huang\* Bill Yuchen Lin He Jiang\* Liyuan Liu\* Xiang Ren
{olan,huan183,yuchen.lin, jian567,xiangren}@usc.edu, ll2@illinois.edu
†Computer Science Department, University of Southern California
$^{\ddagger}$ Computer Science Department, University of Illinois at Urbana-Champaign
Sequence labeling is a fundamental task for a range of natural language processing problems. When used in practice, its performance is largely influenced by the annotation quality and quantity, and meanwhile, obtaining ground truth labels is often costly. In many cases, ground truth labels do not exist, but noisy annotations or annotations from different domains are accessible. In this paper, we propose a novel framework Consensus Network (CONNET) that can be trained on annotations from multiple sources (e.g., crowd annotation, cross-domain data). It learns individual representation for every source and dynamically aggregates source-specific knowledge by a context-aware attention module. Finally, it leads to a model reflecting the agreement (consensus) among multiple sources. We evaluate the proposed framework in two practical settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation. Extensive experimental results show that our model achieves significant improvements over existing methods in both settings. We also demonstrate that the method can apply to various tasks and cope with different encoders.
Sequence labeling is a general approach encompassing various natural language processing (NLP) tasks including part-of-speech (POS) tagging (Ratnaparkhi, 1996), word segmentation (Low et al., 2005), and named entity recognition (NER) (Nadeau and Sekine, 2007). Typically, existing methods follow the supervised learning paradigm, and require high-quality annotations. While gold standard annotation is expensive and
time-consuming, imperfect annotations are relatively easier to obtain from crowdsourcing (noisy labels) or other domains (out-of-domain). Despite their low cost, such supervision usually can be obtained from different sources, and it has been shown that multi-source weak supervision has the potential to perform similar to gold annotations (Ratner et al., 2016).
Specifically, we are interested in two scenarios: 1) learning with crowd annotations and 2) unsupervised cross-domain model adaptation. Both situations suffer from imperfect annotations, and benefit from multiple sources. Therefore, the key challenge here is to aggregate multi-source imperfect annotations for learning a model without knowing the underlying ground truth label sequences in the target domain.
Our intuition mainly comes from the phenomenon that different sources of supervision have different strengths and are more proficient with distinct situations. Therefore they may not keep consistent importance during aggregating supervisions, and aggregating multiple sources for a specific input should be a dynamic process that depends on the sentence context. To better model this nature, we need to (1) explicitly model the unique traits of different sources when training and (2) find best suitable sources for generalizing the learned model on unseen sentences.
In this paper, we propose a novel framework, named Consensus Network (CONNET), for sequence labeling with multi-source supervisions. We represent the annotation patterns as different biases of annotators over a shared behavior pattern. Both annotator-invariant patterns and annotator-specific biases are modeled in a decoupled way. The first term comes through sharing part of low-level model parameters in a multi-task learning schema. For learning the biases, we decouple them from the model as the transformations
\*The first two authors contributed equally. Our code can be found at https://github.com/ INK-USC/ConNet .
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2134-2146 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
(b)
Figure 1: Illustration of the task settings for the two applications in this work: (a) learning consensus model from crowd annotations; (b) unsupervised cross-domain model adaptation.
on top-level tagging model parameters, such that they can capture the unique strength of each annotator. With such decoupled source representations, we further learn an attention network for dynamically assigning the best sources for every unseen sentence through composing a transformation that represents the agreement among sources (consensus). Extensive experimental results in two scenarios show that our model outperforms strong baseline methods, on various tasks and with different encoders. CONNET achieves state-of-the-art performance on real-world crowdsourcing datasets and improves significantly in unsupervised cross-domain adaptation tasks over existing works.
There exists three threads of related work with this paper, which are sequence labeling, crowdsourcing and unsupervised domain adaptation.
Neural Sequence Labeling. Traditional approaches for sequence labeling usually need significant efforts in feature engineering for graphical models like conditional random fields (CRFs) (Lafferty, 2001). Recent research efforts in neural network models have shown that end-to-end learning like convolutional neural networks (CNNs) (Ma and Hovy, 2016a) or bidirectional long short-term memory (BLSTMs) (Lample et al., 2016) can largely eliminate human-crafted features. BLSTM-CRF models have achieved promising performance (Lample et al., 2016) and are used as our base sequence tagging model in this paper.
Crowd-sourced Annotation. Crowd-sourcing has been demonstrated to be an effective way of fulfilling the label consumption of neural models (Guan et al., 2017; Lin et al., 2019). It collects annotations with lower costs and a higher speed from non-expert contributors but suffers from some degradation in quality. Dawid and
Skene (1979) proposes the pioneering work to aggregate crowd annotations to estimate true labels, and Snow et al. (2008) shows its effectiveness with Amazon's Mechanical Turk system. Later works (Dempster et al., 1977; Dredze et al., 2009; Raykar et al., 2010) focus on Expectation-Maximization (EM) algorithms to jointly learn the model and annotator behavior on classification.
Recent research shows the strength of multitask framework in semi-supervised learning (Lan et al., 2018; Clark et al., 2018), cross-type learning (Wang et al., 2018), and learning with entity triggers (Lin et al., 2020). Nguyen et al. (2017); Rodrigues and Pereira (2018); Simpson et al. (2020) regards crowd annotations as noisy gold labels and constructs crowd components to model annotator-specific bias which were discarded during the inference process. It is worth mentioning that, it has been found even for human curated annotations, there exists certain label noise that hinders the model performance (Wang et al., 2019).
Unsupervised Domain Adaptation. Unsupervised cross-domain adaptation aims to transfer knowledge learned from high-resource domains (source domains) to boost performance on low-resource domains (target domains) of interests such as social media messages (Lin et al., 2017). Different from supervised adaptation (Lin and Lu, 2018), we assume there is no labels at all for target corpora. Saito et al. (2017) and Ruder and Plank (2018) explored bootstrapping with multitask tri-training approach, which requires unlabeled data from the target domain. The method is developed for one-to-one domain adaptation and does not model the differences among multiple source domains. Yang and Eisenstein (2015) represents each domain with a vector of metadata domain attributes and uses domain vectors to train the model to deal with domain shifting, which is highly dependent on prior domain knowledge.
(Ghifary et al., 2016) uses an auto-encoder method by jointly training a predictor for source labels, and a decoder to reproduce target input with a shared encoder. The decoder acts as a normalizer to force the model to learn shared knowledge between source and target domains. Adversarial penalty can be added to the loss function to make models learn domain-invariant feature only (Fernando et al., 2015; Long et al., 2014; Ming Harry Hsu et al., 2015). However, it does not exploit domain-specific information.
3 Multi-source Supervised Learning
We formulate the multi-source sequence labeling problem as follows. Given  $K$  sources of supervision, we regard each source as an imperfect annotator (non-expert human tagger or models trained in related domains). For the  $k$ -th source data set  $S^{(k)} = \{(\mathbf{x}_i^{(k)},\mathbf{y}_i^{(k)})\}_{i = 1}^{m_k}$ , we denote its  $i$ -th sentence as  $\mathbf{x}_i^{(k)}$  which is a sequence of tokens:  $\mathbf{x}_i^{(k)} = (x_{i,1}^{(k)},\dots ,x_{i,N}^{(k)})$ . The tag sequence of the sentence is marked as  $\mathbf{y}_i^{(k)} = \{y_{i,j}^{(k)}\}$ . We define the sentence set of each annotators as  $\mathcal{X}^{(k)} = \{\mathbf{x}_i^{(k)}\}_{i = 1}^{m_k}$ , and the whole training domain as the union of all sentence sets:  $\mathcal{X} = \bigcup_{k = 1}^{(K)}\mathcal{X}^{(k)}$ . The goal of the multi-source learning task is to use such imperfect annotations to train a model for predicting the tag sequence  $\mathbf{y}$  for any sentence  $\mathbf{x}$  in a target corpus  $\mathcal{T}$ . Note that the target corpus  $\mathcal{T}$  can either share the same distribution with  $\mathcal{X}$  (Application I) or be significantly different (Application II). In the following two subsections, we formulate two typical tasks in this problem as shown in Fig. 1.
Application I: Learning with Crowd Annotations. When learning with crowd-sourced data, we regard each worker as an imperfect annotator  $(S^{(k)})$ , who may make mistakes or skip sentences in its annotations. Note that in this setting, different annotators tag subsets of the same given dataset  $(\mathcal{X})$ , and thus we assume there are no input distribution shifts among  $\mathcal{X}^{(k)}$ . Also, we only test sentences in the same domain such that the distribution in target corpus  $\mathcal{T}$  is the same as well. That is, the marginal distribution of target corpus  $P_{\mathcal{T}}(\mathbf{x})$  is the same with that for each individual source dataset, i.e.  $P_{\mathcal{T}}(\mathbf{x}) = P_k(\mathbf{x})$ . However, due to imperfectness of the annotations in each source,  $P_{k}(\mathbf{y}|\mathbf{x})$  is shifted from the underlying truth  $P(\mathbf{y}|\mathbf{x})$  (illustrated in the top-left part of Fig. 1). The multi-source learning objective here is to learn a model  $P_{\mathcal{T}}(\mathbf{y}|\mathbf{x})$  for supporting infer
ence on any new sentences in the same domain.
Application II: Unsupervised Cross-Domain Model Adaptation. We assume there are available annotations in several source domains, but not in an unseen target domain. We assume that the input distributions  $P(\mathbf{x})$  in different source domains  $\mathcal{X}^{(k)}$  vary a lot, and such annotations can hardly be adapted for training a target domain model. That is, the prediction distribution of each domain model  $(P_{k}(\mathbf{y}|\mathbf{x}))$  is close to the underlying truth distribution  $(P(\mathbf{y}|\mathbf{x}))$  only when  $\mathbf{x} \in \mathcal{X}^{(k)}$ . For target corpus sentences  $\mathbf{x} \in \mathcal{T}$ , such a source model  $P_{k}(\mathbf{y}|\mathbf{x})$  again differs from underlying ground truth for the target domain  $P_{\mathcal{T}}(\mathbf{y}|\mathbf{x})$  and can be seen as an imperfect annotators. Our objective in this setting is also to jointly model  $P_{\mathcal{T}}(\mathbf{y},\mathbf{x})$  while noticing that there are significant domain shifts between  $\mathcal{T}$  and any other  $\mathcal{X}^{(k)}$ .
4 Consensus Network
In this section, we present our two-phase framework CONNET for multi-source sequence labeling. As shown in Figure 2, our proposed framework first uses a multi-task learning schema with a special objective to decouple annotator representations as different parameters of a transformation around CRF layers. This decoupling phase (Section 4.2) is for decoupling the model parameters into a set of annotator-invariant model parameters and a set of annotator-specific representations. Secondly, the dynamic aggregation phase (Section 4.3) learns to contextually utilize the annotator representations with a lightweight attention mechanism to find the best suitable transformation for each sentence, so that the model can achieve a context-aware consensus among all sources. The inference process is described in Section 4.4.
4.1 The Base Model: BLSTM-CRF
Many recent sequence labeling frameworks (Ma and Hovy, 2016b; Misawa et al., 2017) share a very basic structure: a bidirectional LSTM network followed by a CRF tagging layer (i.e. BLSTM-CRF). The BLSTM encodes an input sequence  $\mathbf{x} = \{x_{1}, x_{2}, \ldots, x_{n}\}$  into a sequence of hidden state vectors  $\mathbf{h}_{1:n}$ . The CRF takes as input the hidden state vectors and computes an emission score matrix  $\mathbf{U} \in \mathbb{R}^{n \times L}$  where  $L$  is the size of tag set. It also maintains a trainable transition matrix  $\mathbf{M} \in \mathbb{R}^{L \times L}$ . We can consider  $\mathbf{U}_{i,j}$  is the score of labeling the tag with id  $j \in \{1, 2, \ldots, L\}$  for  $i^{th}$
Figure 2: Overview of the CONNET framework. The decoupling phase constructs the shared model (yellow) and source-specific matrices (blue). The aggregation phase dynamically combines crowd components into a consensus representation (blue) by a context-aware attention module (red) for each sentence  $x$ .
word in the input sequence  $\mathbf{x}$ , and  $\mathbf{M}_{i,j}$  means the transition score from  $i^{th}$  tag to  $j^{th}$ .
The CRF further computes the score  $s$  for a predicted tag sequence  $\mathbf{y} = \{y_1, y_2, \dots, y_k\}$  as
$$
s (\mathbf {x}, \mathbf {y}) = \sum_ {t = 1} ^ {T} \left(\mathbf {U} _ {t, y _ {t}} + \mathbf {M} _ {y _ {t - 1}, y _ {t}}\right), \tag {1}
$$
and then tag sequence  $\mathbf{y}$  follows the conditional distribution
$$
P (\mathbf {y} | \mathbf {x}) = \frac {\exp s (\mathbf {x} , \mathbf {y})}{\sum_ {\mathbf {y} \in Y _ {\mathbf {x}}} \exp s (\mathbf {x} , \mathbf {y})}. \tag {2}
$$
4.2 The Decoupling Phase: Learning annotator representations
For decoupling annotator-specific biases in annotations, we represent them as a transformation on emission scores and transition scores respectively. Specifically, we learn a matrix  $\mathbf{A}^{(k)}\in \mathbb{R}^{L\times L}$  for each imperfect annotator  $k$  and apply this matrix as transformation on  $\mathbf{U}$  and  $\mathbf{M}$  as follows:
$$
s ^ {(k)} (\mathbf {x}, \mathbf {y}) = \sum_ {t = 1} ^ {T} \left(\left(\mathbf {U A} ^ {(k)}\right) _ {t, y _ {t}} + \left(\mathbf {M A} ^ {(k)}\right) _ {y _ {t - 1}, y _ {t}}\right). \tag {3}
$$
From this transformation, we can see that the original score function  $s$  in Eq. 1 becomes an source-specific computation. The original emission and transformation score matrix  $\mathbf{U}$  and  $\mathbf{M}$  are still shared by all the annotators, while they both are transformed by the matrix  $\mathbf{A}^{(k)}$  for  $k$ -th annotator. While training the model parameters in this phase, we follow a multi-task learning schema. That is, we share the model parameters for BLSTM and CRF (including  $\mathbf{W}$ ,  $\mathbf{b}$ ,  $\mathbf{M}$ ), while updating  $\mathbf{A}^{(k)}$  only by examples in  $S_{k} = \{\mathcal{X}^{(k)}, \mathcal{Y}^{(k)}\}$ .
The learning objective is to minimize the negative log-likelihood of all source annotations:
$$
\mathcal {L} = - \log \sum_ {k = 1} ^ {K} \sum_ {i = 1} ^ {| \mathcal {X} ^ {(k)} |} P \left(\mathbf {y} _ {i} ^ {(k)} \mid \mathbf {x} _ {i} ^ {(k)}\right), \tag {4}
$$
$$
P (\mathbf {y} _ {i} ^ {(k)} | \mathbf {x} _ {i} ^ {(k)}) = \frac {\exp s ^ {(k)} (\mathbf {x} _ {i} ^ {(k)} , \mathbf {y} _ {i} ^ {(k)})}{\sum_ {\mathbf {y} ^ {\prime}} \exp s ^ {(k)} (\mathbf {x} , \mathbf {y} ^ {\prime})}. (5)
$$
The assumption on the annotation representation  $\mathbf{A}^{(k)}$  is that it can model the pattern of annotation bias. Each annotator can be seen as a noisy version of the shared model. For the  $k$ -th annotator,  $\mathbf{A}^{(k)}$  models noise from labeling the current word and transferring from the previous label. Specifically, each entry  $\mathbf{A}_{i,j}^{(k)}$  captures the probability of mistakenly labeling  $i$ -th tag to  $j$ -th tag. In other words, the base sequence labeling model in Sec. 4.1 learns the basic consensus knowledge while annotator-specific components add their understanding to predictions.
4.3 The Aggregation Phase: Dynamically Reaching Consensus
In the second phase, our proposed network learns a context-aware attention module for a consensus representation supervised by combined predictions on the target data. For each sentence in target data  $\mathcal{T}$ , these predictions are combined by weighted voting. The weight of each source is its normalized  $F_{1}$  score on the training set. Through weighted voting on such augmented labels over all source sentences  $\mathcal{X}$ , we can find a good approximation of underlying truth labels.
For better generalization and higher speed, an attention module is trained to estimate the relevance of each source to the target under the supervision of generated labels. Specifically, we compute the sentence embedding by concatenating the last hidden states of the forward LSTM and the backward LSTM, i.e.  $\mathbf{h}^{(i)} = [\overrightarrow{\mathbf{h}}_T^{(i)};\overleftarrow{\mathbf{h}}_0^{(i)}]$ . The attention module inputs the sentence embedding and outputs a normalized weight for each source:
$$
\mathbf {q} _ {i} = \operatorname {s o f t m a x} \left(\mathbf {Q h} ^ {(i)}\right), \text {w h e r e} \mathbf {Q} \in \mathbb {R} ^ {K \times 2 d}. \tag {6}
$$
where  $d$  is the size of each hidden state  $\mathbf{h}^{(i)}$ . Source-specific matrices  $\{\mathbf{A}^{(k)}\}_{k = 1}^{K}$  are then aggregated into a consensus representation  $\mathbf{A}_i^*$  for sentence  $\mathbf{x}_i\in \mathcal{X}$  by
$$
\mathbf {A} _ {i} ^ {*} = \sum_ {k = 1} ^ {K} q _ {i, k} \mathbf {A} ^ {(k)}. \tag {7}
$$
In this way, the consensus representation contains more information about sources which are more related to the current sentence. It also alleviates the contradiction problem among sources, because it could consider multiple sources of different emphasis. Since only an attention model with weight matrix  $\mathbf{Q}$  is required to be trained, the amount of computation is relatively small. We assume the base model and annotator representations are well-trained in the previous phase. The main objective in this phase is to learn how to select most suitable annotators for the current sentence.
4.4 Parameter Learning and Inference
CONNET learns parameters through two phases described above. In the decoupling phase, each instance from source  $S_{k}$  is used for training the base sequence labeling model and its representation  $\mathbf{A}^{(k)}$ . In the aggregation phase, we use aggregated predictions from the first phase to learn a lightweight attention module. For each instance in the target corpus  $\mathbf{x}_i \in \mathcal{T}$ , we calculate its embedding  $\mathbf{h}_i$  from BLSTM hidden states. With these sentence embeddings, the context-aware attention module assigns weight  $\mathbf{q}_i$  to each source and dynamically aggregates source-specific representations  $\{\mathbf{A}^{(k)}\}$  for inferring  $\hat{\mathbf{y}}_i$ . In the inference process, only the consolidated consensus matrix  $\mathbf{A}_i^*$  is applied to the base sequence learning model. In this way, more specialist knowledge helps to deal with more complex instances.
4.5 Model Application
The proposed model can be applied to two practical multi-sourcing settings: learning with crowd annotations and unsupervised cross-domain model adaptation. In the crowd annotation learning setting, the training data of the same domain is annotated by multiple noisy annotators, and each annotator is treated as a source. In the decoupling phase, the model is trained on noisy annotations, and in the aggregation phase, it is trained with combined predictions on the training set. In the cross-domain setting, the model has access to
unlabeled training data of the target domain and clean labeled data of multiple source domains. Each domain is treated as a source. In the decoupling phase, the model is trained on source domains, and in the aggregation phase, the model is trained on combined predictions on the training data of the target domain. Our framework can also extend to new tasks other than sequence labeling and cope with different encoders. We will demonstrate this ability in experiments.
Our method is also incorporated as a feature for controlling the quality of crowd-annotation in annotation frameworks such as AlpacaTag (Lin et al., 2019) and LEAN-LIFE (Lee et al., 2020).
We evaluate CONNET in the two aforementioned settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation. Additionally, to demonstrate the generalization of our framework, we also test our method on sequence labeling with transformer encoder in Appendix B and text classification with MLP encoder in Section 5.5.
Crowd-Annotation Datasets. We use crowd-annotation datasets based on the 2003 CoNLL shared NER task (Tjong Kim Sang and De Meulder, 2003). The real-world datasets, denoted as AMT, are collected by Rodrigues et al. (2014) using Amazon's Mechanical Turk where F1 scores of annotators against the ground truth vary from  $17.60\%$  to  $89.11\%$ . Since there is no development set in AMT, we also follow Nguyen et al. (2017) to use the AMT training set and CoNLL 2003 development and test sets, denoted as AMTC. Overlapping sentences are removed in the training set, which is ignored in that work. Additionally, we construct two sets of simulated datasets to investigate the quality and quantity of annotators. To simulate the behavior of a non-expert annotator, a CRF model is trained on a small subset of training data and generates predictions on the whole set. Because of the limited size of training data, each model would have a bias to certain patterns.
Cross-Domain Datasets. In this setting, we investigate three NLP tasks: POS tagging, NER and text classification. For POS tagging task, we use the GUM portion (Zeldes, 2017) of Universal Dependencies (UD) v2.3 corpus with 17 tags and 7
Methods	AMTC	AMT
Precision(%)	Recall(%)	F1-score(%)	Precision(%)	Recall(%)	F1-score(%)
CONCAT-SLM	85.95(±1.00)	57.96(±0.26)	69.23(±0.13)	91.12(±0.57)	55.41(±2.66)	68.89(±1.92)
MVT-SLM	84.78(±0.66)	62.50(±1.36)	71.94(±0.66)	86.96(±1.22)	58.07(±0.11)	69.64(±0.31)
MVS-SLM	84.76(±0.50)	61.95(±0.32)	71.57(±0.04)	86.95(±1.12)	56.23(±0.01)	68.30(±0.33)
DS-SLM (Nguyen et al., 2017)	72.30*	61.17*	66.27*	-	-	-
HMM-SLM (Nguyen et al., 2017)	76.19*	66.24*	70.87*	-	-	-
MTL-MVT (Wang et al., 2018)	81.81(±2.34)	62.51(±0.28)	70.87(±1.06)	88.88(±0.25)	65.04(±0.80)	75.10(±0.44)
MTL-BEA (Rahimi et al., 2019)	85.72(±0.66)	58.28(±0.43)	69.39(±0.52)	77.56(±2.23)	67.23(±0.72)	72.01(±0.85)
CRF-MA (Rodrigues et al., 2014)	-	-	-	49.40*	85.60*	62.60*
Crowd-Add (Nguyen et al., 2017)	85.81(±1.53)	62.15(±0.18)	72.09(±0.42)	89.74(±0.10)	64.50(±1.48)	75.03(±1.02)
Crowd-Cat (Nguyen et al., 2017)	85.02(±0.98)	62.73(±1.10)	72.19(±0.37)	89.72(±0.47)	63.55(±1.20)	74.39(±0.98)
CL-MW (Rodrigues and Pereira, 2018)	-	-	-	66.00*	59.30*	62.40*
CONNET (Ours)	84.11(±0.71)	68.61(±0.03)	75.57(±0.27)	88.77(±0.25)	72.79(±0.04)	79.99(±0.08)
Gold (Upper Bound)	89.48(±0.32)	89.55(±0.06)	89.51(±0.21)	92.12(±0.31)	91.73(±0.09)	91.92(±0.21)
Table 1: Performance on real-world crowd-sourced NER datasets. The best score in each column excepting Gold is marked bold. * indicates number reported by the paper.