Title: LEARNING TO CONTEXTUALLY AGGREGATE MULTI-SOURCE SUPERVISION FOR SEQUENCE LABELING

Abstract: Sequence labeling is a fundamental framework for various natural language processing problems including part-of-speech tagging and named entity recognition. Its performance is largely influenced by the annotation quality and quantity in supervised learning scenarios. In many cases, ground truth labels are costly and time-consuming to collect or even non-existent, while imperfect ones could be easily accessed or transferred from different domains. A typical example is crowdsourced datasets which have multiple annotations for each sentence which may be noisy or incomplete. Additionally, predictions from multiple source models in transfer learning can be seen as a case of multi-source supervision. In this paper, we propose a novel framework named Consensus Network (CONNET) to conduct training with imperfect annotations from multiple sources. It learns the representation for every weak supervision source and dynamically aggregates them by a context-aware attention mechanism. Finally, it leads to a model reflecting the consensus among multiple sources. We evaluate the proposed framework in two practical settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation. Extensive experimental results show that our model achieves significant improvements over existing methods in both settings. 1 

1. INTRODUCTION:

Sequence labeling is a fundamental framework for various natural language processing (NLP) tasks including part-of-speech (POS) tagging (Ratnaparkhi, 1996) , noun phrase chunking (Sang & Buchholz, 2000) , word segmentation (Low et al., 2005) , and named entity recognition (NER) (Nadeau & Sekine, 2007) . Typically, existing methods follow the supervised learning paradigm, and require high-quality annotations. While ground truth labels are expensive and time-consuming, imperfect annotations are much easier to obtain from crowdsourcing (noisy labels) or other domains (out-ofdomain). To alleviate or even address the problem of noise and incompleteness, it is important and beneficial to learn from multiple sources. Specifically, we are interested in two typical application scenarios: 1) learning with crowd annotations and 2) unsupervised cross-domain model adaptation, which are detailed in Sec. 2 and Sec. 3. The key challenge of learning with multi-source supervision is to aggregate annotators for learning a model without knowing the underlying ground truth label sequences in the target domain. Many attempts have been made in generalizing multiple informative sources to an out-of-domain distribution for a range of tasks, including multi-class classification (Sheshadri & Lease, 2013) , object detection (Su et al., 2012) and information extraction (Liu et al., 2017) . However, most of the prior works (Nguyen et al., 2017; Wang et al., 2019; Peng & Dredze, 2016; Yang et al., 2017; Chen & Cardie, 2018) choose to use simple heuristic-based methods of aggregating source models for performing on a target corpus without carefully calibration.

Our intuition is mainly from the common phenomenon that each source of supervision has distinct strength in different inputs, and thus they should not keep consistent importance in aggregating supervisions when inferring tag sequences for new inputs. Aggregating multiple sources for a specific input should be a dynamic process depending on the sentence context rather than a fixed way. To In this paper, we propose a novel framework, named Consensus Network (CONNET), for sequence labeling with multi-source supervisions. We represent the annotation patterns as different biases of annotators over a shared behavior pattern. Both annotator-invariant patterns and annotator-specific biases are modeled in a decoupled way. The first term through sharing part of low-level model parameters in a multi-task learning schema. For learning the biases, we decouple them from the model as the transformations on top-level tagging model parameters, such that they can capture the unique strength of each annotator. With such decoupled source representations, we further learn an attention network for dynamically assigning the best sources for every unseen sentence through composing a transformation that represents the consensus. Extensive experimental results in two scenarios show that our model always outperforms strong baseline methods. CONNET achieves the state-of-the-art performance on real-world crowdsourcing datasets and improve significantly in most unsupervised cross-domain adaptation tasks over existing works. In addition to sequence labeling, it also shows its effectiveness on text classification tasks.

2. RELATED WORK:

Neural Sequence Labeling. Traditional approaches for sequence labeling usually need significant efforts in feature engineering for graphical models like hidden markov models (HMMs) (Rabiner, 1989) and conditional random fields (CRFs) (Lafferty, 2001) . Recent research efforts in neural network models have shown that end-to-end learning like convolutional neural networks (CNNs) (Ma & Hovy, 2016b) or bidirectional long short-term memory (BLSTMs) (Lample et al., 2016) can largely eliminate human-crafted features. Together with a final CRF layer, these BLSTM-CRF models have achieved promising performance and are used as our base sequence tagging model in this paper.

Crowd-sourced Annotation. Crowd-sourcing has been demonstrated to be an effective way of fulfilling the label consumption of neural models (Guan et al., 2017) . It collects annotations with lower costs and a higher speed by non-expert contributors but suffers from some degradation in quality. Dawid & Skene (1979) proposes the pioneering work to aggregate crowd annotations to estimate true labels, and Snow et al. (2008) shows its effectiveness with Amazon's Mechanical Turk system. Later works (Dempster et al., 1977; Dredze et al., 2009; Raykar et al., 2010) focus on Expectation-Maximization (EM) algorithms to jointly learn the model and annotator behavior on classification problems. Recent research shows the strength of multi-task framework in semisupervised learning (Lan et al., 2018; Clark et al., 2018) , and cross-type learning (Wang et al., 2018) . Nguyen et al. (2017) and Rodrigues & Pereira (2018) regards crowd annotations as noisy versions of glod labels and constructs crowd components to model annotator-specific bias which were discarded during the inference process.

Unsupervised Domain Adaptation. Unsupervised cross-domain adaptation aims to transfer knowledge learned from high-resource domains (source domains) to boost performance on low-resource domains (target domains). Different from supervised adaptation (Lin & Lu, 2018), we assume there is no labels at all for target corpora. Saito et al. (2017) and Ruder & Plank (2018) explored bootstrapping with multi-task tri-training approach. The method is developed for one-to-one domain adaptation and does not model the differences among multiple source domains. Yang & Eisenstein (2015) represents each domain with a vector of metadata domain attributes and uses domain vectors to train the model to deal with domain shifting. Ghifary et al. (2016) uses an auto-encoder method by jointly training a predictor for source labels, and a decoder to reproduce target input with a shared encoder. The decoder acts as a normalizer to force the model to learn shared knowledge between source and target domains. Adversarial penalty can be applied to the loss function to make models learn domain-invariant feature only (Fernando et al., 2015; Ming Harry Hsu et al., 2015) .

3. LEARNING WITH MULTI-SOURCE SUPERVISION:

We consider the multi-source sequence labeling problem as follows. There are N sources of supervision, each source can be regarded as an imperfect annotator (non-expert human tagger or models trained in related domains). For the k-th source data set

S (k) = {(x (k) i , y (k) i )} m k i=1 , we denote its i-th sentence as x (k)

i which is a sequence of tokens:

x (k) i = (x (k) i,1 , • • • , x (k) i,N ).

The tag sequence of the sentence is thus to be y

(k) i = {y (k)

i,j }. We define the sentence set of each annotators as

X (k) = {x (k) i } m k i=1 ,

and the whole training domain as the union of all sentence sets: X = (K) k=1 X (k) . The goal of the multi-source learning task is to use such imperfect annotations to train a model for predicting the tag sequence y for any sentence x in a target corpus T . Note that the target corpus T can either share the same distribution with X (Application I) or be significantly different (Application II). In the following two subsections, we formulate two typical tasks in this problem as shown in Fig. 1 .

Application I: Learning with Crowd Annotations. When learning with crowd-sourced sequence labeling data, we regard each worker as an imperfect annotator (S (k) ), who may make mistakes or skip sentences in its annotations. Note that for crowd-sourcing data, different annotators tag subsets of the same given dataset (X ), and thus we assume there are no input distribution shifts among X (k) . Also, we only test sentences in the same domain such that the distribution in target corpus T is the same as well. That is, the marginal distribution of target corpus P T (x) is the same with that for each individual source dataset, i.e. P T (x) = P k (x). However, due to imperfectness of the annotations in each source, the P k (y|x) has obvious shift from the underlying truth P (y|x) (illustrated in the top-left part of Fig. 1 ). The multi-source learning objective here is to learn a model P T (y|x) for supporting inference on any new sentences in the same domain.

Application II: Unsupervised Cross-Domain Model Adaptation. We assume that we have wellannotated data in some source domains while having no labels in the target domain at all. Following the formulation, we claim that the input distributions P (x) in different source domains X (k) vary a lot, and fitting those annotations can only generalize to in-domain samples. That is, P k (y|x) ≈ P (y|x) only for x ∈ X (k) . For target corpus sentences x ∈ T , such a source model P k (y|x) again differs from underlying ground truth for the target domain P T (y|x) and can be seen as an imperfect annotators. Our objective in this setting is also to jointly model P T (y, x) while noticing that there are significant domain shifts between T and any other X (k) .

4. PROPOSED APPROACH: CONSENSUS NETWORK:

In this section, we present our two-phase framework CONNET for multi-source sequence labeling. As shown in Figure 2 , our proposed framework first uses a multi-task learning schema with a special objective to decouple annotator representations as different parameters of a transformation around CRF layers. This decoupling phase (Section 4.2) is for decoupling the model parameters into a set of annotator-invariant model parameters and a set of annotator-specific representations. Secondly, the dynamic aggregation phase (Section 4.3) learns to contextually utilize the annotator representations with a lightweight attention mechanism to find the best suitable transformation for each sentence, so that the model can achieve a context-aware consensus among all sources. The inference process is described in Section 4.4.

4.1. THE BASE MODEL FOR SEQUENCE LABELING: BLSTM-CRF:

Many recent sequence labeling frameworks (Ma & Hovy, 2016a; Misawa et al., 2017) share a very basic structure: a bidirectional LSTM network followed by a CRF tagging layer (i.e. BLSTM-CRF). The BLSTM encodes an input sequence x = {x 1 , x 2 , . . . , x n } into a sequence of hidden state vectors h 1:n . The CRF takes as input the hidden state vectors and computes an emission score matrix U ∈ R n×L where L is the size of tag set. It also maintains a trainable transition matrix M ∈ R L×L . We can consider U i,j is the score of labeling the tag with id j ∈ {1, 2, ..., L} for i th word in the input sequence x, and M i,j means the transition score from i th tag to j th .

The CRF further computes the score s for a predicted tag sequence y = {y 1 , y 2 , ..., y k } as

EQUATION

then tag sequence y follows the conditional distribution

P (y|x) = exp s(x, y) y∈Yx exp s(x, y)

.

(2)

4.2. THE DECOUPLING PHASE: LEARNING ANNOTATOR REPRESENTATIONS:

For decoupling annotator-specific biases in annotations, we represent them as a transformation on emission scores and transition scores respectively. Specifically, we learn a matrix A (k) ∈ R L×L for each imperfect annotator k and apply this matrix as transformation on U and M as follows:

EQUATION

From this transformation, we can see that the original score function s in Eq. 1 becomes an annotatorspecific computation. The original emission and transformation score matrix U and M are still shared by all the annotators, while they both are transformed by the matrix A (k) for k-th annotator.

While training the model parameters in this phase, we follow a multi-task learning schema. That is, we share the model parameters for BLSTM and CRF (including W, b, M), while updating

A (k) only by examples in S k = {X (k) , Y (k) }.

The assumption on the annotation representation A (k) is that it can model the pattern of annotation bias. Each annotator can be seen as a noisy version of the shared model. For the k-th annotator, A (k) models noise from labeling the current word and transferring from the previous label. Specifically, each entry A (k) i,j captures the probability of mistakenly labeling i-th tag to j-th tag. In other words, the base sequence labeling model in Sec. 4.1 learns the basic consensus knowledge while annotatorspecific components add their understanding to predictions.

4.3. THE AGGREGATION PHASE: DYNAMICALLY REACHING CONSENSUS:

In the second phase, our proposed network learns a context-aware attention module for a consensus representation supervised by combined predictions on the target data. For each sentence in target data T , we use the model obtained from the decoupling phase to make predictions, and combine predictions using weighted voting. The weight of each source is its normalized F 1 score on the training set. Through weighted voting on such augmented labels over all source sentences X , we can find a good approximation of underlying truth labels.

For better generalization and higher speed, an attention module is trained to estimate the relevance of each source to the target under the supervision of generated labels. Specifically, we compute embedding of each sentence by concatenating the last hidden states of the forward LSTM and the backward LSTM, i.e.

h (i) = [ - → h (i) T ; ← - h (i) 0 ]

. The attention module takes as input the sentence embedding and outputs a normalized weight for each source:

EQUATION

)

where d is the size of each hidden state h (i) . Source-specific matrices {A (k) } K k=1 are then aggregated into a consensus representation A * i for sentence x i ∈ X by

EQUATION

In this way, the consensus representation contains more information about sources which are more related to the current type of sentence. It also alleviates the contradiction problem among sources, because it could consider multiple sources of different emphasis. Since only an attention model with weight matrix Q is trained, the amount of computation is relatively small. We assume the sequence labeling model and annotator representations are well-trained in the previous phase. The main objective in this phase is to learn how to select most suitable annotators for the current sentence.

4.4. PARAMETER LEARNING AND INFERENCE OF CONNET:

CONNET learns parameters through two phases described above. In the first decoupling phase, each training instance from source S k is used for training the base sequence labeling model and its specific representation A (k) . In the second aggregation phase, we use aggregated predictions from the first phase to learn a lightweight attention module. For each instance in the target corpus x i ∈ T , we calculate its embedding h i from BLSTM hidden states. With these sentence embeddings, the context-aware attention module assigns weight q i to each source and dynamically aggregates source-specific representations {A (k) } for inferring ŷi . In the inference process, only the consolidated consensus matrix A * i is applied to the base sequence learning model. In this way, more specialist knowledge helps to deal with more complex instances.

4.5. MODEL APPLICATION:

The proposed model can be applied to multi-sourcing learning. Here we describe the application in two practical settings: learning with crowd annotations and unsupervised cross-domain model adaptation. In the crowd annotation learning setting, the training data of the same domain is annotated by multiple noisy annotators, and each annotators is treated as a source. In the decoupling phase, the model is trained on noisy annotations, and in the aggregation phase, it is trained with combined predictions on the training set. In the cross-domain setting, the model has access to unlabeled training data of the target domain and clean labeled data of multiple source domains. Each domain is treated as a source. In the decoupling phase, the model is trained on source domains, and in the aggregation phase, the model is trained on combined predictions on the training data of the target domain.

Our framework can also potentially extend to new tasks other than sequence labeling with different encoders. For example, if we use an MLP as encoder, we can transform its hidden representation by multiplying with crowd/consensus matrices. The transformed representation will then used by a decoder to make predictions. We will demonstrate this ability in experiments.

5. EXPERIMENTS:

We evaluate CONNET in the two aforementioned settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation. Additionally, to demonstrate the generalization of our framework, we also test our method on sequence labeling with transformer encoder in Appendix B and text classification with MLP encoder in Section 5.5. Cross-Domain Datasets. In this setting, we investigate three NLP tasks: POS tagging, NER and text classification. For POS tagging task, we use the GUM portion (Zeldes, 2017) of Universal Dependencies (UD) v2.3 corpus with 17 tags and 7 domains: academic, bio, fiction, news, voyage, wiki, and interview. For NER task, we select the English portion of the OntoNotes v5 corpus (Hovy et al., 2006) . The corpus is annotated with 9 named entities with data from 6 domains: broadcast conversation (bc), broadcast news (bn), magazine (mz), newswire (nw), pivot text (pt), telephone conversation (tc), and web (web). Multi-Domain Sentiment Dataset (MDS) v2.0 (Blitzer et al., 2007) is used for text classification, which is built on Amazon reviews from 4 domains: books, dvd, electronics, and kitchen. Since the dataset only contains word frequencies for each review without raw texts, we follow the setting in Chen & Cardie (2018) considering 5,000 most frequent words and use the raw counts as the feature vector for each review.

5.2. EXPERIMENT SETUP:

For sequence labeling tasks, we follow Liu et al. (2018) to build the BLSTM-CRF architecture as the base model. The dimension of character-level, word-level embeddings and BLSTM hidden layer are set as 30, 100 and 150 respectively. For text classification, We use an MLP with a hidden size of 100 as encoder and a linear classification layer for predicting their labels. The dropout with a probability of 0.5 is applied to the non-recurrent connections for regularization. The network parameters are randomly initialized and updated by stochastic gradient descent (SGD). The learning rate is initialized as 0.015 and decayed by 5% for each epoch. The training process stops early if no improvements in 15 continuous epochs and selects the best model on the development set. For the dataset without a development set, we report the performance on the 50-th epoch. For each experiment, we report the average performance of 3 runs with different random initialization.

5.3. COMPARISON WITH BASELINE METHODS:

We compare our models with multiple baselines, which can be categorized in two groups: wrapper methods and joint models. To demonstrate the theoretical upper bound of performance, we also train the base model using ground-truth annotations in the target domain (Gold).

A wrapper method consists of a label aggregator and a deep learning model. These two components could be combined in two ways: (1) aggregating labels on crowd-sourced training set then feeding the generated labels to a Sequence Labeling Model (SLM) (Liu et al., 2017) ; (2) feeding multi-source data to a Multi-Task Learning (MTL) (Wang et al., 2018) model then aggregating multiple predicted labels. We investigate multiple label aggregation strategies. CONCAT considers all crowd annotations as gold labels. MVT does majority voting on the token level, i.e., the majority of labels {y k i,j } is selected as the gold label for each token x i,j . MVS is conducted on the sequence level, addressing the problem of violating Begin/In/Out (BIO) rules. DS (Dawid & Skene, 1979) , HMM (Nguyen et al., 2017) and BEA (Rahimi et al., 2019) induce consensus labels with probability models.

In contrast with wrapper methods, joint models incorporate multi-source data within the structure of sequential taggers and jointly model all of the individual annotators. CRF-MA models CRFs with Multiple Annotators by EM algorithm (Rodrigues et al., 2014) . Nguyen et al. (2017) augments the LSTM architecture with crowd vectors. These crowd components are element-wise added to the tags scores (Crowd-Add) or concatenated to the output of the hidden layer (Crowd-Cat). These two methods are the most similar to our extraction phase. We implemented them and got better results than reported. CL-MW applies a crowd layer to a CNN-based deep learning framework (Rodrigues & Pereira, 2018) . Tri-Training uses bootstrapping with multi-task Tri-Training approach for unsupervised one-to-one domain adaptation (Saito et al., 2017; Ruder & Plank, 2018) .

Methods:



5.4. PERFORMANCE ON LEARNING WITH CROWD ANNOTATIONS:

Tab. 1 shows the performance of aforementioned methods and our CONNET on two real-world datasets, i.e. AMT and AMTC. We can see that CONNET outperforms all other methods on both datasets significantly on F 1 score, which shows the effectiveness of dealing with noisy annotations for higher-quality labels. Although CONCAT-SLM achieves the highest precision, it suffers from low recall. All existing methods have the high-precision but low-recall problem. One possible reason is that they try to find the latent ground truth and throw away illuminating annotator-specific information. So only simple mentions can be classified with great certainty while difficult mentions fail to be identified without sufficient knowledge. In comparison, CONNET pools information from all annotations and focus on matching knowledge to make predictions. It makes the tagging model be able to identify more mentions and get a higher recall.

It is enlightening to analyze whether the model decides the importance of annotators given a sentence. In Fig. 4 we visualize test F1 score of all annotators on each tag, and attention weights q i in Eq. 4 for 4 sampled sentences containing different entity types. Obviously, the 2nd sample sentence with ORG has higher attention weights on 1st, 5th and 33rd annotator who are best at labeling ORG. More details and cases are shown in Appendix A.1. We also investigate multiple variants and conduct ablation study in Appendix A.2.

To analyze the impact of annotator quality, we split the origin train set into {5, 10, 15, 30, 50} folds and train a CRF model on each fold whose reliability could be represented as r = {1/5, 1/10, 1/15, 1/30, 1/50} because a model with less training data would have stronger bias and less generalization. For each setting, we randomly select 5 trained models as the simulated annotators to annotate the whole training set. When the reliability level of all annotators is too low, i.e. 1/50, only the base model is used for prediction without annotator representations. The models are then trained with simulated annotations. Shown in Fig. 3 (a), CONNET achieves significant improvements over MVT-SLM and competitive performance as Crowd-Cat. Our model shows its effectiveness when annotators are less reliable.

Regarding the influence of annotator quantity, we split the train set into 50 subsets to train 50 models (r = 1/50) respectively and randomly select {5, 10, 15, 30, 50} models as simulated annotators to annotate the whole training set. The models are then trained with simulated annotations. Fig. 3(b) shows CONNET is superior to baselines and able to well deal with many annotators while there is no obvious relationship between the performance and the number of annotators in baselines. We can see the performance of our model increases as the number of annotators and, regardless of the number of annotators, our method consistently outperforms than other baselines. Annotator ID To better understand the effect and benefit of CONNET, we do some case study on AMTC real-world dataset with 47 annotators. We look into some more instances to investigate the ability of attention module to find right annotators in Fig. 5 and Tab. 3. Sentence 1-12 contains a specific entity type respectively while 13-16 contains multiple different entities. Compared with expertise of annotators, we can see that the attention module would give more weight on annotators who have competitive performance and preference on the included entity type. Although top selected annotators for ORG has relatively lower expertise on ORG than PER and LOC, they are actually the top three annotators with highest expertise on ORG.

5.5. PERFORMANCE ON CROSS-DOMAIN ADAPTATION:

the attention scores are averaged for each source-target pair. Fig. 6(b ) shows all the source-to-target average attention scores. We can see that some domains can contribute to other related domains. For example, bn (broadcast news) and nw (newswire) are both about news and they contribute to each other; bn and bc (broadcast conversation) are both broadcast and bn contributes to bc; bn and nw both contributes to mz (magzine) probably because they are all about news; wb (web) and tc (telephone conversation) almost make no positive contribution to any other, which is reasonable because they are informal texts compared to others and they are not necessarily related to the other. Overall the attention scores can make some sense to human, and it suggests that the attention is meaningful and contributing to the model. To demonstrate the generalization of our framework, we re-implement CONNET and some baselines (MTV-SLM, Crowd-Add, Gold) with Transformer-CRF as the base model. Specifically, the base model takes Transformer as the encoder for CRF, which has shown its effectiveness in many NLP tasks (Vaswani et al., 2017; Devlin et al., 2018) . Transformer models sequences with self-attention and eliminates all recurrence. Following the experimental settings from Vaswani et al. (2017) , we set the number of heads for multi-head attention as 8, the dimension of keys and values as 64, and the hidden size of the feed-forward layers as 1024. We conduct experiments with crowd-annotation dataset AMTC on NER task and cross-domain dataset UD on POS task, which are described in Section 5.1. Results are shown in Table 5 . We can see our model outperforms over other baselines in both tasks and applications.

:

Code and data have been uploaded and will be published up to the acceptance of the paper.

Figures and Tables:

Figure 1: Illustration of the task settings for the two applications in this work: (a) learning consensus model from crowd annotations; (b) unsupervised cross-domain model adaptation.

Figure 2: CONNET Overview. The decoupling phase constructs the shared model (yellow) and source-specific matrices (blue). The aggregation phase dynamically combines crowd components into a consensus representation (blue) by a context-aware attention module (red) for each sentence x.

Datasets. We use crowd-annotation datasets based on the 2003 CoNLL shared NER task(Sang & De Meulder, 2003). The real-world datasets, denoted as AMT, are collected byRodrigues et al. (2014) using Amazon's Mechanical Turk where F1 scores of annotators against the ground truth vary from 17.60% to 89.11%. Since there is no development set in AMT, we also followNguyen et al. (2017) to use the AMT training set and CoNLL 2003 development and test sets, denoted as AMTC. Overlapping sentences are removed in the training set, which is ignored in that work. Additionally, we construct two sets of simulated datasets to investigate the quality and quantity of annotators. To simulate the behavior of a non-expert annotator, a CRF model is trained on a small subset of training data and generates predictions on the whole set. Because of the limited size of training data, each model would have a bias to certain patterns.

Figure 3: Performance on simulated crowd-sourced NER datasets with (a) 5 annotators on different reliability levels; (b) various numbers of annotators whose reliability r = 1/50.

Figure 4: Visualizations of (a) the expertise of annotators; (b) attention weights for sample sentences. More cases and details are described in Appendix A.1.

Figure 5: Visualizations of (a) the expertise of annotators; (b) attention weights for additional sample sentences to Fig. 4. Details of samples are described in Tab. 3.

Performance on real-world crowd-sourced NER datasets. (* indicates number reported by the paper.)

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td>AMTC</td><td/><td/><td/><td>AMT</td></tr><tr><td/><td/><td/><td colspan="3">Precision(%)</td><td>Recall(%)</td><td colspan="2">F1-score(%)</td><td colspan="2">Precision(%)</td><td>Recall(%)</td><td>F1-score(%)</td></tr><tr><td colspan="2">CONCAT-SLM</td><td/><td colspan="8">85.95(±1.00) 57.96(±0.26) 69.23(±0.13) 91.12(±0.57) 55.41(±2.66) 68.89(±1.92)</td></tr><tr><td>MVT-SLM</td><td/><td/><td colspan="8">84.78(±0.66) 62.50(±1.36) 71.94(±0.66) 86.96(±1.22) 58.07(±0.11) 69.64(±0.31)</td></tr><tr><td>MVS-SLM</td><td/><td/><td colspan="8">84.76(±0.50) 61.95(±0.32) 71.57(±0.04) 86.95(±1.12) 56.23(±0.01) 68.30(±0.33)</td></tr><tr><td colspan="3">DS-SLM (Nguyen et al., 2017)</td><td/><td>72.30  *</td><td/><td>61.17  *</td><td/><td>66.27  *</td><td>-</td><td>-</td><td>-</td></tr><tr><td colspan="3">HMM-SLM (Nguyen et al., 2017)</td><td/><td>76.19  *</td><td/><td>66.24  *</td><td/><td>70.87  *</td><td>-</td><td>-</td><td>-</td></tr><tr><td colspan="3">MTL-MVT (Wang et al., 2018)</td><td colspan="8">81.81(±2.34) 62.51(±0.28) 70.87(±1.06) 88.88(±0.25) 65.04(±0.80) 75.10(±0.44)</td></tr><tr><td colspan="3">MTL-BEA (Rahimi et al., 2019)</td><td colspan="8">85.72(±0.66) 58.28(±0.43) 69.39(±0.52) 77.56(±2.23) 67.23(±0.72) 72.01(±0.85)</td></tr><tr><td colspan="3">CRF-MA (Rodrigues et al., 2014)</td><td/><td>-</td><td/><td>-</td><td/><td>-</td><td>49.40  *</td><td>85.60  *</td><td>62.60  *</td></tr><tr><td colspan="3">Crowd-Add (Nguyen et al., 2017)</td><td colspan="8">85.81(±1.53) 62.15(±0.18) 72.09(±0.42) 89.74(±0.10) 64.50(±1.48) 75.03(±1.02)</td></tr><tr><td colspan="3">Crowd-Cat (Nguyen et al., 2017)</td><td colspan="8">85.02(±0.98) 62.73(±1.10) 72.19(±0.37) 89.72(±0.47) 63.55(±1.20) 74.39(±0.98)</td></tr><tr><td colspan="4">CL-MW (Rodrigues &amp; Pereira, 2018)</td><td>-</td><td/><td>-</td><td/><td>-</td><td>66.00  *</td><td>59.30  *</td><td>62.40  *</td></tr><tr><td colspan="2">CONNET (Ours)</td><td/><td colspan="8">84.11(±0.71) 68.61(±0.03) 75.57(±0.27) 88.77(±0.25) 72.79(±0.04) 79.99(±0.08)</td></tr><tr><td colspan="2">Gold (Upper Bound)</td><td/><td colspan="8">89.48(±0.32) 89.55(±0.06) 89.51(±0.21) 92.12(±0.31) 91.73(±0.09) 91.92(±0.21)</td></tr><tr><td/><td>80.00</td><td/><td/><td/><td/><td/><td/><td>60.00</td><td/></tr><tr><td/><td>70.00</td><td/><td/><td/><td/><td/><td/><td>55.00</td><td/></tr><tr><td>F1 (%)</td><td>60.00</td><td/><td/><td/><td/><td/><td>F1 (%)</td><td>45.00 50.00</td><td/></tr><tr><td/><td/><td>MVT-SLM</td><td/><td/><td/><td/><td/><td/><td/><td>MVT-SLM</td></tr><tr><td/><td>50.00</td><td>Crowd-Cat</td><td/><td/><td/><td/><td/><td>40.00</td><td/><td>Crowd-Cat</td></tr><tr><td/><td/><td>ConNet</td><td/><td/><td/><td/><td/><td/><td/><td>ConNet</td></tr><tr><td/><td>40.00</td><td/><td/><td/><td/><td/><td/><td>35.00</td><td/></tr><tr><td/><td>1/5</td><td>1/10</td><td>1/15</td><td>1/30</td><td>1/50</td><td/><td/><td>5</td><td>10</td><td>15</td><td>30</td><td>50</td></tr><tr><td/><td/><td colspan="3">(a) Reliability Level</td><td/><td/><td/><td/><td colspan="2">(b) Annotator Number</td></tr></table>

Performance on cross-domain adaptation. The average score for all domains is reported for each task. The best score in each column that is significantly (p < 0.05) better than the second-best is marked bold, while those are better but not significantly are underlined. Detailed results can be found in Appendix A.3.The average performance of each method on each task is shown in Tab. 2. More detailed results on each target domain can be found in Appendix A.3. We report the accuracy for POS tagging and text classification, and report the chunk-level F1 score for NER. We can see that CONNET achieves the highest average score in all tasks. MTL-MVT is similar to our decoupling phase without the attention module, which performs much worse. It shows that naively doing unweighted voting does not work well. The attention can be viewed as implicitly doing weighted voting on the feature level. This demonstrates the importance of having such a module to assign weights to all domains based on the input sentence. The Tri-Training model trained on the concatenated training data from all sources performs worse than CONNET, which suggests that it is important to have a multi-task structure to model the difference among domains. We analyze the OntoNotes dataset to show that the attention scores generated by the model are meaningful. Details can be found in Appendix A.4.6 CONCLUSIONIn this paper, we present CONNET for learning a sequence tagger from multi-source supervision. It could be applied in two practical scenarios: learning with crowd annotations and cross-domain adaptation. In contrast to prior works, CONNET learns fine-grained representations of each source which are further dynamically aggregated for every unseen sentence in the target data. Experiments show that our model is superior to previous crowd-sourcing and unsupervised domain adaptation sequence labeling models. The proposed learning framework also shows promising results on other NLP tasks like text classification.A ANALYSIS OF CONNET WITH BLSTM ENCODER A.1 CASE STUDY ON LEARNING WITH CROWD ANNOTATIONS

Table content: <table><tr><td/><td>Overall</td></tr><tr><td/><td>PER</td></tr><tr><td>(a)</td><td>ORG</td></tr><tr><td/><td>LOC</td></tr><tr><td/><td>MISC</td></tr><tr><td/><td>0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46</td></tr><tr><td/><td>0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46</td></tr></table>

Performance on cross-domain adaptation. The best score (except the Gold) in each column that is significantly (p < 0.05) better than the second best is marked bold, while those are better but not significantly are underlined. * means the results from the paper.B RESULT OF CONNET WITH TRANSFORMER ENCODER

Table content: <table><tr><td>Methods</td><td/><td>AMTC</td><td/><td>UD</td></tr><tr><td/><td>Precision(%)</td><td>Recall(%)</td><td>F1-score(%)</td><td>Accuracy(%)</td></tr><tr><td>MVT-SLM</td><td colspan="3">72.21(±1.63) 51.72(±3.58) 60.21(±1.87)</td><td>87.23(±0.51)</td></tr><tr><td colspan="4">Crowd-Add (Nguyen et al., 2017) 75.32(±1.41) 50.80(±0.30) 60.68(±0.67)</td><td>88.20(±0.36)</td></tr><tr><td>CONNET (Ours)</td><td colspan="3">76.86(±0.33) 56.43(±3.32) 65.05(±2.32)</td><td>89.27(±0.31)</td></tr><tr><td>Gold (Upper Bound)</td><td colspan="3">81.24(±1.25) 80.52(±0.37) 80.87(±0.79)</td><td>90.45(±0.71)</td></tr></table>

Performance of methods with Transformer-CRF as the base model on crowd-annotation NER dataset AMTC and cross-domain POS dataset UD.

Table content: <table/>

Peer review (iclr2020):
## Updated review I have read the rebuttals. The new version of the paper is clearer and the new baseline experiments are a good addition. ## Original review This paper presents an approach to train a neural networks-based model for sequence modelling using labels from different sources. The proposed approach explicitly models the annotator and uses an attention model to select the best aggregation method. The model is used on two scenarios: learning with crowd annotation and cross-domain adaptation. For the first scenario, noisy annotators are simulated with models trained on subsets of the data, the proposed model is compared with related works and is shown to achieve the highest f-1 score. For the second scenario, different domains in three NLP tasks are used and the model is shown to yield the best performance. I think this paper should be accepted, for the following reasons: - The approach is novel as far as I can tell, and the approach of learning to aggregate labels is significant, as it could also be applied to tasks where inter-annotators agreement is a problem. - The experiments are convincing and show the potential of the proposed approach. - The comparison with related works is thorough. Detailed comments - I don*t understand the notion of *normalized expertise* in Section 5.4, can the authors briefly describe it in the paper ? - The paper is not easy to read, for instance the first paragraph of Section 5 contains critical information to understand the experiments, maybe it should be moved the Section 4 and developed more, typically in two subsections *Application to crowd annotation* and *Application to cross-domain* for example. - Typos: - Section 2, 3rd paragraph *for traget corpora* -> *target* - Same paragraph: *Yang & Eisenstien (2018) represented* -> *represents* to be consistent - Section 4.1: *BiLSTM-CRF* -> *BLSTM-CRF*