Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling
Authors: Ouyu Lan*
Xiao Huang*
Bill Yuchen Lin
He Jiang
Xiang Ren
Venue:  (2020)
Keywords: crowdsourcing
domain adaptation
sequence labeling
named entity recognition
weak supervision

Abstract: Sequence labeling is a fundamental framework for various natural language processing problems including part-of-speech tagging and named entity recognition. Its performance is largely influenced by the annotation quality and quantity in supervised learning scenarios.  In many cases, ground truth labels are costly and time-consuming to collect or even non-existent,  while imperfect ones could be easily accessed or transferred from different domains. A typical example is crowd-sourced datasets which have multiple annotations for each sentence which may be noisy or incomplete.   Additionally,  predictions from multiple source models in transfer learning can be seen as a case of multi-source supervision.  In this paper, we propose a novel framework named Consensus Network (CONNET) to conduct training with imperfect annotations from multiple sources.   It learns the representation for every weak supervision source and dynamically aggregates them by a context-aware attention mechanism.  Finally, it leads to a model reflecting the consensus among multiple sources.  We evaluate the proposed framework in two practical settings of multi-source learning:  learning with crowd annotations and unsupervised cross-domain model adaptation. Extensive experimental results show that our model achieves significant improvements over existing methods in both settings.

Under review as a conference paper at ICLR 2020
LEARNING TO CONTEXTUALLY AGGREGATE MULTI- SOURCE SUPERVISION FOR SEQUENCE LABELING
Anonymous authors
Paper under double-blind review
ABSTRACT
Sequence labeling is a fundamental framework for various natural language pro- cessing problems including part-of-speech tagging and named entity recognition. Its performance is largely influenced by the annotation quality and quantity supervised learning scenarios. In many cases, ground truth labels costly and time-consuming to collect or even non-existent, while imperfect ones could be easily accessed transferred from different domains. A typical example is crowd- sourced datasets which have multiple annotations for each sentence which may be noisy or incomplete. Additionally, predictions from multiple source models in transfer learning can be seen as case of multi-source supervision. In this paper, we propose novel framework named Consensus Network (CONNET) conduct training with imperfect annotations from multiple sources. It learns the repre- sentation for every weak supervision source and dynamically aggregates them by a context-aware attention mechanism. Finally, it leads to model reflecting the consensus among multiple sources. We evaluate the proposed framework two practical settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation. Extensive experimental results show that model achieves significant improvements existing methods in both settings.
INTRODUCTION
Sequence labeling is a fundamental framework for various natural language processing (NLP) tasks including pfrt-of-speech (POS) tagging (Ratnaparkhi (996), noun phrase chunking (Sang & Buch- holz, 2000). word segmentation (Low et al., 2005), and named entity recognition (NER) (Nadeau & Sekine 2007). Typically, existing methods follow the supervised learning paradigm, and require high-quality annotations. While ground truth labels are expensive and time-consuming, imperfect annotations are much casier to obtain from crowdsoureing (noisy labels) or other domains (out-of- domain). To alleviate or even address the problem of noise and incompleteness, it is important and beneficial to learn from multiple sources.
Specifically, we are interested in two typical application scenarios: 1) learning with crowd anno- tations and 2) unsupervised cross-domain model adaptation, which are detailed in Sec. 2 and Sec. The key challenge of learning with multi-source supervision is to aggregate annotators for learning model without knowing the underlying ground truth label sequences in the target domain. Many attempts have been made generalizing multiple informative sources to out-of-domain distribution for a range of tasks, including multi-class classification (Sheshadri & Lease| (013). ob-
ject detection (Su al.) 2012) and information extraction (Liu et I 2017). However, most of the prior works (Nguyen et al. 2017) Wang et al. 2019 Peng & Dredze 2016 Yang et al. 2017 Chen & Cardie 2018) choose
simple heuristic-bart based methods of aggregating source models for performing on target corpus without carefully calibration.
Our intuition is mainly from the common phenomenon that each source of supervision has distinct strength in different inputs, and thus they should not keep consistent importance in aggregating su- pervisions when inferring tag sequences for new inputs. Aggregating multiple sources for a specific input should be dynamic process depending on the sentence context rather than fixed way. To
"oode and data have heen uploaded and will be published up to the acceptance of the paper.

---

Under review as a conference paper at ICLR 2020
g(
g(xx)
y()
y(
y(x)
yo
A
(ylx)
(ylx)
(vlx)
P(y|x)
(yly)
(2(y)x)
Px (vlx)
Pr P+(ylx)
x(2)
(4)
x
X--X
x-x x-x(())
X-X((K)
X-T
(b)
Figure 1:
Illustration of the task settings for the two applications in this work: (a) learning consensus model from crowd annotations; (b) unsupervised cross-domain model adaptation.
better model this nature, we need to (1) explicitly model the unique traits of different sources when training and (2) find best suitable sources for generalizing the learned model on unseen sentences.
In this paper, we propose a novel framework, named Consensus Network (CONNET), for sequence labeling with multi-source supervisions. We represent the annotation patterns as different biases of annotators over a shared behavior pattern. Both ianotator-invarian patterns and anmotator-specific biases are modeled in decoupled way. The first term through sharing part of low-level model parameters in a multi-task learning schema. For learning the biases, we decouple them from the model as the transformations on top-level tagging model parameters, such that they can capture the unique strength of each annotator. With such decoupled source representations, we further learn an attention network for dynamically assigning the best sources for every unseen sentence through composing transformation that represents the consensus. Extensive experimental results in two scenarios show that our model always outperforms strong baseline methods. CONNET achieves the state-of-the-an ahrt performance unsupervised cross-domain adaptation tasks over existing works. In addition to sequence labeling, it also shows its effectiveness text classification tasks.
crowdsourcing datasets and improve significantly in most
2 RELATED WORK
Neural Sequence Labeling, Traditional approaches for sequence labeling usually need significant efforts in feature engineering for graphical models like hidden markov models (HMMs) (Rabiner) 1989) and conditional random fields (CRFs) (Lafferty] (001). Recent research efforts in neural net- work models have shown that end-to-end learning like convolutional neural networks (CNNs) (Ma & Hovy 2016b) or bidirectional long short-term memory (BLSTMs) (Lample et al.) 2016) can largely climinate human-crafted features. Together with a final CRF layer, these BLSTM CRF models have achieved promising performance and are used as our base sequence tagging model in this paper.
Crowd-sourced Annotation. Crowd-sourcing has been demonstrated to be an effective way of fulfilling the label consumption of neural models (Guan et al., 2017). It collects annotations with lower costs and a higher speed by non-expert contributors but suffers from some degradation in quality. Dawid & Skene (1979) proposes the pioneering work to aggregate crowd annotations to estimate true labels, and Snow et al. (2008) shows its effectiveness with Amazon's Mechanical Turk system. Later works (Dempster al. 1977 Dredze al.) 2009 Raykar al.) (2010) focus on Expectation-Maximzzaine (EM) algorithms to jointly learn the model and annotator behavior on classification problems. Recent research shows the strength of multi-task framework in semi- supervised learning (Lan et al.) 2018 Clark et al. 2018), and cross-type learning (Wang et al., 2018). Nguyen al. (2017) and Rodrigues & Pereiral (2018) regards crowd annotations as noisy versions of glod labels and constructs crowd components to model annotator-specific bias which were discarded during the inference process.
Unsupervised Domain Adaptation. Unsupervised cross-domain adaptation aims to transfer knowl- edge learned from high-resource domains (source domains) to boost performance on low-resource domains (target domains). Different from supervised adaptation (Lin & Lu 2018), we assume there is no labels at all for target corpora. Saito et al. (2017) and Ruder & Plank (2018) explored boot- strapping with multi-task tri- tri-training approach. The method is developed for one-to-one domain adaptation and does not model the differences among multiple source domains. Yang & Eisenstein 2015) represents each domain with vector of metadata domain attributes and uses domain vectors
2

---

Under review as a conference paper at ICLR 2020
auto-encoder method by jointly training a predictor for source labels, and a decoder to reproduce target input with a shared encoder. The decoder acts normalizer to force the model to learn shared knowledge between source and target domains. Adversarial penalty can be applied to the loss function to make models learn domain-invariant feature only (Fernando et al. 2015 Ming Harry Hsu et al. 20155.
to train the model deal with domain shifting. Ghifary al. (2016)
LEARNING WITH MULTI-SOURCE SUPERVISION
We consider the multi-source sequence labeling problem follows. There are N sources of super- vision, each source can be regarded as an imperfect annotator (non-expert human tagger or models trained related domains). For the k-th source data set ( = {(x N. ()) we denote its i-th sentence x; (k) which is sequence of tokens: x(*) (= ,ri,N) The tag sequence of the sen- tence is thus to be y. [ul,)). We define the sentence set of each annotators as x {x (2) ym
(k)
and the whole training domain as the union of all sentence sets: X U x²()) The goal of the multi-source learning task to use such imperfect annotations to train a model predicting tag sequence y for any sentence in target corpus T. Note that the target corpus T can either share the same distribution with (Application I) or be significantly different (Application II), In the following two subsections, formulate two typical tasks in this problem shown in Fig.
Application Learning with Crowd Annotations. When learning with crowd-sourced sequence labeling data, we regard each worker as an imperfect annotator (S(R),, who may make mistakes or skip sentences in its annotations. Note that for crowd-sourcing data, different annotators tag subsets of the same given dataset (2'), and thus we assume there are no input distribution shifts among (k)) Also, we only test sentences in the same domain such that the distribution in target corpus T the same as well. That is, the marginal distribution of target corpus Px(x) is the same with that for each individual source dataset, i.e. (x()) = (x). However, due to imperfectness of the annotations in each source, the Pg. (y (x) has obvious shift from the underlying truth P (illustrated in the top-left part of Fig. [ The multi-source learning objective here is to learn a model Pr (x) for supporting inference any new sentences in the same domain.
Application II: Unsupervised Cross-Domain Model Adaptation. We assume that we have well- annotated data in some source domains while having labels in the target domain at all. Following the formulation, we claim that the input distributions P(x) in different source domains 2 vary a lot, and fitting those annotations can only generalize to in-domain samples. That is, Px(y(x) so ([y|x) only for x E (*)) For target corpus sentences x E T, such a source model P& (y|y) again differs from underlying ground truth for the target domain (y(y() and can be seen as an imperfect annotators. Our objective in this setting is also to jointly model Py (y, x) while noticing that there
are significant domain shifts between T and any other (a))
PROPOSED APPROACH: CONSENSUS NETWORK
In this section, we present our two-phase framework CONNET for multi-source sequence label- ing. As shown in Figure? 2 our proposed framework first uses multi-task learning schema with special objective to decouple annotator representations as different parameters of a transformation around CRF layers. This decoupling phase (Section 4.2 is for decoupling the model parameters into a set of annotator-invariant model parameters and a set of annotator-specific representations. Secondly, the dynamic aggregation phase (Section 4.3) learns to contextually utilize the annotator representations with a lightweight attention mechanism to find the best suitable transformation for each sentence, so that the model can achieve a context-aware consensus among all sources. The inference process is described in Section
4.1 THE BASE MODEL FOR SEQUENCE LABELING: BLSTM-CRF
Many recent sequence labeling frameworks (a & Hovy 2016a Misawa et al. 2017) share a very basic structure: a bidirectional LSTM network followed by CRF tagging layer (i.c. BLSTM- CRF). The BLSTM encodes an input sequence x = {x) 22, xn into a sequence of hidden state vectors hi:-- The CRF takes input the hidden state vectors and computes an emission score
3

---

Under review as a conference paper at ICLR 2020
prediction (g(t)
Weighted Voting
prediction (9)
Annotator [A )
CRF
Attention (Q)
Consensus A
CRF
BLSTM
BI.STM
sentence (x/)
source ID (R)
sentence (X)
Decoupling Phase
Aggregation Phase
Figure 2: CONNET Overview. The decoupling phase constructs the shared model (yellow) and source-specific matrices (bluc). The aggregation phase dynamically combines crowd components into consensus representa- tion (blue) by context-aware attention module (red) for each
matrix U Rux!. where L is the size of tag set. It also maintains trainable transition matrix M E LXX, We can consider U; is the score labeling the tag with id j E {1, 2..,.... L} for tth
word in the input sequence x, and M means the transition score from jthh tag to j
The CRF further computes the score for predicted tag sequence y {y1, y2, ... yke
T
x(x,)) y)
[U +My 1-n):
then tag sequence y follows the conditional distribution
exps(xxy))
P(y\x) Eyer, exp sxx
4.2 THE DECOUPLING PIIASE: LEARNING ANNOTATOR REPRESENTATIONS
For decoupling annotator-specific biases in annotations, we represent them as transformation on emission scores and transition scores respectively. Specifically, we learn a matrix A (k) e R Ax for each imperfect annotator & and apply this matrix transformation U and M as follows:
(10) Σ ((UA ( (MA (K) )
From this transformation, specific computation. The original emission and transformation score matrix U and M are still shared by all the annotators, while they both are transformed by the matrix A (k) for k-th annotator. While training the model parameters in this phase, we follow multi-task learning schema. That is, share the model parameters for BLSTM and CRF (including W, b, M), while updating A (k) only by examples in Sg (x)()) ((()).
that the original score function s in Eq becomes an annotator-
The assumption on the annotation representation A is that it can model the pattern of annotation bias. Each annotator can be seen as a noisy version of the shared model. For the k-th annotator, A (k) models noise from labeling the current word and transferring from the previous label. Specifically, each entry A i.j captures the probability of mistakenly labeling i-th tag to j-th tag. In other words, the base sequence labeling model in Sec. 4.1 learns the basic consensus knowledge while annotator- specific components add their understanding to predictions.
4.3 THE AGGREGATION PHASE: DYNAMICALLY REACHING CONSENSUS
In the second phase, our proposed network learns a context-award attention module for a consensus representation supervised by combined predictions the target data. For each sentence in target data T, we use the model obtained from the decoupling phase to make predictions, and combine predictions using weighted voting. The weight of each source its normalized Fi score the
4

---

Under review as a conference paper at ICLR 2020
training set. Through weighted voting such augmented labels over all source sentences X, can find a good approximation of underlying truth labels.
For better generalization and higher speed, attention module is trained to estimate the relevance of each source to the target under the supervision of generated labels. Specifically, we compute embedding of each sentence by concatenating the last hidden states of the forward LSTM and the backward LSTM, i.c. h()) = h h (
The attention module takes as input the sentence emhedding and outputs a normalized weight for each source:
q softmax (Chima
(h()), where Q E RxXx
where d is the size of each hidden state h() Source-specific matrices {A Jk=1 are then aggre- gated into a consensus representation A for sentence x; E X by
A = A
(5)
1=1
In this way, the consensus representation contains more information about sources which are more related the current type of sentence. It also alleviates the contradiction problem among sources, because it could consider multiple sources of different emphasis. Since only an attention model with weight matrix Q is trained, the amount of computation is relatively small. We assume the sequence labeling model and annotator representations are well-trained in the previous phase. The main objective in this phase is learn how select most suitable annotators for the current sentence.
4.4 PARAMETER LEARNING AND INFERENCE OF CONNET
CONNET learns parameters through two phases described above. In the first decoupling phase, each training instance from source Sş is used for training the base sequence labeling model and its specific representation A (R), In the second aggregation phase, we use aggregated predictions from the first phase to learn lightweight attention module. For each instance in the target corpus X; E T, we calculate its embedding h, from BLSTM hidden states. With these sentence em- beddings, the context-aware attention module assigns weight q, to each source and dynamically aggregates source-specific representations (A {*) for inferring ý:- In the inference process, only the consolidated consensus matrix A is applied the base sequence learning model. In this way, more specialist knowledge helps to deal with more complex instances.
4.5 MODEL. APPLICATION
The proposed model can be applicd to multi-sourcing learning. Here we describe the application in two practical settings: learning with crowd annotations and unsupervised cross-domain model adap- tation. In the crowd annotation learning setting, the training data of the same domain is annotated by multiple noisy annotators, and each annotators is treated as source. In the decoupling phase, the model is trained on noisy annotations, and in the aggregation phase, it is trained with combined pre- dictions on the training In the cross-domain setting, the model has access to unlabeled training data of the target domain and clean labeled data of multiple source domains. Each domain is treated as source. the decoupling phase, the model is trained on source domains, and the aggregation phase, the model is trained on combined predictions on the training data of the target domain.
Our framework can also potentially extend new tasks other than sequence labeling with different encoders. For example, if we use an MLP as encoder, we can transform its hidden representation by multiplying with crowd/consensus matrices. The transformed representation will then used by a decoder to make predictions. We will demonstrate this ability in experiments.
5 EXPERIMENTS
We evaluate CONNET in the two aforementioned settings multi-source learning: learning with crowd annotations and unsupervised cross domain model adaptation. Additionally, demonstrate the generalization of our framework, we also test our method on sequence labeling with transformer encoder in Appendix B and text classification with MLP encoder in Section 5.5
5

---

Under review as a conference paper at ICLR 2020
5.1 DATASETS
datasets based on the 2003 CoNLL shared NER task (Sang & De Meulder 2003). The real-world datasets, denoted as AMT, are collected by |Rodrigues et al. (2014) using Amazon's s Mechanical Turk where FI scores of annotators against the ground truth vary from 17.60% to 89.1 19. Since there no development set in AMT, we also follow Nguyen et al. 2017) to use the AMT training set and CoNLL 2003 development and test sets, denoted as AMTC. Overlapping sentences are removed in the training set, which is ignored in that work. Additionally, we construct two sets of simulated datasets to investigate the quality and quantity of annotators. To simulate the behavior of a non non-expert annotator, a CRF model trained on small subset of training data and generates predictions size of training data, each model would have a bias to certain patterns.
Crowd-Annotation Datasets, We use
the whole set. Because of the limited
Cross-Domain Datasets. In this setting, we investigate three NLP tasks: POS tagging, NER and text classification. For POS tagging task, we use the GUM portion (Zeldes) 2017) of Universal Dependencies (UD) v2.3 corpus with 17 tags and 7 domains: academic, bio, fiction, news, voyage, wiki, and interview. For NER task, we select the English portion of the OntoNotes v5 corpus (Hovy al. 2006). The corpus is annotated with named entities with data from 6 domains: broadcast conversation (bc), broadcast news (bn), magazine (mz), newswire (nw). pivot text (pt). telephone conversation (tc), and web (web). Multi-Domain Sentiment Dataset (MDS) v2.0 (Blitzer et al.) 2007) is used for text classification, which is built on Amazon reviews from 4 domains: books, dvd, electronics, and kitchen. Since the dataset only contains word frequencics for each review without raw texts, we follow the setting in Chen & Cardie (2018) considering 5,000 most frequent words and use the raw counts as the feature vector for each review.
5.2 EXPERIMENT SETUP
For sequence labeling tasks, we follow Liu et al. (2018) to build the BLSTM-CRF architecture as the base model. The dimension character-level, word-level embeddings and BLSTM hidden layer are set as 30, 100 and 150 respectively. For text classification, We use MLP with a hidden size of 100 as encoder and a linear classification layer for predicting their labels. The dropout with probability of 0.5 is applied to the non-recurrent connections for regularization. The network parameters are randomly initialized and updated by stochastic gradient descent (SGD). The learning rate is initialized 0.015 and decayed by 5% for each epoch. The training process stops early if no improvements in 15 continuous epochs and selects the best model on the development set. For the dataset without development set, we report the performance on the 50-th epoch. For each experiment, we report the average performance of 3 runs with different random initialization.
5.3 COMPARISON WITH BASELINE METHODS
We compare models with multiple baselines, which can be categorized in two groups: wrapper methods and joint models. To demonstrate the theoretical upper bound of performance, we also train the base model using ground-truth annotations in the target domain (Gold)
wrapper method consists of a label aggregator and a deep learning model. These two components could be combined in two ways: (1) aggregating labels on crowd-sources training set then feeding the generated labels to Sequence Labeling Model (SLM) (Liu et al. 2017);; (2) feeding multi-source data to Multi-Task Learning (MTL) (Wang et al. 2018) model then aggregating multiple predicted labels. We investigate multiple label aggregation strategies. CONCAT considers all crowd annota- tions as gold labels. MVT does majority voting on the token level, i.e.. the majority of labels (y: is selected as the gold label for each token X..j. MVS is conducted on the sequence level, addressing the problem of violating Begin/In/Out (BIO) rules. DS (Dawid Skene 1979), HMM (Nguyen al., 2017) and BEA (Rahimi et al. 2019) induce consensus labels with probability models.
In contrast with wrapper methods, joint models incorporate multi-source data within the structure of sequential taggers and jointly model all of the individual annotators. CRF RMA models CRFs with Multiple Annotators by EM algorithm (Rodrigues et al., 2014). Nguyen et al. (2017) augments the LSTM architecture with crowd vectors. These crowd components scores (Crowd-Addl or concatenated to the output of the hidden layer (Crowd-Cas CCt)). These two methods are the most similar our extraction phase. We implemented them and got better results
element- wise added to the tags

---

Under review as a conference paper at ICLR 2020
Methods
AMTC
AMT Recall(S)
Precision(
Recall(3))
Fl (k)
Precision(%)
Fl (core(S))
CONCAT-SLM
85.95(+1.000
$7,96(+0.26) 26)
69.23 (3)
55.41 0.557
66)
68 68.89(=1.92))
MVT-SLM
84.78(20.66)
71.9620.666 1.36
69.64(20.31) 00.17 16.96(11.22) 71.37:00.04)
MVS-SLM
84. 84.76(+0.50)
61 (5(0.)))))
(6.95(=1112)
00.23(=0.01)
68 68.30
DS.SIM (gguyen al. 2017
72.30'
61.17"
66.27
HMM-SLM [egueen 1 2017
76.19*
66.24
70.87
MTI.MVT
Wang
81.8((+2.34)
62.51(+0.28)
L0.87(+L066
88.88(-0.25)
65.04(-0.80)
75.10/-0.44
MTL.BBAA [ahhim el al
2019
85.72(100.66)
58.28(10.43)
69.33 19.39()
77.56(11.225
67.23(
12.01(10.85)
CRF-MA Rodrigues
2014
49,40"
85,60'
62.60'
Crowd-Aud Nguven & a.
2017
62.15(++0188)
42.09:0042)
89 0.1100
64. (.)30
75.03(=1.02)
Crowd-Cat Iggoyen
85.02(+0.98)
12.1977))37 12.73(+1.10)
89, 77777)0047
63.55(=1 (,20)
74.39=0098)
CI-MW (Riangnes & Fereira
2018
66.00 66.00"
59.30'
62.40'
CONNET (Ours)
84.11(20.71)
68.61(+0.03)
75.57(=0.27)
12.79(=0.04) 08.7))
70.99)00.08)
Gold (Upper Bound)
19.48(10.52)
89.55(10.06)
89.51(10.21)
(2.12(10.31)
91.73(
91.92( 100.11)
Table 1: Performance on real-world crowd-sourced NER datasets. (* indicates number reported by the paper.)
80.00
60.00
55,00
70.00
50.00
60.00
F1
MVT-SIM
MVT-SLM
50.00
Croned-Cat
40.00
Crowd-Cat
ConNet
ConNet
40.00
35.00
1/10 1/15
1/30
10
15
50
(o) Reliability Level
(b) Annotator Number
Figure 3: Performance on simulated crowd NER datasets with (a) annotators on different reliability
levels; (b) various numbers of annotators whose reliability r = 1/50.
than reported. CL-MW applies a crowd layer to a CNN- deep learning framework (Rodrigues) & Pereira| 2018). Tri-Trainime Traininn uses bootstrapping with multi-task Tri- Training approach for unsupervised one-to-one domain adaptation (Saito et al. 2017 Ruder & Plank 2018
5.4 PERFORMANCE ON LEARNING WITH CROWD ANNOTATIONS
Tab. T shows the performance of aforementioned methods and our CONNET on two real-world datasets, í.e. AMT and AMTC. We can see that CONNET outperforms all other methods on both datasets significantly on Fl score, which shows the effectiivness of dealing with noisy annotations for higher-quality labels. Although CONCAT- SLM achieves the highest precision, it suffers from low recall. All existing methods have the high-precision but low-recall problem. One possible reason is that they try to find the latent ground truth and throw away illuminating annotator-specins information. So only simple mentions can be classified with great certainty while difficult mentions fail to be identified without sufficient knowledge. In comparison, CONNET pools information from all annotations and focus on matching knowledge to make predictions. It makes the tagging model be able to identify more mentions and get higher recall.
It is enlightening to analyze whether the model decides the importance of annotators given a sen- tence. In Fig. 4 we visualize test Fl score of all annotators on each tag, and attention weights qa in Eq. for 4 sampled sentences containing different entity types. Obviously, the 2nd sample sen- tence with ORG has higher attention weights on 1st, 5th and 33rd annotator who are best at labeling ORG. More details and cases are shown in Appendix A.T. We also investigate multiple variants and conduct ablation study in Appendix A.2
To analyze the impact of annotator quality, we split the origin train set into (5. 10, 15, 30, 50} folds and train a CRF model on each fold whose reliability could be represented as r (1/5, 1/10, 1/15, 1/30, 1/50] because model with less training data would have stronger bias and less generalization. For each setting, we randomly select 5 trained models as the simulated annotators to annotate the whole training set. When the reliability level of all annotators is too low, i.e. 1/50, only the base model is used for prediction without annotator representations. The mod- els are then trained with simulated annotations. Shown in Fig. a)). CONNET achieves significant
7

---

Under review as a conference paper at ICLR 2020
Ovenk
LOC
NISC
15
10
3000000
-05
--03
16 20 25 26
Annolator ID
Figure 4: Visualizations of (a) the expertise of annotators; (h) attention weights for sample sentences. More and details are described in Appendix A.T
improvements over NVT-SLM and competitive performance as Crowd-Cat. Our model shows its effectiveness when annotators are less reliable.
Regarding the influence of annotator quantity, we split the train set into 50 subsets to train 50 models (r 1/50) respectively and randomly select {5, 10, 15, 30,50) models simulated annotators to annotate the whole training set. The models are then trained with simulated annotations. Fig. Bb) shows CONNET is superior baselines and able well deal with many annotators while there is no obvious relationship between the performance and the number of annotators in baselines. We see the performance of our model increases the number of annotators and, regardless of the number of annotators, our method consistently outperforms than other baselines.
5.5 PERFORMANCE ON CROSS -OMMAIN ADAPTATION
Methods
POS Tagging
Named Entity Recognition
Text Clussification
CONCAT-SLM
02.11(22.00)
61.24(2:092))
79.41(20.022
MTL-MVT Wang al 2018
90.73( 29)
60:44:20:45)
77.54(=0.00)
MTL.-EAA
91.71(+00066
52.15(+0.58)
78.01(=00.28)
Crowd-Add
Neuven al.
2017
91.36/10.11 14)
39.30(:2/.44)
79.30(=9.21)
Crowd-Cal
Nguven
91.94(10.08)
62.14(110.8)
Tri- Traiing
(Kuur & Plank, 2018
91. 91.93(00001)
61.67(200.11)
80.58(=0.022)
CoNNET (Ours)
(2.33(+-0.17)
63.32(
$1.55( 00004)
Gold (Upper Bound)
92.88(=0..14
68.61(=0.64)
83.22(=0.19)
Performance on cross-domain adaptation. The average score for all domains is reported for cach task. The best score in each column that significantly 0.05) better than the sccond-best is marked bold, while those are better but not significantly are underlined. Detailed results can be found Appendix A.3
Table 2:
The average performance of each method on each task is shown in Tab. 2 More detailed results on each target domain can be found in Appendix [A.3] We report the accuracy for POS tagging and text classification, and report the chunk-level Fl score for NER. We can see that CONNET achieves the highest average score in all tasks. MTL-MVT is similar to our decoupling phase without the attention module, which performs much worse. It shows that naively doing unweighted voting does not work well. The attention can be viewed as implicitly doing weighted voting on the feature level. This demonstrates the importance of having such module to assign weights all domains based on the input sentence. The Tri-Training model trained on the concatenated training data from all sources performs worse than CONNET, which suggests that it is important have multi-task structure model the difference among domains. We analyze the OntoNotes dataset to show that the attention scores generated by the model are meaningful. Details can be found in Appendix A.4
CONCLUSION
In this paper, we present CONNET for learning sequence tagger from multi-source supervision. could be applied two practical scenarios: learning with crowd annotations and cross-domain adaptation. In contrast to prior works, CONNET leams fine-grained representations of each source which are further dynamically aggregated for every unseen sentence in the target data. Experiments show that model is superior to previous crowd-soureing and unsupervised domain adaptation sequence labeling models. The proposed learning framework also shows promising results on other NLP tasks like text classification.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
John Blitzer, Mark Dredze, and Fernando Pereira.
Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pp. 440-447, 2007.
Xilun Chen and Claire Cardie, Multinomial adversarial networks for multi-domain text classifica- tion. arXiv preprint arXiv: 1802.056449 2018.
Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc V Le. Semi-supervised se- quence modeling with cross-view training. arXiv preprint 1r09.
2018.
A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society. Series C (Applied Statisttics), 28(1):20-28 1979. ISSN 00359254, 14679876. URLhttp. www. o
A. P. Dempster, N. M. Laird, and D. Rubin. Maximum likelihood from incomplete data via the em algorithm. JOURNAL OF THE ROYAL STATISTICAL SOCIETY, SERIES B, 39(1):1-38, 1977.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1eto.0880005 2018.
Mark Dredze, Partha Pratim Talukdar, and Koby Crammer. Sequence learning from data with mul- tiple labels. In Workshop Co-Chairs, pp. 39, 2009.
Basura Fernando, Tatiana Tommasi, and Tinne Tuytelaars. Joint cross-domain classification and subspace learning for unsupervised adaptation. Puttern Recognition Letters, 65:60-66, 2015.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep reconstruction-caasffccaaii networks for unsupervised domain adaptation. In European Con- ference on Computer Vision, pp. 597-613. Springer, 2016.
Melody Y. Guan, Varun Gulshan, Andrew M. Dai, and Geoffrey E. Hinton. Who said what: Modeling individual labelers improves classification. CoRR, abs/1703.08774, 2017. URL
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. Ontonotes: The 90) % solution. In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, 2006.
John Lafferty. Conditional random fields: Probabilistic models for segmenting and labeling quence data. In ICML, pp. 282-289. Morgan Kaufmann, 2001.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. arXiv preprint 1601v 2016.
Ouyu Lan, Su Zhu, and Kai Yu. Semi-supervised training using adversarial multi-task learning for spoken language understanding. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6049-6053, IEEE, 2018.
Bill Yuchen Lin and Wei Lu. Neural adaptation layers for cross-domain named entity recognition. In EMNLP, 2018
Liyun Liu, Xiang Ren, Qi Zhu, Shi Zhi, Huan Gui, Heng Ji, and Jiawei Han. Heterogeneous supervision for relation extraction: A representation learning approach. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 46-56. Association for Computational Linguistics, 2017. doi: 10. 10.116653/1/DD1--0005 URL http: http::////a//eb
oxt/aatthelogy/117-1000
Liyuan Liu, Xiang Ren, Jingbo Shang, Jian Peng, and Jiawei Han. Efficient Contextualized Repre- sentation: Language Model Pruning for Sequence Labeling. In EMNLP, 2018.

---

Under review as a conference paper at ICLR 2020
Jin Kiat Low, Hwee Tou Ng. and Wenyuan Guo. A maximum entropy approach to chinese word segmentation. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, 2005.
Xuezhe Ma and Eduard Hovy. End-to-end sequence labeling via bi-directional Istm-cuns-crf, In Proceedings the 54th Anmual Meeting of the Association Computational Linguistics (Volume I: Long Papers), pp. 1064-1074, Association for Computational Linguistics, 201 6a. doi: 10. 18653/v1/P16-111 1101. URL http:
/ //a:: web. httpb//ggg/ancoolgg/yP..e..ee
Xuezhe Ma and Eduard Hovy. End-to-end sequence labeling via bi-directional lstm-cnns-erf. arXiv preprint arXiv: arXiv:/603.00 1603.01344 2016b.
Tzu Ming Harry Hsu, Wei Yu Chen, Cheng-An Hou, Yao-Hung Hubert Tsai, Yi-Ren Yeh, and Yu- Chiang Frank Wang. Unsupervised domain adaptation with imbalanced cross-domain data. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4121-4129, 2015.
Shotaro Mísawa, Motoki Taniguchi, Yasuhide Miura, and Tomoko Ohkuma. Character-based bidi- rectional LSTM-CRE with words and characters for Japanese named entity recognition. In Proceedings of the First Workshop on Subword and Character Level Models in NLP, pp. 97- 102, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.11653//1//11-44114
URLhtts:://wwe: http::://wwwwww.o www www.wb.wwwb............ wwt.a//ww....rraaaaooooooo/////.
David Nadeau and Satoshi Sekine. A survey of named entity recognition and classification. Lingvis- ticae Investigationes, 30(1):3-260 2007.
An Thanh Nguyen, Byron Wallace, Junyi Jessy Li, Ani Nenkova, and Matthew Lease. Aggre- gating and predicting sequence labels from crowd annotations. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 299-309, Association for Computational Linguistics, 2017. doi: 10.18653/11/117-102.. URL
http:// org/ anthalogyhhheaaeo 1028
Nanyun Peng and Mark Dredze. Multi-task domain adaptation for sequence tagging. arXiv preprint arXiv:1608.02668) 02689, 2016.
L. Rabiner. A tutorial on hidden markov models and selected applications speech recognition. Proceedings of the IEEE. 77(2) Feb 1989. ISSN 0018-9219 doi: 10.1109/5.1866.
Afshin Rahimi, Yuan Li, and Trevor Cohn. Massively multilingual transfer for ner. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pp. 151-164, 2019.
Adwait Ratnaparkhi. A maximum entropy model for part-of-speech tagging. In Conference on Empirical Methods in Natural Language Processing, 1996.
Vikas C Raykar, Shipeng Yu, Linda Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 11 (Apr):1297-1322, 2010.
Filipe Rodrigues and Francisco C. Pereira. Deep learning from crowds. In AAAI, 2018.
Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. Sequence labeling with multiple annotators. Machine Learning, 95(2):165-181, May 2014. ISSN 1573-0565. doi: 10.1007/
$10994-013-5411- URL htLhhtts::/ /ttp:/tppp//.o.oo////oo// 0ttp://.....00000000000000000000
Sebastian Ruder and Barbara Plank. Strong baselines for neural semi-supervised learning under domain shift. arXiv preprint arXiv:1804.09530. 2018.
Kuniaki Saito. Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri tri-training for unsupervised domain adaptation. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3988-29975 .MLR. org, 2017.
Erik F Sang and Sabine Buchholz. Introduction to the conll-2000 shared task: Chunking. arXiv preprint cs/0009008, 2000.
10

---

Under review as a conference paper at ICLR 2020
Erik F Sang and Fien De Meulder. Introduction to the con1l-2003 shared task: Language- independent named entity recognition. arXiv preprint cs/0306050, 2003.
Aashish Sheshadri and Matthew Lease. SQUARE: A Benchmark for Research on Computing Crowd Consensus. In Proceedings of the 1st AAAI Conference on Human Computation (HCCMP). pp. 156-164, 2013. URL http:///rr sheshadri.
edu/ sqqurre/documeee
Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast-but but is
good?: Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods Natural Language Processing, EMNLP "08, pp. 254- 263, Stroudsburg, PA, USA, 2008. Association for Computational Linguisties. URL http:
7/al. acm. org/ http:///........................ 1613751
Hao Su, Jia Deng, and Li Fei-Fei. Crowdsourcing annotations for visual object detection. In Work- shops al the Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Xuan Wang. Yu Zhang, Xiang Ren, Yuhao Zhang, Marinka Zitnik, Jingbo Shang, Curtis Langlotz, and Jiawei Han. Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformatics, pp. bty869, 2018.
doi: 10. 10.10//bionfoomatts/btys3999 URL http::/dxx
doi.
Xuan Wang, Yu Zhang, Xiang Ren, Yuhao Zhang, Marinka Zitnik, Jingbo Shang, Curtis Langlotz, and Jiawei Han. Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformaties, 35(10):1745-17)) 2019.
Yi Yang and Jacob Eisenstein. Unsupervised multi-domain adaptation with feature embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 672-682 2015.
Zhilin Yang, Ruslan Salakhutdinov, and William W Cohen. Transfer learning for sequence tagging with hicrarchical recurrent networks. arXiv preprint arXiv.1003.06345 2017.
Amir Zeldes. The gum corpus: creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):881-612, 2017.

---

Under review as a conference paper at ICLR 2020
A ANALYSIS OF CONNET WITH BLSTM ENCODER
A.1 CASE STUDY ON LEARNING WITH CROWD ANNOTATIONS
Orocc
0s
LOC
2.
155
1.0
Annolator
Figure 5: Visualizations of (a) the expertise of annotators; (b) attention weights for additional sample sentences to Fig. Details samples are described in Tab.
Defender [PER Hassan Abbas] rose intercept a long ball into the area in the 84th minute
but only managed to divert into the top corner of [PER Bitar] 's goal
[ORG Plymouth] 4 [ORG Exeter]
Hosts [LOC UAE] play [LOC Kuwait] and [LOC South Korea] take on [LOC Indonesia] on Saturday in Group A matches
3
The former [MISC
Soviet] republic was playing in an MISC Asian Cup] finals tic for
4
the first time..
[PER Bitar] pulled off fine saves whenever they did
[PER Coste] said he had approached the player two months ago about comeback.
7
[ORG Goias] 1 [ORG Gremio] 3
8
[ORG Portuguesa] [ORG
Atletico Mineiro] 0
9
[LOC Melbourne] 1996-12-08
Friday for their friendly against [LOC Scotland] at [LOC Murrayfield] more than a year after the 30-year-old wing announced he was retiring following differences over selection.
10
11
Scoreboard in the MMISC World Series]
12
- Sheffield Shield] score
13
He ended the [MISC World Cup] on the wrong note [PER Coste] said.
14
Soccer [ORG Leeds] [PER Bowyer] fined for part in fast-food fracas
15
[ORG Rugby Union PER Cuttitta] back for [LOC Italy] after year.
[LOC Australia] gave [PER Brian Lara] another reason to be miserable when they beat [LOC West Indies] by five wickets in the opening [MISC World Series] limited overs
16
match on Friday.
Table 3: Sample instances in Fig. and Fig. with NER annotations including PER (red), ORG (blue), LOC (violet) and MISC (orange).
To better understand the effect and benefit of CONNET, dataset with 47 annotators. We look into some more instances to investigate the ability of attention module to find right annotators in Fig. 5 and Tab. 3 Sentence 1-12 contains a specific entity type respectively while 13-16 contains multiple different entities. Compared with expertise of annotators, we can see that the attention module would give more weight on annotators who have competitive performance and preference the included entity type. Although top selected annotators for ORG has relatively lower expertise on ORG than PER and LOC, they are actually the top three annotators with highest expertise on ORG.
some case study on AMTC real-world
12

---

Under review as a conference paper at ICLR 2020
A.2 ABLATION STUDY ON CONNET
To investigate the effectiveness of our proposed method, multiple variants of the decoupling phase and aggregation phase are evaluated the AMT dataset, shown in Fig. 6(a]
Decoupling Phase We tried three different approaches to incorporate source-specific representation in the decoupling phase (DP). CRF means the traditional approach without annotator matrices as
Eq. I while DP (1+2) is described in Eq. 3 Similarly, DP (1) only applies source representations
A (R) to the emission score U while D2 (2) only transfers the transition matrix M. They directly
use latent representations U and M in the decoupling phase (not augmented by A (k)) for prediction.
We can observe from the result that both variants that model the annotator- mmnotator-specifi information
improve the result. The underlying model keeps more consensus knowledge we extract annotator- specific bias sentence encoding and label transition.
Aggregation Phase We compare four methods generating supervision targets in the aggregation phase (AP). OMV uses majority voting of original annotations targets for training, while PMV sub- stitutes them with model prediction learned from DP. AMV extends the model by using all prediction, while ANV uses majority voting weighted by each annotator's 's training F1 score. The results show the effectiveness of AWV, which could well approximate the ground truth and supervise the attention module estimate the expertise of annotator on the current sentence. We can also infer labels on the test set by conducting AWV predictions of the underlying model with each annotator- specific components. But it leads to heavy computation-consuuman test F1 score is 77. (7..08) To investigate the importance of extracted source-specific compo- nents, we trained traditional BLSTM-CRE model by the supervision of the same AMV labels. Its result is 78.93 78.93(+10.3)
and unsatisfying performance, whose
which is lower than our CONNET.
These experiments convey four messages: (1) Using learned model prediction better than directly using original annotations as its target; (2) It is desirable use all model predictions instead of just confine each annotator on their data set. This means that the learned representation of indi- vidual annotator generalize data it didn't annotate. (3) Weighting aggregating models by their performance beneficial to overall accuracy. (4) The attention model could learn to estimate annotators' ability based the context.
80
77.44
79,08
78,73
79.99
tc bn wb nw
0.8
F1(%)
75
72,16
70,53
70
68,89
68.67
0.0
-0.8
mz bc
-1.6
CRF
DP(1)
DP(3)
DP(++))
AP(OMV)
AP(PM))
AP(AMV)
AP(AWV)
nw
wbb
bn
tc
bc
mz
(a)
(b)
Figure 6: (a) Performance of CONNET variants for ablation study; (b) Heatmap of averaged attention scores from each source domain to each target domain.
A.3 DETAILED RESULTS FOR CROSS-DOMAIN ADAPTATION
The results of each task on each domain are shown Tah. We can see that CONNET performs the best most of the domains and achieves the highest average for all tasks.
A.4 MULTI-DOMAIN: ANALYSIS
We analyzed the attention scores generated by the attention module on the OntoNotes dataset. For each sentence in the target domain collected the attention score of each source domain, and finally
13

---

Under review as a conference paper at ICLR 2020
the attention scores averaged for each source-target pair. Fig. 6(b) (hosswowsss all the source-to tourcett-taarget target average attention scores. We can see that some domains can contribute to other related domains. For example, bn (broadcast news) and (newswire) are both about news and they contribute to each other; bn and be (broadcast conversation) are both broadcast and bn contributes to be; bn and both contributes (telephone conversation) almost make no positive contribution to any other, which is reasonable because they are informal texts compared to others and they are necessarily related the other. Overall the attention scores can make some sense to human, and it suggests that the attention is meaningful and contributing to the model.
(magzine) probably because they all about news; wb (web) and to
Task Corpus
Multi-Domain POS Tagging: Universal Dependencies v2.3 GUM
Target Domain
academic
bio
fiction
voyage
wiki
interview
AVG Aco(S)
CONCAT
92.68
92.12
93.05
90.79
92.38
92.32
91. .4
92.11(+0.07)
MTL-MVT Wang al.
2018
92.42
90.59
91.16
89.69
90.75
90.29
90.21
90.73(#0.29)
MTL-BEA Kanini et
2019
92.87
91.88
91.90
91.03
91.67
91.31
91.29
91.71(=0.06)
Crowd-Add Nguven et
92.58
91.91
91.50
90.73
91.74
90.47
90.61
91.36(20.14)
Crowd-Ca Cat Nguyen el
2017
92.71
91.71
92.48
91.15
92.35
91.97
91.22
91.94(+0.08)
Tri- Tri-Training Kiver & Plank
2018
92.84
92.15
92.51
91.40
92.35
91.29
91.00
91.03(=0.01)
CONNET
92.97
92.25
23.15
91.06
92.52
92.74
91.66
92.33(20.17)
Gold (Upper Bound)
92.64
93.10
93.15
91.33
93.09
94.67
92.20
92.88(20.14)
Task & Corpus
Multi-Domain NER: OntoNotes v5.0
English
Target Domain
wb
bn
tc
bc
mz
AVG ((
CONCAT
68.23
32.96
77.25
53.66
72.74
62.61
61.24(+:0922)
MTL-MVT Wang et al
2018
65.74
33.25
76.80
53.16
69.77
63.91
60.44(2:.46) 45)
MII.-BAA Kahimi
32.62
47.83
52.68
52.15(++0.58)
Crowd-Ade Add Neuven et 2017
45.76
32.51
26.47
52.94
28.12
39.30((t-4444)
Crowd-Cat Nguven et al.
2017
68.95
32,61
78.07
53.41
74.22
65.55
62.14(::0.89)
Tri-Truinig [Ruder & Plank
2018
69.68
33.41
79.62
47.91
70.85
68.53
61.67(1:0.1)
CONNET
71.31
34,06
79.66
52.72
71.47
70.71
63.32(++0.81)
Gold (Upper Bound)
84.70
46.98
83.77
52.57
70.58
68.61(11064))
Task & Corpus
Multi-Domaie Text Classification: MDS
Target Domain
books
dvd
electronics
kitchen
AVG Acc(%)
CONCAT
75.68
77.02
81.87
83.07
79.41(++0.02)
MTL.-MVT (Wag al.
2018
74.92
74.43
79.33
81.47
77.54(+0.06)
MTL-BEA
Rahimi et
2019
74.88
74.60
79.73
82.82
78.01(110.28)
Crowd-Add Nguyen
2017
75.72
77.35
81.25
82.90
79.30(+9.211
Crowd-Cat Nguyen cl al
201
76.45
77.37
81.22
83.12
79.54(+-0.25
Tii-Traning [Rider & Plank
2018
78.75
78.45
81.95
83.17
80.58(+:0.02)
CONNET
77.58
84.12
83.45
81.55(40.04)
Gold (Upper Bound)
78.78
86.21
85.76
83.22(+0.19)
Table 4: Performance on cross-domain adaptation. The best score (except the Gold) in each column that is significantly (p < 0.05) better than the second best is marked bold, while those are better but significantly are underlined. means the results from the paper.
B RESULT OF CONNET WITH TRANSFORMER ENCODER
AMTC
UD
Methods
Precision( %)
Recall(%)
F1-score({)
Accuracy($)) (%)
MVT-SLM
51.72(13.558 12.21(11.63)
60.21(11.87)
87.23(10.51)
Crowd-Add (gguyen et al. 2017)
75.42(+11441
50.80(10.30)
60.68(10067)
88.20(10.36)
CONNET (Ours)
76.36(100.3)
56.43(+3.32)
65.05(+12.32)
89.27(10.31)
Gold (Upper Bound)
81.24(+1.25)
80.87(110.79) 80.52(+00.7))
90.45(100771)
Table 5: Performance of methods with Transformer-CBB the base model AMTC and cross-domain POS dataset UD.
NER datasct
14

---

Under review as a conference paper at ICLR 2020
To demonstrate the generalization of our framework, we re-imllement CONNET and some baselines (MTV-SLM, Crowa-Add, Gold) with Transformer- as the base model. Specifically, the base model takes Transformer the encoder for CRF, which has shown its effectiveness in many NLP tasks (Vaswani et al. 2017 Devlin et al. 2018). Transformer models sequences with self-attention and eliminates all recurrence. Following the experimental settings from Vaswani et al. (2017),
set the number of heads for multi-head attention as 8, the dimension of keys and values as 64, and the hidden size of the feed-forward layers as 1024. We conduct experiments with crowd-annotation dataset AMTC on NER task and cross-domain dataset UD on POS task, which are described in Section 5.1 Results are shown in Table We can see our model outperforms over other baselines in both tasks and applications.
15


Reviews:
[{"review_id": "Bkg21BVsYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes ConNet, a new label aggregation method for sequence labeling tasks, including crowd-annotation and cross-domain model adaptation. The model consists of a decoupling phase, which learns annotator-specific transforming matrices A, and an aggregation phase with an attention module. Extensive experimental results demonstrate the superiority of the proposed model over baselines. \nThe paper is generally well-written and easy to follow, and the results seem convincing, so I think it can inspire other works on this topic. My main concern on the paper is its generalization. Crowdsourcing usually involves lots of annotators, and some of them only give very few labels. In these situations, the proposed model introduces lots of new parameters (A, Q), which may cause difficulty during training. So the positive results in Fig 3(b) are very important to dispel my worry, which requires more explanation. \nBelow are some detailed questions:\n-      How do you calculate the sentence embeddings h_i during training?\n-      Have you introduced some regularization terms on A and Q?\n-      What are the most important hyper-parameters for this model, and how to tune?\n-      Can this method be extended to new tasks other than sequence labeling?\n-      Can you compare your method with the aggregation method used in on-the-job learning paper [1]?\n-      92.33 in tab 2 shouldn’t be bold.\n \n[1] Werling K , Chaganty A , Liang P , et al. On-the-Job Learning with Bayesian Decision Theory[J]. Computer Science, 2015.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJlZKk6pKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Updated review\n\nI have read the rebuttals. The new version of the paper is clearer and the new baseline experiments are a good addition. \n\n## Original review\n\nThis paper presents an approach to train a neural networks-based model for sequence modelling using labels from different sources. The proposed approach explicitly models the annotator and uses an attention model to select the best aggregation method. The model is used on two scenarios: learning with crowd annotation and cross-domain adaptation. For the first scenario, noisy annotators are simulated with models trained on subsets of the data, the proposed model is compared with related works and is shown to achieve the highest f-1 score. For the second scenario, different domains in three NLP tasks are used and the model is shown to yield the best performance.\n\nI think this paper should be accepted, for the following reasons:\n- The approach is novel as far as I can tell, and the approach of learning to aggregate labels is significant, as it could also be applied to tasks where inter-annotators agreement is a problem.\n- The experiments are convincing and show the potential of the proposed approach. \n- The comparison with related works is thorough.\n\nDetailed comments\n- I don't understand the notion of \"normalized expertise\" in Section 5.4, can the authors briefly describe it in the paper ?\n- The paper is not easy to read, for instance the first paragraph of Section 5 contains critical information to understand the experiments, maybe it should be moved the Section 4 and developed more, typically in two subsections \"Application to crowd annotation\" and \"Application to cross-domain\" for example.\n- Typos:\n    - Section 2, 3rd paragraph \"for traget corpora\" -> \"target\"\n    - Same paragraph: \"Yang & Eisenstien (2018) represented\" -> \"represents\" to be consistent\n    - Section 4.1: \"BiLSTM-CRF\" -> \"BLSTM-CRF\"\n ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xyg9XT5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a method to learn how to aggregate weak supervision sources in the context of sequence labeling. In particular, the model has two main steps: i) it learns a source dependent transformation and ii) it learns a mechanism to combine them. \n\nIn general, the paper is well organized but it is not easy to follow. It was not clear to me how the authors combine the source dependent representation in Section 4.2 with the aggregation phase presented in Section 4.3. In particular, it is confusing to me how the model combines during the training Eq. 3 and Eq. 5. Similarly, it is not clear to me how in Eq. 6 the model can calculate attention coefficients based only on information from the sentence embedding (h^(i)).\n\nThis work assumes a BLSTM-CRF architecture as a baseline, but it does not explore alternative approaches, such as transformer.   \n\nIn terms of the experiments, authors evaluate the resulting model using two application settings: combining noisy crowd annotations (AMT) and unsupervised cross-domain model adaptation. Results are encouraging, the proposed method is able to outperforms several recent works in terms of F1 metric for the case of AMT and accuracy for the case of cross-domain adaptation. Qualitative results also shows reasonable performance. The supplemental material also include an ablation study.\n\nIn summary, the proposed method is interesting and results seem to be encouraging, however, there are parts of the proposed method that are not clear to me. I rate the paper as weak reject. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the revised paper was better and improved the reviewers assessment of the work, the paper is just below the threshold for acceptance. The authors are strongly encouraged to continue this work.
Decision: Reject