SlowGuess/ACL_2020_OCR huggingface dataset:
domains: academic, bio, fiction, news, voyage, wiki, and interview. For NER task, we select the English portion of the OntoNotes v5 corpus (Hovy et al., 2006). The corpus is annotated with 9 named entities with data from 6 domains: broadcast conversation (bc), broadcast news (bn), magazine (mz), newswire (nw), pivot text (pt), telephone conversation (tc), and web (web). Multi-Domain Sentiment Dataset (MDS) v2.0 (Blitzer et al., 2007) is used for text classification, which is built on Amazon reviews from 4 domains: books,DVD, electronics, and kitchen. Since the dataset only contains word frequencies for each review without raw texts, we follow the setting in Chen and Cardie (2018) considering 5,000 most frequent words and use the raw counts as the feature vector for each review.
For sequence labeling tasks, we follow Liu et al. (2018) to build the BLSTM-CRF architecture as the base model. The dimension of character-level, word-level embeddings and LSTM hidden layer are set as 30, 100 and 150 respectively. For text classification, each review is represented as a 5000-d vector. We use an MLP with a hidden size of 100 for encoding features and a linear classification layer for predicting labels. The dropout with a probability of 0.5 is applied to the nonrecurrent connections for regularization. The network parameters are updated by stochastic gradient descent (SGD). The learning rate is initialized as 0.015 and decayed by  $5\%$  for each epoch. The training process stops early if no improvements in 15 continuous epochs and selects the best model on the development set. For the dataset without
a development set, we report the performance on the 50-th epoch. For each experiment, we report the average performance and standard variance of 3 runs with different random initialization.
5.3 Compared Methods
We compare our models with multiple baselines, which can be categorized in two groups: wrapper methods and joint models. To demonstrate the theoretical upper bound of performance, we also train the base model using ground-truth annotations in the target domain (Gold).
A wrapper method consists of a label aggregator and a deep learning model. These two components could be combined in two ways: (1) aggregating labels on crowd-sourced training set then feeding the generated labels to a Sequence Labeling Model (SLM) (Liu et al., 2017); (2) feeding multi-source data to a Multi-Task Learning (MTL) (Wang et al., 2018) model then aggregating multiple predicted labels. We investigate multiple label aggregation strategies. CONCAT considers all crowd annotations as gold labels. MVT does majority voting on the token level, i.e., the majority of labels  $\{\mathbf{y}_{i,j}^{k}\}$  is selected as the gold label for each token  $\mathbf{x}_{i,j}$ . MVS is conducted on the sequence level, addressing the problem of violating Begin/In/Out (BIO) rules. DS (Dawid and Skene, 1979), HMM (Nguyen et al., 2017) and BEA (Rahimi et al., 2019) induce consensus labels with probability models.
In contrast with wrapper methods, joint models incorporate multi-source data within the structure of sequential taggers and jointly model all individual annotators. CRF-MA models CRFs with Multiple Annotators by EM algorithm (Rodrigues et al., 2014). Nguyen et al. (2017) augments the LSTM
Figure 3: Visualizations of (a) the expertise of annotators; (b) attention weights for sample sentences. More cases and details are described in Appendix A.1.
architecture with crowd vectors. These crowd components are element-wise added to tags scores (Crowd-Add) or concatenated to the output of hidden layer (Crowd-Cat). These two methods are the most similar to our decoupling phase. We implemented them and got better results than reported. CL-MW applies a crowd layer to a CNN-based deep learning framework (Rodrigues and Pereira, 2018). Tri-Training uses bootstrapping with multi-task Tri-Training approach for unsupervised one-to-one domain adaptation (Saito et al., 2017; Ruder and Plank, 2018).
5.4 Learning with Crowd Annotations
Performance on real-world datasets. Tab. 1 shows the performance of aforementioned methods and our CONNET on two real-world datasets, i.e. AMT and  $\mathrm{AMTC}^2$ . We can see that CONNET outperforms all other methods on both datasets significantly on  $F1$  score, which shows the effectiveness of dealing with noisy annotations for higher-quality labels. Although  $\mathrm{CONCAT - SLM}$  achieves the highest precision, it suffers from low recall. Most existing methods have the high-precision but low-recall problem. One possible reason is that they try to find the latent ground truth and throw away illuminating annotator-specific information. So only simple mentions can be classified with great certainty while difficult mentions fail to be identified without sufficient knowledge. In comparison, CONNET pools information from all annotations and focus on matching knowledge to make predictions. It makes the model be able to identify more mentions and get a higher recall.
Case study. It is enlightening to analyze whether the model decides the importance of annotators given a sentence. Fig. 3 visualizes test F1 score of all annotators, and attention weights  $\mathbf{q}_i$  in Eq. 6
Figure 4: Performance of CONNET variants of decoupling phase (DP) and aggregation phase (AP).
(a) Reliability level
Figure 5: Performance on simulated crowd-sourced NER data with (a) 5 annotators with different reliability levels; (b) different numbers of annotators with reliability  $r = 1 / 50$ .
(b) Number of Annotators
for 4 sampled sentences containing different entity types. Obviously, the 2nd sample sentence with ORG has higher attention weights on 1st, 5th and 33rd annotator who are best at labeling ORG. More details and cases are shown in Appendix A.1.
Ablation study. We also investigate multiple variants of two phases on AMT dataset, shown in Fig. 4. We explore 3 approaches to incorporate source-specific representation in the decoupling phase (DP). CRF means the traditional approach as Eq. 1 while DP  $(1 + 2)$  is for our method as Eq. 3. DP (1) only applies source representations  $\mathbf{A}^{(k)}$  to the emission score  $\mathbf{U}$  while DP (2) only transfers the transition matrix  $\mathbf{M}$ . We can observe from the result that both variants can improve the result. The underlying model keeps more consensus knowledge if we extract annotator-specific bias on sentence encoding and label transition. We also compare 4 methods of generating supervision targets in the aggregation phase (AP). OMV uses ma
2We tried our best to re-implement the baseline methods for all datasets, and left the results blank when the re-implementation is not showing consistent results as in the original papers.
Task & Corpus	Multi-Domain POS Tagging: Universal Dependencies v2.3 - GUM
Target Domain	academic	bio	fiction	news	voyage	wiki	interview	AVG Acc(%)
CONCAT	92.68	92.12	93.05	90.79	92.38	92.32	91.44	92.11(±0.07)
MTL-MVT (Wang et al., 2018)	92.42	90.59	91.16	89.69	90.75	90.29	90.21	90.73(±0.29)
MTL-BEA (Rahimi et al., 2019)	92.87	91.88	91.90	91.03	91.67	91.31	91.29	91.71(±0.06)
Crowd-Add (Nguyen et al., 2017)	92.58	91.91	91.50	90.73	91.74	90.47	90.61	91.36(±0.14)
Crowd-Cat (Nguyen et al., 2017)	92.71	91.71	92.48	91.15	92.35	91.97	91.22	91.94(±0.08)
Tri-Training (Ruder and Plank, 2018)	92.84	92.15	92.51	91.40	92.35	91.29	91.00	91.93(±0.01)
CONNET	92.97	92.25	93.15	91.06	92.52	92.74	91.66	92.33(±0.17)
Gold (Upper Bound)	92.64	93.10	93.15	91.33	93.09	94.67	92.20	92.88(±0.14)
Task & Corpus	Multi-Domain NER: OntoNotes v5.0 - English
Target Domain	nw	wb	bn	tc	bc	mz	AVG F1(%)
CONCAT	68.23	32.96	77.25	53.66	72.74	62.61	61.24(±0.92)
MTL-MVT (Wang et al., 2018)	65.74	33.25	76.80	53.16	69.77	63.91	60.44(±0.45)
MTL-BEA (Rahimi et al., 2019)	58.33	32.62	72.47	47.83	48.99	52.68	52.15(±0.58)
Crowd-Add (Nguyen et al., 2017)	45.76	32.51	50.01	26.47	52.94	28.12	39.30(±4.44)
Crowd-Cat (Nguyen et al., 2017)	68.95	32.61	78.07	53.41	74.22	65.55	62.14(±0.89)
Tri-Training (Ruder and Plank, 2018)	69.68	33.41	79.62	47.91	70.85	68.53	61.67(±0.31)
CONNET	71.31	34.06	79.66	52.72	71.47	70.71	63.32(±0.81)
Gold (Upper Bound)	84.70	46.98	83.77	52.57	73.05	70.58	68.61(±0.64)
Task & Corpus	Multi-Domain Text Classification: MDS
Target Domain	books	dv	electronics	kitchen	AVG Acc(%)
CONCAT	75.68	77.02	81.87	83.07	79.41(±0.02)
MTL-MVT (Wang et al., 2018)	74.92	74.43	79.33	81.47	77.54(±0.06)
MTL-BEA (Rahimi et al., 2019)	74.88	74.60	79.73	82.82	78.01(±0.28)
Crowd-Add (Nguyen et al., 2017)	75.72	77.35	81.25	82.90	79.30(±9.21)
Crowd-Cat (Nguyen et al., 2017)	76.45	77.37	81.22	83.12	79.54(±0.25)
Tri-Training (Ruder and Plank, 2018)	77.58	78.45	81.95	83.17	80.29(±0.02)
CONNET	78.75	81.06	84.12	83.45	81.85(±0.04)
Gold (Upper Bound)	78.78	82.11	86.21	85.76	83.22(±0.19)
Table 2: Performance on cross-domain data The best score (except the Gold) in each column that is significantly  $(p < 0.05)$  better than the second best is marked bold, while those are better but not significantly are underlined.
jority voting of original annotations, while PMV substitutes them with model prediction learned from DP. AMV extends the model by using all prediction, while AWV uses majority voting weighted by each annotator's training  $F1$  score. The results show the effectiveness of AWV, which could augment training data and well approximate the ground truth to supervise the attention module for estimating the expertise of annotator on the current sentence. We can also infer labels on the test set by conducting AWV on predictions of the underlying model with each annotator-specific components. However, it leads to heavy computation-consuming and unsatisfying performance, whose test  $F1$  score is  $77.35(\pm 0.08)$ . We can also train a traditional BLSTM-CRF model with the same AMV labels. Its result is  $78.93(\pm 0.13)$ , which is lower than CONNET and show the importance of extracted source-specific components.
Performance on simulated datasets. To analyze the impact of annotator quality, we split the
origin train set into  $z$  folds and each fold could be used to train a CRF model whose reliability could be represented as  $r = 1 / z$  assuming a model with less training data would have stronger bias and less generalization. We tried 5 settings where  $z = \{5, 10, 15, 30, 50\}$  and randomly select 5 folds for each setting. When the reliability level is too low, i.e.  $1 / 50$ , only the base model is used for prediction without annotator representations. Shown in Fig. 5(a), CONNET achieves significant improvements over MVT-SLM and competitive performance as Crowd-Cat, especially when annotators are less reliable.
Regarding the annotator quantity, we split the train set into 50 subsets  $(r = 1 / 50)$  and randomly select  $\{5, 10, 15, 30, 50\}$  folds for simulation. Fig. 5(b) shows CONNET is superior to baselines and able to well deal with many annotators while there is no obvious relationship between the performance and annotator quantity in baselines. We can see the performance of our model
Figure 6: Heatmap of averaged attention scores from each source domain to each target domain.
increases as the number of annotators and, regardless of the number of annotators, our method consistently outperforms than other baselines.
5.5 Cross-Domain Adaptation Performance
The results of each task on each domain are shown in Tab. 2. We can see that CONNET performs the best on most of the domains and achieves the highest average score for all tasks. We report the accuracy for POS tagging and classification, and the chunk-level  $F1$  score for NER. We can see that CONNET achieves the highest average score on all tasks. MTL-MVT is similar to our decoupling phase and performs much worse. Naively doing unweighted voting does not work well.
The attention can be viewed as implicitly doing weighted voting on the feature level. MTL-BEA relies on a probabilistic model to conduct weighted voting over predictions, but unlike our approach, its voting process is independent from the input context. It is probably why our model achieves higher scores. This demonstrates the importance of assigning weights to domains based on the input sentence.
Tri-Training trains on the concatenated data from all sources also performs worse than CONNET, which suggests the importance of a multi-task structure to model the difference among domains. The performance of Crowd-Add is unstable (high standard deviation) and very low on the NER task, because the size of the crowd vectors is not controllable and thus may be too large. On the other hand, the size of the crowd vectors in Crowd-Cat can be controlled and tuned. These two methods leverage domain-invariant knowledge only but not domain-specific knowledge and thus does not have comparable performance.
5.6 Analyzing Learned Attention
We analyzed the attention scores generated by the attention module on the OntoNotes dataset. For
each sentence in the target domain we collected the attention score of each source domain, and finally the attention scores are averaged for each source-target pair. Fig. 6 shows all the source-to-target average attention scores. We can see that some domains can contribute to other related domains. For example, bn (broadcast news) and nw (newswire) are both about news and they contribute to each other; bn and bc (broadcast conversation) are both broadcast and bn contributes to bc; bn and nw both contributes to mz (magazine) probably because they are all about news; wb (web) and tc (telephone conversation) almost make no positive contribution to any other, which is reasonable because they are informal texts compared to others and they are not necessarily related to the other. Overall the attention scores can make some sense. It suggests that the attention is aware of relations between different domains and can contribute to the model.
In this paper, we present CONNET for learning a sequence tagger from multi-source supervision. It could be applied in two practical scenarios: learning with crowd annotations and cross-domain adaptation. In contrast to prior works, CONNET learns fine-grained representations of each source which are further dynamically aggregated for every unseen sentence in the target data. Experiments show that our model is superior to previous crowd-sourcing and unsupervised domain adaptation sequence labeling models. The proposed learning framework also shows promising results on other NLP tasks like text classification.
This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, United States Office Of Naval Research under Contract No. N660011924033, and NSF SMA 18-29268. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNl, IARPA, or the U.S. Government. We would like to thank all the collaborators in USC INK research lab for their constructive feedback on the work.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proc. of ACL, pages 440-447.
Xilun Chen and Claire Cardie. 2018. Multinomial adversarial networks for multi-domain text classification. In Proc. of NAACL-HLT, pages 1226-1240, New Orleans, Louisiana. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proc. of EMNLP, pages 1914-1925, Brussels, Belgium. Association for Computational Linguistics.
A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20-28.
A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society. Series B, 39(1):1-38.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL-HLT, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Mark Dredze, Partha Pratim Talukdar, and Koby Crammer. 2009. Sequence learning from data with multiple labels. In ECML/PKDD Workshop on Learning from Multi-Label Data.
Basura Fernando, Tatiana Tommasi, and Tinne Tuyte-laars. 2015. Joint cross-domain classification and subspace learning for unsupervised adaptation. Pattern Recognition Letters, 65:60-66.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. 2016. Deep reconstruction-classification networks for unsupervised domain adaptation. In European Conference on Computer Vision, pages 597-613. Springer.
Melody Y. Guan, Varun Gulshan, Andrew M. Dai, and Geoffrey E. Hinton. 2017. Who said what: Modeling individual labelers improves classification. In AAAI.
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The  $90\%$  solution. In Proc. of NAACL-HLT.
John Lafferty. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. of ICML, pages 282-289. Morgan Kaufmann.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proc. of NAACL-HLT, pages 260-270, San Diego, California. Association for Computational Linguistics.
Ouyu Lan, Su Zhu, and Kai Yu. 2018. Semi-supervised training using adversarial multi-task learning for spoken language understanding. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6049-6053. IEEE.
Dong-Ho Lee, Rahul Khanna, Bill Yuchen Lin, Jamin Chen, Seyeon Lee, Qinyuan Ye, Elizabeth Boschee, Leonardo Neves, and Xiang Ren. 2020. Leanlife: A label-efficient annotation framework towards learning from explanation. In Proc. of ACL (Demo).
Bill Y Lin, Frank Xu, Zhiyi Luo, and Kenny Zhu. 2017. Multi-channel bilstm-crf model for emerging named entity recognition in social media. In Proc. of ACL Workshop, pages 160-165.
Bill Yuchen Lin, Dong-Ho Lee, Ming Shen, Ryan Moreno, Xiao Huang, Prashant Shiralkar, and Xiang Ren. 2020. Triggerner: Learning with entity triggers as explanations for named entity recognition. In ACL.
Bill Yuchen Lin, Dongho Lee, Frank F. Xu, Ouyu Lan, and Xiang Ren. 2019. Alpacatag: An active learning-based crowd annotation framework for sequence tagging. In Proc. of ACL (Demo).
Bill Yuchen Lin and Wei Lu. 2018. Neural adaptation layers for cross-domain named entity recognition. In Proc. of EMNLP.
Liyuan Liu, Xiang Ren, Jingbo Shang, Jian Peng, and Jiawei Han. 2018. Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling. In Proc. of EMNLP.
Liyuan Liu, Xiang Ren, Qi Zhu, Shi Zhi, Huan Gui, Heng Ji, and Jiawei Han. 2017. Heterogeneous supervision for relation extraction: A representation learning approach. In Proc. of EMNLP, pages 46-56. Association for Computational Linguistics.
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. 2014. Transfer joint matching for unsupervised domain adaptation. In Proc. of CVPR, pages 1410-1417.
Jin Kiat Low, Hwee Tou Ng, and Wenyuan Guo. 2005. A maximum entropy approach to chinese word segmentation. In Proc. of SIGHAN Workshop.
Xuezhe Ma and Eduard Hovy. 2016a. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proc. of ACL, pages 1064-1074, Berlin, Germany. Association for Computational Linguistics.
Xuezhe Ma and Eduard Hovy. 2016b. End-to-end sequence labeling via bi-directional LSTM-cnns-crf. In Proc. of ACL, pages 1064-1074. Association for Computational Linguistics.
Tzu Ming Harry Hsu, Wei Yu Chen, Cheng-An Hou, Yao-Hung Hubert Tsai, Yi-Ren Yeh, and Yu-Chiang Frank Wang. 2015. Unsupervised domain adaptation with imbalanced cross-domain data. In Proc. of ICCV, pages 4121-4129.
Shotaro Misawa, Motoki Taniguchi, Yasuhide Miura, and Tomoko Ohkuma. 2017. Character-based bidirectional LSTM-CRF with words and characters for Japanese named entity recognition. In Proc. of ACL Workshop, pages 97-102, Copenhagen, Denmark. Association for Computational Linguistics.
David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. *Lingvisticae Investigationes*, 30(1):3-26.
An Thanh Nguyen, Byron Wallace, Junyi Jessy Li, An Nenkova, and Matthew Lease. 2017. Aggregating and predicting sequence labels from crowd annotations. In Proc. of ACL, pages 299-309. Association for Computational Linguistics.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for ner. In Proc. of ACL, pages 151-164.
Adwait Ratnaparkhi. 1996. A maximum entropy model for part-of-speech tagging. In Proc. of EMNLP.
Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher R. 2016. Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 29.
Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research, 11(Apr):1297-1322.
Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. 2014. Sequence labeling with multiple annotators. Machine Learning, 95(2):165-181.
Filipe Rodrigues and Francisco C Pereira. 2018. Deep learning from crowds. In Thirty-Second AAAI Conference on Artificial Intelligence.
Sebastian Ruder and Barbara Plank. 2018. Strong baselines for neural semi-supervised learning under domain shift. In Proc. of ACL, pages 1044-1054, Melbourne, Australia. Association for Computational Linguistics.
Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. 2017. Asymmetric tri-training for unsupervised domain adaptation. In Proc. of ICML, pages 2988-2997. JMLR.org.
Edwin Simpson, Jonas Pfeiffer, and Iryna Gurevych. 2020. Low resource sequence tagging with weak labels. In AAAI 2020.
Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast—but is it good?: Evaluating non-expert annotations for natural language tasks. In Proc. of EMNLP, EMNLP '08, pages 254-263, Stroudsburg, PA, USA. Association for Computational Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proc. of NAACL-HLT, pages 142-147.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Xuan Wang, Yu Zhang, Xiang Ren, Yuhao Zhang, Marinka Zitnik, Jingbo Shang, Curtis Langlotz, and Jiawei Han. 2018. Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformatics, page bty869.
Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. 2019. Crossweigh: Training named entity tagger from imperfect annotations. In EMNLP/IJCNLP.
Yi Yang and Jacob Eisenstein. 2015. Unsupervised multi-domain adaptation with feature embeddings. In Proc. of NAACL-HLT, pages 672-682.
Amir Zeldes. 2017. The gum corpus: creating multilayer resources in the classroom. *Language Resources and Evaluation*, 51(3):581-612.
A Analysis of ConNet with BLSTM Encoder
A.1 Case study on learning with crowd annotations
To better understand the effect and benefit of CONNET, we do some case study on AMTC real-world dataset with 47 annotators. We look into some more instances to investigate the ability of attention module to find right annotators in Fig. 7 and Tab. 3. Sentence 1-12 contains a specific entity type respectively while 13-16 contains multiple different entities. Compared with expertise of annotators, we can see that the attention module would give more weight on annotators who have competitive performance and preference on the included entity type. Although top selected annotators for ORG has relatively lower expertise on ORG than PER and LOC, they are actually the top three annotators with highest expertise on ORG.
B Result of ConNet with Transformer Encoder