SlowGuess/ACL_2020_OCR huggingface dataset:
Majority voting (MV) The simplest method for aggregating outputs is majority voting, i.e. outputting the most frequent label among the ones predicted by each labelling function. However, specialised labelling functions will output  $\mathsf{O}$  for most tokens, which means that the majority label is typically  $\mathsf{O}$ . To mitigate this problem, we first look at tokens that are marked with a non-O label by at least  $T$  labelling functions (where  $T$  is a hyperparameter tuned experimentally), and then apply majority voting on this set of non-O labels.
Snorkel model The Snorkel framework (Ratner et al., 2017) does not directly support sequence labelling tasks as data points are required to be independent. However, heuristics can be used to extract named-entity candidates and then apply labelling functions to infer their most likely labels (Fries et al., 2017). For this baseline, we use the
three functions nnp_detector, proper_detector and compound_detector (see Appendix A) to generate candidate spans. We then create a matrix expressing the output of each labelling function for each span (including a specific "abstain" value to denote the absence of prediction) and run the matrix-completion-style approach of Ratner et al. (2019) to aggregate the predictions from all functions.
mSDA is a strong domain adaptation baseline (Chen et al., 2012) which augments the feature space of a model with intermediate representations learned using stacked denoising autoencoders. In our case, we learn the mSDA representations on the unlabeled source and target domain data. These 800 dimensional vectors are concatenated to 300 dimensional word embeddings and fed as input to a two-layer LSTM with a skip connection. Finally, we train the LSTM on the labeled source data and test on the target domain.
AdaptaBERT This baseline corresponds to a state-of-the-art unsupervised domain adaptation approach (AdaptaBERT) (Han and Eisenstein, 2019). The approach first uses unlabeled data from both the source and target domains to domain-tune a pretrained BERT model. The model is finally task-tuned in a supervised fashion on the source domain labelled data (Ontonotes). At inference time, the model makes use of the pretraining and domain tuning to predict entities in the target domain. In our experiments, we use the cased-version of the base BERT model and perform three fine-tuning epochs for both domain-tuning and task-tuning. We additionally include an ensemble model, which averages the predictions of five BERT models finetuned with different random seeds.
Mixtures of multinomials
Following the notation from Section 3.2, we define  $Y_{i,j,k} = \mathrm{I}(P_{i,j,k} = \max_{k' \in \{1,\dots,S\}} P_{i,j,k'})$  to be the most probable label for word  $i$  by source  $j$ . One can model  $\mathbf{Y}_{ij}$  with a Multinomial probability distribution. The first four baselines (the fifth one assumes Markovian dependence between the latent states) listed below use the following independent, i.e.  $p(s_i, s_{i-1}) = p(s_i)p(s_{i-1})$ , mixtures of Multinomials model for  $\mathbf{Y}_{ij}$ :
$$
\begin{array}{l} \mathbf {Y} _ {i j} | \boldsymbol {p} _ {j} ^ {\boldsymbol {s} _ {i}} \quad \stackrel {{i n d}} {{\sim}} \quad \text {M u l t i n o m i a l} (\boldsymbol {p} _ {j} ^ {\boldsymbol {s} _ {i}}), \\ \begin{array}{c c c} \boldsymbol {s} _ {i} & \overset {\text {i n d}} {\sim} & \operatorname {M u l t i n o m i a l} (\boldsymbol {\sigma}). \end{array} \\ \end{array}
$$
Accuracy model (ACC) (Rodrigues et al., 2014) assumes the following constraints on  $p_j^{s_i}$ :
$$
p _ {j k} ^ {s _ {i}} = \left\{ \begin{array}{l} \pi_ {j}, \text {i f} s _ {i} = k, \\ \frac {1 - \pi_ {j}}{J - 1} s _ {i} \neq k. \end{array} \right.
$$
Here, for each labelling function it is assumed to have the same accuracy  $\pi_{j}$  for all of the tokens.
Confusion vector (CV) (Nguyen et al., 2017a) extends ACC by relying on separate success probabilities for each token label:
$$
p _ {j k} ^ {s _ {i}} = \left\{ \begin{array}{l} \pi_ {j k}, \text {i f} s _ {i} = k, \\ \frac {1 - \pi_ {j k}}{J - 1} s _ {i} \neq k. \end{array} \right.
$$
Confusion matrix (CM) (Dawid and Skene, 1979) allows for distinct accuracies conditional on the latent states, which results in:
$$
p _ {j k} ^ {s _ {i}} = \pi_ {j k} ^ {s _ {i}}. \tag {8}
$$
Sequential Confusion Matrix (SEQ) extends the CM model of Simpson and Gurevych (2019), where an "auto-regressive" component is included in the observed part of the model. We assume dependence on a covariate indicating that the label has not changed for a given source, i.e.:
$$
p _ {j k} ^ {s _ {i}} = \operatorname {l o g i t} ^ {- 1} \left(\mu_ {j k} ^ {s _ {i}} + \mathrm {I} \left(Y _ {i - 1, j, k} ^ {T} = Y _ {i, j, k} ^ {T}\right) \beta_ {j k} ^ {s _ {i}}\right).
$$
Dependent confusion matrix (DCM) combines the CM-distinct accuracies conditional on the latent states of (8) and the Markovian dependence of (3).
The evaluation results are shown in Tables 1 and 2, respectively for the CoNLL 2003 data and the crowd-annotated sentences. The metrics are the (micro-averaged) precision, recall and  $F_{1}$  scores at both the token-level and entity-level. In addition, we indicate the token-level cross-entropy error (in log-scale). As the labelling functions are defined on a richer annotation scheme than the four labels of ConLL 2003, we map GPE to LOC and EVENT, FAC, LANGUAGE, LAW, NORP, PRODUCT and WORK_OF_ART to MISC. The results for the ACC and CV baselines are not included as the parameter estimation did not converge and hence did not provide reliable posteriors over parameters.
Table 1 further details the results for subsets of labelling functions. Of particular interest is the contribution of document-level functions, boosting the
entity-level  $F_{1}$  from 0.702 to 0.716. This highlights the importance of these relations in NER.
The last line of the two tables reports the performance of the sequence labelling model (Section 3.3) trained on the aggregated labels. We observe that its performance remains close to the HMM-aggregated labels. This shows that the knowledge from the labelling functions can be injected into a standard neural model without substantial loss.
Although not shown in the results due to space constraints, we also analysed whether the informative priors described in Section 3.2 influenced the performance of the aggregation model. We found informative and non-informative priors to yield similar performance for CoNLL 2003. However, the performance of non-informative priors was very poor on the Reuters and Bloomberg sentences ( $F_{1}$  at 0.12), thereby demonstrating the usefulness of informative priors for small datasets.
We provide in Figure 3 an example with a few selected labelling functions. In particular, we can observe that the Ontonotes-trained NER model mistakenly labels "Heidrun" as a product. This erroneous label, however, is counter-balanced by other labelling functions, notably a document-level function looking at the global label frequency of this string through the document. We do, however, notice a few remaining errors, e.g. the labelling of "Status Weekly" as an organisation.
Figure 4 illustrates the pairwise agreement and disagreement between labelling functions on the CoNLL 2003 dataset. If both labelling functions make the same prediction on a given token, we count this as an agreement, whereas conflicting predictions (ignoring O labels), are seen as disagreement. Large differences may exist between these functions for specific labels, especially MISC. The functions with the highest overlap are those making predictions on all labels, while labelling functions specialised to few labels (such as legal_detector) often have less overlap. We also observe that the two gazetteers from Crunchbase and Geonames disagree in about  $15\%$  of cases, presumably due to company names that are also geographical locations, as in the earlier Komatsu example.
In terms of computational efficiency, the estimation of HMM parameters is relatively fast, requiring less than 30 mins on the entire CoNLL 2003 data. Once the aggregation model is estimated, it
Model:	Token-level	Entity-level
P	R	F1	CEE	P	R	F1
Ontonotes-trained NER	0.719	0.706	0.712	2.671	0.694	0.620	0.654
Majority voting (MV)	0.815	0.675	0.738	2.047	0.751	0.619	0.678
Confusion Matrix (CM)	0.786	0.746	0.766	1.964	0.713	0.700	0.706
Sequential Confusion Matrix (SEQ)	0.736	0.716	0.726	2.254	0.642	0.668	0.654
Dependent Confusion Matrix (DCM)	0.785	0.744	0.764	1.983	0.710	0.698	0.704
Snorkel-aggregated labels	0.710	0.661	0.684	2.264	0.714	0.621	0.664
mSDA (OntoNotes)	0.640	0.569	0.603	2.813	0.560	0.562	0.561
AdaptaBERT (OntoNotes)	0.693	0.733	0.712	2.280	0.652	0.736	0.691
AdaptaBERT (Ensemble)	0.704	0.754	0.729	2.103	0.684	0.743	0.712
HMM-agg. labels (only NER models)	0.658	0.720	0.688	2.653	0.642	0.599	0.620
HMM-agg. labels (only gazetteers)	0.759	0.394	0.518	3.678	0.687	0.367	0.478
HMM-agg. labels (only heuristics)	0.722	0.771	0.746	1.989	0.718	0.683	0.700
HMM-agg. labels (all but doc-level)	0.714	0.778	0.744	1.878	0.713	0.693	0.702
HMM-agg. labels (all functions)	0.719	0.794	0.754	1.812	0.721	0.713	0.716
Neural net trained on HMM-agg. labels	0.712	0.790	0.748	2.282	0.715	0.707	0.710
Table 1: Evaluation results on CoNLL 2003. MV=Majority Voting, P=Precision, R=Recall, CEE=Cross-entropy Error (lower is better). The results are micro-averaged on all labels (PER, ORG, LOC and MISC).
Model:	Token-level	Entity-level
P	R	F1	CEE	P	R	F1
OntoNotes-trained NER	0.793	0.791	0.792	2.648	0.694	0.635	0.664
Majority voting (MV)	0.832	0.713	0.768	2.454	0.699	0.644	0.670
Confusion Matrix (CM)	0.816	0.702	0.754	2.708	0.667	0.636	0.652
Sequential Confusion Matrix (SEQ)	0.741	0.630	0.682	3.261	0.535	0.547	0.540
Dependent Confusion Matrix (DCM)	0.819	0.706	0.758	2.702	0.673	0.641	0.656
mSDA (OntoNotes)	0.749	0.751	0.750	2.501	0.618	0.684	0.649
AdaptaBERT (OntoNotes)	0.799	0.801	0.800	2.351	0.668	0.734	0.699
AdaptaBERT (Ensemble)	0.813	0.815	0.814	2.265	0.682	0.748	0.713
HMM-aggregated labels (all functions)	0.804	0.823	0.814	2.219	0.749	0.697	0.722
Neural net trained on HMM-agg. labels	0.805	0.827	0.816	2.448	0.749	0.701	0.724
Table 2: Evaluation results on 1094 crowd-annotated sentences from Reuters and Bloomberg news articles. The results are micro-averaged on 8 labels (PERSON, NORP, ORG, LOC, PRODUCT, DATE, PERCENT, and MONEY).
can be directly applied to new texts with a single forward-backward pass, and can therefore scale to datasets with hundreds of thousands of documents. This runtime performance is an important advantage compared to approaches such as AdaptaBERT (Han and Eisenstein, 2019) which are relatively slow at inference time. The proposed approach can also be ported to other languages than English, although heuristic functions and gazetteers will need to be adapted to the target language.
This paper presented a weak supervision model for sequence labelling tasks such as Named Entity Recognition. To leverage all possible knowledge sources available for the task, the approach uses a broad spectrum of labelling functions, including data-driven NER models, gazetteers, heuristic functions, and document-level relations between entities. Labelling functions may be specialised to recognise specific labels while ignoring oth
Figure 3: Extended example showing the outputs of 6 labelling functions, along with the HMM-aggregated model.
Figure 4: Pairwise agreement (left) and disagreement (right) between the labelling functions on the CoNLL 2003 data with labels PER, ORG, LOC, MISC, normalized by total number of labelled examples.
ers. Furthermore, unlike previous weak supervision approaches, labelling functions may produce probabilistic predictions. The outputs of these labelling functions are then merged together using a hidden Markov model whose parameters are estimated with the Baum-Welch algorithm. A neural sequence labelling model can finally be learned on the basis of these unified predictions.
Evaluation results on two datasets (CoNLL 2003 and news articles from Reuters and Bloomberg) show that the method can boost NER performance by about 7 percentage points on entity-level  $F_{1}$ . In particular, the proposed model outperforms the unsupervised domain adaptation approach through contextualised embeddings of Han and Eisenstein (2019). Of specific linguistic interest is the contribution of document-level labelling functions, which take advantage of the internal coherence and narrative structure of the texts.
Future work will investigate how to take into account potential correlations between labelling
functions in the aggregation model, as done in e.g. (Bach et al., 2017). Furthermore, some of the labelling functions can be rather noisy and model selection of the optimal subset of the labelling functions might well improve the performance of our model. Model selection approaches that can be adapted are discussed in Adams and Beling (2019); Hubin (2019). We also wish to evaluate the approach on other types of sequence labelling tasks beyond Named Entity Recognition.
The research presented in this paper was conducted as part of the innovation project "FinAI: Artificial Intelligence tool to monitor global financial markets" in collaboration with Exabel AS<sup>4</sup>. Additionally, this work is supported by the SANT project (Sentiment Analysis for Norwegian Text), funded by the Research Council of Norway.
4www.exabel.com
Stephen Adams and Peter A Beling. 2019. A survey of feature selection methods for Gaussian mixture models and hidden Markov models. Artificial Intelligence Review, 52(3):1739-1779.
Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 724-728, Minneapolis, Minnesota. Association for Computational Linguistics.
Stephen H. Bach, Bryan He, Alexander Ratner, and Christopher Ré. 2017. Learning the structure of generative models without labeled data. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, pages 273-282. JMLR.org.
Jeremy Barnes, Roman Klinger, and Sabine Schulte im Walde. 2018. Projecting embeddings for domain adaption: Joint modeling of sentiment analysis in diverse domains. In Proceedings of the 27th International Conference on Computational Linguistics, pages 818-830, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440-447, Prague, Czech Republic. Association for Computational Linguistics.
John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 120-128, Sydney, Australia. Association for Computational Linguistics.
Daniel Braun, Adrian Hernandez Mendez, Florian Matthes, and Manfred Langen. 2017. Evaluating natural language understanding services for conversational question answering systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 174-185, Saarbrücken, Germany. Association for Computational Linguistics.
Minmin Chen, Zhixiang Xu, Kilian Q. Weinberger, and Fei Sha. 2012. Marginalized denoising autoencoders for domain adaptation. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML'12, pages 1627-1634, USA. Omnipress.
Jason P.C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4:357-370.
Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256-263, Prague, Czech Republic. Association for Computational Linguistics.
A. P. Dawid and A. M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied Statistics, 28(1):20-28.
Leon Derczynski, Kalina Bontcheva, and Ian Roberts. 2016. Broad twitter corpus: A diverse named entity recognition resource. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1169-1179, Osaka, Japan. The COLING 2016 Organizing Committee.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. 2014. Using structured events to predict stock price movement: An empirical investigation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1415-1425, Doha, Qatar. Association for Computational Linguistics.
Jason Fries, Sen Wu, Alex Ratner, and Christopher Ré. 2017. Swellshark: A generative model for biomedical named entity recognition without labeled data.
Johanna Geiß, Andreas Spitz, and Michael Gertz. 2018. Neckar: A named entity classifier for wikidata. In *Language Technologies for the Challenges of the Digital Age*, pages 115–129, Cham. Springer International Publishing.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Scott Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In AAAI.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, pages 513-520, USA. Omnipress.
Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2):203-225.
Barbara J. Grosz and Candace L. Sidner. 1986. Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3):175-204.
Honglei Guo, Huijia Zhu, Zhili Guo, Xiaoxun Zhang, Xian Wu, and Zhong Su. 2009. Domain adaptation with latent semantic association for named entity recognition. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 281-289, Boulder, Colorado. Association for Computational Linguistics.
Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4237-4247, Hong Kong, China. Association for Computational Linguistics.
Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120-1130, Atlanta, Georgia. Association for Computational Linguistics.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. 2016. Harnessing deep neural networks with logic rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2410-2420, Berlin, Germany. Association for Computational Linguistics.
Aliaksandr Hubin. 2019. An adaptive simulated annealing EM algorithm for inference on nonhomogeneous hidden Markov models. In Proceedings of the International Conference on Artificial Intelligence, Information Processing and Cloud Computing, pages 1-9.
Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pages 619-627, La Palma, Canary Islands. PMLR.
Vijay Krishnan and Christopher D. Manning. 2006. An effective two-stage model for exploiting non-local dependencies in named entity recognition. In Proceedings of the 21st International Conference on
Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 1121-1128, Sydney, Australia. Association for Computational Linguistics.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270, San Diego, California. Association for Computational Linguistics.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Soren Auer, and Christian Bizer. 2015. Dbpedia - a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167-195.
Bill Yuchen Lin and Wei Lu. 2018. Neural adaptation layers for cross-domain named entity recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2012-2022, Brussels, Belgium. Association for Computational Linguistics.
Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003-1011, Suntec, Singapore. Association for Computational Linguistics.
Diego Mollá, Menno van Zaanen, and Daniel Smith. 2006. Named entity recognition for question answering. In Proceedings of the Australasian Language Technology Workshop 2006, pages 51-58, Sydney, Australia.
An T Nguyen, Byron C Wallace, Junyi Jessy Li, An Nenkova, and Matthew Lease. 2017a. Aggregating and predicting sequence labels from crowd annotations. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2017, page 299. NIH Public Access.
An Thanh Nguyen, Byron Wallace, Junyi Jessy Li, An Nenkova, and Matthew Lease. 2017b. Aggregating and predicting sequence labels from crowd annotations. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 299-309, Vancouver, Canada. Association for Computational Linguistics.
Nanyun Peng and Mark Dredze. 2017. Multi-task domain adaptation for sequence tagging. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 91-100, Vancouver, Canada. Association for Computational Linguistics.