SlowGuess/ACL_2020_OCR huggingface dataset:
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Lawrence R. Rabiner. 1990. A tutorial on hidden markov models and selected applications in speech recognition. In Alex Waibel and Kai-Fu Lee, editors, Readings in Speech Recognition, pages 267-296. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
Jonathan Raiman and Olivier Raiman. 2018. Deep-type: Multilingual entity linking by neural type system evolution. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5406-5413.
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147-155, Boulder, Colorado. Association for Computational Linguistics.
Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid training data creation with weak supervision. Proc. VLDB Endow., 11(3):269-282.
Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2019. Snorkel: rapid training data creation with weak supervision. The VLDB Journal.
Alan Ritter, Luke Zettlemoyer, Mausam, and Oren Etzioni. 2013. Modeling missing data in distant supervision for information extraction. Transactions of the Association for Computational Linguistics, 1:367-378.
Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. 2014. Sequence labeling with multiple annotators. Mach. Learn., 95(2):165-181.
Juan Diego Rodriguez, Adam Caldwell, and Alexander Liu. 2018. Transfer learning for entity recognition of novel classes. In Proceedings of the 27th International Conference on Computational Linguistics,
pages 1974-1985, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Esteban Safranchik, Shiying Luo, and Stephen H. Bach. 2020. Weakly supervised sequence tagging from noisy rules. In AAAI Conference on Artificial Intelligence (AAAI).
Omer Sagi and Lior Rokach. 2018. Ensemble learning: A survey. WIREs Data Mining and Knowledge Discovery, 8(4):e1249.
Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. 2015. Domain adaption of named entity recognition to support credit risk assessment. In Proceedings of the Australasian Language Technology Association Workshop 2015, pages 84-90, Parramatta, Australia.
Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named entity tagger using domain-specific dictionary. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2054-2064, Brussels, Belgium. Association for Computational Linguistics.
Edwin D. Simpson and Iryna Gurevych. 2019. A Bayesian approach for sequence tagging with crowds. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1093-1104, Hong Kong, China. Association for Computational Linguistics.
Benjamin Strauss, Bethany Toma, Alan Ritter, Marie-Catherine de Marneffe, and Wei Xu. 2016. Results of the WNUT16 named entity recognition shared task. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT), pages 138-144, Osaka, Japan. The COLING 2016 Organizing Committee.
Amber Stubbs, Christopher Kotfila, and Ozlem Uzuner. 2015. Automated systems for the de-identification of longitudinal clinical narratives. Journal of Biomedical Informatics, 58(S):S11-S19.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Arata Ugawa, Akihiro Tamura, Takashi Ninomiya, Hiroya Takamura, and Manabu Okumura. 2018. Neural machine translation incorporating named entity. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3240-3250, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Hai Wang and Hoifung Poon. 2018. Deep probabilistic logic: A unifying framework for indirect supervision. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1891-1902, Brussels, Belgium. Association for Computational Linguistics.
Limin Wang, Shoushan Li, Qian Yan, and Guodong Zhou. 2018. Domain-specific named entity recognition with document-level optimization. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 17(4):33:1-33:15.
R. Weischedel, E. Hovy, M. Marcus, Palmer M., R. Belvin, S. Pradhan, L. Ramshaw, and N. Xue. 2011. OntoNotes: A large training corpus for enhanced processing. In Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation. Springer.
Marc Wick. 2015. Geonames ontology.
Vikas Yadav and Steven Bethard. 2018. A survey on recent advances in named entity recognition from deep learning models. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2145-2158, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. 2019. Tener: Adapting transformer encoder for name entity recognition. arXiv preprint arXiv:1911.04474.
Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. In International Conference on Learning Representations.
Jianfei Yu and Jing Jiang. 2016. Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 236-246, Austin, Texas. Association for Computational Linguistics.
Joey Tianyi Zhou, Hao Zhang, Di Jin, Hongyuan Zhu, Meng Fang, Rick Siow Mong Goh, and Kenneth Kwok. 2019. Dual adversarial neural transfer for low-resource named entity recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3461-3471, Florence, Italy. Association for Computational Linguistics.
Yftah Ziser and Roi Reichart. 2017. Neural structural correspondence learning for domain adaptation. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 400-410, Vancouver, Canada. Association for Computational Linguistics.
A Labelling functions
B Label matching problem
The baseline models relying on mixtures of multinomials have to address the so-called label matching problem, which needs some extra care.
The following approach was employed in the experiments from Section 4:
- First, we put strong initial values to the probabilities  $\sigma$  of individual classes based on the frequency of appearance of these classes in the most reliable labelling function. This is expected to increase the probability of EM exploring the mode around the initialised values.
- Second, we perform post-processing and set the labels to the states corresponding to the labels with the highest pairwise correlations to the latent labels from one of the three options:
1. the most reliable labelling function (Ontonotes-trained NER);
2. the majority voting labelling function;
3. the suggested Dirichlet dependent mixture model.
Additionally, if this highest correlation is below the threshold of 0.1 the O label is assigned to the corresponding state. We empirically observed that the label matching technique that performed best was to map the states to the labels produced by the majority voter (based on the pairwise correlations).
In Table 4, we provide the detailed results distributed by NER label for the CoNLL data 2003 which were presented in micro-averaged form in Table 1 of the main paper.
Label	Frequency	Model	Token-level	Entity-level
P	R	F1	P	R	F1
LOC	30.3 %	Ontonotes-trained NER	0.767	0.812	0.788	0.764	0.800	0.782
Majority voting (MV)	0.740	0.839	0.786	0.739	0.828	0.780
Confusion Matrix	0.721	0.895	0.798	0.714	0.890	0.792
Sequential Confusion Matrix	0.681	0.856	0.758	0.664	0.848	0.744
Dependent Confusion Matrix	0.718	0.890	0.794	0.710	0.886	0.788
Snorkel-aggregated labels	0.634	0.855	0.728	0.676	0.747	0.710
HMM (only NER models)	0.601	0.825	0.696	0.650	0.733	0.690
HMM (only gazetteers)	0.707	0.632	0.668	0.694	0.630	0.660
HMM (heuristics)	0.715	0.870	0.784	0.745	0.832	0.786
HMM (all but doc-level)	0.701	0.862	0.774	0.724	0.838	0.776
HMM (all functions)	0.726	0.859	0.786	0.738	0.839	0.786
NN trained on HMM	0.736	0.851	0.790	0.734	0.850	0.788
PER	28.7 %	Ontonotes-trained NER	0.850	0.833	0.842	0.787	0.741	0.764
Majority voting (MV)	0.915	0.871	0.892	0.831	0.775	0.802
Confusion Matrix	0.891	0.921	0.906	0.806	0.834	0.820
Sequential Confusion Matrix	0.849	0.879	0.864	0.730	0.789	0.758
Dependent Confusion Matrix	0.892	0.920	0.906	0.806	0.834	0.820
Snorkel-aggregated labels	0.816	0.903	0.858	0.769	0.717	0.742
HMM (only NER models)	0.837	0.860	0.848	0.770	0.744	0.756
HMM (only gazetteers)	0.917	0.452	0.606	0.835	0.391	0.532
HMM (heuristics)	0.836	0.933	0.882	0.791	0.799	0.794
HMM (all but doc-level)	0.859	0.917	0.888	0.814	0.782	0.798
HMM (all functions)	0.857	0.947	0.900	0.820	0.826	0.822
NN trained on HMM	0.856	0.946	0.898	0.814	0.824	0.818
ORG	26.6 %	Ontonotes-trained NER	0.536	0.517	0.526	0.437	0.306	0.360
Majority voting (MV)	0.725	0.512	0.600	0.610	0.434	0.508
Confusion Matrix	0.698	0.613	0.652	0.571	0.537	0.554
Sequential Confusion Matrix	0.632	0.590	0.610	0.485	0.515	0.500
Dependent Confusion Matrix	0.696	0.613	0.652	0.567	0.536	0.552
Snorkel-aggregated labels	0.512	0.639	0.568	0.519	0.496	0.508
HMM (only NER models)	0.516	0.549	0.532	0.425	0.333	0.374
HMM (only gazetteers)	0.648	0.304	0.414	0.512	0.235	0.322
HMM (heuristics)	0.566	0.625	0.594	0.549	0.501	0.524
HMM (all but doc-level)	0.565	0.631	0.596	0.551	0.494	0.520
HMM (all functions)	0.542	0.665	0.598	0.545	0.527	0.536
NN trained on HMM	0.539	0.665	0.596	0.537	0.519	0.528
MISC	14.4 %	Ontonotes-trained NER	0.676	0.599	0.636	0.702	0.583	0.636
Majority voting (MV)	0.861	0.187	0.308	0.809	0.193	0.312
Confusion Matrix	0.895	0.319	0.470	0.850	0.332	0.478
Sequential Confusion Matrix	0.850	0.320	0.464	0.791	0.333	0.468
Dependent Confusion Matrix	0.893	0.318	0.468	0.844	0.330	0.474
Snorkel-aggregated labels	0.852	0.398	0.542	0.863	0.400	0.546
HMM (only NER models)	0.667	0.544	0.600	0.708	0.518	0.598
HMM (only gazetteers)	0.745	0.011	0.022	0.594	0.008	0.016
HMM (heuristics)	0.842	0.499	0.626	0.850	0.478	0.612
HMM (all but doc-level)	0.714	0.596	0.650	0.781	0.575	0.662
HMM (all functions)	0.814	0.571	0.672	0.830	0.565	0.672
NN trained on HMM	0.852	0.577	0.688	0.866	0.583	0.696
Table 4: Detailed evaluation results on the CoNLL2003 dataset.