SlowGuess/NAACL_2021_OCR huggingface dataset:
for Low-Resource NLP, pages 12-18, Melbourne. Association for Computational Linguistics.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. 2018. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML.
Zhanming Jie, Pengjun Xie, Wei Lu, Ruixue Ding, and Linlin Li. 2019. Better modeling of incomplete annotations for named entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 729-734. ACL.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. *NAACL*, pages 260–270.
Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. 2020. Bond: Bert-assisted open-domain named entity recognition with distant supervision. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1054-1064.
Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, pages 63-70.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-cnns-crf. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1064-1074.
Stephen Mayhew, Snigdha Chaturvedi, Chen-Tse Tsai, and Dan Roth. 2019. Named entity recognition with partially annotated training data. In Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 645-655.
Minlong Peng, Xiaoyu Xing, Qi Zhang, Jinlan Fu, and Xuan-Jing Huang. 2019. Distantly supervised named entity recognition using positive-unlabeled learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2409-2419.
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147-155.
Herbert Robbins and Sutton Monro. 1951. A stochastic approximation method. The annals of mathematical statistics, pages 400-407.
Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Tetsuro Sasada, Shinsuke Mori, Tatsuya Kawahara, and Yoko Yamakata. 2016. Named entity recognizer trainable from partially annotated data. Communications in Computer and Information Science, 593:148-160.
Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named entity tagger using domain-specific dictionary. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2054-2064.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
Charles Sutton and Andrew McCallum. 2006. An introduction to conditional random fields for relational learning. Introduction to statistical relational learning, 2:93-128.
Denny Vrandecic and Markus Krötzsch. 2014. Wiki-data: a free collaborative knowledgebase. Communications of the ACM, 57(10):78-85.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Yaosheng Yang, Wenliang Chen, Zhenghua Li, Zhengqiu He, and Min Zhang. 2018. Distantly Supervised NER with Partial Annotation Learning and Reinforcement Learning. COLING, pages 2159-2169.
A Dataset Processing
A.1 Artificially Perturbed Dataset
The gold annotations of training data are perturbed by lowering the recall and precision following Mayhew et al. (2019). Firstly, we randomly select an entity from the whole entity set and tag all of its occurrences to 'O'. We repeat this operation until the recall decreases to 0.5. Then, we randomly tag some tokens/spans to a random entity label to decrease the precision to 0.9. The detailed data statistics are shown in Table 2.
A.2 Distarntly Supervised Dataset
All distantly supervised datasets in our experiments are the same as those in Liang et al. (2020). The distant labels are generated by external knowledge bases (e.g. Wikidata Vrandecić and Krötzsch, 2014) and gazetteers collected from multiple online resources. Specifically, the entity candidates are first detected by POS tagger (NLTK Loper and Bird, 2002). Next, the ambiguous candidates are filtered out by the Wikidata query service. Then, they match the entities with words in multi-resources gazetteers to get their entity types. Additional rules are used to get the entity labels of the unmatched tokens. The detailed data statistics are shown in Table 2.
B.1 Model Structure and Implementation
For all the experiments with LSTM, we use the same word embeddings as Lample et al. (2016). We use the character-level LSTM with hidden size 25 to produce character-level word embeddings. The concatenation of the two embeddings are fed into BiLSTM with hidden size 100. We also apply the dropout (Srivastava et al., 2014) between layers, with a rate of 0.5. The model is optimized using Stochastic Gradient Descent (Robbins and Monro, 1951) with a learning rate of 0.01.
For experiments with BERT, we use the BERT-base (Devlin et al., 2018) as our encoder. The implementation is based on the codebase Hugging-Face Transformers (Wolf et al., 2020). The dropout rate is set to 0.2. The model is optimized using Adam (Kingma and Ba, 2014) with an initial learning rate of 3e-5.
Figure 5: Results of self-training.
B.2 Hyper-Parameters
There are two important hyper-parameters in our model as the positive noise rate  $\tau_{p}$  and the negative noise rate  $\tau_{n}$ . Based on our observation, the initial noise rates are various in different datasets. However, since our model has the ability to handle the noise, the noise rates are relatively stable after the first iteration of self training. Therefore, we empirically set  $\tau_{p}$  and  $\tau_{n}$  to 0.005 and 0.15 for all experiments from the second iteration. For the first iteration, we report the results of two strategies as follows:
Oracle. 'Oracle' means that we use the gold noise ratio (unavailable in real settings) of positive noise rate  $\tau_{p}$  and negative noise rate  $\tau_{n}$ . The strategy is only applicable for artificially perturbed datasets since the complete annotation is known.
Searched. 'Searched' means that we search the two hyper-parameters for best performance on the development set. We search two parameters separately since we assume  $\tau_{n}$  and  $\tau_{p}$  are independent. The search ranges from 0.0 to 0.2 with an interval of 0.01. We determine the two parameters with the best development result on different datasets.
C Analysis of Self Training
The self training is borrowed from Jie et al. (2019) and not our main contribution. However, to be self-contained, we also report the results of self training in Figure 5. Our method (both local and global) outperforms the baseline by a large margin at the first iteration, which indicates we have a better base model of handling noise. Also, all curves raise in the first several iterations and maintain stable relatively in the subsequent iterations.
Dataset	Training	Dev	Test
#entity	#sent	#entity	#sent	#entity	#sent
English	23,499	14,041	5,942	3,250	5,648	3,453
Spanish	18,796	8,322	4,338	1,914	3,559	1,516
German	11,851	12,152	4,833	2,867	3,673	3,005
Dutch	13,344	15,806	2,616	2,895	3,941	5,195
CoNLL	-	14,041	-	3,250	-	3,453
Tweet	-	2,393	-	999	-	3,844
Webpage	-	385	-	99	-	135
Wikigold	-	1,142	-	280	-	274