SlowGuess/NAACL_2022_OCR huggingface dataset:
In this Section, we describe our experimental setup (Section 5.1), the datasets<sup>10</sup> we use to train (Section 5.2) and evaluate (Section 5.3) our methodology, and finally the results obtained (Section 5.4).
In our experiments, we evaluate the quality of our data in two different settings:
1. In order to compare our dataset against previous state-of-the-art automatically-created datasets, we map our fine-grained annotations to the coarse-grained classes used by these datasets. Then, we train the mBERT + BiLSTM + CRF model introduced in Section 4.4 on both our dataset and the other abovementioned datasets. Finally, we compare the performance of the corresponding systems on gold-standard benchmarks for NER. Appendix A provides class mapping details;
2. To measure the quality of our fine-grained annotations, we manually annotate a random
10 All the datasets used are freely available for research purposes, except for OntoNotes 5.0 (LDC license).
Train	Test	CONLL	WIKIGOLD	ONTONOTES	BSNLP
EN	ES	NL	DE	EN	EN	RU	PL
WIKIANN	56.85	53.55	55.76	44.39	57.05	36.43	51.85	53.50
WIKINER	73.05	75.07	74.75	64.03	81.98	71.16	65.99	62.31
WIKINEURAL	76.94	77.87	77.40	64.02	82.42	71.98	66.50	62.44
MULTINERD W/O SELF-IMPROVEMENT	69.75	71.04	70.58	57.99	76.88	65.12	60.41	58.43
MULTINERD (OUR WORK)	77.11	78.20	77.84	65.22	83.11	72.45	67.39	62.94
Table 4: Span-based micro  $\mathrm{F}_{1}$  scores obtained by training a reference NER system on different automatically-created datasets (i.e. WikiANN, WikiNER, WikiNEuRal, MultiNERD) and testing on common NER benchmarks.
sample of 1K English sentences, and compare the annotations produced by our methodology with the corresponding ground truths.
We implement our model with PyTorch using the Transformers library (Wolf et al., 2019) to load the weights of BERT-base-multilingual-cased (mBERT), and train each model configuration with an early stopping strategy using a patience value of 5. We use Adam optimizer (Kingma and Ba, 2015) with learning rate of  $10^{-3}$  and a cross-entropy loss criterion. We repeat each training on 10 different seeds, fixed across experiments, and report the mean of their span  $\mathbf{F}_1$  scores computed with the official conllegal script. Further details are provided in Appendix B.
5.2 Training Data
We train our reference model with four different silver-standard datasets:
- MultiNERD: the resource created using the steps described in Section 4 from Wikipedia and Wikinews articles<sup>11</sup>, with  $n = 30K$  and  $t = 8$ . It covers 10 languages: Chinese, Dutch, English, French, German, Italian, Polish, Portuguese, Russian and Spanish. Statistics are shown in Table 3.
- WikiNEuRal (Tedeschi et al., 2021b): the current best-performing approach for NER silver data creation. It covers 9 languages (i.e. Dutch, English, French, German, Italian, Polish, Portuguese, Russian and Spanish), and sentences are extracted from Wikipedia.
- WikiNER (Nothman et al., 2013): a high-quality automatically-derived dataset for NER from Wikipedia. It covers the same languages as WikiNEuRal.
- WikiANN $^{12}$  (Pan et al., 2017): a massive dataset for NER consisting of Wikipedia documents annotated in 282 languages.
All datasets are tagged with the four standard entity types (PER, ORG, LOC, MISC), except for WikiANN which does not contain the MISC label. Indeed, when evaluating the WikiANN dataset, only the PER, ORG and LOC classes are evaluated.
5.3 Test Data
Common Benchmarks For our first setting (Section 5.1), we use 5 common gold-standard test sets:
- CoNLL-2002 and 2003 (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003): a well-known corpus of NER-annotated newswire articles for Dutch, English, German and Spanish;
- WikiGold (Balasuriya et al., 2009): a set of human-annotated English Wikipedia articles.
- OntoNotes 5.0 (Pradhan et al., 2012): a popular dataset for NER that includes texts from different textual genres and multiple domains.
BSNLP-2017 (Piskorski et al., 2017): a notable set of manually-annotated articles for Slavic languages.
All datasets use coarse-grained entity types (i.e. PER, ORG, LOC and MISC), except OntoNotes, which uses fine-grained entities that we convert to coarse-grained classes. Appendix A provides additional details about this mapping. For the WikiGold dataset, which is limited in size, we use the entire dataset as test material. For other datasets, instead, we use the official validation and test splits. Finally, all datasets are converted to BIO format.
12The WikiANN version used in this study corresponds to the dataset of Rahimi et al. (2019), which supports 176 of the 282 languages from the original WikiANN corpus (https://huggingface.co/datasets/wikiann).
<sup>11</sup>We use the April 2021 snapshot for both Wikipedia and Wikinews dumps, sampling random articles.
Figure 1: Confusion matrix of our silver-standard annotations compared to the corresponding ground truths.
Manual Annotation For our second experimental setting (Section 5.1), due to the absence of NER benchmarks that use our set of categories (Section 3), we conduct a manual evaluation to assess the quality of our dataset. Specifically, we randomly select<sup>13</sup> a sample of 1K English sentences, preannotated with the NER tags produced using our methodology (Section 4), and confirm or replace the annotations associated with each token in the dataset. The resulting gold-standard dataset is used to analyze the quality of our silver-standard data.
Coarse-Grained Evaluation In our first setting (Section 5.1), we measure the effectiveness of our methodology by comparing the quality of the data produced against that of other datasets created using previous state-of-the-art strategies for NER silver-data creation (i.e. datasets listed in Section 5.2). Since past approaches focused on coarse-grained entities, we can compare the quality only for such entity types. The results obtained are shown in Table 4. Although our dataset covers a wider range of categories than its competitors, it nevertheless outperforms all of them on all tested
Class	P	R	F1	Support
ANIM	0.72	0.52	0.60	60
BIO	0.78	0.66	0.71	32
CEL	0.96	0.96	0.96	25
DIS	0.99	0.88	0.93	78
EVE	0.96	0.93	0.94	27
FOOD	0.83	0.44	0.58	122
INST	0.90	0.53	0.67	34
LOC	0.99	0.99	0.99	262
MEDIA	0.95	0.95	0.95	41
O	0.99	1.00	0.99	11823
ORG	0.97	0.95	0.96	59
PER	0.99	0.99	0.99	217
PLANT	0.73	0.40	0.52	47
MYTH	0.88	0.96	0.92	23
TIME	0.99	0.96	0.98	171
VEHI	0.82	0.85	0.84	27
ALL	0.90	0.81	0.85	13048
Table 5: Evaluation of the MultiNERD dataset on our manually-annotated English test set.
datasets and languages. We attribute this advancement mainly to the self-improvement algorithm introduced in Section 4.4, which iteratively refines the annotations using a better model at each iteration. To demonstrate the impact of our algorithm, we construct baseline versions of MULTINERD for
<sup>13</sup>We ensure that the dataset contains a sufficient number  $n$  of instances for each NER class. We set  $n = 20$ . Statistics are provided in the "Support" column of Table 5.
DE, EN, ES, NL, PL and RU with the same sizes as the corresponding refined versions, but without using our enhancement procedure. As can be observed from Table 4, the refined versions provide an average improvement of almost  $7\mathrm{F}_1$  points. In addition, the wider number of textual genres covered by MULTINERD leads to more robust systems.
Fine-Grained Evaluation Although the coarse-grained evaluation conducted in the previous Section demonstrated that our MULTINERD methodology creates high-quality annotations, independently of the language, it is not sufficient to understand how our annotation pipeline performs on fine-grained classes. Indeed, to measure this, we use a sample of 1K English sentences manually-annotated with fine-grained entities, as explained in Section 5.3, and report the results in Table 5. As expected, the PER, ORG and LOC classes are among the best-performing classes. Similarly, celestial bodies, diseases, events and media also have very high performance, thanks to their occurrences being almost always linked in Wikipedia and Wikinews articles (high recall) and easily distinguishable (high precision). In contrast, animals, biological entities, foods and plants have a high-degree of confusion (lower precision), and are very often not linked (low recall). To better explain this, we report in Figure 1 the confusion matrix of the silver-standard annotations produced by our approach compared to the gold-standard ones. As an example, it can be observed that animals and plants are often confused with each other, mainly because their scientific names are morphologically very close. Similarly, animals and plants are also confused with foods (e.g. Alaskan salmon and Quinoa), and vice versa.
Even though the quality of the annotations produced by our approach for any particular one of the 10 languages covered herein is strongly dependent on the quality of the corresponding Wikipedia and Wikinews dumps, we expect comparable performance on all other languages, as suggested by the statistics in Table 3 which show strong consistency across languages.
In this work we introduced MULTINERD, a novel resource for training robust multilingual and fine-grained Named Entity Recognition (and Disambiguation) systems. To create it, we presented a new language-agnostic strategy for generating
high-quality silver-standard NER and ED annotations. This strategy uses a knowledge-based semantic classifier to automatically annotate Wikipedia and Wiknews articles, and then iteratively enhances the annotations produced by means of a self-improvement algorithm which builds upon neural models. Our experiments showed that MULTIN-ERD outperformed previous state-of-the-art dataproduction methods across all tested languages and domains, while covering a much wider set of NER categories. Additionally, we also included image URLs in our dataset to encourage the development of multimodal NER and ED systems. This visual information could also be exploited to further improve the quality of the annotations by assembling the predictions of NLP and Computer Vision models. We release our MULTIN-ERD dataset and software at https://github.com/Babelscape/multinerd.
Rami Al-Rfou, Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. 2015. POLYGLOT-NER: massive multilingual named entity recognition. In Proceedings of the 2015 SIAM International Conference on Data Mining, Vancouver, BC, Canada, April 30 - May 2, 2015, pages 586-594. SIAM.
Chinatsu Aone, Mary Okuowski, and James Gorlinsky. 1998. Trainable, scalable summarization using robust nlp and machine learning. pages 62-66.
Bogdan Babych and Anthony Hartley. 2003. Improving machine translation quality with automatic named entity recognition. In Proceedings of the 7th International EAMT workshop on MT and other language technology tools, Improving MT through other language technology tools, Resource and tools for building MT at EACL 2003.
Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R. Curran. 2009. Named entity recognition in Wikipedia. In Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources (People's Web), pages 10-18, Suntec, Singapore. Association for Computational Linguistics.
Nancy Chinchor and Patricia Robinson. 1997. Muc-7 named entity task definition. In Proceedings of the 7th Conference on Message Understanding, volume 29, pages 1-21.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147, Copenhagen, Denmark. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ralph Grishman and Beth Sundheim. 1996. Message Understanding Conference- 6: A brief history. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics.
Eduard Hovy, Roberto Navigli, and Simone Paolo Ponzetto. 2013. Collaboratively built semi-structured content and artificial intelligence: The story so far. Artificial Intelligence, 194:2 - 27. Artificial Intelligence, Wikipedia and Semi-Structured Resources.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Pedro Henrique Martins, Zita Marinho, and André F. T. Martins. 2019. Joint learning of named entity recognition and entity linking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 190-196, Florence, Italy. Association for Computational Linguistics.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
Diego Mollá, Menno van Zaanen, and Daniel Smith. 2006. Named entity recognition for question answering. In Proceedings of the Australasian Language Technology Workshop 2006, pages 51-58, Sydney, Australia.
David Mueller, Nicholas Andrews, and Mark Dredze. 2020. Sources of transfer in multilingual named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8093-8104, Online. Association for Computational Linguistics.
David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Linguisticae Investigationes, 30(1):3-26.
Roberto Navigli, Michele Bevilacqua, Simone Conia, Dario Montagnini, and Francesco Ceccconi. 2021. Ten years of babelnet: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4559-4567. International Joint Conferences on Artificial Intelligence Organization. Survey Track.
Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217 - 250.
Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James Curran. 2013. Learning multilingual named entity recognition from wikipedia. Artificial Intelligence, 194:151-175.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
Jakub Piskorski, Lidia Pivovarova, Jan Šnajder, Josef Steinberger, and Roman Yangarber. 2017. The first cross-lingual challenge on recognition, normalization, and matching of named entities in Slavic languages. In Proceedings of the 6th Workshop on Balto-Slavic Natural Language Processing, pages 76-85, Valencia, Spain. Association for Computational Linguistics.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1-40, Jeju Island, Korea. Association for Computational Linguistics.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151-164, Florence, Italy. Association for Computational Linguistics.
Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An experimental study. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1524-1534, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Simone Tedeschi, Simone Conia, Francesco Cecconi, and Roberto Navigli. 2021a. Named Entity Recognition for Entity Linking: What works and what's next. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2584-2596, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Simone Tedeschi, Valentino Maiorca, Niccolò Campolungo, Francesco Ceccconi, and Roberto Navigli. 2021b. WikiNEuRal: Combined neural and knowledge-based silver data creation for multilingual NER. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2521-2533, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Chen-Tse Tsai, Stephen Mayhew, and Dan Roth. 2016. Cross-lingual named entity recognition via wikification. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 219–228, Berlin, Germany. Association for Computational Linguistics.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013. OntoNotes Release 5.0.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442–6454, Online. Association for Computational Linguistics.
A NER Classes
Additional Details The OntoNotes 5.0 dataset distinguishes between locations, facilities and geopolitical entities – which are closely related – while the LOC class in our set clusters them. Moreover, we ignore the cardinals, ordinals, quantities and monetary values because their instances are not linked in Wikipedia, and their identification requires language-specific solutions. We also remove
Our Class	CoNLL class	C/NE
PER	PER	NE
ORG	ORG	NE
LOC	LOC	NE
ANIM	O	C
BIO	MISC	C
CEL	O	C/NE
DIS	MISC	C
EVE	MISC	NE
FOOD	O	C/NE
INST	MISC	NE
MEDIA	MISC	NE
PLANT	O	C
MYTH	PER	NE
TIME	O	C/NE
VEHI	MISC	NE
Table 6: The second column shows the CoNLL tags corresponding to our 15 NER classes (included the O tag), while the third column specifies whether for a given class we consider only Concepts (C), Named Entities (NE), or both (C/NE), as valid entities.
the PHY class (i.e. physical phenomena) due to its extreme rarity (i.e.  $0.4 \text{‰}$  of entities belong to the PHY class according to Tedeschi et al., 2021a).
Mappings In order to measure the quality of our data against that of previous state-of-the-art automatically-generated datasets, which cover only the four coarse-grained entity types, we need to map our 15 classes to those entities. To do so, we analyze how elements belonging to our classes are labeled in the CoNLL dataset. The resulting class mapping is provided in Table 6. Similarly, for evaluating system performance on the OntoNotes dataset, we map its 18 classes to the 4 CoNLL classes. In this case, we use the mapping provided by Tedeschi et al. (2021b). Once mapped, as in Tedeschi et al. (2021a), the datasets are refined through a NER classifier trained on WikiNER.
Entity or Concept? Some of the entities that we are interested in (e.g. animals and diseases) are labeled as concepts in BabelNet, hence we need to accurately specify for which categories concepts are allowed. Our choices are reported in Table 6.
All model training was carried out on an NVIDIA GeForce RTX 3090 architecture. It required  $\sim 45$  s/epoch on the CoNLL and WikiANN datasets,
Hyperparameter name	Value
number of Bi-LSTM layers	2
LSTM hidden size	512
batch size	128
learning rate	0.001
dropout	0.5
gradient clipping	1.0
adam β1	0.9
adam β2	0.999
adam ε	1e-8
Table 7: Hyperparameter values of the reference model.
whereas it required  $\sim 6$  min/epoch on the WikiNER, WikiNEuRal and MultiNERD datasets. The full list of hyperparameter values of the best performing model – the one used to compare the quality of the different datasets – is shown in Table 7.