SlowGuess/NAACL_2022_OCR huggingface dataset:
Simone Tedeschi
Babelscope & Sapienza University of Rome
tedeschi@babelscape.com
Roberto Navigli
Named Entity Recognition (NER) is the task of identifying named entities in texts and classifying them through specific semantic categories, a process which is crucial for a wide range of NLP applications. Current datasets for NER focus mainly on coarse-grained entity types, tend to consider a single textual genre and to cover a narrow set of languages, thus limiting the general applicability of NER systems. In this work, we design a new methodology for automatically producing NER annotations, and address the aforementioned limitations by introducing a novel dataset that covers 10 languages, 15 NER categories and 2 textual genres. We also introduce a manually-annotated test set, and extensively evaluate the quality of our novel dataset on both this new test set and standard benchmarks for NER. In addition, in our dataset, we include: i) disambiguation information to enable the development of multilingual entity linking systems, and ii) image URLs to encourage the creation of multimodal systems. We release our dataset at https://github.com/Babelscope/multinerd.
Named Entity Recognition (NER) represents a milestone in information extraction, and its aim is to identify and classify key information in unstructured texts, i.e. named entities (Nadeau and Sekine, 2007). It is widely used in a broad spectrum of downstream applications, like machine translation (Babych and Hartley, 2003), question answering (Mollá et al., 2006), automatic text summarization (Aone et al., 1998), and entity linking (Martins et al., 2019), inter alia.
With the advent of pretrained language models like BERT (Devlin et al., 2019) or LUKE (Yamada et al., 2020) – this latter with a particular focus on named entities – the NER field observed astonishing results on conventional benchmarks (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003). However, such benchmarks are limited
in size, cover a single textual genre, and are available only for a narrow set of languages. Moreover, they focus on coarse-grained entity types, and often overlook more complex entities like titles of books, songs and movies. These latter entities are not simple nouns and can be both syntactically and semantically ambiguous. Specifically, they can assume the form of any linguistic constituent (e.g. Singin' in the Rain) which makes them difficult to extract. Indeed, in the last decade, the OntoNotes 5.0 dataset (Weischedel et al., 2013) has become very popular thanks to its high quality, size and fine-grained categories. Nevertheless, it covers only 3 languages, namely, English, Arabic and Chinese.
Since the manual creation of training data for NER is expensive and time-consuming – especially when many languages have to be covered – several studies have tried to address data scarcity by producing training data automatically (Nothman et al., 2013; Al-Rfou et al., 2015; Tsai et al., 2016; Pan et al., 2017), recently showing that automatically-generated annotations can boast a quality comparable to that of manually-created ones (Tedeschi et al., 2021b). Unfortunately, although these studies have considered a wider range of languages, they have still focused on coarse-grained entities and on a single textual genre, i.e. encyclopedic texts from Wikipedia<sup>1</sup> (Hovy et al., 2013).
In this paper, inspired by the success of the OntoNotes 5.0 dataset and by recent achievements in automatic data creation, we fill the aforementioned gaps and propose the following novel contributions:
1. We design a new language-agnostic methodology for automatically generating high-quality and fine-grained NER annotations by exploiting the texts of Wikipedia and Wikinews<sup>2</sup>;
<https://en.wikipedia.org/
2It is a free-content multilingual wiki containing news articles (https://en.wikinews.org/), as opposed to the encyclopedic articles contained in Wikipedia.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 801 - 812
2. We introduce a novel automatically-created benchmark for NER that covers 10 languages, 15 entity types and 2 textual genres, together with a small manually-curated test set for the English language;
3. We extensively evaluate the quality of the data produced on both our manually-annotated test set and standard benchmarks for NER.
Additionally, although in this work we focus on NER, we also contribute to the entity disambiguation (also known as entity linking) task, i.e. the task of linking entities mentioned in texts with their corresponding entry in a knowledge base. Specifically, for a given entity, we provide disambiguation information together with its NER tag in order to enable training, validation and testing of multilingual entity linking models. Finally, we also include image URLs to encourage the creation of multimodal systems. To enable comparability on our benchmark, we release our data and software at https://github.com/Babelscope/multinerd.
2.1 Gold-Standard Data
High-quality annotations are essential for both learning and evaluation of NER systems. Indeed, in the past few decades a large number of NER datasets have been proposed. Initially, the MUC-6 and MUC-7 shared tasks focused on entity names (i.e. persons, locations and organizations), temporal expressions (i.e. dates and times) and number expressions (i.e., currency values and percentages), but only English newswire articles were considered (Grishman and Sundheim, 1996; Chinchor and Robinson, 1997).
A few years later, different datasets were derived from Reuters News for the CoNLL-2002 and 2003 shared tasks on language-independent Named Entity Recognition (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003), covering four different languages (i.e. Dutch, English, German and Spanish). However, these datasets were limited in size and only four coarse-grained entity types were considered: Person, Location, Organization and Miscellaneous<sup>3</sup>. Nonetheless, these datasets are still widely used to benchmark NER systems.
Balasuriya et al. (2009) claimed that NER was needed in many domains beyond newswire texts,
and introduced WikiGold, a manually-annotated dataset derived from Wikipedia articles. Even so, WikiGold covered coarse-grained entities, was limited in size and considered only the English language. Following the same motivation as Balasuriya et al. (2009), Ritter et al. (2011) introduced a dataset for English tweets using 10 NER classes.
Another considerable step forward was made by Weischedel et al. (2013), who introduced OntoNotes 5.0. This dataset covered 18 fine-grained classes, multiple genres (e.g. newswire and weblogs), and multiple languages (English, Chinese, and Arabic). Thanks to its high quality, it is one of the most widely used datasets for NER.
Finally, another notable dataset was proposed for the WNUT 2017 shared task on emerging and rare entities, covering different textual genres (tweets, YouTube comments, Reddit and StackExchange posts) (Derczynski et al., 2017). However, only the English language and 6 categories were considered.
2.2 Silver-Standard Data
Although the OntoNotes 5.0 dataset constitutes a valuable resource for training and evaluating multilingual and fine-grained NER systems, its applicability is limited to the three languages it covers. Indeed, in the last decade, with the aim of scaling NER to a wider set of languages, more interest has been devoted to automatic data creation.
The first successful attempt in this direction was made by Nothman et al. (2013) who produced the WikiNER dataset. They proposed a strategy for automatically creating multilingual training data for NER by exploiting the texts of Wikipedia and its hypertext organization. In addition, they also used redirect-base heuristics to infer more namedentity mentions. By applying this methodology, they covered 9 languages, but they still focused on the standard coarse-grained entity types.
Adopting a similar strategy, Pan et al. (2017) introduced WikiANN, a language-independent framework for extracting entities from documents. Their procedure was made up of two main steps: i) classify entries in the English Wikipedia into specific entity types, and ii) propagate the annotations to other languages by applying cross-lingual transfer. This procedure yielded massive corpora consisting of 282 languages, but with lower annotation quality and a focus on persons, locations, organizations and geo-political entities.
Finally, Tedeschi et al. (2021b) proposed
3Miscellaneous entities are entities that do not belong to the Person, Location and Organization categories.
WikiNEuRal, an annotation pipeline that effectively combined recent pretrained language models with knowledge-based approaches, and produced high-quality annotations for NER in 9 languages by exploiting Wikipedia texts. Surprisingly, the authors showed that their methodology, in 2 out of 3 settings, produced annotations with a quality even higher than that of manual ones. However, again, only coarse-grained entities were considered.
Despite automatic methods achieved high annotation quality and covered many languages, all of them focused on coarse-grained entities and on a single textual source: Wikipedia. On the other hand, gold-standard datasets focused mainly on the English language. Additionally, none of them included disambiguation information. Evidently, a unified effort to obtain a large-scale multilingual, multi-genre and fine-grained resource for Named Entity Recognition and Disambiguation is still missing.
3 NER Classes
Our 15 NER classes are a subset of the newly introduced 18 classes of Tedeschi et al. (2021a) designed to reduce the intrinsic sparsity of the Entity Linking task. Specifically, they are: Person (PER), Location (LOC), Organization (ORG), Animal (ANIM), Biological entity (BIO), Celestial Body (CEL), Disease (DIS), Event (EVE), Food (FOOD), Instrument (INST), Media (MEDIA), Plant (PLANT), Mythological entity (MYTH), Time (TIME) and Vehicle (VEHI).
We prefer these classes to the OntoNotes ones, because they cover a wider range of macro categories. For instance, the OntoNotes' PRODUCT class – which groups very heterogeneous entities – is split into FOOD, VEHI and INST. Over and beyond these, our new set contains animals, plants, biological entities, celestial bodies, diseases and mythological entities that are not present in OntoNotes. Table 1 provides textual descriptions and examples of instances concerning our proposed classes. Further details about NER classes are provided in Appendix A.
4 MultiNERD
In this Section we describe our language-agnostic strategy for automatically generating a fine-grained and multilingual resource to train robust NER and ED systems. Specifically, our methodology widely extends previous state-of-the-art strategies, and it is
characterized by the following five steps: i) preprocessing of Wikipedia and Wikinews articles (Section 4.1), ii) identification of entities (Section 4.2), iii) tagging the identified entities with the NER labels (Section 4.3), iv) propagation of the annotations (Section 4.3), v) enhancement of the annotations (Section 4.4).
4.1 Wikitext Preprocessing
Wikipedia and Wikinews articles provide plenty of manually-curated information that can be exploited for the automatic annotation of sentences, i.e. Wikilinks<sup>4</sup>. However, in addition to Wikilinks, articles may contain elements (e.g. images, tables, formulas and lists) and sections (e.g. see also, references, further readings) that do not correspond to well structured text; therefore, we remove them with the intent of reducing noise. This step converts articles to plain texts containing only Wikilinks.
4.2 Entity or Concept?
Wikilinks provide potential entity mentions. Indeed, some of them correspond to entities (e.g. *Elon Musk*), and others correspond to concepts (e.g. *Table*). In order to distinguish between them, we take advantage of the one-to-one linkage between Wikipedia and BabelNet $^5$  (Navigli and Ponzetto, 2012; Navigli et al., 2021) and exploit the concept-vs.-entity categorization provided therein. Although it is evident that for some categories we are only interested in entities in the strictest sense (e.g. PER, ORG and LOC), we need to relax this constraint for other classes (e.g. ANIM, PLANT, FOOD and DIS). Thus, in order to extract animals (e.g. Labrador Retriever), plants (e.g. Pinus), food (e.g. Carbonara) and diseases (e.g. Alzheimer's disease), among others, we also need to consider elements that are labeled as concepts in BabelNet. This step tells us which Wikilinks have to be annotated with an entity type, and which of them have to be discarded. The full list of design choices is provided in Appendix A.
4.3 Tagging Wikipedia and Wikinews Articles
Semantic Classifier We now aim at providing each (remaining) Wikilink in a Wikipedia (or Wikinews) article with a category  $c \in C$ , where  $C$
<sup>4</sup>A Wikilink is a link placed inside an article that points to a Wikipedia page.
5BabelNet (https://babelnet.org) is a wide multilingual semantic network that integrates both lexicographic and encyclopedic knowledge from different sources. We use BabelNet 5.0.
Class	Description	Examples
PER	People.	Ray Charles, Jessica Alba, Leonardo DiCaprio, Roger Federer, Anna Massey.
ORG	Associations, companies, agencies, institutions, nationalities and religious or political groups.	University of Edinburgh, San Francisco Giants, Google, Democratic Party.
LOC	Physical locations (e.g. mountains, bodies of water), geopolitical entities (e.g. cities, states), and facilities (e.g. bridges, buildings, airports).	Rome, Lake Paiku, Chrysler Building, Mount Rushmore, Mississippi River.
ANIM	Breeds of dogs, cats and other animals, including their scientific names.	Maine Coon, African Wild Dog, Great White Shark, New Zealand Bellbird.
BIO	Genus of fungus, bacteria and protoctists, families of viruses, and other biological entities.	Herpes Simplex Virus, Escherichia Coli, Salmonella, Bacillus Anthracis.
CEL	Planets, stars, asteroids, comets, nebulae, galaxies and other astronomical objects.	Sun, Neptune, Asteroid 187 Lamberta, Proxima Centauri, V838 Monocerotis.
DIS	Physical, mental, infectious, non-infectious, deficiency, inherited, degenerative, social and self-inflicted diseases.	Alzheimer's Disease, Cystic Fibrosis, Dilated Cardiomyopathy, Arthritis.
EVE	Sport events, battles, wars and other events.	American Civil War, 2003 Wimbledon Championships, Cannes Film Festival.
FOOD	Foods and drinks.	Carbonara, Sangiovese, Cheddar Beer Fondue, Pizza Margherita.
INST	Technological instruments, mechanical instruments, musical instruments, and other tools.	Spitzer Space Telescope, Commodore 64, Skype, Apple Watch, Fender Stratocaster.
MEDIA	Titles of films, books, magazines, songs and albums, fictional characters and languages.	Forbes, American Psycho, Kiss Me Once, Twin Peaks, Disney Adventures.
PLANT	Types of trees, flowers, and other plants, including their scientific names.	Salix, Quercus Petraea, Douglas Fir, Forsythia, Artemisia Maritima.
MYTH	Mythological and religious entities.	Apollo, Persephone, Aphrodite, Saint Peter, Pope Gregory I, Hercules.
TIME	Specific and well-defined time intervals, such as eras, historical periods, centuries, years and important days. No months and days of the week.	Renaissance, Middle Ages, Christmas, Great Depression, 17th Century, 2012.
VEHI	Cars, motorcycles and other vehicles.	Ferrari Testarossa, Suzuki Jimmy, Honda CR-X, Boeing 747, Fairey Fulmar.
Table 1: Descriptions and instance examples of our NER classes.
is the set of the NER classes introduced in Section 3. To do this, we introduce a Semantic Classifier that exploits the one-to-one correspondence between Wikipedia pages and BabelNet synsets. We start by manually-annotating 300 synsets to cover as many high-order concepts of the WordNet $^6$  (Miller, 1995) nominal taxonomy – which is a subset of the BabelNet taxonomy – as possible. For instance, we label the following high-level synsets as follows:
- animal (bn:00004222n)  $\rightarrow$  ANIM;
company (bn:00021286n)  $\rightarrow$  ORG;
- town (bn:00077773n)  $\rightarrow$  Loc;
Then, to propagate these annotations to all other synsets in WordNet, we descend through its taxonomy by following hyponymy and has-instance relationships (i.e. parent-to-child relations). For example, all the children of animal (bn:00004222n), e.g. dog
(bn:00015267n), inherit the ANIM tag. This step yields 40k high-quality annotated synsets.
At this point, to annotate a Wikilink  $l$  in a Wikipedia (or Wikinews) article  $w$ , we retrieve its corresponding synset  $s$  from BabelNet, and we follow hypernymy relations (child-to-parent relations) until one or more of the 40k synsets in the expanded set is reached<sup>7</sup>. Here, we distinguish between two possible cases:
1. When a single ancestor is reached, or when all the ancestors share the same class, the corresponding annotation is just inherited. For instance, starting from Apple Inc. (bn:03739345n) and climbing the taxonomy, we find only company (bn:00021286n) at distance 1, hence Apple Inc. inherits the ORG annotation.
2. When two or more ancestors have discordant annotations, then the highest-scoring class<sup>8</sup>
7We employ a Breadth-First Search with max depth  $= 2$  If more classes have the same score, the corresponding synset is not annotated.
<sup>6</sup>We start from WordNet synsets because they are manually curated.
Token	Label	BabelNet ID	Wikidata ID	Wikipedia ID	Wikipedia Title	Definition	Image URL
Neither	O	-	-	-	-	-	-
George	B-PER	bn:00040218n	Q2643	12731	George Harrison	English rock star...	george_harrison.jpg
Harrison	I-PER	-	-	-	-	-	-
nor	O	-	-	-	-	-	-
Ringo	B-PER	bn:00067735n	Q2632	25832	Ringo Starr	Rock star and drummer...	ringo_starr.jpg
Starr	I-PER	-	-	-	-	-	-
was	O	-	-	-	-	-	-
involved	O	-	-	-	-	-	-
in	O	-	-	-	-	-	-
the	O	-	-	-	-	-	-
recording	O	-	-	-	-	-	-
.	O	-	-	-	-	-	-
Table 2: Example of sentence in our MultiNERD dataset. Together with the NER type (in BIO format), each entity is provided with the following additional information: BabelNet synset, Wikidata ID, Wikipedia ID, Wikipedia title, definition and image url.
is assigned. Formally, for each NER class  $c \in C$ , the score is computed as follows:
$$
\operatorname {s c o r e} (c) = \sum_ {a \in A _ {c}} \frac {1}{d (a)}, \tag {1}
$$
where  $A_{c}$  is the set of all the ancestors of synset  $s$  with tag  $c$ , and  $d(a)$  is a function that returns the distance of  $a$  from  $s$  in the Babel-Net taxonomy. As an example, consider the synset Bill Gates (bn:00010401n). When climbing the taxonomy, at distance 2, the hominid (bn:00044571n) synset is reached, which is classified as ANIM. However, the Bill Gates' synset is also child of human (bn:00044576n), computer scientist (bn:00021495n) and magnate (bn:00008639n), resulting in a highest score for the PER class.
This procedure allows us to label each Wikilink in a given article with a NER class.
Tag Propagation Wikipedia and Wikinews guidelines specify that only the first mention to a certain article has to be linked. This implies that tagging only Wikilinks leads to sparse annotations. To cope with this issue, we employ a simple yet effective exact-match heuristic in which for each Wikilink  $l$ , with an associated class  $c$ , we assign the class  $c$  to all the expressions  $e_i$  in the same document of  $l$  such that  $e_i = l \lor e_i \in \operatorname{syn}(l)$ , where  $\operatorname{syn}(l)$  is a function that returns the synonyms of  $l$  from BabelNet. Finally, the annotations are converted to BIO format<sup>9</sup>.
The above-described methodology allows us to have at the same time, for a given entity, both the
Algorithm 1 Self-Improvement Algorithm
Inputs: Corpus of raw Wikipedia and Wikinews documents W
Parameters:Integer  $n$  ，Integer  $t$
Output: MultiNERD Dataset D
1:  $A\gets \{w_1,\ldots ,w_n\}$ $w_{i}\in W$
2:  $D\gets$  annotate(A)
3:  $M_D \gets \text{train}(D)$
4: for i  $\leftarrow 1,\dots ,t$  do
5:  $\hat{A} \gets \{w_1', \ldots, w_n'\}$ ,  $w_i' \in W$ ,  $A \cap \hat{A} = \emptyset$
6:  $A = A\cup \tilde{A}$
7:  $D = \text{annotate}(A, M_D)$
8:  $M_D = \text{train}(D)$
9: end for
10: return  $D$
NER annotation and the BabelNet synset. Then, as already mentioned, through BabelNet we can easily access other resources, and retrieve the corresponding Wikipedia and Wikidata pages. Hence, in our dataset we include disambiguation information from the three above-mentioned knowledge bases. Additionally, for a given entity, we include also the corresponding definition and the main image from Wikidata, where this latter can be used to develop multimodal NER and entity linking systems. An instance of our dataset is provided in Table 2.
4.4 Annotation Enhancement
The above steps enable multilingual and fine-grained annotations to be created. However, these annotations are derived automatically and, therefore, they may contain errors.
Tedeschi et al. (2021b) improved the quality of the annotations by combining them with the
<sup>9</sup>The BIO tagging scheme (short for Beginning, Intermediate, Out) is a popular format for handling spans of tokens.
DE	EN	Es	FR	IT	NL	PL	PT	RU	Zh
Statistics	PER	79.2K	75.8K	70.9K	89.6K	75.3K	56.9K	66.5K	54.0K	43.4K
ORG	31.2K	33.7K	20.6K	28.2K	19.3K	21.4K	29.2K	13.2K	21.5K
LOC	72.8K	78.5K	90.2K	90.9K	98.5K	78.7K	100.0K	124.8K	75.2K
ANIM	11.5K	15.5K	10.5K	11.4K	8.8K	34.4K	19.7K	14.7K	7.3K
BIO	0.1K	0.2K	0.3K	0.1K	0.1K	0.1K	0.1K	0.1K	0.1K
CEL	1.4K	2.8K	2.4K	2.3K	5.2K	2.1K	3.3K	4.2K	1.2K
DIS	5.2K	11.2K	8.6K	3.1K	6.5K	6.1K	6.5K	6.8K	1.9K
EVE	4.0K	3.2K	6.8K	7.4K	5.8K	4.7K	6.7K	5.9K	2.8K
FOOD	3.6K	11.0K	7.8K	3.2K	5.8K	5.6K	3.3K	5.4K	3.2K
INST	0.1K	0.4K	0.6K	0.7K	0.8K	0.2K	0.6K	0.6K	1.1K
MEDIA	2.8K	7.5K	8.0K	8.0K	8.6K	3.8K	4.9K	9.1K	11.3K
MYTH	0.8K	0.7K	1.6K	2.0K	1.8K	1.3K	1.3K	1.6K	0.6K
PLANT	7.8K	9.5K	7.6K	4.4K	5.1K	6.3K	6.6K	9.2K	4.8K
TIME	3.3K	3.2K	45.3K	27.4K	71.2K	31.0K	44.1K	48.6K	22.8K
VEHI	0.5K	0.5K	0.3K	0.6K	0.6K	0.4K	0.7K	0.3K	0.5K
O (OTHER)	2.4M	3.1M	3.8M	3.8M	4.2M	2.7M	2.5M	3.4M	2.0M
Sentences	156.8K	164.1K	173.2K	176.2K	181.9K	171.7K	195.0K	177.6K	129.0K
Tokens	2.7M	3.6M	4.3M	4.3M	4.7M	3.0M	3.0M	3.9M	2.3M
Avg. sentence length	17.7	21.7	24.6	24.5	25.7	17.7	15.3	21.9	16.7
Avg. NEs per sentence	1.4	1.5	1.6	1.6	1.7	1.5	1.5	1.7	1.3
Table 3: Statistics concerning the data produced.
predictions of a Transformer-based neural classifier (mBERT + Bi-LSTM + CRF, Mueller et al., 2020). Unfortunately, this strategy requires pre-existing annotated data in the same set of languages and with the same NER tags in order to train the NER classifier, and these are not available in our case. To cope with this issue, we employ the same Transformer-based architecture but drop the requirement of pre-existing training data by introducing a general, straightforward iterative strategy to jointly improve both the performance of the neural model and the quality of the data produced. Algorithm 1 illustrates the procedure. Essentially, it starts by taking a set  $A$  of  $n$  articles from  $W$  (line 1), and annotating them with the steps described in Sections 4.1-4.3 (line 2). Then, it uses the obtained annotated dataset  $D$  to train a neural model  $M_D$  (line 3). Here the iterative step begins: i) another (disjoint) set  $\hat{A}$  of  $n$  articles is taken from  $W$  (line 5), ii) a larger set  $A$  is obtained by concatenating the new set  $\hat{A}$  with the previous set  $A$  (line 6), and consequently iii) a larger dataset  $D$  is obtained, but this time using also the model  $M_D$  to validate the annotations produced by the steps in Sections 4.1-4.3 (line 7), and finally iv) a better model  $M_D$  is trained on the new (larger and more accurate) dataset  $D$  (line 8). Steps (i)-(iv) are repeated  $t$  times, where  $t$  is used to regulate the size and the quality of the final dataset  $D$ .
In the annotate  $(A, M_D)$  function, the neural model  $M_D$  is used to refine the annotations and
reduce noise. Specifically, if the NER class of an entity predicted by the neural model is different from the one assigned by the knowledge-based approach (Section 4.3), the corresponding sentence is discarded.