MultiCoNER v2: a Large Multilingual dataset for Fine-grained and Noisy Named Entity Recognition
Authors: Besnik Fetahu
Zhiyu Chen
Sudipta Kar
Oleg Rokhlenko
Shervin Malmasi
Venue: EMNLP 2023 Findings (2023)
Keywords: ner
multilingual ner
fine-grained ner
noisy ner

Abstract: We present MULTICONER V2, a dataset for fine-grained Named Entity Recognition covering 33 entity classes across 12 languages, in both monolingual and multilingual settings. This dataset aims to tackle the following practical challenges in NER: (i) effective handling of fine-grained classes that include complex entities like movie titles, and (ii) performance degradation due to noise generated from typing mistakes or OCR errors. The dataset is compiled from open resources like Wikipedia and Wikidata, and is publicly available. Evaluation based on the XLM-RoBERTa baseline highlights the unique challenges posed by MULTICONER V2: (i) the fine-grained taxonomy is challenging, where the scores are low with macro-F1=0.63 (across all languages), and (ii) the corruption strategy significantly impairs performance, with entity corruption resulting in 9% lower performance relative to non-entity corruptions across all languages. This highlights the greater impact of entity noise in contrast to context noise.

MultiCoNER v2: Large Multilingual dataset for Fine-grained and Noisy Named Entity Recognition
Besnik Fetahu
Zhiyu Chen
Sudipta Kar Oleg Rokhlenko
Shervin Malmasi
Amazon.com, Inc. Seattle, WA, USA
(besnikf, zhiyuche, sudipkar, olegro, malmasi i } amaaon. com
Abstract
English voico of genoral wade justica unlimbed
Chinese
We present MULTICONER v2, dataset for fine-grained Named Entity Recognition cov- ering 33 entity classes across 12 languages, in both monolingual and multilingual settings. This dataset aims to tackle the following prac- tical challenges in NER: (i) effective handling of fine-grained classes that include complex entities like movie titles, and (ii) performance degradation due to noise generated from typ- ing mistakes or OCR errors.
Farsi
German song
vorgestell
barrace
escrll por vicentl blasco ibatter:
Swedish Jomn hombion amer kansk ker basiel semmmitt
Actist
Hindl Ale Rati serrel go
Figure 1: Example sentences from MULTICONER v2.
The dataset is compiled from open resources like Wikipedia and Wikidata, and is publicly available.' Evaluation based on the XLM- RoBERTa baseline highlights the unique chal- lenges posed by MULTICONER v2: (i) the fine-grained taxonomy is challenging, where the scores are low with macro-F1=0.63 (across all languages), and (ii) the corruption strat- egy significantly impairs performance, with entity corruption resulting in 9% lower perfor- mance relative non-entity corruptions across all languages. This highlights the greater im- pact of entity noise in contrast to context noise.
biguation steps are needed in many practical sys- tems. Furthermore, models trained on datasets such as CoNLL03 perform significantly worse on unseen entitics or noisy texts (Meng et al., 2021).
1.1 Real-world Challenges in NER
Outside the news domain used in datasets like CoNLLO3, there are many challenges for NER. We categorize the challenges (cf. Table 5 in Ap- pendix A) typically encountered in NER by sev- eral dimensions: (i) available context around en- tities, (ii) named entity surface form complex- ity, (iii) frequency distribution of named entity types, (iv) multilinguality, (v) fine-grained entity types, and (vi) noisy text containing typing errors. MULTICONER v2 represents all of these chal- lenges, with a focus on (v) and (vi).
1 Introduction
Named Entity Recognition (NER) is a core task in Natural Language Processing that involves identi- fying entities and recognizing their type (e.g., per- son or location). Recently, Transformer-bart based NER approaches have achieved new state-of-the- art (SOTA) results well-known benchmark datasets like CoNLLO3 and OntoNotes (Devlin al., 2019). Despite these strong results, as shown by Malmasi et al. (2022a), there remain a number of practical challenges that are not well represented by existing datasets, including low context, diverse domains, missing casing informa- tion, and text with grammatical and typographical errors. Furthermore, wide use of coarse- graiied taxonomies often contribute lower utility of NER systems, additional fine-grained disam-
Fine-Grained Entity Types Most datasets use coarse types (c.g., PERSON, LOCATION in CoNLL03), with each type subsuming a wide range of entities with different properties. For in- stance, for LOCATION, different entities can ap- pear different contexts and have diverging sur- face form token distributions, which can lead to challenges in terms of NER, c.g., airports as part of LOCATION are often named after notable per- sons, thus for sentences with low context, NER models can confuse them with PERSON entities.
We introduce a fine-grained named entity taxon- omy with 33 classes, where coarse grained types are further broken down into fine-grained types.
pat / beg my wnreeaatee awd tr mooner Lpu / /hougging ang Cave. daa Mull CoNER mul Loones__2 the fance
fru

---

Additionally, we introduce types from the medical domain, presenting a further challenge for NER. Fine-grained NER poses challenges like distin- guishing between different fine-grained types of a coarse type. For example, correctly identifying fine-grained types (c.g., SCIENTIST, ARTIST) of a PERSON entity requires effective representation of the context and often external world knowledge. Since two instances belonging to different types may have the same name, the context needed to correctly disambiguate the types.
2 MULTICONER v2 Dataset Overview
MULTICONER v2 was designed to address the challenges described in $1.1. It contains 12 lan- guages, including multilingual subsets, with to- tal of 2.3M instances with 32M tokens and a to- tal of 2.2M entities (IM unique). While focused on different challenges, we adopted the data con- struction process described of the original MUL- TICONER (v1) corpus Malmasi et al. (2022a) and details about the process are provided in Ap- pendix B. While both the vI and v2 datasets fo- cus on complex entities, MUL.ICONER v2 intro- duces fine- taxonomy, adds realistic noise as well as 5 new languages.
Noisy Text Existing NER datasets mostly con- sist of well-formed sentences. This creates a gap in terms of training and evaluation, and datasets assume that models are applied only on well- formed content. However, NER models are typ- ically used in web settings, exposed to user- created content. We present a noisy test set in MULTICONER v2, representing typing errors af- fected by keyboard layouts. This noise, impacting named entity and context tokens, presents chal- lenges for accurately identifying entity spans.
Next we describe the fine-grained NER taxon- omy construction and noise generation processes.
2.1 NER Taxonomy
MULTICONER v2 builds on top of the WNUT 2017 (Derczynski et al., 2017) taxonomy entity types, with the difference that it groups GROUP and CORPORATION types into one,² and intro- duces the MEDICAL type. Next, we divide coarse classes into fine- -ne-grained sub-types. Table 1 shows the 33 fine-grained classes, grouped across 6 coarse grained types. Our taxonomy is inspired by DBpedia's ontology (Auer et al., 2007), where subset of types are chosen fine-grained types that have support in encyclopedic resources.
Contributions Our contributions through this work can be summarized follows:
1. We created taxonomy of 33 fine-grained NER classes across six coarse categorics (PERSON, LOCATION, GROUP, PRODUCT, CREATIVEWORK, and MEDICAL). We use the taxonomy to develop a dataset for 12 lan- guages based on this taxonomy. Our exper- iments show that NER models have large performance gap 14% macro-FI) in identi- fying fine- ane-iedi classes versus coarse ones.
This taxonomy allows us capture fine-grained differences entity types, where the entity con- text plays a crucial role in distinguishing the different types, as well as cases where external knowledge can help where context may be insuffi- cient PRIVATECOR vs. PUBLICCORP).
2.2 Languages and Subsets
2. We implement techniques to mimic web text errors, such as typos, to enhance and test NER model resilience to noise.
MULrICONER v2 includes 12 languages:
Bangla (BN), Chinese (2H), English (EN), Farsi (FA), French (FR), German (DE),
Hindi (HI), Italian (IT), Portuguese (PT), Spanish (ES), Swedish (SV), Ukrainian (UK). The chosen languages span diverse typology, writing systems, and range from well-resourced (e.g., EN) to low-resourced (e.g., FA, BN).
3. Experimentally we show that noise can sig- nificantly impact NER performance. On av- erage, across all languages, the macro-Fi score for corrupted entities is approximately 9% lower than that for corrupted context, em- phasizing the greater influence of entity noise as opposed to context noise.
Monolingual Subsets Each of the 12 languages has their own subset with data from all domains, consisting of total of 2.3M instances.
MULTICONER v2 has been used as the main dataset for the MULTICONER v2 shared task (Fe- tahu et al., 2023) at SemEval-2023.
"The original definition is amhiguous in the difference be- tween CORPORATION and GROUP entity.

---

FER (Person)
LOC (Location)
CRF (Group)
PROD (Product)
CN (Creative Nork) MED (Medical)
ARTIST
FACILITY
CLUTHING
ARTWORK
ANNTOMICALSTRUCCI
ATHLETE
HUMANSETTLEMENT
CARMANURACTURER
DRINK
MUSICALWORK
DISEASE
STATION
MUSICALURP
FOOD
SOFTWARE
PROCEDURE
POLITICIAN
OTHERLOC
ORG
VEHICLE
WORUALWOKK
MEDICATTION/AACCINN
SCIENTIST
PRIVATECORP PUBLICCORP
OTHERPROD
WRITTEN WORK
SYMPTOM
SPORTSMANAGER
OTICRPER
SPORTSGRP
Table I: MULTICONER v2 fine-grained NER taxonomy. There are 33 fine-grained classes, grouped across 6 coarse types.
Multilingual Subset This subset is designed for evaluation of multilingual models, and should be used under the assumption that the language for each sentence is unknown. The training and de-
Corruption Strategy
Prob.
Adds neighbor letter keyboard next to 0.1 random token letter.
Replaces random word letter with its 0.2 keyboard-neighbor letter. Replaces random word letter with another 0.2 visually similar one.
velopment set from the monolingual splits are merged, while, for the test set, we randomly se- lect at most 35,000 samples resulting in a total of 358,668 instances. The test set consists of subset of samples that incorporate simulated noise.
Skips random word letter. Swaps random consecutive word letters. Repeats random letter.
0.2
Table 2:
Corruption strategies and the probability with which they are applicd.
2.3 Noise Generation
To emulate realistic noisy scenarios for NER, we developed different corruption strategies that re- flect typing errors by users. We considered two main strategies: (i) replace characters in a token with neighboring characters (mimicking typing rors) on given keyboard layout for a target lan- guage. and (ii) replace characters with visually similar characters (mimicking incorrectly visually recognized characters, e.g., OCR).
from a truncated Poisson distribution (A = and < 3). Since the minimum of 5 can be zero, we use 8' = d 1 as the final corruption count.
Corruptions can apply to entity or non-entity (context) tokens. For ZH, we corrupt a context or entity character with 15% and 85% probabil- ity respectively. Using the corruption dictionary, we replace the original character at random with a visually or phonologically similar character from identified candidates. For the other languages, the probability of corrupting entity tokens is set 80%. This is lower given that each letter in entities can be corrupted and the number of candidates in entities more than that in ZH. On a word cho- sen for corruption, we perform a corruption action based on probabilities shown in Table 2. Examples of corrupted sentences are shown in Figure 2.
We applied the noise only on the test set for a subset of languages (EN, ZH, IT, ES. FR, PT and SV). The noise can corrupt context tokens and entity tokens. For ZH, the corruption strate- gies are applied the character level. We fol- lowed Wang et al. (2018)'s approach that auto- matically extracts visually similar characters using OCR tools and phonologically similar characters using ASR tools. We used their constructed dic- tionary* to match targeting characters.
For the other six languages, we developed word-level corruption strategies based common typing mistakes, utilizing language specific key- board layouts.5 Table 2 summarizes the designed corruptions and their probabilities of occurrence.
EN
31o species was desoribed by cielich brandis after the forester boun (oouddloo).
ES
sambién conocado por probado ampilmmente distrbbuddo jclsarbu bu doj victor hugo.
FR
trovvaiont tamiral dument cluvillo
vaggla soggloman moogaommy poi negt
gustemata.
Noise Sampling We introduce synthetic noise into 30% of the test set. For each corrupted sen- tence, the number of corrupted tokens d is sampled
PT obe osmo um subredtule to (subviilule) (e senivoole cachaeg.
SV
med seactio och
"To emulate realistic scenario, where models trained clean text, but are exposed directly to user input. aecpes http://ww......tt..../.tooooooe. Goncratioo amm/ddmmy//utttnaa.....
Figure 2: Examples of corrupted sentences. The red tokens are the corrupted versions of the blue tokens.
*We extended the keyboard layouts to include 7 lan- /typo.
guages: htgpe///gitnne coa/

---

3
NER Model Performance
gap of up 34% in Fl scores, seen with CREATIVEWORK and PERSON, indicating the challenges of introducing fine-grained types not seen at the coarse level (Malmasi et al., 2022b).
We assess if MULTICONER v2 is challenging (ef. 11.1) by training and testing with XLM-RoBERTa (Conneau et al., 2020, XLMR).
The gap can be attributed to two primary com- ponents: (i) the incorrect classification of entities under inappropriate sub-categories broad cate- gories, and (ii) recall problems where certain enti- ties are missed (assigned the O tag).
Metries We measure the model performance us- ing Precision (P), Recall (R), and FI, where for Fl we distinguish between micro/macro averages.
3.1 Results
Misclassifications Figure 3 shows the distribu- tion of predicted coarse tags for MULTI. Most of the misclassifications are for the GROUP class, which is often misclassified with LOCATION PERSON. Yet, this does not reveal how the models handle the specific classes within coarse type. To understand the challenges presented by our taxonomy for NER models, we further in- spect misclassifications within the coarse type GROUP in Figure 7 of Appendix D. There are several types that are misclassified, e.g.. PUBLICCORP as ORG, PRIVATECORP with ORG PUBLICCORP. On the other hand, MUSICALGRP is often misclassified with types from other coarse types, such as ARTIST or VISUALWORK. This highlights some of the is- sues and challenges that NER models face when exposed to more fine-grained types.
Table 3 shows the results obtained for the XLMR baseline. The results show the micro-F1 scores achieved on the individual NER coarse classes (See Appendix C for fine-graine fine-grained performance). Across all subsets, XLMR achieves the highest performance of
for HI, and lowest micro-F1=0.61 for EN, ES and FA. The reason for the higher Fl scores on HI (or BN second highest score) is due to the smaller test set size, since the data is translated and is more scarce. For the rest, the Fl scores are typically lower, this is is mainly associated with the larger test sizes.
Lang.
Macra-F1
Micro-F1
Lang.
Macro-F1
MLCrO-FL
BK
0.68
0.74
II
0.58
0.63
ER
0.53
0.61
57
0.57
0.70
0.53
0.54
0.65
PA
F6.
0.56
0.64
Recall Issues Figure 3 shows the rate of enti- ties marked with the O tag. Consistent recall is- sues affect the MEDICAL, CREATIVEWORK, and PRODUCT classes, often with more than 20% of entities being missed. Across languages the recall can vary, with even higher portions of missed en- tities (c.g., FA, ZH, EN with more than 30+% MEDICAL entities missed, cf. Figure 6).
Table 3: XLMR performance on MULTICONER v2.
CW
80
LoC
84.5
2.8
Ground Truth
60
PER
0.7
0.6
92.0
1.2
0.1
0.1
5.3
GRP
1.6
3.7
2.8
78.1
0.9
0.1
12.9
40
3.3 Impact of Corruption Strategies
MED PROD
Figure 4 shows the impact of noise level on the macro-FI score, where we strong negative correlation between noise level and performance (Spearman's 's correlation across different languages is أ = -0.998). For ZH, the impact is higher than for the rest languages and the macro-F1 decreases faster: with 50% of test data corrupted, the score decreases upproximately by 9%, while for the rest it drops only between 4% and 5.5%.
1.3
2.0
1.1
3.7
65.3
1.3
25.2
20
0.3
0.6
0.4
0.5
1.9
72.3
24.0
CW
LOC
PER
GRP PROD
MED
Prediction
Figure 3: Coarse grained errors on the MULTI test set.
3.2 Impact of Fine-Grained Taxonomy
Table 4 shows the impact of corruption strate- gies on context and entity tokens, targeting exclu- sively the respective tokens. For both variants, the number of corruptions per sampled with À =
Performance varies greatly across fine-grained types within a coarse type (Table 10). Using MULTI as a representative baseline, we note a

---

Performance Corrupted Data
future work. Expansion to other more challeng- ing domains, such as search queries (Fetahu et al., 2021), can help explore fine-grained NER within even shorter inputs.
0.59
Nacro-FF M Scores
0.57
Finally, evaluating the performance of this data using zero-shot and few-shot cross-lingual models (Fetahu et al., 2022) even unsupervised NER models (lovine al., 2022) can help better un- derstand the role of data quantity building effec- tive NER systems.
0.55
0.53
10% 15% 30% 35%
50%
Percentage of Noise in set
Figure 4: Noise impact on performance. X-axis shows the rate of noise, and Y-axis shows the macro-F1 score. Our final test applies noise 30% of sentences.
References
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In The Semantic Web, 6th International Se- mantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007 + ASWC 2007, Busan, Ko- rea, November 11-15, 2007, volume 4825 of Lec- ture Notes in Computer Science, pages 722-735. Springer.
Corruption Target
KN
ESS
FR
IT
PT
SV
UK
ZH
Context
0.48
0.51
0.50
0.54
0.51
0.55
0.54
0.55
Eaiity
0.22
0,42
045
0,46
Table 4: macro-F1 scores when all the test data is corrupted context entity tokens.
and 8 < 4 papeer-arammters.. Results show that entity token corruption has notably detrimental effect on performance, in contrast context token corruption. On average across all languages, the macro-F1 score for corrupted entities is approx- imately 9% lower than the performance on cor- rupted context, emphasizing the higher impact of entity noise compared to context noise. Finally, Table 11 shows the baseline's performance break- down the clean and noisy subsets of the test set.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle moyer, and Veselin Stoyamov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 8440 8451. Associ- ation Computational Linguistics.
Leon Derezynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task novel and emerging entity recogni- tion. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147.
4 Conclusions and Future Work
We presented MULTICONER v2, a new large- scale dataset that represents a number of current challenges in NER. The results from XLMR indi- cate that our dataset is difficult, with macro-F1 score of only 0.63 the MULTI test set.
Jacob Devlin, Ming-Wes Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In NAACL- HLT((1) Association for Com- putational Linguistics.
The results presented illustrate that the MULTICONER v2, with its detailed NER tax- onomy and noisy test sets, poses substantial challenges to large pre-trained language models' performance. It is our hope that this resource will help further research for building better NER systems. This dataset can serve a benchmark for evaluating NER methods that infuse external entity knowledge, as well assessing NER model robustness on noisy scenarios.
Besnik Fetahu, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2021. Gazetteer enhanced named entity recognition for code-mixed web queries. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Infor- mation Retrieval, pages 1677-1681.
Besnik Fetahu, Anjie Fang. Oleg Rokhlenko, and Shervin Malmasi. 2022. Dynamic gazetteer inte- gration in multilingual models for trossllingual and cross-domain named entity recognition. In Proceed- ings of the 2022 Conference of the North Ameri- can Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2777-2790.
We envision several directions for future work.
The extension of MULTICONER v2 to additional languages is the most straightforward direction for

---

Besnik Fetahu, Sudipta Kar, Zhiyu Chen, Oleg Rokhlenko, and Shervin Malmasi. 2023. SemEval- 2023 Task 2: Fine-grained Multilingual Named En- tity Recognition (MultiCoNER 2). In Proceedings of the 17th International Workshop on Semantic Evaluation 2023). Association for Com- putational Linguistics.
Andrea lovine, Anjie Fang. Besnik Fetahu, Oleg Rokhlenko, and Shervin Malmasi. 2022. Cyclener: unsupervised training approach for named entity recognition. In Proceedings of the ACM Web Con- ference 2022, pages 29 16-2924.
Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022a. MultiCoNER: A large-scale multilingual dataset for complex named entity recognition. In Proceedings of the 29th In- ternational Conference Computational Linguis- tics, pages
Gyeongju, Republic of Ko- rea. International Committee on Computational Lin- guistics.
Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022b. SemEval-2022 task II: Multilingual complex named entity recog- nition (MultiCoNERR) In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1412-1437, Seattle, United States. Association for Computational Linguistics.
Stephen Mayhew, Tatiana Tsygankova, and Dan Roth. 2019. ner and pos when nothing is capitalized. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP) pages 6256- 6261, Hong Kong, China. Association for Computa- tional Linguistics.
Tao Meng, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2021. GEMNET: effective gated gazetteer representations for recognizing complex entities in low-context input. In Proceedings of the 2027 Con- ference of the North American Chapter of the As- sociation for Computational Linguisties: Human Language Technologies. NAACL-HLT 2021, Online, June 6-11, 2021, pages 1499-1512. Association for Computational Linguistics.
Erik Tjong Kim Sang and Fien De Meulder. 2003. In- troduction to conll-2003 shared task: Language- independent named entity recognition. In Proceed- ings the Seventh Conference Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Dingmin Wang. Yan Song, Jing Li, Jialong Han, and Haisong Zhang. 2018. A hybrid approach to auto- matie corpus generation for chinese spelling check. Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2517-2527.

---

represent contributions and enrichment of the cur- rent NER data landscape, represent common types encountered in NER research. We generate the en- tity to type association at the ine-grained level, given that the association at the coarse level can be inferred automatically.
Appendix
A NER Challenges & Motivation
Table 5 lists the key challenges that are introduced in MULTICONER v2 for NER model. The chal- lenges range from low context sentences, complex entities with additionally fine-grained NER taxon- omy, syntactic issues such as lowercasing as well as noisy context and entity tokens.
For each of the fine fine-grained types we manually identify specifie Wikidata types, from where we then query the associated entities. Wikidata types and entity type associations are done by the Wiki- data community, and thus, often there are redun- dant entity types (with similar semantics). From the initially identified types and their entities, we expand our set of Wikidata types that are associ- ated to a fine-grained type in our taxonomy, by ad- ditionally taking into account other Wikidata types that the entities are associated with. From the identified Wikidata types for a tie-ggraned type in our taxonomy, we traverse the entire Wikidata type graph, and obtain all the entities that are as- sociated with those types and all of their children.
B Dataset Construction
This section provides a detailed description of the methods used to generate our dataset. Namely, how sentences are selected from Wikipedia, and additionally how the sentences are tagged with the corresponding named entities and how the under- lying fine-grained taxonomy is constructed. Addi- tionally, the different data splits are described, as well as the license of our dataset.
Table 6 shows the detailed stats about our pro- posed MULTICONER v2 dataset.
Finally, this resulted in 16M Wikidata entities, whose distribution across the different coarse and fine-grained types is shown in Table 7 and 8.
B.1 Fine-Grained NER Taxonomy
Quality Filtering: Finally, once we have ob- tained all the possible Wikidata entities that are associated with the fine-grained types in NER taxonomy, we need ensure that the associations are not ambiguous.
We use our fine-grained NER taxonomy, namely, the entities associated with each type as part of the automated data generation process. To generate entity associations to our fine-grained taxonomy we make use of Wikidata. The entity type graph in Wikidata is noisy, containing cycles, and often types are associated with different parent types, thus, making it challenging to use the data is for generating fine-grained and clean taxonomies for NER. We carry out series of steps that we explain in the following.
We tackle the problem ambiguity, namely Wiki- data entities being associated with different types, causing them to be in different NER classes at the same time. To resolve such ambiguous cases, we compute the distribution of entities that are asso- ciated with a set of fine-grained NER type, and based on a small sample analysis, manually deter- mine with which NER type they should be associ- ated.
Type Cycles Wikidata type graph contains cy- cles, which are not suitable for taxonomy gener- ation. We first break the cycles through a depth- first search approach, by following the children of a given type, and whenever given type has chil- dren that are already visited (encountered during the DFS), those edges are cut out.
B.2 Sentence Extraction
MULTICONER v2 is constructed from encyclo- pedic resources such as Wikipedia, with focus on sentences with low context, sampled using heuris- tics that identify sentences that represent the NER challenges we target in Table 5. Figure 5 shows the steps from our data construction workflow, which for a given language it extracts sentences from the corresponding Wikipedia locale, and applies se- ries of steps to generate the NER data. This pro- cess is performed for the following languages: FA, FR, EN, ES, IT, PT, SV. For the rest languages
Entity to Fine-Grained Type Association: Our taxonomy consists of 6 coarse types, PERSON, LOCATION, GROUP, MEDICAL, PRODUCT, CREATIVEWORK, and 33 of fine-grained types. The coarse types represent classes that we aim to have in our taxonomy, and with the exception of MEDICAL and PRODUCT, which at the same time

---

Challenge Fine-grained Entities
Description
The entity type can be different based on the context. For example, creative work entity "Bacry Petter the Sorcerer's Stone" could be book a film, depending on the context. Gazetteer based models would not work for typos (e.g., "aony xperla" + "sony xpris") or spelling enrors (c.g.. "ford cargo" "sord marg>") in entities, degrading significantly their performance.
Noisy NER
Ambiguous Entities and Contexts
Some NLs are ambiguous: they are not always entities, e.g., "Inside Out". "Among Us", and "Bonanza" may refer to NEs (a movie, video game. and TV show) in some contexts, but not in others. Such NEs often resemble regular syntactic constituents.
Surface Features
features are large drivers of success in NER (Mayhew et al., 2019), but short imputs (ASR, queries) often lack such surface features. An uncased evaluation is needed to assess model performance.
Capitalization/pnnttuee
Table 5: Challenges addressed by MULTICONER v2.
Class
Split
EN
DE
FA
FR
ES
UK
SV
BI
BN
28
XT
PT
Multi
train
9,294
5,508
8,006
9,295
8,360
6,441
7,695
3,609
3,778
4,862
10,387
8,241
85,476
PER
413
445
194
239
4,487
137,681
11,299
115,868
141,401
125,379
96,864
111,157
5,736
6,935
9.095
160,598
120,413
180,880
trinn
4,084
2,466
3,661
5,438
3,606
2,907
3,714
1,646
1,981
2,264
5,048
3,839
40,654
CW
dev
21.5
127
184
268
183
146
200
90
103
112
267
206
2,101
test
62,126
4,777
53,034
84,952
55,459
43,291
54,806
2,804
3,640
4,369
79,873
58,245
87,030
train
4,224
2,815
3,209
3,745
3,632
3,204
3,459
2,273
2,227
2,696
3,416
3,788
38,688
GRP
180
195
151
194
145
2,093
60,026
4,418
38,807
52,987
50,259
39,709
46,929
3,897
3,651
4,715
46,271
48,994
73,226
train
4,353
2,269
5,086
4,723
4,651
5,458
7,176
2,487
2,457
2,470
4,446
4.794
50,370
LOC
dev
197
117
267
242
230
294
370
133
127
129
248
250
2,604
test
67,901
5,306
70,907
73,373
72,996
84,643
111,879
7,172
7,375
6,170
68,564
70,923
117,257
train 1,935
1,571
2,049
1,946
1,989
2,258
1,989
1,420
1,384
1,529
1,770
1,927
21,767
PROD
dev
107
100
112
86
1.124
27,580
1,643
18,212
28,274
28,469
30,071
22,686
1,611
1,493
1,869
22,887
21,115
35,545
train 1,559
1,322
1,651
1,230
1,669
1,688
1,381
1,435
1,396
1,407
1,376
1,850
17,964
MED
dev
76
62
85
64
81
86
70
70
63
75
76
88
896
test 22,491
1,434
15,287
17,208
23.812
20,796
13,702
1,979
1,919
1,781
19,029
21,062
29.553
train 16,778
9,785
16,321
16,548
16,453
16,429
16,363
9,632
9,708
9,759
16,579
16,469
170,824
dev
855
857
851
856
507
506
8,895
249,980
20,145
219,168
249,786
246,900
238,296
231,190
18,399
19,859
20,265
247,881
229,490
358.668
Table 6: MULTICONER v2 dataset statistics for the different languages for the Train/Dew/Test splits. For cach NER class we show the total number of entity instances per class on the different data splits. The bottom three rows show the total number of sentences for each language.
Coarse Class #Entities
ratio
can be mapped into their corresponding Wiki- data entity equivalent. In this way we ensure that the entities in sentence are marked with high quality to their corresponding fine-grained types (cf. Table 8).
PERSON
9,586.716
57%
CREATIVEWORK
3,146,607
18.7%
LOCATION
3,010.448
17.9%
GROUP
607,420
3.5%
PRODUCT
396.611
2.4%
MEDICAL
29.968
0.18%
2. Sentence Filtering:
We remove sentences that are shorter than 28 characters and longer than 128 characters. In this way, we ensure that the sentences contain complete clauses," and furthermore, by filtering out longer sentences (more than 128 characaters) we ensure that the remaining sentences have low context, making NER more challenging. Finally, we addition- ally filter out sentences that contain capitalized nouns (proper nouns), which are not interlinked
Table 7: Gazetteer stats at the coarse named entity type level.
(BN, ZE, DE, HI), we apply machine translation to obtain the data, described below.
In the following we explain the required steps to generate the final output of sentences tagged with the corresponding entity types.
Sentence Extraction: From the cleaned Wikipedia pages, we extract sentences, which contain hyperlinks that point to entities that
*i.c,, sentences shorter than 28 characters may contain only entity names, and as such are not suitable for NER

---

human annotators in terms of the translated sen- Coarse Clas9
Ploe-Grained Class
1t sea
ratloo
BN, ZH, and is very high, with over 90% translation accuracy (i.c., accuracy as measured by
OTRERPER
4,345,007
25,8955
SCIENTIST
1,901,338
11.333%
ARTIST
1,392,664
8.301%
PERSON
CLERIC
134,078
tence retaining the semantic meaning and as well 0.799%
SPORTSMANAGER
48,741
0.291%
ATHLETE
1,003,554
5.881%
have a correct syntactic structure in the target lan- guage).
Poi IILIIICIAN
762,544
4.145%
2,470,918
14.727%
OTHERLOC
256,644
1.53%
LOCATION
HUMANSETTLEMENT
86,773
0.517%
B.4
Data Splits
ORG
392,390
2.339%
To ensure that obtained NER results on this dataset reproducible, create three predefined sets for training, development and testing. The enti-
MUSICALGRP
106,543
0.636%
SPORTSGRP
83,902
0.5%
GROUP
PUBLICCORP
12,570
0.739
PRIVATECORP
1,888
0011%
CARMANUFACTUURRR
9,528
0087%
AEROSRACEMLNUUFACTUREER
499
0.00%%
ties each class are distributed following a root
OTHERPROD
215,848
1.275%
square normalization, which takes into account the actual distribution of a given entity class. Ad- ditionally we set a minimum number of entities from class that need to appear in each split. This combination allows not to heavily bias the data splits towards the more popular classes (e.g. WRITTENWORKE WORK). Table shows detailed statis- tics for each of the 13 subtasks and data splits.
VERICLE
144,555
0862%
PRODUCT
FOOD
17,791
0.106%
CLOTHING
11,804
0.09%%
DRINK
8,813
0.53%
WHITHNWODKK
19366.177
8.143%
ARTWORK
575,665
3431%
CREATIVEWORK
540,773
3.223%
MUSICALWORK
539,878
3.218%
0.519%
DESPASE:
14,627
0.87%
ATTICCATOO/VACCIIN
0.35%
MEDICAL
PROICALLROCCEDURE
4,285
0.026%
ANTTOXICALSTRUCTURE
4,186
0.025%
SYMPTOM
0.005%
Training Data For the training data split, we limit the size to approx. 16k sentences. The number of instances was chosen to com- parable to well-known NER datasets such as CoNLL03 (Sang and De Meulder, 2003). Note that in the case of the Multi subset, the training split contains all the instances from the individual language splits.
Table 8: Gazetteer stats the coarse named entity type level.
to entities. This done for languages that fol- low capitalization rules for proper nouns (e.g. EN and ES). This presents an important step, since if a proper noun is not interlinked to an entity, however it represents one, this will ar- tificially inflate false positive scores for NER models.
Development Data We randomly sample around ~800 instances, & a reasonable amount of data for assessing model generalizability.
Test Data Finally, the testing set represents the remaining instances that are not part of the training or development set. To avoid exceedingly large test sets, we limit the number of instances in the test set to be around at most 250k sentences (cf. Table 6). The only exception is for the Multi. which
B.3 Sentence Translation
For the languages, for which we cannot automat- ically extract sentences and their corresponding tagged entities (e.g. DE uses capitalization for all nouns, thus, we cannot accurately distinguish nouns against proper nouns), we apply automatic translation to generate two portions of our data. For four languages (BN, DE, HI, 2H), Wikipedia sentences are translated from the EN data.
generated from the language specific test splits, and was downsampled to contain at most 35k from each monolingual test set, result- ing in a total of 358k instances. The larger test set sizes are ideal to assess the generalizability and ro- bustness of models on unseen entities.
We use the Google Translation API to per- form our translations. The input texts may contain known entity spans or slots.
To prevent the spans from being translated, leveraged the notranslate attribute to mark them and prevent from being translated. The trans- lation quality in the different languages such as
B.5 License, Availability, and File Format
The dataset is released under a CC BY-SA 4.0 li- cense, which allows adapting the data. Details
"With the exception of the translated languages, DE, HT, BN, and ZH, where the data is scaree.
"https://cl oud. google.coo.ooooo.oooooooo com/transatt

---

[8]
Wikppdia
Wikipecia Parsing
Sentences
Interlink Parsing
Servencee Filtering
Dota Sampling
Dump
Wikidata Dump
NER Data
Figure 5: An overview of the different steps involved in extracting the MULTICONER v2 data from Wikipedia dumps, similar to MULTICONER.
about the license available the Creative Commons website. 9
grained classes and languages. We note high di- vergence in terms of performance for the different fine-grained types within coarse type.
The data distributed using the commonly used BIO tagging scheme in CoNLLO3 format (Sang and De Meulder, 2003). The complete dataset will be available for download.
Performance on Noise and Clean Test Subsets: From Table 1I, we can see that the simulated noise can have huge impact on the performance of the baseline model. On average, the Macro-F1 score noisy subset is about 10% lower than that the clean subset. this section, we study the impact of different corruption strategies, and amount of corruption on the test set. Note that, differently from the fixed amount of noise of 30% applied on the test set, here we assess the range of noise with up to 50% of the test data.
B.6 Noise Generation
Original Character
Visually Similar Ones
Phonetically Similar Ones
T.453
5,
Table 9: Examples of visually and phonetically similar zt characters.
Finally, as part of the data construction, we gen- erate a subset of corrupted test set (30%) that adds noise to either the context or entity tokens as de- scribed in in $2.3. The purpose of the noisy test set is to assess the robustness of NER models, trained clean data, in spotting correctly the named enti- ties that may contain either context or entity noise. The noise mimics typing noise or noise that may be attributed to OCR, where characters are misrce- ognized with visually similar characters.
D Fine Grained NER Error Analysis
Here we show the errors that the XLMR base- line makes at the fine-grained type level. Fig- ure 7 shows the misclassification of the base- line approach. Here, we notice that within the GROUP type there are several fine-grained types that are often confused with other types within GROUP, e.g. PUBLICCORP often misclassified as ORG, or PRIVATECORP often misclassified with ORG or PUBLICCORP. On the other hand, MUSICALGRP often misclassified with other fine-grained types from other coarse types, such as ARTIST or VISUALWORK.
For all the languages the noise is applied at the token level, where random characters within a to- ken replaced with either character that comes from typing errors (neighboring characters on given keyboard layout for specific language) or visually similar characters. For ZH is applied at character level, this is due to the language struc- ture. Table 9 shows examples of visually similar and phonetically similar characters in ZH. Figure shows examples of corrupted sentences from seven different languages.
The error analysis highlights some of the issues that NER models face when exposed to more fine- grained types. Furthermore, through this analysis, provide insights the challenges presented by the newly proposed NER taxonomy and areas of improvement in terms of NER performance.
Fine Grained NER Performance
Performance Fine-grained Classes: Ta- ble 10 shows the results for the different fine-
nntrpes / Foreat vecornons ceg/ Loeesea bby 50/4.0

---

DE
EN
105
FR
SV
IT
2111
P7
P7
3N
UK
MULTI
ANATOMICALSTRUCTURE
0.52
0.64
0.60
0.63
0.58
0.47
0.77
0.81
0.67
DISEASE
0.62
0.59
0.65
0.56
0.66
0.63
0.54
0.82
0.85
0.66
0.68
MED
REDDICATIOO/VACCCI
0.81
0.65
0.67
0.65
0.62
0.68
0.64
0.81
0.81
0.74
0.73
MEDICALPROCEDURE
0.77
0.53
0.54
0.55
0.55
0.58
0.58
0.57
0,52
0.79
0.84
0.54
0.63
SYMPTOM
0.55
0.40
0.37
0.49
0.40
0.41
0.41
0.32
0.44
0.79
0.83
0.46
0.55
ARTIST
0.74
0.72
0.69
0.76
0.72
0.81
0.68
0.72
0,72
0.69
0.71
0.69
0,76
ATHLETE
0.75
0.73
0.68
0.70
0.66
0.33
0.72
0.64
0.49
0.71
0.77
0.76
0.74
CLERIC
0.56
0.44
0.50
0.54
0.50
0.61
0.39
0.55
0.42
0.71
0.65
0.54
0.56
POLITICIAN
0.58
0.49
0.54
0.59
0.48
0.52
0.54
0.53
0.64
0.69
0.49
0.57
SCIENTIST
0.41
0.37
0.34
0.42
0.34
0.40
0.46
0,29
0.55
0.42
0.42
SPORTS MANAGRR
0.47
0.48
OTHERPER
0.43
0,47
0.42
0.44
0.45
0.38
0.52
0.54
0.44
FACILITY
0.69
0.58
0.53
0.60
0.67
0.66
0.60
0.57
0.53
0.70
0.70
0.57
0.64
HUMANSETTLEMINT
0.88
0.83
0.79
0.78
0.91
0.82
0.75
0.83
0.75
0.86
0.88
0.85
0.84
OTHERLOC
0.54
0.45
0.38
0.43
0.88
0.43
0.57
0.65
0.30
0.81
0.75
0.55
0.57
STATION
0.78
0.69
0.66
0.71
0.73
0.66
0.82
0.68
0.75
0.87
0.83
0.70
0.73
FOOD
0.67
0.47
0.50
0.49
0.58
0.48
0.57
0.53
0.52
0.65
0.75
0.55
0.57
DRINK
0.66
0.45
0.55
0.50
0.59
0.53
0.43
0.58
0.48
0.82
0.87
0.56
0.60
SOPTWARE
0.77
0.59
0.64
0.70
0.68
0.55
0.70
0.60
0.82
0.82
0.73
0.74
PROD
VEHICLE
0.68
0.48
0.76
0.55
0.54
0.37
0.48
0.36
0.27
0.34
0.84
OTHERPROD
0.62
0.40
0.44
0.49
0,58
0.48
0.52
0.58
0.53
0.58
0.65
0.51
0.57
MUSICALGRP
0.73
0.57
0.63
0.64
0.70
0.76
0.61
0.67
0.60
0.68
0.87
0.74
0.72
PRIVATECORP
0.64
0.19
0.33
0.38
0.14
0.36
0.70
0.00
0,26
0.95
0.91
0.13
0.54
CARMANIFACTURER
0.69
0.51
0.59
0.57
0.55
0.62
0.59
0.52
0.64
0.70
0.76
0.65
0.63
GRP
ALROSPACEMANNTTACUUREE
0.69
0.41
0.32
0.44
0.16
0.21
0.56
0.16
0.62
0.08
0.17
0.26
0.54
PUBLICCORP
0.61
0.49
0.61
0.55
0.53
0.59
0.51
0.70
0.55
0.70
0.80
0.67
0.67
ORG
0.67
0.52
0.54
0.52
0.59
0.52
0.61
0.58
0.55
0.82
0.84
0.62
0.61
SPORTSGRP
0.86
0.73
0.70
0.71
0.78
0.75
0.78
0.74
0.82
0.90
0.94
0.81
0.78
ARTWORK
0.37
0.25
0.50
0.46
0.07
0.10
0.06
0.12
0.22
0.43
VISUALWORK
0.73
0.62
0.79
0.71
0.87
0.63
0.64
0.73
0.71
0.74
0.69
0.77
WRITTENWORE WORK
0.76
0.58
0.59
0.66
0.61
0.56
0.64
0.56
0.48
0.74
0.76
0.60
0.64
MUSICALWORK
0.77
0.67
0.59
0.59
0.67
0.76
0.54
0.67
0.53
0.61
0.71
0.55
0.70
macro-F1 mi cra-F1
0.67
0.53
0.53
0.56
0.57
0.58
0.58
0.54
0.52
0.68
0.72
0.57
0.63
0.71
0.61
0.61
0.64
0.70
0.70
0.63
0.65
0.61
0.74
0.77
0.66
0.69
Table 10: XLM-RoBERTa baseline results the MULTICONER v2 test set measured by the Fl score for the different NER tags.

---

2N
2.5
FR
3V
IT
2H
PT
clean
noisy
clean
poisy
clean
muisy
clean
poisy
clean
nuisy
clean
boisy
clean
moisy
ANATOMIC AATOMICALSTRIICUUREE
0.61
0.55
0.62
0.51
0.54
0.47
0.67
0.57
0.62
0.56
0.70
0.44
0.60
0.53
MGDICATION/VACING
0.70
0.62
0.58
0.68
0.56
0.71
0.57
0.68
0.57
0.67
0.54
0.72
0.61
MEDICAL MEDICALPROCTTDURT PROCTDDUE
0.55
0.49
0.57
0.47
0.58
0.48
0.60
0.42
0.61
0.52
0.65
0.44
0.59
0.53
SYMPTOM
0.43
0.32
0.41
0.26
0.54
0.35
0.45
0.30
0.42
0.37
0.52
0.23
0.35
0.22
0.73
ATHLETE
0.74
0.70
0.69
0.66
0.71
0.66
0.67
0.64
0.84
0.80
0.76
0.63
0.66
0.61
CLERIC
0.46
0.39
0.52
0.47
0.55
0.51
0.51
0.47
0.63
0.58
0.46
0.26
0.58
0.48
PER
POLITICIAN
0.50
0.46
0.52
0.46
0.55
0.51
0.60
0.56
0.50
0.45
0.57
0.42
0.56
0.50
SPORTSMANAGER
0.50
0.46
0.55
0.53
0.52
0.48
0.36
0.35
0.64
0.61
0.62
0.36
0.48
0.44
OTHERPER
0.43
0.38
0.41
0.37
0.45
0.40
0.49
0.44
0.43
0.40
0.49
0.33
0.46
0,43
FACILITY
0.62
0.50
0.57
0.45
0.63
0.53
0.71
0.57
0.69
0.60
0.49
HUMANSETTLEMENT
0.85
0.93
0.87
0.77
10C
STATION
0.73
0.61
0.68
0.59
0.75
0.62
0.78
0.60
0.70
0.58
0.88
0.67
0.71
0.60
OTHERLOC
0.49
0.36
0.41
0.29
0.48
0.32
0.96
0.65
0.47
0.32
0.60
0.51
0.69
0.55
0.45
0.44
DRINK
0.48
0.38
0.55
0.37
0.63
0.48
0.58
0.52
0.64
0.38
SOFTWARE
0.62
0.53
0.76
0.63
0.68
0.57
0.74
0.61
0.70
0.61
0.59
0.48
0.74
0.61
VEHICLE
0.44
0.36
0.49
0.43
0.44
0.37
0.57
0.45
0.51
0.43
0.68
0.36
0.47
0.33
OTHERPROD
0.42
0.36
0.47
0.38
0.51
0.43
0.61
0.50
0.50
0.44
0.58
0.61
0.52
MUSICALGRP
0.60
0.50
0.66
0.55
0.68
0.57
0..73
0.63
0.78
0.71
0.67
0.49
0.69
0.61
PRIVATECORP
0.20
0.16
0.31
0.36
0.40
0.32
0.11
0.19
0.36
0.33
0.77
0.48
0.010
0.00
CARMANUFACTURER
GRP
PACTTURR
0.45
0.32
0.37
0.22
0.48
0.36
0.18
0.12
0.23
0.17
0.00
0.00
0.20
0.09
PUBLICCORP
0.52
0.41
0.65
0.51
0.58
0.48
0.57
0.45
0.63
0.51
0.60
0.40
0.73
0.63
ORG
0.57
0.57
0.44
0.64
0.49
0.55
0.62
0.48
SPORTSGRP
0.77
0.76
0.70
0.68
WORK
0.39
0.37
0.26
0.22
0.39
0.31
0.28
0.19
0.54
0.41
0.00
0.08
0.07
0.05
VISUALWORK
0.63
0.55
0.65
0.57
0.81
0.75
0.74
0.64
0.89
0.84
0.67
0.55
0.67
0.59
WRITTENWORS
0.57
0.47
MUSKCALWORK
0.69
0.63
0.61
0.53
0.62
0.55
0.70
0.62
0.78
0.49
0.69
0.61
Table 11: XIM-RRoERTa baseline results on the MULTICONER v2 test set measured by the macro Fl score for the different NER tags.

---

2.5
1.8
1.8
23
80
2.1
1.2
210
LOC
1.2
9.8
12
3.5
70
LOC
82.
49
40
22
1.7
09
14
9.4
0.4
499
31
85
14
92
0.
94
50
Counne Trumt
REN
3.0
6.5
63
92
0.1
32
Gruus
40
GRP
PAC
5.6
1.2
51.5
1.8
52.5
20
2.5
2.5
1.5
572
24
133
MED PALO
2.3
231
1.2
4.7
54.7
14
15.0
20
CW
FEN
GIP mibo MED
LOC
PEI
GIP
pnco
LOG
FEN
GRP mice MED
Prediction
Fridite
Pridition
(a) BN
(b) ZH
(C) EN
LOC QU
62.7
0.7
2-2
1.4
3.5
0.1
13
0.6
2/4
1.8
3.
0.
22
F4.
2.5
2.3
4.2
1.3
0.2
20
50
40
LOC
0.5
26.9
9:6
9.3
0.8
00
Cruus Thut PER
0.6
4LS
1.4
933
0.0
5.1
40
Cround Truf
PER
6.5
S43
93
0,1
25
0.8
3.0
40
DRP
3.4
3.5
47
40
PADO
3.3
575
3.3
MED PADI
2.4
12.0
20
0.2
1.2
11.1
10
MED
6.7
0.6
66.8
0.8
0.2
16.1
9.
CV
LOC
FER
GIP
Pico
MED
CW
LOC
FEIL
GAP
pnoo
ViD
0
cW
LOC
FER
Git
pico
MED
Fredittion
Frediciin
Prediction
(d)
(e) FR
(f) DE
3.7
1.3
2.4
2.4
000
3.8
LOC
ML.
1.5
0.4
0.3
2.2
28
145
LOV
82.
1.8
0.2
319
60
PER
40
Gruun
Oround CRF
12
2.6
2.0
87-
3.8
0.1
61
2.1
3.6
4.0
91-9
24
0.1
15.3
40
3.0
5.0
42
115
14
0.1
15
40
MED MROD
MED
0.8
0.7
0.1
3.0
87.1
10.0
MED
0.6
0.5
2.4
2.7
15.6
10:1
0.2
0.7
0.5
0.3
1.8
15.9
305
CTP PRO0
GAP
LOC
MED
(g) HI
(h) IT
PT
LOC
1.4
22.1
3.7
5.6
1.3
0.3
15
LOC
0.2
89.8
0.5
2.4
0.8
0.3
6.0
PER LOC
0.6
649
1.5
1.5
0.7
0.1
10
5.
Gound
40
Trufh GRP
40
2.7
5.6
72.5
3.4
0.8
14.
7.2
78.
10
3.6
4.3
3.3
75.8
1.4
0.1
31.9
4.5
587
5.0
3.5
MED PRLD
2.7
14
26.6
MED
13.3
mio
GAP piio
LOC
mnc
Frediciion
Frediction
Prediction
( ES
(k) SV
(0) UK
Figure 6: Error analysis for the baseline approach. The plots on the x-axis show the predict coarse class, while the y-axis shows the ground-trun -tuth class. The scores represent the ratio of entities of specific class and its predicted class. The diagonal represent correct class predictions.

---

Vualea Work
2.5
7.0
00
94
0.
63
00
01
02
94
92
P4.
Withenllook-
3.4
54.
4.6
0.
00
3/4
17
0.
231 0. 02 6.6
00
0.0
02
00
58
Software 9.5
6.1
2:2
F1.6
61
0.
0.
62
0.
02 9/
05
06 9. 04 3.6 00 02 5.
11 01 92 0.0
01 02 36
9.
01 66 911
22/
Visua Work
43.0
Facilty
0.0
9.0
NumorScttlement
3.0
0.0 5.2
0.1 0.2 3.5 S76
0. 0.5
0.0
02
0.1
02 0.3 0.0 0.5 2.5 0.0 0. 61 01 00
01
0.0
61
01
01 0.0 00 0. 0.0
Station
9.1 0.0 0.0
0. 303
03.7
0.0
9.0
0.
1.7
00
06
9.0
0.
11.4
-70
Sportoterager--e 3.0
62
10.5
0.0
0.0
AtNirte 3. 6.0 51 0.8 02 51 61 00
2.
32
16.7
3.2
8.0
3.2
6.1
A5
0. 00
11
50
04
0.0
51
0.0
00
03
00
0.0 04
00 00 9.
600
Cleric 0.0
61
0.
660
1.0
0.
0.
9.
9.1
Pullkdan 3.0 0.
84
0.0 03 022 08 01 6.
02
85 3.0
14.455 45.9
1. 23.3
3.2
0.0
011
8.0 011 01 91
0.0
6.0
0.0
000 9.0
00 0.0
0.0
Scientst
08 0.7 03
0.1
1.2
15.7
51.
3.1
0.1
0.6
04
03
9.0
0.
0.2
5.8
10
030-00 3.4 0.8 32
0.0
3.1
011
55.7
09
0.
94
PHWWHCor-- 0.3 61 7.6 16 1.4 3.7 07
0.
0.
02 34
10 25 02 0.5
P52-17
1.0 3.5
00.3
09 2.4
13
02
15
0.0 01
0. 50 94
0.0 12.3
Sporstap -A 01 05
03
9.1
0.
3.5
76.8
0.
01
9.0
0.
PutlcCorr-- 011 00 33 04 89 02 0.0
89 9.8
07 02 02
21.0
2.5
2.9
47.0
02
16
811
0.0 0.6
9.0
CuMtinfazuuer 9.0 00 51 03 84 12 00 0. 9.8
0.
0.1
7.9
92
42
3.4
54.3
0.1
02
6.9
39
9.1
00
0.6
0.0
-30
ckithing-04
0.0 922
0.0
3.1
00
02
57.9
Drnk-322 0. 54 02 04 57 13 0. 0.0 01 3.4 01 00 0.0 0.6 2.1 011 01
50
34 01
53
45.7
05
12.3
1.5
0.0
0.0
0.1
01
23.2
-20
WHNc-002 5.2 67 1.
9.1
011
2.1
3.1
20
01
03.2
0.2
0.1
27.0
-A
07 68
0.0 3/8
01
0.0
6.8
0.6
00
3.8
1.8
Medication- 3.0 00 86 0.1 00 61 04 00 00
00
62
00
00
86
00
88
01
05
08
13
68
06
01
0.
-10
Symptom-
01
02
ArddomisaiSmuut-e -0 01 83
0.0
0.0
01
0.
00
02
00
01
3/0
00
88
02
01
0.0
00
8.0
0.
00
32
03
0.0
1.0
08
0.4
12
0.5
629
51.3
Mecicafhhootte..o 01 03 0.
0.
0.1
6.8
0.
C.0
60
1.7
0.8
0.0
42.4
Onston
other PER
CRG
Muato GD
Food
090090
Anstamica Soutt-
Frndction
Figure Fine-grained NER predictions the XLM-RoBERP baseline on EN test set.

---

Vualea Work 75.0
1.4
34
07
00
94
02
63
00
00
03
02
60
5.8
Withenllook-
3.4
73.2
32
04
00
3/4
14
0.4
2.8
01
61
61
03
61
03
14.
Software 9.0
02
599
76.6
1:1 99 06
0.
62
03
60
9/
02 06 9. 0.0 1/2 90 03 52 30 03 95 0.0
00
65 25 00 92 00 66 98
83
Visua Work
Facilty
0.1
0.0
NumorScttlement
3.0
011 5.2
0.1
0.2 1.4 48 0.6
5.5
0.0
01
022
03 08 0.0 0.5 2.1 01 02 5.0 04 01
03 10 0.0 61
02 01 011 01 5. 0.1
Station
3.0 0.0 0.0
0.0
41.3
0.0
9.4
0.0
0.4
07
9.0
0.
Sportoterager--e 3.0
0.0
C4.4
00
0.0
0.0
0.0
1.3
AtNirte 3. 6.0 5.5 0.8 61 5. 02 00
2.
53
76-0
34
54
1.6
0.8
5.
01 01
63
5.
01 61
51
0.0
00
03
00
0.0
04
00 0. 9.8 2.7
Cleric 0.3 63 0.0
0.
614
6.9
06
0.
9.
Pultkdan 0.1 01 84 0.0 0.1 022 0.7 00 8.0
0.7
92
2.8
9.
61.0
92 13.4
3.1 0.1 0.0
8.0 02 01 91 0.0
0.0
6.0 011 00
9.0
00 0.0
0.1
Scientst
1. 0.0 0.
08
81
k2
13.4
0.3
0.0 08 08
08
9.0
0.0
0.0
0.6
010-02 32
27
32
0.
1.0
02
07
02
6.8
PHWWHCor-- 34 6.0 56 07 07 57 8. 0.0
0.
0. 34
00
00
04
0.0
6.0
11600
0.00000
57
36.
0.
04
00
00
15
0.0 04
0. 50 94
Sporstap 0.1 55 61 04
1.
3/
03
2.4
21.3
02
0.
04
9.0
0.8
40
AuticCorp 3. 00 8.6 02 00
06
9.8
00
12 11.
06
2.0
72.8
0.0
1.4
9.8
0.0 0.6 0.0
CuMtinfazurer 9. 00 85 0. 8.8 0. 00 0. 9.8
00 066 07
3.4
00
8.6
62.6
0.0
0.0
2.1
07
9.0
00 0.6 0.0
2.7
clitting-
0.0
0.0 3.4
0.0
0.
00
37.6
9.0
Drnk--3 0. 55 0. 0.0 3.7 34 0.0 00 0.0 00 3.4 00 00 0. 0.0 0.0
0.0 0.0
5.0
0. 00
0.2
$1-4
5.
0.0
5.
00
0.0
94
0.0
WHNc-005 51 0.0 0.
0. 9/4
0.
0.5
0.5
18
9.2
0.
20
Food- 3A 06 63 0. 0. 03
0.0 3/ 03 0.0 6.
06 00 0.6
06
88
0.
23.0
3.4
6.1
13.5
Medication- 3.0 00 86 0.. 00 83 00 00 00 88
00
00 03 00 00 86
00
88
03
05
05
A3
03
0.6
0.0
11.6
Symptom 3.0
0.0
00
AradomisaiSmuut-e -30 02 8.0
0.0
0.0
0.2
02
00
00
00
3.3
00
88
03
0.0
00
8.0
0.
00
32
07
0.0
03
02
12
022
05
56
Mecicafhhoottee.o 34 32 0.
0.
12
60
04
28
Onston
Scon
other PER
CRG
Nuato Ga
Food
090390
Enstarica Soutt-
Frndction
Figure Fine-grained NER predictions the SLM-RoBERTI baseline on DE test set.

---

Vualea Work .3.4
3.5
13
01
00
94
0.
63
0.
01
02
P4.
Withenllook
2.4
541
21
0.
00
33
8.1
1.8
0.
02
00
02
61
03
02
81
10
Software 1.2
0.0
1.4
6844
211
61
0.1
00
08
0.
9/
06
0.6
0.1
2.8
01 02 56
14
01
14
0.0
00
02 28
60 9.1 00 66
98
158
Visua Work
Facitty
NumorSettlement
0.2
0:2 0.5
0.0
0.2 2.8 14.
0.
0.0
0.0
02
0.3
0. 1.6 0.0
5.2
00
5.0
01
01
5.8
011
01 61 01 08 01
01 50 0.0 00 10.9
Station 0.1 0.0 0.2
0.0
05.0
0.0
9.2
0.1
1.0
01
0A
91
0. 17.9
Sportoterager--o 0.1
0.0
11.0 20.3
0.0
0.0
0.0
0.0
AtNirte 0.1 6.0 531 0.8 02 5.3 8.
00
2.
80
644
34
54.6
3.5
63
01
14
5.0
0.
61
35
0.0
00
03
00
0.0
04
00 00 9.
Cleric 0.2
0.0
07
50.9
0.4
63
0.
01
0.
Pultkdan 0.1 011 02 0.0 0.1 022 00
8.
1.0
37
24
15.
44.9
111
12:4
0.6
0.0
011
8.0 0.0 01 9.2
0.0
6.0
0.0
000
9.0
00 0.0
0.0
Scientst 0.1 03 09 01
05
1.3
10.4
1633 20.7
0.5
0.0
03
04
08
911
0.6
0.0
0.0-033 3.3 05
0.0
031
011
54.5
09
1.0
9.
PHWWHCor-- 34 6.0 55 44 04 54 04 0.
0.
0.0
344
13
00
08
1.5
0.8
54.0
1.3
0.7
09
5.8
04
04
04
1.
0.0 3/1 0. 50 94
-40
Sporstap 12 53 02 14
0.
9.1
0.
2.7
24.3
13
02
9.0
0.
10.4
PutliCoor-- 033 00 3.6 06 55 01
01
02
9.8
18 02 0.4
114
2.0
27
60.7
03
12
811
0.0 0.6 9.0
CuMtinfozuuer 0.3 0.1 53 0. 83 00
0.
9.0
033 07 1.0 3.0
35
233
66-2
0.3
01
4.9
111
9.1
00
0.6 0.0
ckithinp-3
0.0
0.0
08
24.
27
Dr1k--33 02 53 011 011 0.5 03 00 0.0 0. 3.4
05
03 0.0 02 3.1 0.4 01
50
04
02
3.2
51.4
53 05
12.4
23
01 0.0
0.1
01 24.4
WHNc--02 53 02
0. 9.
2.3
45
0.8
0.2
0.1
23.9
0.0
0.0
0.
03
0.
8.0
00
07
04
0.8
Medication- 3.0 02 51 00 61
00
00
02
00
86
00
81
03
06
06
16
12
03
02
23.4
-10
Symptom-
00
00
01
Anddomisiiimuu--e 01 01 83
0.0 0.0
011
0.
81
00
00
02
80
02
01
0.0
00
8.0
0.
00
3.
14
0.0
03
13
21
1.1
08
514
23A
Mecicafhoowtaee.o 01 23 0.
0.
0.1
60
13
0.
Onston
othe PER
CRG
Muato Go
Food
090090
Anstamica Soutt-
Frndction
Figure Fine-grained NER predictions the SLM-RoBERTI baseline on E.S test set.

---

Vuatea Work 59.0 24
4.0
03
00
0.
633
00
01
0A
237
Withenklook-
344
04.0
2.0
02
0.0
34
0.8
0.0
13
0.
00
000
02
00
230
Software 9.8
0.0
1.4
62.0
62
02
0.0
0.
9/
06
63 9.1 06 6.9 91 02 52
02
54
0.0
01
02
5.0
6.0 9. 00 66 98
21.
Visua Work
P2.5
Facilty
0.
0.
NumorScttlement
0.1
0.1 5.2
0.0
0.1 2. 18.
10
0.8
0.0
01
011
02
0.0 0.9 5.5 0.0
0.
61 01 01 02
01
61 02
01 0.0
00 0.
0.0
00
10.5
Station 3.0 0.0 0.0
0.0
07.3
0.0
9.0
0.
1.2
01
08
9.0
06
0.
13.2
-60
Sportoterager--o 3.0
0.0
56.4
260
0.0
9.
AtNiate 0.1 6.0 5.0 0.0 61 5.3 02 00
0.
9.
34
82
1.3
0.4
7.5
03
0.
08
5.0
01
02
53
0.0
00
02
01
0.0 04
00 0. 9.8
2.0
Cleric 0.2 65
0.5
14.1
0.4
6.7
06
01
9.
Pultkdan 0.1 022 83 0.0 0.1 83 09 0.0
6.
1.6
255
2.5
123
51.6
18.3
0.9
0.0
011
8.0
011 01 91
0.0
6.1 011
00 9.0
00 0.0
0.0
Scientst 022 01 04 04 04
1.0
1.2
15.4
18.4
25.5
0.6
0.0
08
05
077
9.0
0.6
0.0
030-00
02 35
32
0.1
53.4
0.
94
PHWWHCor--- 02 0.0 555 0.8 03 5. 03
03 02 0.
0.0 34
17 25
03
0.5
350
52
03
7.5
30.
4.0
2.8
03
00
0.5
04
0.0 04
0. 50 94
Sporstap 0. 0. 53 04 34
0.
9/
0.
3.3
67.1
344
03
9.0
0.
14.
PutlcCorp 3.0 00 02 03 02
01
05 9.8
02
14.
3.4
502
0.0
1.
011
0.0 0.6 9.0
-30
CuMtinfazuuer 9.0 0. 53 011 03
01
9.0 05 0. 0.
3.7
19
23
3.0
66.
0.1
0.0
28
9.1
00
0.6
0.0
clitting-22
01 02
9.0
0.0
086
02
50.
Drnk-33 02 54 01. 011 5.5 23 0.0 34 0.0 02 3.4 01 0.0 0.0 04 3.7 02
01
50
02 01
52
51.4
54
1.5
30.7
1.2
0.
0.0
0.1
01 04.2
WHNc-002 0.2 02 08
0.1 1.5
30
011
0.2
0.
53.7
63.
61 04
0.0
0.0
03
8.0
01
0.
1.5
0.4
30.0
-10
Medication- 300 02 51 00 61 02 00
00
00
64
00 00 88
00
81
01
15
1.8
16
68
06
0.0
0.
232
64
Symptom- -2
0.
AnddomisiiSmud--e 011 04 52 0.0
0.0
0.2
011
00
07
00
00
00
80
03
81
0.0
00
8.0
0.
00
15
0.
022
35
14
1.0
15
556
54.0
Mecicafhoowaaee.o 42 03 03 0.0
0.
0.
6.8
0.
2.1
60
21
1.3
33.0
Onston
Scon
other PER
CRG
Muato Go
Food
Medicaa
040090
Anstamica Soutt-
Fredction
Figure Fine-grained NER predictions for XLM-RoBERTA baseline on the FR test set.

---

Mualca Work- -3.6 2.4
22
02
00
0.
63
00
00
02
94
92
16.7
Withenllook
06
08
00
33
25
0.
2.0
0.0
02 6.6
00
02
61
0.
61
81
Software 14
6.1
3.0
67.3
12 9.9
0.1
0.
0.
04
9/
09
066
0.1 02 3/2 90 0.5 63 18 0.0 51 0.0
00
61
27 6.0 9.1 00 66 98
19.
Visua Work
Facilty
9.0
NumorScttlement
0.1 0.1 5.5 0.0
0.1 32 05 04 5.5
0.0
0.5
0.1
05 0.7 0.0 0.8 1.5 0.
5.0 01 00
01
0.0
61
01
01 0.0
01 0.
0.0
Station 9.0 0.3 5.2
0.0
62.
0.0
9.0
0.0
1.2
01
9.0
06
0.1
19.2
Sportoterager-- 3.1.
0.0
02
$3.0 36-
0.0
0.0
AtNiate 0.1 6.0 5.5 0.0
61 5.3 01 00
24
23
34
55
13
63
01
04
5.0
04 0.0
51
0.0
00
03
00
0.0
04
00 0. 9.8
10
Cleric 0.0
0.
02-3
0.4
0.
0.
0.0
Pultkdan 0.1 02 855 0.0 0.2 822 0.0
8.0
0.
36
2.7
189
47.9
1.8
16.3
6.5
0.0
0.0
8.0
0.0 00 9.8
0.0
0.0
6.1 011 00 9.0 00 0.0
0.0
Scientst 0.2 02 0.6 05 02
05
1.4
14.6
57 23.0
0.5
0.1 0.8
02
03
9.1
0.0
0.0
010-02 3.0
0.0
3.1
0.1
67.4
09
0.
9.
27.9
PHWNHCor-- 34 0.0 3.1 8 0.0 53 84
00
0. 31 34
00
00
04
0.0
3.4
66.7
11
3.1
0.0
33
00
00
23
0.0 94
11 50 94
0.0 12.4
Sporstap 12 01 06
3/
01
1.0
21.8
2.6
0.0
02
9/0
0.8
PutlcCorp 01 00 85 03 85 01 01 66 988 11 03
153
0.4
53
56.
02
1.
9.8
0.0
0.6
0.0
CuMtinfazurer 9.0 00 51 01 611 06 01 01 9.8 02 03 08
6.6
95
79
6.5
3.5
66.9
0.1
01
26
9.1
01
0.6
0.0
-30
ckithinp-4
922
01
06
00
2.5
Drnk--311 01 53 011 0.7 54 077 0. 0.0 00 3.4 02 03 0. 05 1.5 00
0.0
5. 15 00
0.2
51.2
0.2
05
7.5
23
00
01 0.2 0.0
23.3
WHNc-002 01 5.1 02 08
9.
1.2
1.3 22
22.2
922
0.0
22.0
02.
07 03
0.0
01 3/
0.0
655
00
15
1.0
Medication- 3.1. 00 51 00 62 00
00
86
01
02
00
01 86
00
88
06
14
06
67
68
08
03
0.6
23.
Symptom-
0.
00
ArddomisiiSmuut-o -0 01 85 0.0
0.0
0.
00
00
00
3/0
00
80
01
03
0.0
00
8.0
00
00
32
22
0.0
0.1
15
08
0.4
11
0.4
601
52.3
Mecicafhhootaee.o 01 02 03 0.
0.
0.
0.4
2.1
00
1.
18
0.7
$3.0
Onston
other PER
CRG
Nuato Go
brikk
Food
Medhcaa
040090
Anstamica Soutt-
Fredction
Figure Fine-grained NER predictions for XLM-RoBERTA baseline on the IT test set.

---

Muatea Work-to. 66.5
54
25
00
0.
61
00
01
03
22.
Withenllook-
3.0
61
3.
26
1.9
61
01
000
06
98
00
81
Software 9.8
0.0
1.4
65.
93
02
0.0
00
02
9/ 05 06 0. 3.4 91 0. 63
11 01 54
0.0
00
62
68
6.0 9. 00 66
98
178
Visua Work
Facitty
0.1
NumorScttlement
0.1 0.1 5.2
0.1
0.1 2. 04. 04 011
01.
0.2 02 0.4 0.0 0. 1.1 0.
5.0
00
00
03
000
61
01
01
0.0
00 00 011
5.0
-70
Station 3.0 0.0 0.0
0.0
14.3
0.0
9.0
0.1
6.5
00
9.0
0.
12.9
Sportoterager-- 3.1
0.0
02
52.1
01
0.0
9.
AtNiate 0.1 6.0 53 0.1 02 51 02 00
2.
74 06.2
3.4
16.
1.3
1.0
01
0.0
5.0
0. 61
51
0.0
00
03
01
0.0 0.
00 00 98
Cleric 0.0 65
06
567
1.0
0.
00
9.1
Pullkdan 3.0 011 81 0.0 0.1 8.5 0.0
0.0
1.
2.0
122
50.7
20.5
0.2
0.0
011
8.0 0.0 00 9.8
0.0
6.
011
00 9.0
00 0.0
0.0
Scientst 9.1 07 05 02
15
0.9
15.4
28.0
0.3
011
04
02
04
9.0
0.
0.0
030-00
03
3.4
02
51.3
09
1.0
94
011
PHWWHCCr--- 34 6.0 53 0.8 0.0 38 0. 0.0
0.
0.0
344
0. 00 04 0.0
13.
42
0.0
5.
32.5
0.0
00
00
00
0.0 04
0. 50 94 57.5
Sporstap 4A 01 02
08
9/
0.
1.1
76.1
03
9.0
0.
10.4
PutlcCoor 011 00 87 02 02
0.0 01 9.8 04 86 03
73
1.5
3.
103
25
0.0
03
12
011
00 0.6 9.0
CuMtinfazurer 9.0 00 86 00 844 03 01
011
9.8 04 06 01
1.5
39
4.6
76-0
0.1
0.0
2.8
211
9.0
00
0.6 0.0
ckithinp-4
0.0
02
06
24.0
Drik-332 03 51 0. 01 3.2 04 0.0 011 311
02 00 011 0.0 0.8
0.5 03
51
14
02 51
17.57.
08
6.6
1.0
01
0.2
0.7 01 23.1
WHNc-003 0.3.6 55 0.0
9.1
0.5
27
0.
022
0.8
23.2
611 05
0. 3/8
0.0
6.
0.
00
8.6
07
08
2.7
0.
1.5
Medication- 3.0 011 86 00 82 00
01
00
00
00
88
01
88
03
04
06
67
70
07
1.4
22.0
-10
Symptom- 9.1
00
ArddomisaiSmuu--o -0 01 83 0.0
0.0
0.4
00
00
00
3/0
02
80
02
01
0.0
00
8.0
00
00
32
07
14
08
022
07
61
525
Mecicafhhowtaee.o 34 01 52 0.
0.
0.1
6.3
2.1
60
1.8
1.3
1.5
34.0
Onston
other PER
CRG
Muato Go
brikk
Food
Medhcam
040090
Anstamica Soutt-
Fredction
Figure Fine-grained NER predictions for XLM-RoBERTA baseline on the test set.

---

Vuatea Work 469 2.5
67
04
00
0.
61
00
01
07
94
02
412
Withenllook-
3.
157
2.4
04
00
33
19
81
13
0.
00
000
01
00
10
Software
9.1
0.0
3:3
00.0
1.4
92
02
0.
0.
9/
02
06
0.1
3.4
90 0.0 5.6 10 01 92
0.0
00
69 1.5 0.0 9. 00 66 98
31.
Visua Work
Facilty
07
NumorScttlement
3.0
0.0 5.2
0.1 0.1 577 16.0 01 6.2
0.0
0.0
0.0
00
6. 0.0 02 1.0 0.0
6.0
01 00
01
000
0.0
00
9.0
00 0.0 0.0 200
Station
3.0 0.0 0.0
02
03.3
0.0
9.0
0.
6.3
00
02
9.0
0.
22.2
Sportoterager--o 3.0
63
15.4
0.0
00
94
AtNirte 3. 6.0 53 0.2 67 5.3 01 00
2.
125
17.0
34
36.6
1.4
466
01 0A
03
5.
01 0.
53
00
03 02
0.0 04
00 00 9.
0.0 37.4
Cleric 0.0 54
0.1
04.9
6.9
63
06
00
9.
66.3
Pultkdan 0.0 011 833 0.0 0.2 022 0.0
8.0
04
05
1.7
9.
44.4
13.9
0.3
0.0
0.0
8.0 0.1 00 9.8
0.0
6.0 0.0
00
9.0
00 0.0 0.0
21.2
-50
Scientst 9.1 0.5 05
02
1.0
97
51.7
23.4
0.4
0.0
0.8
08
03
9.0
0.
0.2
23.4
0.0-0.0
02
0.0
34
0.1
51.2
04
9.
PHWWHCCrp-- 34 6.0 56 30 0.0 3.2 84 06
0.
0. 34
0.
00
0.0
7.5
26.4
0.0
6:3
07.4
61
04
00
00
04
0.6 04
0. 50 94 P3.7
Sporstap 1A 53 61 11
03
9/
0.1
7.0
24.0
0.6
01
9/2
0.
17.
PutlcCoor 3. 00 84 22 01
02
0.0
00 9.8 03 06 0.4
93
0.5
42
96.5
12
02
09
9.4
00 0.6 9.0
10
CuMtinfazuuer 9.0 00 51 00 82 00 0.0
9.8
02 0.
1.7
01
27
67.3
0.0
0.0
11
9.0
00
0.6 0.8
165
clithin--0
0.0 3.4
0.0
0.
$2.0
9.0
0.7
Drnk-33 01 53 01 01 5. 04 0.0
0.0 01 3.4 03 00 0.0
0.4
0.0
01
14 00
5.
56.2
011
5.4
1.5
00
61 98 0.0 23.7
WHCc-002 0...1.13 01 04
01 9/4
0.4
01
06
0.0
563
00.
92
0.
23.7
0.4 04
61 3/8
01
0.0
0.0
0.
8.6
00
04
3.4
Medication- 322 00 86 00 88 00 00
00
11
61
00
00 86
00
88
01
06
04
1.0
660
0.
63
0.1
247
100
Symptom-
01
23.0
AnddomisiiSmuut-e 011 02 83 01 011
011
00
81
00
00
00
88
02
01
0.0
00
8.0
0.
00
3.
07
0.0
03
03
03
0.6
19
46
Mecicafhhowtaee.o 34 21 02
0.
ea
0.4
C.
60
05
35.2
Onston
other PER
CRG
Nuata Go
brikk
Food
040090
Anstamica Soutt-
Fredction
Figure Fine-grained NER predictions for XLM-RoBERTA baseline on the test set.

---

Vualea Work 59.3 23
22
0.
00
03
6.
00
00
00
60
63
Withenllook-
3.1
760
0.6
11
00
3,4
11
0.4
655
0. 07 8.6
61
0.0
000
00
00
142
Software 9.0
0.0
59
#5.
03 9.8
0.
0.
68
0.
6. 9/ 01 0.6 0.5 03 1/ 90 0.0 5. 9.8 00
94 0.0 00 00 08
911 03 66
98
8.4
Visua Work
Facilty
NumorScttlement 3.0
0.0 5.2 0.1 0.1 33 20 05 02
011
02
01
04 0.4 0.0 03 6.5 00 00 5.0 00 01 0.5 00 01 61 01 01 02 01 5. 0.0
5.7
Station 3.0 0.0 0.5
0.0
94-4
0.0
9.0
0.
6.0
00
00
9.0
0.
0.
Sportoterager--e 3.0
0.5
07
15.
16.
361
03
00
0.0
0.0
03
10.7
AtNiate 3. 6.0 5. 0.8 02 53 08 00
24 1. 01-3
33
2.6
17
144 01 09
05
5.0
0.
61
54 0.0
00
03
01
61 04
00 0. 9.8
Cleric 0.0
0.0
05
913
1.4
6.9
0.
00
9.
Pullkdan 3.0 011
82
0.0 0.0 8.5 13 01 61
12
17
2.1
60
21.0
55
65
0.0
0.0
8.0
0.0
02 9.8 0.0
0.0
6. 0.0
00 011
00 0.0
0.0
01
Scientst 0.0 05 077 0.
0.
1.4
14
K2
0.0
0.0
14
00
9.0
0.
0.0
030
0.0 22
01
32
011
02:3
02
0.0
04
021
PHWWHCor--- 34 6.0 56 0.8 0.0 53 84
0.0
00 0.
00 34
00
00 04
0.0
1.
04
0.0
5.0
0.
00
0.
00
0.0
0.0 94
0. 50 94
Sporstap 1A 01 06 04
61
0.
6.1
41.0
02
0.
06
9.0
0.
PutlcCoor- 3. 00 85 00 00
0.0
00 3.5 02 06 02
42
00
0.0
8.0
F1.8
0.0
02
09
8.8
0.0 0.6 9.0
CuMtinfozuurrr 9.0 00 85 00 87 07 00
0. 9.8
03 13
6.0
00
8.6
93.3
0.0
0.0
6.9
00
9.0
00 0.6 0.0
3.3
clithing-00
0.0
0.0 3.4
0.0
88
00
9.0
Drnk-33 00 55 0.0 0.0 577 0.0 0. 00 0.0 00 3.4 00 00 0. 00 0.0
0. 0.0 5.0 0. 00 0.2
25.2
0.0
29
00
0.
04 0.0
WHNc-00 22. 85 0.
0. 9/4
0.5
0.5
0.6
30
0.0
155
9/2
0.
Food- 3A
0. 07
0.0 3/
0.
63
06 00 8.6
02
04
03
0.4
15A
Medication- 3.0 00 86 0.2 00 62 00 00
02
05
02
00 00 86
00
88
02
02
1.0
05
A1
05
02
9.8
12.
6.
Symptom
0.0
00
00
AradomisalSmuut-e -0 00 8.0
0.0
0.0
0.3
0.
0.
88
00
00
02
82
00
8.0
0.0
00
8.0
0.
00
32
0.
0.0
03
1.0
10
0.4
0.
15
10.1
Mecicafhhootuee.o 34 28 0.0
0.
ea
6.3
0.
C.
00
0.
0.0
Onston
other PER
CRG
Nuato Ga
brikk
Food
090090
Anstarica Soutt-
Fredction
Figure Fine-grained NER predictions for XLM-RoBERTA baseline on the HI test set.

---

Vuatea Work-77. 70.3
1.6
35
00
94
00
0.
6.
04
00
00
94
60
04
15A
Wittenklook-
750
3.6
14
00
3/5
16
8.1
2.0
0.
00
0.0
000
0.
02
02
04
Software
9.3
0.0
2.6
$1.4
07 45 0.5
0.
0.
9.
01
63
0.1 0. 0.2 90
0.0 5.6 16
02 53 0.0 00
02 28
9. 00 66 98
85
Visua Work
Facilty
0.0
022
NumorScttlement
0.1 0.0 5.5 0.1 0.5 30 80.9 01 6.
0.0
0.0
011
04
0.4
0.0 0.3 1.2 0. 02 5.0 01 01 01 01 0.0 02
02 01 0.0 02 5.0 0.1 0.
Station 3.0 0.0 0.0
0.0
92.
0.0
9.0
0.
6.3
00
00
9.0
0.0
Sportoterager--o 3.0
0.0
1.0
96.6
0.0
1.0
0.0
0.0
0.0
0.
1.0
AtNiate 3. 6.0 51 0.1 04 51 02 00
0.
62
61.4
3.4
41
3.6
1.4
38 03 0A 12
5.0
02 61
51 61
00
03 04
61 04
00 0. 9.8
2.8
Cleric 0.0 1.2 0.0
04
60.9
0.4
6.4
0.
0.
9.
Pullkdan 0.0 03
88
0.0 0.4 84 28 0.0
6.
05
21
1.3
12: 92.4
6.5
0.0
6.1
02
933
0.0
01
6.1 0.0 01
0.0
01 0.0
0.1
Scientst 0. 84 0.0 11
00
28
9.4
5.0
64.5
13.5
0.0
0.0
1.1
00
00
00
9.0
0.6
0.0
5.8
0.0
08
19
0.0
34
0.1
01.7
0A
94
0.0
PHWWHCor--- 34 6.0 58 8 0.0 53 84
0.
0.
00 34
0.
00
0.
0.0
1.2 55
0.0
5.0
0A
1.6
04
0.0
00
0.0
0.0 04
0. 500 94
0.0
Sporstap 1A 05 03
0.
0.
1.6
44.
02
06
9.0
0.
AuticCoor- 30 00 84 11 02 02 02 00 9.8 0.0
0.4
4.0
08
8.0
F1.
13
8.8
02
0.6 0.0
CuMtinfozuuer 1.2 00 85 0.0 00 12 00 12 9.0 00 066 00
6.0
00
8.6
99.3
0.0
00
9.0
00 0.6 0.0
clithing-04
0.0
0.0 3.4
00
88
0.0
9.0
0.0
Drnk-33 00 55 0.0 0.0 0.3 08 00 00 0.0 00 3.4 00 00 0. 000 0.0
0.0
0.0 5.0 0. 00
5.8
05.2
0.
0.0
5.5
9.
00
0.0
2.5 0.0 2.5
WHNc-00 22. 55 0.0
0. 9/4
0.0
8.5
00
0.0
9.2
0.0
Food- 3.4
044 08
0. 3/8
0.0
6.0
06 00
8.6
02
0.
0.0
16.
Medication- 3.0 02 86 0.0 00 6 88 00 02 88
00
66
00
00 86
00
82
02
04
6.
1.0
17
63
0.4
Symptom 3.0
0.0
00
00
AradomisaiSmuut-e -00 02 82 0.0 0.4
04
00
00
04
3/0
02
88
02
00
8.
0.0
00
8.0
00
00
32
0.4
0.0
111
0.
0.7
18
525
Mecicafhhoottee.o 34 32 07
0.
ea
C.0
60
04
00
Ontoon
other PER
CRG
Nuato Go
brikk
Food
090090
Enstarica Soutt-
Medso
Fredction
Figure Fine-grained NER predictions for XLM-RoBERTA baseline on the BN test set.

---

Muatea Work
54.00-0000 1.
2.0
0.6
01
9.3
01
61
00
01
01
94
60
01
21.
Wittenllook-
05
3.0
0.7
0.
3/5
0.0
1.
0.
03 69
00
0.0
61
02
01
01
02
232
Software 0.6
0.0
1:2
$8.6
93
0.1
0.
68
0. 6.0 9.
06
63 0.1
066
90 02 5.6
966
01 93 0.0 02 63 24 65 9. 02 66 98
05
94.
Visua Work
-70
Facilty
0.0
02
011
NumorScttlement
0.2
0:1 0.8
0.1 0.4 2.8 15.4 0. 0.2
0.0
02
3.5
05
0.7 0.0 1.0 1.4 00
88 5.0 80 01 02
01
02 02
04 02
00 0.0 0.1 00
15.2
Station 3.0 0.0 0.2
0.0
15.0
0.0
9.0
0.
677
00
9.0
06
0.3 10.3
-60
Sportoterager--o 3.0
0.0
50.0
0.0
0.0
0.0
0.0
0.
AtNirte 0.1 6.0 53
0.1 03 51 03 00
43
06.2
3.
61
3.0
03
03
03
5.0
03
61
53
6.1
00
03
00
0.0 04
00 00 9.
Cleric 0.3 1/2 0.0
03
50.5
6.
63
0.
0.
0.3
-0
Pultkdan 0.3 04 844
011 0.1 8.5 29 0.0
8.0
0.7
15
9.2
12.5 44.4
17 14.9
0.7 0.1 011 8.0
00
9.4
0.0
6.0
011
0.0
9.0
02 0.0 0.1 00 11.9
Scientst 0.0 0.5 05 12
00
0.0
4.
15:3
41.7
0.5
0.0
0.8
00
00
9.0
06
0.8 11.3
030-00 3.5 03 32
0.0
$2.4
1.0
94
0.
FHWNHCoR-- 34 14 3/4 27 07 5. 04 0.0 00 0.
0.0
34
14
00
0.0
6.0
50.0
0.0
5.0
36.9
0.0
50
00
00
14
0.0 04
0. 50 94
Sporstap -2 53 02 04
04
0.
2.1
20.0
0.4
02
9/0
0.
PutlcCoor-- 03 00 8.6 2 02 33 17 0.0
0.0
00 92
85
02
65
08
8.8
F2.6
24
9.8
0.0 0.6 0.2
141
CuMtinfazuuer 9.0 00 8.6 0.
00 0.
9.0
06 00
66
00
8.6
64.7
0.0
0.6
06
9.0
00 0.6 0.0 17.9
ckittinp-3
0.0
0.0 3.4
00
88
00
63
0.
9.0
Drnk-33 0. 58 0. 0.0 5.3 14 0. 00 0.0 00 3.4 00 00 0. 00 0.0 0. 0.0 5. 0. 00 0.
42.0
0.8
0.0
4.0
3.5
08
88
0.8
0.0 43.0
WHic-00 53 04
0. 5.2
0.
03
0.5
10
0.4
9/4
0.0
22.0
Food- -A
0.0 04
0.0
3/
0.0
0.0
0.0
0. 00 8.6
00
04
1.4
10
Medication- a.1. 00 51 0.1 00 61 04 00
00
61 11 00
01
63
00 01 86
00
88
03
06
0.
16
68
0.6
0.4
639
63
00
Symptom
0.0
00
0.0
ArddomisiiSmuut-e -22 00 82
02
0.0
0.3
02
00
00
00
3/0
03
0.
00
8.0
0.0
00
8.0
0.
00
32
0.6
0.0
03
09
18
0.3
05
625
23A
Mecicafhhowtaee.o 34 23 0.
0.
6.4
60
07
0.0
1.8
35.0
Onston
other PER
CRG
Nuato Ga
Food
090090
Anstamica Stutt-
Fredction
Figure Fine-grained NER predictions for XLM-RoBERTA baseline on the ZH test set.

---

Mualca Work-too 50.9
02
2.5
0.
02
00
94
0.
61
00
01
01
94
01
43.0
Withenllook-
3.
02
61.4
13 0.4
00
33
15
8.0
69
0.
00
000
01
00
Software 0.6
0.0
3.4
17.8
07 61
0.0
0.
68
0. 0.
9/
02
06
9. 0. 67 90 01 5.6 11 01 93 0.0
00
02 28
6.0 9.
00
66
91 0. 137
Visua Work
Facilty
0.0
88
9.0
NumorScttlement
3.0
0.0 5.2 0.0 0.1 1.4 26 12 02 0.1
04
9.0
04
0.7 0.0 0.7 0.5 0.0
5.0
00
00
01
000000000000000
0.0
61
0.0 00
9.0
00 0. 0.0
Station
9.1 0.0 0.1
03
15.4
0.0
9.0
0.0
6-2
00
9.0
0.
12.0
Sportoterager--e 3.0
0.0
$2.6
00
00
AtNiate 3. 6.0 5.5
0.8 0.0 5.3 8. 0.
2.
10.
A1-3
34
41
8.9
03
1.0
0.0
61
5.0
0. 0.0
34
0.0
00
03
00
0.0
04
00 00 9.
Cleric 0.1 54 0.
06
147
0.4
63
0.
00
9.
Pultkdan 0.1 0. 83 0.0 0.0 83 11 0.0 8.0
16
85
1.9
15
44.9
1.1 19:2
0.6 0.0
0.0
8.0
0.0
00
9.8
0.0
0.0
6.0 0.0
00 9.0
00 0.0
0.0
Scientst 02 044 05 02
1.
1.0
23.2
0.2
0.0
83
01
03
9.0
0.0
0.0
010-02 3.5
38
0.0
32
0.
53.
05
021
13.0
PHWNHCor-- 34 6.0 56 155 0.0 5. 15
0. 04
0.
00 34
0.
00
0.0
0.0
25
0.0
5.0
35.4
15
04
46
80
00
15
0.0 0.
0.
50 94
0.0
13.5
400
Sporstap -A 01 02 13
1.6
3/
0.
1.2
01.8
03
0.0
06
9.0
0.8
AuticCoor-- 033 00 02 03 01
01
03 9.8 02 63 09
25
12
3.5
680
0.0
0.0
12
811
0.0
0.6
0.0
CuMtinfazuuer 9.1. 02 86 00 88 00 0.1
9.8
02 06 00
1.5 11
07
1.4
72.2
0.0
011
10
18
9.0
00
0.6
0.0
clithing-25
0.0 922
0.0
01
00
20.8
Drnk-33 01 53 0.0 0.0 0.5 05 00 00 0.0 01 3.4 01 01 0.0 0. 0.4 0.0 0.0
5.
0. 01
41.4
03 02
123
0.8
0.
0.0 0.2 0.0 21.4
WHic--02 0.1 01 01 03
0.
011
0.4
13
0.2
0.0
19.2
02
61
0.
80 3/ 01
0.0
6-1
0. 00 8.6
00
05
0.7
Medication- 3.1. 01 51 00 8 00 01
88
01
03
00
00
86
00
88
02
07
04
16
79
07
0.6
9.1
150
Symptom- 300
00
Arddomisiitmuut-o -0 01 83 0.0
0.0
0.2
00
00
01
00
88
00
8.0
0.0
00
8.0
0.
00
07
0.0
03
07
18
23.3
Mecicafhoowaaee.o 34 00 24 0.
0.
ea
0.4
0.
60
1.0
0.7
0.8
33.4
Onston
other PER
CRG
Nuato Go
Food
040090
Aestamica Soutt-
Medica
Fredction
Figure Fine-grained NER predictions for XLM-RoBERTA baseline on the UK test set.

---

Vusica Work 64.1
05
2.5
26
02
03
00
0.0
01
91
21
03
00 02 63 00
02
5.
0.
01
32
01
00
6.1 02
00 98 00 66
9.8 00
230
AtWork-1.2
33.0
12
12
00
93
0.
67
02
1.0
9/
45.9
100
2.0
03
9.0
Sohwerr--03 3.5
0.
3:3
70.4
1.0 5.3
03
00
02
0.
02
04
00
02 02 1.5 01 033
61 17 01 53 00
000
63 31 01 9. 00
00 51 01 152
-50
Visua Work
26.6
Paciity 0.1
0.0
2.8
01
00
9.
17.4
NumanStttement 0.0 0.1 5-2
3:4 66
04
0.0
02
3.5 00 05 66
01
91 0.0 00
01
80 00 0. 9.
Otheroc-30 02
0.0
0.
61
00
00
10
9.0
01 03
SportMtangger 0.0 0.5
01 01
+1.
3.0
044
61
14
01
00
9.0
06
0.2
Clork: 3.0 04 0.0 0.1 0.5 0.7 00 0.0
04
04
51.
9. 11.0
0. 15.5
0. 0.0
5.0
0.0
00 51
0.
0.0
60 0.0
0.0 9.0
09
0.0 0.0 0.
Ar1 53 03
03
16.0
0.8
63
00
00
9.0
0.
Pot0dan
09
00
9.0
00
Selentot-344 01 51 01 02 5. 0.0
01
0. 2.0
0.1
10.5
9.5
405526.
0.4
0.0
011
53
03
04 51
00
03 03 00 0.4
01 50 98
01C-03
03 33
0.0
614
11
05
94
HhatsCorp 0.1. 0.0 59 06 66 00 00
96
01
01 09
72.00
64
1.5
01
01
00 98
dersspaceHoru -30 00 86 00
00
02
6.0
32
0.0
21
86
01 80 08
NateCerr 34 01 08
02
30
2.
0.1
077
9.0
06
0.2
-00
MARCARSAP 13 00 54 01 11 0.5 94 00 00 0.0 05 3.0
01 0.0 02 2.1 01 08
51
0.3
01
14.0
0.0
01
02
01
0.0
09
0.0 0,0 0.
10.2
Cltthing-0
0.
06
9/8
0.
0.8
0.
00.8
2.
0.1
1.3
67.4
Drink
02
00
3.0
0.
02
93.2
0.7
0.1
-20
WHice 1. 61 51 011 61 0.8 84 02 03 00 03 3.4 02
83 011 09 3.4 00 08
544
02 30
53
02
01
60
01
0.1
0.1
0.0
0.0
9.
0.0
23.0
0.2
01
0.0
3/
0.0
08
0.0
88
01
06
3.8
Melcadionnoo -00 00 51 00 00 00
09
63
00 88
01
03
07
05
70.
08
0.
19.4
Symptom 01 01 03 00 00
59
73
Arakemmtaaimmut 3. 01 0.0
08
0.3
0.2
00.6
23.3
Walu Wor.
Voan Work
Spo Spor MaggRnngg
5000100 tat
othoree
Chooag
brikk
fadd
Aratum ctiiaat
Mediaa
Figure 18; Fine- Dine-grained NER predictions for the 3LM-RoBERTa baseline on the MULTI test set.


Reviews:
[{"review_id": "YlwmdtchIt", "reviewer": "Reviewer_kR1G", "summary": "", "questions": "A. Could you elaborate on the methodological contributions unique to this dataset that aren't extensively covered in Malmasi et al. (2022a)? Aside from the introduction of the noisy test set, in what ways does the dataset presented here diverge from the aforementioned work?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive and insightful feedback on our paper. \n\nIn the following we address the reviewer’s criticism regarding novelty and answer their main question.\n\nIn comparison to the mentioned MultiCoNER work by Malmasi et al. (2022a), our contributions are two-fold. i) developing a fine-grained taxonomy that will scale to 12 languages, where most of the languages are low-resource, and  developing such dataset for those languages is a complex challenge. While fine-grained NER may not be a novel problem, but for many languages, this is the first fine-grained NER dataset with considerable amount of samples. ii) We have introduced the complex techniques for noise generation in different scenarios like typographical errors, OCR noise. Again, this is not a one-size-fits-all technique. We had to adapt different techniques for different languages. These techniques can scale beyond NER problems alone and can be helpful for low-resource languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "To my knowledge, large-scale datasets encompassing noisy entities and context are scarce moreover the introduction of typing / letter similarity noise, barring the references cited by the authors. Introducing such a test set could serve as a valuable asset for a segment of the NER community, potentially fostering the development of more robust solutions for NER models.", "reasons_to_reject": "The framing of the paper's contribution appears to be misleading. This submission primarily presents the construction of a novel multilingual fine-grained NER dataset with the addition of noisy entities and text. Notably, the entire data construction section has been relegated to the appendix, depriving the main body of the paper of crucial information about the dataset and its creation.\n\nUpon a deeper examination of the very long appendix, it is evident that there was significant effort invested in the data construction process, underscored by a detailed pipeline leading to the creation of this large-scale, fine-grained, multilingual dataset. However, the entire process (including some of the plots..) has already been published by Malmasi et al. (2022a) under the title \"MultiCoNER\", which (in my opinion) stands out as a commendable dataset creation paper.\n\nThe authors here neither assert that their dataset is an extension nor a modification of the \"MultiCoNER\" dataset. Still, they posit the creation of a novel multilingual fine-grained NER dataset with noise. Given that the innovation in the dataset creation seems confined mainly to the introduction of typing/OCR noise, the paper's authentic contribution appears to be minor.\nAlthough the minor contribution might resonate with a segment of the NER and IE community, we should evaluate a paper based on its claims. Given the perceived lack of transparency and the mirroring of an already published methodology, the quality of this paper is diminished. It would be appropriate for the paper to be restructured to accurately reflect its unique contribution.", "questions_to_authors": "A. Could you elaborate on the methodological contributions unique to this dataset that aren't extensively covered in Malmasi et al. (2022a)? Aside from the introduction of the noisy test set, in what ways does the dataset presented here diverge from the aforementioned work?", "paper_topic_and_main_contributions": "The authors present a dataset focusing on multilingual fine-grained Named Entity Recognition (NER). This dataset covers a wide range of entity types at varying levels of granularity. The dataset spans a range from English to under-resourced languages like Farsi and Bangla\nA unique feature of this dataset is the introduction of texts that have been deliberately modified to introduce lexical noise. This noise is intended to simulate real-world typing errors or mistakes typically seen in OCR processes.\nIn their evaluation, the authors evaluate XLM-RoBERTa model on their dataset. The results indicate a consistent decline in performance corresponding to the extent of data corruption, for both context and entities.\n\nThe main contribution of this submission is the introduction of a noisy test set designed to replicate errors typical of keyboard typing and OCR (specifically, errors stemming from letter similarities). To the best of my knowledge, there is a lack of large-scale datasets addressing this specific niche, and the discourse on how contemporary NER models perform under such conditions is notably absent in the current literature.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SeD9TnzUw1", "reviewer": "Reviewer_iV2c", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their highly constructive and insightful feedback on our paper. \n\n1) In terms of benchmarking, this dataset is the first to address the following aspects: 1) coverage of diverse languages both in terms of script (Latin, Cyrillic, Arabic, Farsi, Chinese, Hindi, Bangla etc.), 2) mix of low and high resources, and 3) scope of named entities to be detected as part of NER, both in terms of fine-grained NER and as well as other aspects such as noisy entities. To the best of our understanding this is the first work to contribute at this scale and problems at once.\n\n2) We will add more details in the main body of the paper. However, given the nature of Resource track papers, where a lot technical aspects must be taken into account, we have limited ourselves to incorporate in the main body what we determined to be the complete process of the dataset creation. The appendix is rich in additional explanations, examples, results, and algorithms, that can help for reproducibility and further extend our proposed dataset creation mechanism for other researchers. \n\n3) We will incorporate any missing references as suggested by the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Well written paper, good dataset, especially liked the section on evaluation based on noise. \nSupplementary section provides a lot of details. \n", "reasons_to_reject": "Fine-grained NER is a well studied problem, so the novelty of this work is less. I would have loved to see improvement in performance on a regular benchmark by using this dataset which would make the data more valuable.\nDataset details should be part of the main paper even if done in a brief manner as that is the main contribution of the paper.\nNo link to the data provided with the paper.\nSome missing references around noisy and fine-grained NER datasets.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a new dataset for fine-grained multi-lingual NER dataset for which also includes noisy inputs mimicking visual and phonetic errors at the character levels. They run extensive analysis using XLMR model and provide a strong baseline for the dataset along with the impact of different types of noise on evaluation scores. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "https://proceedings.neurips.cc/paper_files/paper/2022/hash/09723c9f291f6056fd1885081859c186-Abstract-Datasets_and_Benchmarks.html \nhttps://xiaoling.github.io/figer/ \nhttps://aclanthology.org/C18-1060/ "}}, {"review_id": "rPTVxvOoFN", "reviewer": "Reviewer_SHVo", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive and insightful feedback on our paper. \n\n1) As pointed to the response for the reviewer iV2c, we wanted to have in the main body what we believed to be the most complete set of steps that were necessary for generating the dataset. The appendix, as pointed out, is in rich in examples, detailed explanations, and results, where we wanted to show the usefulness of this created resource. \n\n   Considering EMNLP’s call for papers, and aspects associated with the Resource track, we believe that given that the paper is self-contained in the main body, the appendix should be considered as a positive aspect rather a reason to reject, since it only shows the utility of the provided resource. Detailed explanations on resource creation are very common to be part of appendices when published at conferences.\n\n2) We thank the reviewer for their suggestion and we will take it under consideration. However, we are still convinced that our dataset paper is highly suited for EMNLP’s resource track.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The dataset approaches realistic challenges such as the size of the textual content (e.g., which can be short textual snippets like Web\nqueries), the characteristics of the entities to be extracted (i.e., capitalization/punctuation features that are missing and the fine-granularity), unseen and emerging entities (e.g., entities that have a fast growth rate such as new songs and movies are released weekly), and the noise that can result from the digitisation process of documents.\n\n- The dataset covers a diverse typology and writing systems, and range from well-resourced (e.g., EN) to low-resourced ones, which is an important contribution.", "reasons_to_reject": "- The paper was submitted as a short paper, however, most of the interesting details are in the Appendix - an astonishing amount of 20 pages! - (such as dataset creation details, use of translation for some languages, sentence tokenization, etc.)\n- While the contribution is important for the information extraction community, the releasing of such a dataset could be more fit into a conference such as LREC, where the authors could take advantage of the maximum page length for including the actual process of the dataset creation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This authors introduce a new dataset, AnonNER, for fine-grained Named Entity Recognition (NER) in 12 languages, with some of the partes of the dataset created though translation of the Englihs one (Bangla, Chinese, German, Hindi). This dataset aims to address two key challenges in NER: effective context representation from short texts to differentiate between 33 fine-grained entity types in complex scenarios, and performance degradation due to noise such as typing mistakes or OCR errors. The dataset is compiled from public resources like Wikipedia and Wikidata.\n\nThe process of building the dataset is identical to MultiCoNER (which indicates that ANON- means anonymized).", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- the tables do not respect the ACL template, the captions are differently positioned, too close to the tables, seems more like it could come from a package clash."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper present a multilingual corpus for Name Entity Recognition tasks comprising 12 languages which allow for  a diverse typology and writing systems, and range from well-resourced to low-resourced ones. 
A fine grained taxonomy is defined leading to 33 entity types. 
The objective of this corpus is to evaluate the robustness of NER system (LLM) given the different languages but also under noisy scenario. Word level and character level corruption strategies have been implemented on the test data. 
It will be good to clarify the link between this paper and the presented to semeval.
Decision: Accept-Findings