Venue: NAACL 2024

# Universal NER: A Gold-Standard Multilingual

Named Entity Recognition Benchmark

 Stephen Mayhew\({}^{\alpha}\) Terra Blevins\({}^{\beta}\) Shuheng Liu\({}^{\gamma}\) Marek Suppa\({}^{\delta_{c}}\)

Hila Gonen\({}^{\beta}\) Joseph Marvin Imperial\({}^{\zeta,\eta}\) Borje F. Karlsson\({}^{\theta}\) Peiqin Lin\({}^{i}\) Nikola Ljubesic\({}^{\kappa}\) LJ Miranda\({}^{\lambda}\) Barbara Plank\({}^{\iota,\mu}\) Arij Riabi\({}^{\nu,\xi}\) Yuval Pintero\({}^{\alpha}\) Duolingo, \({}^{\beta}\)University of Washington, \({}^{\gamma}\)Georgia Institute of Technology,

\({}^{\delta}\)Comenius University in Bratislava, \({}^{\epsilon}\)Cisco, \({}^{\varsigma}\)National University Philippines,

\({}^{\eta}\)University of Bath, \({}^{\theta}\)Beijing Academy of Artificial Intelligence,

LMU Munich, \({}^{\iota}\)Jozef Stefan Institute, \({}^{\lambda}\)Allen Institute for Artificial Intelligence,

IT University of Copenhagen, "Inria Paris, \({}^{\xi}\)Sorbonne Universite, \({}^{\omega}\)Ben-Gurion University stephen@duolingo.com blvnscs.washington.edu

###### Abstract

We introduce Universal NER (UNER), an open, community-driven project to develop gold-standard NER benchmarks in many languages. The overarching goal of UNER is to provide high-quality, cross-lingually consistent annotations to facilitate and standardize multilingual NER research. UNER v1 contains 19 datasets annotated with named entities in a cross-lingual consistent schema across 13 diverse languages. In this paper, we detail the dataset creation and composition of UNER; we also provide initial modeling baselines on both in-language and cross-lingual learning settings. We will release the data, code, and fitted models to the public.1

Footnote 1: [https://www.universalner.org](https://www.universalner.org), UNER v1 available at [https://doi.org/10.7910/DVW/Q8HDL](https://doi.org/10.7910/DVW/Q8HDL)

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4322–4337 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

High-quality data in many languages is necessary for broadly multilingual natural language processing. In named entity recognition (NER), the majority of annotation efforts are centered on English, and cross-lingual transfer performance remains brittle (e.g., Chen et al., 2023b; Ma et al., 2023). Amongst non-English human-annotated NER datasets, while there have been multiple separate efforts in this front (e.g., Agic and Ljubesic, 2014; Plank, 2019; Adelani et al., 2022), these either have disjoint annotation schemes and labels, cover a single language or small set of related languages, or are not widely accessible (e.g., Strassel and Tracey, 2016). For most of the world's languages, the only readily available NER data is the automatically annotated WikiANN dataset (Pan et al., 2017), though this annotation paradigm introduces data quality issues and limits its usefulness for evaluation (Lignos et al., 2022).

To address this data gap, we propose Universal NER (UNER), an open community effort to develop gold-standard named entity recognition benchmarks across many languages. Each dataset in Universal NER is annotated by primarily native speakers on the text of an existing Universal Dependencies treebank (UD; Nivre et al., 2020). Inspired by Universal Dependencies, the overarching philosophy of the UNER project is to provide a shared, universal definition, tagset, and annotation schema for NER that is broadly applicable across languages (Figure 1).

The current version of Universal NER, UNER v1, contains 19 datasets spanning 13 languages (Section 4). To establish performance baselines on UNER, we finetune an XLM-R model on various training configurations (Section 5)and show that while NER transfer performance between European languages is relatively strong, there remains a gap when transferring to different scripts or language typologies.

The goal of the UNER project is to facilitate multilingual research on entity recognition by addressing the need in the multilingual NLP community for standardized, cross-lingual, and manually annotated NER data. With the release of UNER v1, we plan to expand UNER to new languages and datasets, and we welcome all new annotators interested in developing the project.

Figure 1: Parallel sentences annotated with person (PER) and location (LOC) named entities in English (EN), German (DE), Russian (RU), and Chinese (ZH).



Dataset Design Principles

Named entity recognition (NER) is the task of identifying text spans in a given context that uniquely refer to specific _named entities_. The task of NER has a long tradition Grishman (2019) and facilitates many downstream NLP applications, such as information retrieval Khalid et al. (2008) and question answering Molla et al. (2006). Furthermore, successful NER tagging requires a model to reason about semantic and pragmatic world knowledge, which makes the task an informative evaluation setting for testing NLP model capabilities.

As with Universal Dependencies, the goal of Universal NER is to develop an annotation schema that can work in any language. Traditionally, the UD Nivre et al. (2016) and UPOS Petrov et al. (2012) projects have chosen what amounts to the intersection of tags across all language-specific tagsets, keeping the resultant tagset broad and simple. We follow a similar strategy, picking tags that broadly cover the space of proper nouns.

Universal NER's annotation schema emphasizes three coarse-grained entity types: Person (PER), Organization (ORG), and Location (LOC). We provide a short description and an example for each tag.

PerThe PERSON tag includes names of people, real or fictional, but not nominals.

"Mr. Robinson\({}_{\texttt{PRS}}\) smiled at the teacher."

OrgThe ORGANIZATION tag is used for named collections of people.

"The FDA\({}_{\texttt{ORG}}\) announced time travel pills tomorrow."

LocThe LOCATION tag covers all types of named locations.

"I will arise and go now, and go to Innisffee\({}_{\texttt{LOC}}\)"

Figure 1 demonstrates how named entities and their corresponding annotations surface across languages. In some cases (such as in the English and German sentences), the surface forms of named entities are shared. However, often these forms vary--as in the Russian and Chinese examples--which makes entity identification and tagging more challenging, particularly in cross-lingual settings.

Annotation GuidelinesIn preparation for annotation, we developed extensive annotation guidelines,2 using the NorNE project guidelines Jorgensen et al. (2020) as a starting point. Along with tag descriptions, our guidelines include many examples, as well as instructions for dealing with ambiguity and unclear constructions, such as email addresses, pet names, and typographical errors.

Footnote 2: [http://www.universalner.org/guidelines/](http://www.universalner.org/guidelines/)

We expect that the guidelines will be further refined and updated as annotation proceeds. To manage this, we track version numbers and changelogs for different iterations of the guidelines. Each data release will include the corresponding annotation guidelines at the time of release.

## 3 Dataset Annotation Process

Having described the theoretical basis for the tagset, we now discuss the mechanics of annotation.

Sourceing DataWe chose the Universal Dependency corpora as the default base texts for annotation. This jumpstarts the process: there is high coverage of languages, and the data is already collected, cleaned, tokenized, and permissively licensed. Further, by adding an additional annotation layer onto an already rich set of annotations, we not only support verification in our project (Section 4.3) but also enable multilingual research on the full pipeline of core NLP. Since UD is annotated at the word level, we follow a BIO annotation schema (specifically IOB2), where words forming the beginning (inside) part of an X entity (X \(\in\) {PER, LOC, ORG}) are annotated B-X (I-X, respectively), and all other words are given an O tag. For the sake of continuity, we preserve all tokenization from UD.

While UD is the default data source for UNER, we do not limit the project to UD corpora (particularly for languages not currently included in UD). The only criterion for inclusion in the UNER corpus is that the tagging schema matches the UNER guidelines. We are also open to converting existing NER efforts on UD treebanks to UNER. In this initial release, we include four datasets that are transferred from other manual annotation efforts on UD sources (for da, hr, arabizi, and sr).

Sourcing AnnotatorsFor the initial UNER annotation effort, we recruited annotators from the multilingual NLP community through academic networks on social media. Annotators were organized via channels in a Slack workspace. Annotators of the datasets included in UNER thus far are unpaid volunteers. We expect that annotators are native speakers of their annotation language, or are highly proficient, but we did not issue any language tests. For the first release of UNER, the choice of 

the 13 dataset languages is solely dependent on the availability of annotators. As the project continues, we expect that additional languages and datasets will be added as annotators in more languages become available to contribute.

Annotation ToolWe collect annotations for the UD treebanks using TALEN Mayhew and Roth (2018), a web-based tool for span-level sequence labeling.3 TALEN includes an optional feature that propagates annotations - if the user annotates "McLovin" in one section of the document, every other instance of "McLovin" in that document is annotated as well. This significantly speeds up annotation but risks over-annotation mistakes. For example, consider the token "US", which may appear with different senses in contexts such as "The US economy..." or "THEY OFFERED TO BUY US LUNCH!"

Footnote 3: [https://github.com/mayhewsw/talen-react](https://github.com/mayhewsw/talen-react)

Secondary AnnotatorsIn addition to collecting a complete set of annotations from a primary annotator for each dataset, we also gather secondary annotations from another annotator on (at least) a subset of the data in order to estimate inter-annotator agreement (Section 4.2). We aim for at least 5% coverage of each data split with these secondary annotations, although most datasets have significantly more (Table 2). When a document has multiple annotators, we include the labels from the annotator with the most entities annotated in that document in the final dataset. This means a dataset may have multiple annotators, but each document has exactly one. We retain annotator identities in the data files.

Annotation Differences and ResolutionWhen annotators disagreed on annotation decisions or the inter-annotator agreement scores were low, we encouraged them to discuss the disagreements and decide if they were conflicting interpretations of the guidelines or fundamental disagreements. In the former case, annotators came to an agreement on guideline interpretations and updated annotations accordingly. In the latter, the annotations were kept as-is. Not every dataset had this resolution process.

The multilingual nature of this process also highlighted cross-language differences in named entities that affect NER annotation. For instance, most languages in UNER use capitalization as a marker of proper nouns and, therefore, named entities. However, Chinese does not include capitalization in its script, which makes identifying named entities more difficult and time-consuming than in other languages, potentially leading to more annotation errors. Differences in annotating NER across languages also stem from divergent definitions of proper nouns (PROPN) by language and the effects of translation artifacts; these issues are discussed further in Sections 4.3 and 4.4, respectively.

OTHER TagAs a helpful check for annotators, we allow the option of annotating a fourth entity type, Other (OTH), which is not included in the final dataset. This had several purposes: to store annotations that behaved like mentions, but didn't conform to the guidelines of the other tags; to measure potential annotation disagreement on ambiguous cases; and to store an additional layer of annotation. Not all annotators used it, and those that did were sometimes inconsistent. In practice, OTH was most often applied to languages, nationalities, and brands. The OTH tag roughly corresponds to the MISC tag used in CoNLL 2003, which has been described as being "ill-defined" Adelani et al. (2022).

Dataset TransferMost of the included datasets are annotated from scratch using the annotation process detailed above, but a few (da ddt, qaf arabizi, hr and sr set) are transferred from other sources. The Danish ddt annotations are derived from the _News_ portion of the DaN+ dataset Plank et al. (2020); this text corresponds to the Universal Dependencies ddt treebank. The Croatian hr annotations come from the hr500k dataset Ljubesic et al. (2016), half of which, consisting of newspaper and various web texts, was used for producing the Croatian Universal Dependencies hr_set treebank Agic and Ljubesic (2015). The NArabizi arabizi dataset was annotated on UD data using a slightly different NER schema and then automatically converted to the UNER schema. The Serbian sr data come from the SETimes.SR dataset Batanovic et al. (2018), which was used in its fullness to produce the Serbian Universal Dependencies sr_set treebank Samardzic et al. (2017). The original Croatian and Serbian NER annotations were annotated and curated in multiple iterations by various native speakers. However, the annotations added to the UNER dataset were slightly modified to conform to the UNER annotation guidelines; namely, while nationalities and similar groups are annotated as PER in the original dataset, in the UNER dataset such entities are omitted. Finally, we retain the original annotations from existing NER datasets in the "xner" label column.

 



## 4 Universal NER: Statistics and Analysis

This section presents an overview of the Universal NER (UNER) dataset. UNER v1 adds a NER annotation layer to 19 datasets (primarily treebanks from UD). It covers 13 geneologically and typologically diverse languages: Cebuano, Danish, German, English, Croatian, Narabizi, Portuguese, Russian, Slovak, Serbian, Swedish, Tagalog, and Chinese4. Overall, UNER v1 contains ten full datasets with training, development, and test splits over nine languages, three evaluation sets for lower-resource languages (tl and ceb), and a parallel evaluation benchmark spanning six languages.

Footnote 4: Languages sorted by their ISO 639-1/639-2 codes (International Organization for Standardization, 2002, 1998)

### Dataset Statistics

In Table 1, we report the number of sentences, tokens, and annotated entities for each dataset in UNER. The datasets in UNER cover a wide range of data quantities: some provide a limited amount of evaluation data for a commonly low-resourced language, whereas others annotate thousands of training and evaluation sentences.

The datasets in UNER also cover a diverse range of domains, spanning web sources such as social media to more traditional provenances like news text. Table 5 in the appendix presents the complete set of sources for the data and the distribution of NER tags in each dataset, along with references to each original treebank paper. The variety in data sources leads to varied distributions of tags across datasets (Figure 2).

### Inter-Annotator Agreement

We calculate inter-annotator agreement (IAA, Table 2) for each dataset in UNER that was annotated with the above process and for which we have secondary annotations. Table 2 reports agreement as per-label F\({}_{1}\) score, using one annotator as "reference," and the other as "prediction."

ORG vs LOC ConfusionThe agreement on ORG and LOC is generally lower than that on PER. The annotation guidelines allow certain named entities to take either the ORG or LOC tag based on context. In some cases, the context is underspecified, leading to ambiguity. For example, a restaurant is a LOC when you go there to eat, but it is an ORG when it hires a new chef. A city is a LOC when you move there, but it is an ORG when it levies taxes. Officially, it is the

\begin{table}
\begin{tabular}{l l r r r r r r r r r r r r} \hline \hline  & & \multicolumn{4}{c}{Sentences} & \multicolumn{4}{c}{Entities} & \multicolumn{4}{c}{Tokens} \\ \cline{3-17} Lang. & Dataset & Train & Dev & Test & All & Train & Dev & Test & All & Train & Dev & Test & All \\ \hline DA & ddt & 4,383 & 564 & 565 & 5,512 & 3,022 & 379 & 446 & 3,847 & 80,378 & 10,332 & 10,023 & 100,733 \\ EN & ewt & 12,543 & 2,001 & 2,077 & 16,621 & 7,022 & 966 & 1,088 & 9,076 & 204,579 & 25,149 & 25,097 & 254,825 \\ HR & set & 6,914 & 960 & 1,136 & 9,010 & 8,261 & 1,218 & 1,403 & 10,882 & 152,857 & 22,292 & 24,260 & 199,409 \\ PT & bosque & 7,018 & 1,172 & 1,167 & 9,357 & 8,101 & 1,401 & 1,215 & 10,717 & 171,776 & 284,477 & 27,604 & 227,827 \\ QAF & abzizi & 1003 & 139 & 145 & 1287 & 1320 & 204 & 194 & 1718 & 15,522 & 2,124 & 2,118 & 19,764 \\ SK & snk & 8,483 & 1,060 & 1,061 & 1,064 & 2,707 & 636 & 915 & 4,258 & 80,628 & 17,233 & 17,366 & 106,097 \\ SR & set & 3,328 & 536 & 520 & 4,384 & 5,020 & 742 & 847 & 6,609 & 74,259 & 11,993 & 11,421 & 97,673 \\ SV & talbanken & 4,303 & 504 & 1,219 & 6,026 & 967 & 23 & 196 & 1,186 & 66,646 & 9,797 & 20,377 & 96,820 \\ gsd & 3,997 & 500 & 500 & 4,997 & 6,136 & 754 & 767 & 7,657 & 98,616 & 12,012 & 123,291 \\ zh & gsdsimp & 3,997 & 500 & 500 & 4,997 & 6,118 & 753 & 763 & 7,634 & 98,616 & 12,063 & 12,012 & 123,291 \\ \hline DE & pud & – & – & 1,000 & 1,000 & – & – & 1,039 & 1,039 & – & – & 21,331 & 21,331 \\ EN & pud & – & – & 1,000 & 1,000 & – & – & 1,038 & 1,038 & – & – & 21,176 & 21,176 \\ PT & pud & – & – & 1,000 & 1,000 & – & – & 1,099 & 1,099 & – & – & 23,407 & 23,407 \\ RU & pud & – & – & 1,000 & 1,000 & – & – & 1,036 & 1,036 & – & – & 19,355 & 19,355 \\ SV & pud & – & – & 1,000 & 1,000 & – & – & 1,029 & 1,029 & – & – & 19,076 & 19,076 \\ ZH & pud & – & – & 1,000 & 1,000 & – & – & 1,137 & 1,137 & – & – & 21,415 & 21,415 \\ \hline CEB & gja & – & – & 188 & 188 & – & – & 49 & 49 & – & – & 1,295 & 1,295 \\ TL & trg & – & – & 128 & 128 & - & – & 92 & 92 & 92 & – & – & 734 & 734 \\ ugnayan & – & – & 94 & 94 & – & – & 61 & 61 & – & – & 1,097 & 1,097 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Universal NER has broad coverage of named entities in several languages and domains, adding annotations to the development, testing, and training sets from Universal Dependencies Nivre et al. [20].

Figure 2: Distribution of tags in different UNER training sets. zh_gsdsimp has the same distribution as zh_gsd.



city government_ that levies taxes, but common usage allows, for example, "Springfield08G charges a brutal income tax." CoNLL 2003 English also has this ambiguity, with many documents where city names, representing sports teams, are annotated as ORG. We find this ambiguity is particularly common in the en_ewt train and validation splits, primarily in documents in the _reviews_ domain, which are short and very informal (e.g. "we love pamelas").

### Agreement with the Propn Pos Tag

The proper noun (PROPN) part-of-speech tag used in UD represents the subset of nouns that are used as the name of a specific person, place, or object Nivre et al. (2020). We hypothesize that named entities as defined in UNER act roughly as a subset of these PROPN words or phrases, although not a strict subset due to divergent definitions. To test this, we calculate the precision of the UNER annotations against the UD PROPN tags (Table 3, F1 scores reported in Table 4). Overall, precision is relatively high, with a mean precision of 0.761 across datasets. Lower precision is often due to multi-word names containing non-PROPN words (e.g., "Catherine the Great"). The differences in precision can also be due to language-specific PROPN annotation guidelines: for example, while the English PUD treebank tags the United States entity as "United\({}^{\text{\sc ratropn}}\)States\({}^{\text{\sc ratropn}}\)", Russian PUD tags it as "CoedinburghXJJ IIITATOBONON".

### Cross-lingual Agreement in Uner

UNER contains sentence-aligned evaluation sets for six languages German, English, Portuguese, Russian, Swedish, and Chinese) that are annotated on top of the Parallel Universal Dependencies treebanks (PUD; Zeman et al., 2017). Figure 3 summarizes the similarity of the NER annotations across these target languages in PUD.

We find that the overall distribution of tags is similar for the Western European languages (left

\begin{table}
\begin{tabular}{l l r r r r r r r r r r r r} \hline \hline  & & \multicolumn{5}{c}{Train} & \multicolumn{5}{c}{Dev} & \multicolumn{5}{c}{Test} \\ \cline{3-19} Lang. & Dataset & LOC & ORG & PER & \% Docs & LOC & ORG & PER & \% Docs & LOC & ORG & PER & \% Docs \\ \hline da & ddt & .875 & .778 & .959 & 100\% & .917 & .765 & .934 & 100\% & .882 & .805 & .975 & 100\% \\ en & ewt & .696 & .533 & .925 & 20\% & .786 & .640 & .949 & .20\% & .825 & .869 & .969 & 20\% \\ pt & bosque & .928 & .902 & .974 & 11\% & .850 & .885 & .980 & .25\% & .955 & .914 & .975 & 23\% \\ sk & snk & .840 & .743 & .900 & 100\% & .801 & .597 & .770 & 100\% & .837 & .621 & .823 & 100\% \\ sv & talbanken & .857 & .670 & .913 & 100\% & .800 & .461 & .888 & 100\% & .937 & .812 & .871 & 100\% \\ zh & gsd & .800 & .724 & .917 & 14\% & .795 & .661 & .956 & 100\% & .860 & .711 & .944 & 23\% \\ \hline de & pud & – & – & – & – & – & – & – & – & – & .709 & .840 & .812 & .6\% \\ en & pud & – & – & – & – & – & – & – & – & .100 & .936 & .966 & .6\% \\ pt & pud & – & – & – & – & – & – & – & – & .903 & .920 & .985 & 14\% \\ RU & pud & – & – & – & – & – & – & – & – & .719 & .531 & .891 & 100\% \\ sv & pud & – & – & – & – & – & – & – & – & .865 & .735 & .944 & 100\% \\ zh & pud & – & – & – & – & – & – & – & – & .752 & .776 & .971 & 20\% \\ \hline ceb & gja & – & – & – & – & – & – & – & – & .769 & 1.00 & .914 & 71\% \\ TL & trg & – & – & – & – & – & – & – & – & .833 & – & .957 & 100\% \\ ugnayan & – & – & – & – & – & – & – & – & .913 & – & – & 100\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Inter-annotator agreement scores for the datasets annotated natively for the Universal NER project. We don’t report IAA for the datasets adapted from other sources, or from zh_gsdsimp, which has nearly identical annotations to zh_gsd. **% Docs** refers to the percentage of documents annotated by multiple annotators.

\begin{table}
\begin{tabular}{l l c c c} \hline \hline Lang. & Dataset & Train & Dev & Test \\ \hline da & ddt & .709 & .729 & .722 \\ en & ewt & .890 & .895 & .892 \\ hr & set & .683 & .651 & .671 \\ pt & bosque & .864 & .881 & .844 \\ qaf & arabizi & .952 & .960 & .985 \\ sk & snk & .803 & .783 & .688 \\ sk & set & .687 & .631 & .680 \\ sv & talbanken & .766 & .756 & .842 \\ zh & gsd & .605 & .624 & .616 \\ zh & gsdsimp & .601 & .604 & .617 \\ \hline de & pud & – & – & .712 \\ en & pud & – & – & .872 \\ pt & pad & – & – & .749 \\ ru & pud & – & – & .708 \\ sy & pud & – & – & .810 \\ zh & pud & – & – & .634 \\ \hline ceb & gja & – & – & .980 \\ tl & trg & – & – & .958 \\ tl & ugnayan & – & – & .654 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparing the overlap (Precision) between UNER annotations and UD PROPN tags.

 

panel): the English, German, and Swedish annotations contain very similar counts of LOC and PER entities, with slightly more variance in ORG tags. Portuguese has a similar distribution with slightly more LOC entities. However, the Russian and Chinese annotations contain differing distributions from both these languages and each other.

A similar trend occurs in the sentence-level pairwise agreement on entity counts and identities between languages (center). There is relatively high agreement on the number of entities between European languages, with Russian differing slightly more from English, German, Portuguese, and Swedish. However, the Chinese benchmark agrees less frequently: the Chinese annotations match other languages on the number of entities in 50.4% of sentences; the other languages have an average agreement of 71.7-75.6%. Pairwise agreement on the specific entities in a given sentence shows similar behavior, albeit with lower agreement overall (right).

Many of these annotation differences likely stem from the translation process. While the data is aligned at the sentence level, linguistic variation and translator decisions may cause an entity to be added to or removed from the sentence, or the concept may be expressed in a manner that no longer qualifies as a named entity under the annotation guidelines.5 While we cannot directly measure inter-annotator agreement across languages because of the above differences, some variation also undoubtedly stems from annotation differences and errors, just as these cause disagreement between annotators on the same benchmark.

Footnote 5: Consider the phrases: “ 

8, and batch size of 4 in the cases of talbanken and all. All the code we used is adapted from the Huggingface transformers package Wolf et al. (2020).

### Results and Discussion

Figure 4 reports the micro F\({}_{1}\) scores on all test sets when XLM-RLarge is finetuned on different languages. The in-language performance shown on the diagonal on the left of Figure 4 is almost always the highest among all test sets, with a few exceptions such as Simplified Chinese vs Traditional Chinese (zh) and Croatian (hr) vs Serbian (sr). This most likely stems from the fact that both pairs are closely related languages.

We also observe that in most cases (i.e., between European languages), cross-lingual transfer performs well, achieving over .600 F\({}_{1}\). However, transfer results in strikingly low performance on all three Chinese datasets {gsd, gsdsimp, pud}, as well as on the Maghrebi-Arabic-French (QAF) dataset {arabizi}. The results on the Chinese datasets align with observations from previous work Chen et al. (2023); Wu et al. (2020); Bao et al. (2019) that other languages do not transfer well to Chinese. Narabizi is a North-African Arabic dialect written in Latin script that often involves code-switching with French. The lack of similarities between this language and all other languages in our dataset might have resulted in poor transfer performance. Furthermore, Narabizi -- along with Cebuano -- are not included in the pretraining languages for XLM-R, which likely also affects their performance in this setting.

Table 6 (in the Appendix) shows the tag-level performance breakdown. For all languages, F\({}_{1}\) on ORG is always the lowest, and LOC is almost always the second lowest. This likely stems from the similarity between ORG and LOC entities discussed in Section 4.2, whereas the names of people are usually less ambiguous, resulting in the highest F\({}_{1}\) on PER for most datasets. Overall, the trained models finetuned on the UNER datasets exhibit promising results, and we leave further improvements on multi- and cross-lingual NER with these datasets to future work.

Finally, the performance of the model finetuned on all is included in Figure 5. Most a\(11\) F\({}_{1}\) scores are similar to the F\({}_{1}\) scores from individual training sets or lead to a moderate decrease in performance; however, in some cross-lingual cases the joint training improves performance, such as on zh_pud which improved from .410 using a model finetuned on en_ewt to .860. Finetuning on a diverse multilingual dataset helps preserve and even improve the performance on benchmarks in diverse languages.

## 6 Related Work

Adding a NER layer to UDSome single-language efforts have added a manually annotated NER layer to emerging or existing UD data. Agic and Ljubesic (2014) annotated the SETimes.HR dataset with linguistic and NER information, be

Figure 4: Heatmap of micro F\({}_{1}\) scores on test sets with different fine-tuned models. The y-axis indicates the dataset that the model is fine-tuned on, and the x-axis indicates the datasets that the models are evaluated on. **Left**: Model performance on datasets that contains the train, dev and test splits. The highlighted diagonal cells are the in-dataset results. **Center**: Model performance on the PUD datasets. **Right**: Model performance on all other datasets.

 

coming the set_hr UD dataset later (Agic and Ljubesic, 2015). Plank (2019) added a layer of NER to the dev and test portions of the Danish UD treebank (DDT) for cross-lingual evaluation; Plank et al. (2020) fully annotated it with nested NER entities. Hvingelby et al. (2020) annotated the same Danish UD data with a flat annotation scheme.

Other languages have seen efforts in a similar spirit. Jorgensen et al. (2020) added a named entity annotation layer on top of the Norwegian Dependency Treebank, Luoma et al. (2020) built the Turku NER corpus, and Plank (2021) added a layer on top of English EWT. Recently, Muischnek and Muurisep (2023) introduced the largest publicly available Estonian NER dataset. Complementing these efforts, Riabi et al. (2023) added several annotation layers, including NER, to the NArabizi treebank Seddah et al. (2020), a North-African Arabic dialect dataset written in Latin script with a high-level of language variability and code-switching.

Multilingual NER resourcesSeveral benchmark datasets for NER offer coverage for a variety of representative languages. Aside from well-known benchmarks such as CoNLL 2002/2003 Tjong Kim Sang (2002); Tjong Kim Sang and De Meulder (2003), other datasets were built to address a unique need, such as focusing on low-resource languages like LOREIEI Strassel and Tracey (2016) or incorporating particularly challenging annotations, as seen in MultiCoNER Malmasi et al. (2022a, b). MasakhaNER Adelani et al. (2022b) harnessed the _Masakhane_ community to produce gold-standard annotations for ten African languages.

Other datasets for multilingual and non-English NER use a silver-standard annotation process Nothman et al. (2013); Pan et al. (2017); Tedeschi et al. (2021). Nonetheless, CoNLL 2002/2003 remains one of the main benchmarks in multilingual NER. A recent work, also called UNER Alves et al. (2020), attempts to produce silver-standard corpora by propagating English annotations across parallel corpora but with no baseline evaluations. Lastly, another contemporary work called Universal NER Zhou et al. (2023) bears no relation to our effort as it contains no annotation component.

Modeling for multilingual NERSeveral works have explored the task of NER outside of English. The earliest build language-independent methods Cuccerzan and Yarowsky (1999); Lample et al. (2016), _inter alia_). Cross-lingual techniques have also emerged to transfer information between languages, especially from high- to low-resource languages Ruder et al. (2019) or combining model and data transfer across languages Wu et al. (2020). Currently, the standard paradigm for multilingual NER involves finetuning or prompting multilingual language models (e.g., Wu and Dredze, 2020; Muennighoff et al., 2023). UNER supports these modeling efforts by providing gold-standard annotations across various languages.

Community-driven annotation projectsThe field of NLP has been shaped by community-driven annotation projects. One prime example is the Universal Dependencies (UD) project Nivre et al. (2020), precipitated by the earlier introduction of the universal POS tagset McDonald et al. (2013). Extensions and sister projects to UD have emerged (e.g., Savary et al., 2023; Kahane et al., 2021), to which UNER is now added. Another notable endeavor is UniMorph Kirov et al. (2018); McCarthy et al. (2020), which covers 182 languages Batsuren et al. (2022, 2021). The Masakhane Project has also produced several high-quality community efforts Adelani et al. (2021, 2022); Dione et al. (2023),a).

The UNER project follows the same community-driven approach by asking volunteers to contribute annotations for their respective languages.

## 7 Conclusion

We introduce Universal NER (UNER), a gold-standard data initiative covering 13 languages for named entity recognition (NER). The datasets included in UNER v1 cover a wide variety of domains and language families, and we establish initial performance metrics for these benchmarks. UNER opens several opportunities for research

Figure 5: F\({}_{1}\) scores of each UNER test set after finetuning XLM-R\({}_{\text{Large}}\) on all training sets.

 

in NER outside of English and for cross-lingual transfer; in particular, this project provides human-annotated and standardized evaluations for multilingual NER.

After releasing the current version of the UNER project, we plan to expand language coverage and diversity of this effort by both recruiting additional annotators and integrating existing NER datasets when possible. This will also allow us to obtain more robust agreement measures and verify the quality of existing annotations in UNER. In the longer term, our aims for Universal NER include rigorous quality checking of annotation results for robustness and further integration of finetuned models and data analysis tools into the project.

## Limitations

Dataset Domains and LanguagesThe data included in UNER v1 covers a range of domains and languages, depending on the available annotators and datasets in UD (Appendix Table 5). The variance in domains and languages will generally affect the efficacy of cross-lingual learning and evaluation. However, we also provide a standardized, parallel evaluation set for a subset of the languages in UNER. Furthermore, we invite researchers who would like to see additional languages in UNER to join the annotation effort.

Springboarding from Universal DependenciesOur preliminary criterion for languages and data to be included in the current version of UNER is that it should be already in the Universal Dependencies (UD) [20]. This is to ensure the quality of the underlying data and to facilitate research in conjunction with existing UD treebanks, which include part-of-speech tags, tokenization, lemmas, and glosses. However, future iterations of the UNER initiative are open to all languages, especially low-resource ones, regardless of whether they are present in UD.

Number of AnnotatorsThe UNER project relies on crowd-sourcing and community participation for annotation efforts. Thus, the languages included have varying numbers of annotators who have accepted the invitation to contribute. Nonetheless, as reported in Table 2, each language has at least **two** annotators for a subset of its documents and thus a corresponding measure of inter-annotator agreement.

## Ethics Statement

Our annotated data is built on top of Universal Dependencies, an already established data resource. Thus, we do not foresee any serious or harmful issues arising from its content. Interested volunteer annotators who were invited to the project have also been informed of the guidelines as discussed in Section 3 for annotating NER-ready datasets before starting with the process.

## Acknowledgments

This project could not have happened without the enthusiastic response and hard work of many annotators in the NLP community, and for that we are extremely grateful. Annotators additional to authors are: Elyanah Aco, Ekaterina Artemova, Vuk Batanovic, Jay Rhald Caballes Padilla, Chunyuan Deng, Ivo-Pavao Jazbec, Juliane Karlsson, Jozef Kubik, Peter Krantz, Myron Darrel Montefalcon, Stefan Schweter, Sif Sonniks, Emil Stenstrom, Miriam Suppova. We would like to thank Joakim Nivre, Dan Zeman, Matthew Honnibal, Zeljko Agic, Constantine Lignos, and Amir Zeldes for early discussion and helpful ideas at the very beginning of this project. JMI is funded by National University Philippines and the UKRI Centre for Doctoral Training in Accountable, Responsible and Transparent AI [EP/S023437/1] of the University of Bath. Ariji Riabi is funded by the European Union's Horizon 2020 research and innovation programme under grant agreement No. 101021607. Marek Suppa was partially supported by the grant APVV-21-0114.

## References

* Adelani et al. (2018) David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester PalenMichel, Constantine Lignos, Jesujoba Alabi, Shamsuddeem Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Roowweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukibli, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatounata Kabore, Chris Chinney Emezue, Anouluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkhobatep, Victoire Memdijkam Koagne, Allahsera Augustse Tappo, Tebogo Macucwa, Vukosi Marviate, Mboning Tchiaze Elvis, Tsjuddeen Gwadabe, Tosin Adewwanti, Orevaognehia Ahia, Joyce Nakatumba-Nabende, Neo Lerato Mokono, Ignatius Ezeani, Chiamak Chukhwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, 

Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022. MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 4488-4508, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Adelani et al. (2016) David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzanaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Antoulwapo, Catherine Gitau, Derguene Mbaye, Jesuioba Alabi, Seid Muhie Yimam, Tajuddeeen Rabiu Gowadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukitib, Verrah Oriente, Iroro Orife, Davis David, Samba Ngom, Tosin deAdewumi, Paul Ryyson, Mofetoluwa Adeymi, Gerald Murituk, Emmanuel Anebi, Chiamak Chuwuneke, Nkirkuda Ohu, Eric Peter Wariagala, Samuel Overinde, Clemencia Siro, Tobius Saul Bateseal, Temilola Oloyede, Yvonne Wambui, Victor Akindoe, Deborah Nabagerka, Maurice Katussime, Ayodele Awokoya, Mohammadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaghene Aiha, Bonaventure F. P. Dosson, Kelechi Ogueji, Thiieronbrahma DIOP, Apodoulaye Diallo, Adevalae Akinfaderin, Tendai Menengereke, and Salomey Osei. 2021. MasakhaNER: Named entity recognition for African languages. _Transactions of the Association for Computational Linguistics_, 9:1116-1131.
* Agic and Ljubesic (2014) Zeljko Agic and Nikola Ljubesic. 2014. The SETimes.HR linguistically annotated corpus of Croatian. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)_, pages 1724-1727, Reykjavik, Iceland. European Language Resources Association (ELRA).
* Agic and Ljubesic (2015) Zeljko Agic and Nikola Ljubesic. 2015. Universal Dependencies for Croatian (that work for Serbian, too). In _The 5th Workshop on Balto-Slavic Natural Language Processing_, pages 1-8, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA.
* Alves et al. (2020) Diego Alves, Tin Kuculo, Gabriel Amaral, Gaurish Thakkar, and Marko Tadic. 2020. UNER: Universal Named-Entity Recognition Framework. _arXiv preprint arXiv:2010.12406_.
* Aquino et al. (2020) Angelina Aquino, Franz de Leon, and Mary Ann Bacold. 2020. UD_Tagalog-Ugnayan. [https://github.com/UniversalDependencies/UD_Tagalog-Ugnayan](https://github.com/UniversalDependencies/UD_Tagalog-Ugnayan).
* Aranes (2022) Glyd Aranes. 2022. The GJA Cebuano Treebank: Creating a Cebuano Universal Dependencies Treebank. Master's thesis, Ita-Suomen yliopisto.
* Bao et al. (2019) Zuyi Bao, Rui Huang, Chen Li, and Konny Zhu. 2019. Low-resource sequence labeling via unsupervised multilingual contextualized representations. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1028-1039, Hong Kong, China. Association for Computational Linguistics.
* Batanovic et al. (2018) Vuk Batanovic, Nikola Ljubesic, and Tanja Samardzic. 2018. Setimes.SR--A Reference Training Corpus of Serbian. In _Proceedings of the Conference on Language Technologies & Digital Humanities 2018 (JT-DH 2018)_, pages 11-17.
* Batsuren et al. (2021) Khuyagobaatar Batsuren, Gabor Bella, and Fausto Giunchiglia. 2021. MorphyNet: a large multilingual database of derivational and inflectional morphology. In _Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology_, pages 39-48, Online. Association for Computational Linguistics.
* Batsuren et al. (2021) Khuyagobaatar Batsuren, Omer Goldman, Salam Khalifa, Nizar Habash, Witold Kieras, Gabor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yusinus Ganggo Atre, Maria Ryskina, Sabrina Mielke, Elena Budianskaya, Charbel El-Khuasi, Tiago Piemnetl, Michael Gasser, William Abbott Lane, Mohit Raj, Matt Coljer, Jaime Rafael Montoya Samame, Delio Sticinotari Camaiteri, Esau Zumaeta Rojas, Didier Lopez Francis, Arturo Oncevy, Juan Lopez Bautista, Genn Celeste Silva Fillegas, Lucas Torroba Hennigen, Adam Ek, David Guriel, Peter Dirix, Jean-Philippe Bernardy, Andrey Scherpakov, Aziyana Bayyr-ool, Antonios Anastopoulos, Roberto Zariquiev, Karina Sheifer, Sofya Ganiveva, Hilaria Cruz, Ritvan Karahoga, Stella Markantonatou, George Pavlidis, Matvey Plugaryov, Elena Klyachko, Ali Salehi, Candy Angulo, Jatayu Bavi, Andrew Krizhanovsky, Natalia Krizhanovskaya, Elizabeth Salesky, Clara Vania, Sardana Ivanova, Jennifer White, Rowan Hall Maudsay, Josef Valvoda, Ran Zmigrod, Paula Czarnowska, Irene Nikkinarinen, Aelita Salchak, Brijesh Bhatt, Christopher Straughn, Zoey Liu, Jonathan North Washington, Yuval Pinter, Duygu Ataman, Marcin Wolinski, Totot Sukhatjianto, Anna Yablonskaya, Niklas Stoeth, Hossey Dolatian, Zahrob Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Eryaman Arora, Richard J. Hatcher, Ritesh Kumar, Jeremiah Young, Daria Rodionova, Anastasia Yemlena, Taras Andrushko, Igor Marchenko, Polina Mashkovtseva, Alexandra Serova, Emily Prud'hommeaux, Maria Nepominischaya, Fausto Giunchiglia, Eleanor Choroff, Mans Hudden, Mikika Silfverberg, Arya D. McCarthy, David Yarowsky, Ryan Coterell, Reut Tsafaty, and Ekaterina Vladomova. 2022. UniMorph 4.0: Universal Morphology. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 840-855, Marseille, France. European Language Resources Association.
* Chen et al. (2023a) Yang Chen, Chao Jiang, Alan Ritter, and Wei Xu. 2023a. Frustratingly easy label projection for cross-lingual 

transfer. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 5775-5796, Toronto, Canada. Association for Computational Linguistics.
* Chen et al. (2023b) Yang Chen, Vedaant Shah, and Alan Ritter. 2023b. Better Low-Resource Entity Recognition Through Translation and Annotation Fusion. _arXiv preprint arXiv:2305.13582_.
* Conneau et al. (2020) Alexis Conneau, Kartiaky Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Cucerzan and Yarowsky (1999) Silviu Cucerzan and David Yarowsky. 1999. Language independent named entity recognition combining morphological and contextual evidence. In _1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora_.
* de Marneffe et al. (2021) Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. _Computational Linguistics_, 47(2):255-308.
* Bolone et al. (2021) Cheikh M. Bamba Dione, David Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anouluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukibi, Blessing Sibanda, Bonaventure F. P. Dossou, Andissuw Bukula, Rooweither Mabuya, Allahsera Auguste Tappo, Edwin Munkoh-Baubeng, victoire Memdjokam Koagne, Fatourama Ouoba Kabore, Amelia Taylor, Godson Kalipe, Tobogo Macucwa, Vukosi Marivate, Tajudden Gowadabe, Mboning Tchiaze Elvis, Ikechukwu Onyenwe, Gratien Atinodgebe, Tolulope Adelani, Idris Akinade, Olanerwig Samuel, Marien Nahiman, Theogene Musabeyezun, Emile Niyomutakazi, Ester Chimhenga, Kudzai Gotosa, Patrick Mizha, Apelete Agbolo, Seydu Traore, Chinedu Uchechukwu, Aliyu Yusuf, Muhammad Abdullahi, and Dietrich Klakow. 2023a. Masakhapos: Part-of-speech tagging for typologically diverse african languages.
* Damone et al. (2021) Cheikh M. Bamba Dione, David Ifeoluwa Adelani, Peter Nabende, Jesusjobas Alabi, Thapedio Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezeue, Perez Ogayo, Anouluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukibib, Blessing Sibanda, Bonaventure F. P. Dossou, Andissuw Bukula, Roowerthe Tabuya, Allahsera Auguste Tapo, Edwin Munkoh-Bubaberg, Vicotiere Memdjokam Koagne, Fatounata Ouoba Kabore, Amelia Taylor, Godson Kalipe, Tebogo Macucwa, Vukosi Marivate, Tajudden Gowadabe, Mboning Tchiazevlis, Ikechukwu Onyenwe, Gratien Atinodgebe, Tolupole Adelani, Idris Akinade, Olannewaji Samuel, Marien Nahiman, Theogene Musabeyezun, Emile Nigyomutakaji, Ester Chimhenga, Khidevan Motwani, Christineine Gita, Deh Nguyen, Andrei Yuille, Yoshiyuki, and Dietrich Klakow. 2023b. MisakhaPOS: Part-of-speech tagging for typologically diverse African languages. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 10883-10900, Toronto, Canada. Association for Computational Linguistics.
* Grishman (2019) Ralph Grishman. 2019. Twenty-five years of information extraction. _Natural Language Engineering_, 25(6):677-692.
* Hvingelby et al. (2020) Rasmus Hvingelby, Amalie Brogaard Pauli, Maria Barrett, Christina Rosted, Lasse Malm Lidegaard, and Anders Sogaard. 2020. DaNE: A named resource for Danish. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 4597-4604, Marseille, France. European Language Resources Association.
* International Organization for Standardization (1998) International Organization for Standardization. 1998. Codes for the representation of names of languages--Part 2: alpha-3 code. Standard, International Organization for Standardization, Geneva, CH.
* International Organization for Standardization (2002) International Organization for Standardization. 2002. Codes for the representation of names of languages--Part 1: Alpha-2 code. Standard, International Organization for Standardization, Geneva, CH.
* Johannsen et al. (2015) Anders Johannsen, Hector Martinez Alonso, and Barbara Plank. 2015. Universal Dependence for Danish. In _International Workshop on Treebeanks and Linguistic Theories (TLT14)_, page 157.
* Jorgensen et al. (2020) Fredrik Jorgensen, Tobias Asamoe, Anne-Stene Ruud Husv\({\rm\acute{c}}\)g, Lilja Ovrelid, and Erik Velldal. 2020. NorNEx: Annotating named entities for Norwegian. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 4547-4556, Marseille, France. European Language Resources Association.
* Kahane et al. (2021) Sylvian Kahane, Bernard Caron, Emmett Strickland, and Kim Gerdes. 2021. Annotation guidelines of UD and SUD treebanks for spoken corpora: A proposal. In _Proceedings of the 20th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2021)_, pages 35-47, Sofia, Bulgaria. Association for Computational Linguistics.
* Khalid et al. (2008) Mahboob Alam Khalid, Valentin Jijkoun, and Maarten De Rijke. 2008. The impact of named entity normalization on information retrieval for question answering. In _European Conference on Information Retrieval_, pages 705-710. Springer.
* Kirov et al. (2018) Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Geraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sabrina J. Mielke, Arya McCarthy, 

Sandra Kubler, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. UniMorph 2.0: Universal Morphology. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Lample et al. [2016] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 260-270, San Diego, California. Association for Computational Linguistics.
* Lignos et al. [2022] Constantine Lignos, Nolan Holley, Chester Palennic Michel, and Jonne Saleva. 2022. Toward more meaningful resources for lower-resourced languages. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 523-532, Dublin, Ireland. Association for Computational Linguistics.
* Ljubesic et al. [2016] Nikola Ljubesic, Filip Klubicka, Zeljko Agic, and Ivo-Pavao Jazbec. 2016. New inflectional lexicons and training corpora for improved morphov syntactic annotation of Croatian and Serbian. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 4264-4270, Portoroz, Slovenia. European Language Resources Association (ELRA).
* Luoma et al. [2020]Jouni Luoma, Miika Oinonen, Maria Pyykonen, Veronika Laippala, and Sampo Pyysalo. 2020. A broad-coverage corpus for Finnish named entity recognition. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 4615-4624, Marseille, France. European Language Resources Association.
* Ma et al. [2023] Tingting Ma, Qianhui Wu, Huiqiang Jiang, Borje F. Karlsson, Tiejun Zhao, and Chin-Yew Lin. 2023. CoLaDa: A collaborative label denoising framework for cross-lingual named entity recognition. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics.
* Malmasi et al. [2022a] Shervin Malmasi, Anijie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022a. MultiCoNER: A large-scale multilingual dataset for complex named entity recognition. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 3798-3809, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Malmasi et al. [2022b] Shervin Malmasi, Anjie Fang, Bensik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022b. SemEval-2022 task 11: Multilingual complex named entity recognition (MultiCoNER). In _Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)_, pages 1412-1437, Seattle, United States. Association for Computational Linguistics.
* Mayhew and Roth [2018] Stephen Mayhew and Dan Roth. 2018. TALEN: Tool for annotation of low-resource ENtities. In _Proceedings of ACL 2018, System Demonstrations_, pages 80-86, Melbourne, Australia. Association for Computational Linguistics.
* McCarthy et al. [2020] Arya D. McCarthy, Christo Kirov, Matteo Grella, Amrit Nidhi, Patrick Xia, Kyle Gorman, Ekaterina Vylomova, Sabrina J. Milek, Garrett Nicolai, Miikka Silfverberg, Timofey Arkhandelskiy, Nately Krizhanovsky, Andrew Krizhanovsky, Elena Klyachko, Alexey Sorokin, John Mansfield, Valts Ernstreits, Yuval Pinter, Cassandra L. Jacobs, Ryan Cotterell, Mans Hulden, and David Yarowsky. 2020. UniMorph 3.0: Universal Morphology. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 3922-3931, Marseille, France. European Language Resources Association.
* McDonald et al. [2013] Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Takstrom, Claudia Bedini, Wiria Bertomeau Castello, and Jungmee Lee. 2013. Universal Dependency annotation for multilingual parsing. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 92-97, Sofia, Bulgaria. Association for Computational Linguistics.
* Molla et al. [2006] Diego Molla, Menno van Zaanen, and Daniel Smith. 2006. Named entity recognition for question answering. In _Proceedings of the Australasian Language Technology Workshop 2006_, pages 51-58, Sydney, Australia.
* Muennighoff et al. [2023]Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alhamn Fixri Ajfi, Khalil Almubarkas, Samuel Albaniazaid Alvarezfait, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15991-16111, Toronto, Canada. Association for Computational Linguistics.
* Muischnek and Mourisrep [2023] Kadri Muischnek and Kaili Mourisrep. 2023. Named entity layer in Estonian UD treebanks. In _Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaiDa)_, pages 179-184, Torshavan, Faroe Islands. University of Tartu Library.
* Nivre et al. [2016]Joakim Nivre, Marie-Catherine de Marneffe, Filip Gunter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 1659-1666, Portoroz, Slovenia. European Language Resources Association (ELRA).

 

Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajic, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 4034-4043, Marseille, France. European Language Resources Association.
* Nothman et al. (2013) Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R. Curran. 2013. Learning multilingual named entity recognition from Wikipedia. _Artificial Intelligence_, 194:151-175.
* Pan et al. (2017) Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1946-1958, Vancouver, Canada. Association for Computational Linguistics.
* Petrov et al. (2012) Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)_, pages 2089-2096, Istanbul, Turkey. European Language Resources Association (ELRA).
* Plank (2019) Barbara Plank. 2019. Neural cross-lingual transfer and limited annotated data for named entity recognition in Danish. In _Proceedings of the 22nd Nordic Conference on Computational Linguistics_, pages 370-375, Turku, Finland. Linkoping University Electronic Press.
* Plank (2021) Barbara Plank. 2021. Cross-lingual cross-domain nested named entity evaluation on English web texts. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 1808-1815, Online. Association for Computational Linguistics.
* Plank et al. (2020) Barbara Plank, Kristian Norgaard Jensen, and Rob van der Goot. 2020. DaN+: Danish nested named entities and lexical normalization. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 6649-6662, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Qi and Yasuoka (2019) Peng Qi and Koichi Yasuoka. 2019. UD_Chinese-GSDSimp. [https://github.com/UniversalDependencies/UD_Chinese-GSDSimp](https://github.com/UniversalDependencies/UD_Chinese-GSDSimp).
* Rademaker et al. (2017) Alexandre Rademaker, Fabricio Chalub, Livy Real, Claudia Freitas, Eckhard Bick, and Valeria de Paiva. 2017. Universal Dependencies for Portuguese. In _Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017)_, pages 197-206, Pisa, Italy. Linkoping University Electronic Press.
* Riabi et al. (2023) Arij Riabi, Menel Mahamdi, and Djame Seddah. 2023. Enriching the NArabizi treebank: A multifaceted approach to supporting an under-resourced language. In _Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII)_, pages 266-278, Toronto, Canada. Association for Computational Linguistics.
* Ruder et al. (2019) Sebastian Ruder, Anders Sogaard, and Ivan Vulic. 2019. Unsupervised cross-lingual representation learning. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts_, pages 31-38, Florence, Italy. Association for Computational Linguistics.
* Samardzic et al. (2017) Tanja Samardzic, Mirjana Starovic, Zeljkog Agic, and Nikola Ljubesic. 2017. Universal Dependencies for Serbian in comparison with Croatian and other Slavic languages. In _Proceedings of the 6th Workshop on Balto-Slavic Natural Langauge Processing_, pages 39-44, Valencia, Spain. Association for Computational Linguistics.
* Samson and Coltekin (2020) Stephanie Samson and Cagri Coltekin. 2020. UD_Tagalog-TRG. [https://github.com/UniversalDependencies/UD_Taglag-TRG](https://github.com/UniversalDependencies/UD_Taglag-TRG).
* Savary et al. (2023) Agata Savary, Cherifa Ben Khelil, Carlos Ramisch, Voula Giouli, Verginica Barbu Mititelu, Najet Hadj Mohamed, Cvetana Krstev, Chaya Liebeskind, Hongzhi Xu, Sara Stymme, Tunga Gungor, Thomas Pickard, Bruno Guillaume, Eduard Bejck, Archna Bhatia, Mario Canidito, Polona Gantaur, Uxoa Iturriaeta, Albert Gutt, Jolanta Kovalevskaite, Timm Lichte, Nikola Ljubesic, Johanna Monti, Carla Parra Escartin, Mehrnoush Shamsfard, Iveling Stoyanova, Veronika Vincze, and Abigail Walsh. 2023. PARSEME corpus release 1.3. In _Proceedings of the 19th Workshop on Multiword Expressions (MWE 2023)_, pages 24-35, Dubrovnik, Croatia. Association for Computational Linguistics.
* Seddah et al. (2020) Djame Seddah, Farah Essaidi, Amal Fethi, Matthieu Futeral, Benjamin Muller, Pedro Javier Ortiz Suarez, Benoft Sagot, and Abhishek Srivastava. 2020. Building a user-generated content North-African Arabiz treebank: Tackling hell. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1139-1150, Online. Association for Computational Linguistics.
* Shen et al. (2016) Mo Shen, Ryan McDonald, Daniel Zeman, and Peng Qi. 2016. UD_Chinese-GSD. [https://github.com/UniversalDependencies/UD_Chinese-GSD](https://github.com/UniversalDependencies/UD_Chinese-GSD).
* Silveira et al. (2014) Natalia Silveira, Timothy Bozat, Marie-Catherine de Marneffe, Samuel Bowman, Mirian Connor, John Bauer, and Chris Manning. 2014. A gold standard dependency corpus for English. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)_, pages 2897-2904, Reykjavik, Iceland. European Language Resources Association (ELRA).
* Strassel and Tracey (2016) Stephanie Strassel and Jennifer Tracey. 2016. LORELEI language packs: Data, tools, and resources for technology development in low resource languages. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation 

(LREC'16)_, pages 3273-3280, Portoroz, Slovenia. European Language Resources Association (ELRA).
* Tedeschi et al. [2021] Simone Tedeschi, Valentino Maiorca, Niccolo Campolungo, Francesco Cecconi, and Roberto Navigli. 2021. WikiNEuRal: Combined neural and knowledge-based silver data creation for multilingual NER. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 2521-2533, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Tong Kang [2002] Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In _COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
* Tang et al. [2003] Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_, pages 142-147.
* Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacinne Jernite, Julien Plu, Vanwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lihoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Wu et al. [2020a] Qianhui Wu, Zijia Lin, Borje Karlsson, Jian-Guang Lou, and Biqing Huang. 2020a. Single-/multi-source cross-lingual NER via teacher-student learning on unlabeled data in target language. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6505-6514, Online. Association for Computational Linguistics.
* Wu et al. [2020b] Qianhui Wu, Zijia Lin, Borje F. Karlsson, Biqing Huang, and Jian-Guang Lou. 2020b. Unitrans : Unifying model transfer and data transfer for cross-lingual named entity recognition with unlabeled data. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_. International Joint Conferences on Artificial Intelligence Organization.
* Wu and Dredze [2020] Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In _Proceedings of the 5th Workshop on Representation Learning for NLP_, pages 120-130, Online. Association for Computational Linguistics.
* Zeman [2017] Daniel Zeman. 2017. Slovak dependency treebank in universal dependencies. _Journal of Linguistics/Jazkyovedny caospis_, 68(2):385-395.
* Zenman et al. [2017] Daniel Zenman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Girter, Juhani Luotolahit, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Neddouzhko, Silvie Cinkova, Jan Hajic jr., Jaroslava Hlavacova, Vaclava Kettnerova, Zdenka Uresova, Jenna Kanerva, Stina Ojala, Anna Missila, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marmeffe, Manuela Sanguinetti, Maria Simi, Hiroshi Anasyama, Valeria de Paiva, Kira Droganova, Hector Martinez Alonso, Cagri Coltekin, Umut Sulubacak, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim Harris, Katrin Marheinecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkhaky, Zhuoran Yu, Emily Pitler, Saran Leptradit, Michael Mandl, Jesse Kirchner, He Actor Fernandez Alcalde, Jana Strnadova, Esha Banerjee, Ruli Marumus, Antonio Stella, Atsuko Shimada, Soodyoung Kwak, Gustavo Mendonca, Tatiana Lando, Rattima Nitisaroj, and Josie Li. 2017. CoNLL 2017 shared task: Multilingual parsing from raw text to Universal Dependencies. In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_, pages 1-19, Vancouver, Canada. Association for Computational Linguistics.
* Zhou et al. [2023] Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2023. UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition. _arXiv preprint arXiv:2308.03279_.

 

Contributions

**Stephen Mayhew** conception, kickoff, all initial organization, recruitment, and annotation, development of annotation tool, manuscript writing.

**Terra Blevins** annotation, organization, PUD analysis scripts, core manuscript writing.

**Shuheng Liu** annotation, all baseline experiments and analysis.

**Marek Suppa** annotation, PROPN analysis, paper writing, GPU resources.

**Hila Gonen** advising, organization of and feedback on manuscript.

**Joseph Marvin Imperial** facilitated annotations for Tagalog and Cebuano, additions to manuscript for TL/CEB results, limitations, ethics, and conclusion sections.

**Borje F. Karlsson** annotation, manuscript writing and editing, advising.

**Peiqin Lin** annotation, manuscript comments.

**Nikola Ljubesic** preparation and transfer of the HR set and SR set datasets, manuscript comments and edits.

**LJ Miranda** annotation, related work section, comments, edits.

**Barbara Plank** preparation and transfer of the DA ddt dataset, manuscript writing, comments and edits.

**Arij Riabi** preparation and transfer of North African Arabizi dataset, related work section, comments and edits.

**Yuval Pinter** advising, organization of and writing of manuscript.

## Appendix B Additional Dataset Details

In this section, we provide additional statistics and analysis of the datasets included in UNER v1. Table 5 documents the domains included in each dataset along with their distributions of NER tags, and Table 4 presents the F1 overlap score between named entities in UNER and propn tags in the underlying UD treebanks. We also report the full numerical results of our baseline experiments in Table 6.

## Appendix C Dataset Licensing

The Universal Dependencies datasets are licensed under Creative Commons Attribution-ShareAlike (CC BY-SA)9. This license requires that "if you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original." Thus, we distribute all of our datasets under the same license.

Footnote 9: Creativecommons.org/licenses/by-sa/4.0/ 



\begin{table}
\begin{tabular}{l l c r r r r r} \hline \hline  & Source & \multicolumn{3}{c}{Target} & \multicolumn{4}{c}{F\({}_{1}\)} \\ \cline{2-9} Lang. & Dataset & Lang. & Dataset & LOC & ORG & PER & Overall \\ \hline DA & ddt & DA & ddt & .879 & .826 & .924 & .878 \\ EN & ewt & EN & ewt & .871 & .709 & .950 & .858 \\ HR & set & HR & set & .977 & .891 & .970 & .950 \\ SK & snk & SK & snk & .846 & .635 & .882 & .855 \\ PT & bosque & PT & bosque & .882 & .861 & .966 & .904 \\ QAF & arabizi & QAF & arabizi & .780 & .520 & .717 & .711 \\ SR & set & SR & set & .981 & .913 & .983 & .962 \\ SV & talbanken & SV & talbanken & .904 & .742 & .928 & .883 \\ ZH & gsd & ZH & gsd & .906 & .819 & .922 & .896 \\ ZH & gsdsimp & ZH & gsdsimp & .906 & .802 & .925 & .894 \\ \hline  & & DE & pud & .816 & .603 & .893 & .814 \\  & & EN & pud & .785 & .593 & .922 & .805 \\ EN & ewt & PT & pud & .845 & .698 & .914 & .848 \\ RU & pud & .681 & .451 & .875 & .715 \\  & & SV & pud & .887 & .655 & .928 & .872 \\  & & ZH & pud & .465 & .308 & .389 & .410 \\ \hline  & & CEB & gja & .556 & .000 & .842 & .789 \\ EN & ewt & TL & trg & 1.00 & – .923 & .936 \\  & & TL & ugnayan & .857 & – .000 & .783 \\ \hline \hline \end{tabular}
\end{table}
Table 6: The full results of our baseline experiments from finetuning XLM-R\({}_{\text{Largo}}\) on UNER. All scores are reported in micro-F\({}_{1}\). ORG F\({}_{1}\) scores are not reported for the two TL datasets since there are no ORG entities labeled.

\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline  & & & & & Entity Dist. (\%) \\ Data Source & Lang. & Dataset & Domains & LOC & ORG & PER \\ \hline Johannsen et al. (2015) & da & ddt & fiction, news, nonfiction, spoken & 28.0 & 30.8 & 41.2 \\ Silveira et al. (2014) & en & ewt & blog, email, reviews, social, web & 37.8 & 21.8 & 40.4 \\ Agic and Ljubesić (2015) & hr & set & news, web, wiki & 37.4 & 33.0 & 29.6 \\ Rademaker et al. (2017) & pt & bosque & news & 29.5 & 33.9 & 36.6 \\ Seddah et al. (2020) & qaf & arabizi & blog, web, social & 57.5 & 27 & 15.4 \\ Zeman (2017) & sk & snk & fiction, news, nonfiction & 21.2 & 6.2 & 72.6 \\ Samardzić et al. (2017) & sr & set & news & 41.4 & 30.2 & 28.4 \\ McDonald et al. (2013) & sv & talbanken & news, nonfiction & 54.0 & 20.0 & 25.0 \\ Shen et al. (2016) & zh & gsd & wiki & 48.1 & 17.9 & 34.0 \\ Qi and Yasuoka (2019) & & gsdsimp & wiki & 48.0 & 18.0 & 34.0 \\ \hline  & de & pud & news, wiki & 41.3 & 18.5 & 40.2 \\  & en & pud & news, wiki & 39.5 & 21.9 & 38.6 \\ Zeman et al. (2017) & pt & pud & news, wiki & 44.4 & 18.0 & 37.6 \\ Ru & pud & news, wiki & 33.4 & 26.6 & 40.0 \\  & sy & pud & news, wiki & 43.0 & 15.7 & 41.3 \\  & zh & pud & news, wiki & 44.9 & 13.5 & 41.6 \\ \hline Aranes (2022) & ceb & gja & grammar examples & 12.3 & 2.0 & 85.7 \\ Samson and Coltekin (2020) & tl & trg & grammar examples & 10.9 & 0.0 & 89.1 \\ Aquino et al. (2020) & ugnayan & fiction, nonfiction & 47.5 & 0.0 & 52.5 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Domains and distribution of entity types for datasets in UNER. Domains are categorized for the underlying UD datasets at [https://universaldependencies.org/](https://universaldependencies.org/).