# Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark

Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Šuppa, Hila Gonen, Joseph Marvin Imperial, Börje F Karlsson, Peiqin Lin, Nikola Ljubešić, L J Miranda, et al.

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Stephen Mayhew, Terra Blevins, Shuheng Liu, Marek Šuppa, Hila Gonen, et al. Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark. 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Jun 2024, Mexico city, Mexico. 10.7910/DVN/GQ8HDL. hal-04630484

HAL Id: hal-04630484

https://inria.hal.science/hal-04630484v1

Submitted on 1 Jul 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
Universal NER: A Gold-Standard Multilingual
Named Entity Recognition Benchmark

Stephen Mayhew<sup>α</sup> Terra Blevins<sup>β</sup> Shuheng Liu<sup>γ</sup> Marek Šuppa<sup>δε</sup>
Hila Gonen<sup>β</sup> Joseph Marvin Imperial<sup>ζ,η</sup> Börje F. Karlsson<sup>θ</sup> Peiqin Lin<sup>τ</sup>
Nikola Ljubešić<sup>κ</sup> LJ Miranda<sup>λ</sup> Barbara Plank<sup>τ,μ</sup> Arij Riabi<sup>ν,ξ</sup> Yuval Pinter<sup>o</sup>

αDuolingo, βUniversity of Washington, γGeorgia Institute of Technology,

δComenius University in Bratislava, €Cisco, ζNational University Philippines,

ηUniversity of Bath, θBeijing Academy of Artificial Intelligence,

«LMU Munich, κJožef Stefan Institute, λAllen Institute for Artificial Intelligence,

μIT University of Copenhagen, νInria Paris, ξSorbonne Université, °Ben-Gurion University

stephen@duolingo.com

blvns@cs.washington.edu

Abstract

We introduce Universal NER (UNER), an open,
community-driven project to develop gold-
standard NER benchmarks in many languages.
The overarching goal of UNER is to provide
high-quality, cross-lingually consistent annota-
tions to facilitate and standardize multilingual
NER research. UNER v1 contains 19 datasets
annotated with named entities in a cross-lingual
consistent schema across 13 diverse languages.
In this paper, we detail the dataset creation and
composition of UNER; we also provide initial
modeling baselines on both in-language and
cross-lingual learning settings. We will release
the data, code, and fitted models to the public.<fnref n="1" />

# 1 Introduction

High-quality data in many languages is necessary
for broadly multilingual natural language process-
ing. In named entity recognition (NER), the ma-
jority of annotation efforts are centered on En-
glish, and cross-lingual transfer performance re-
mains brittle (e.g., Chen et al., 2023b; Ma et al.,
2023). Amongst non-English human-annotated
NER datasets, while there have been multiple sepa-
rate efforts in this front (e.g., Agić and Ljubešić,
2014; Plank, 2019; Adelani et al., 2022), these ei-
ther have disjoint annotation schemes and labels,
cover a single language or small set of related lan-
guages, or are not widely accessible (e.g., Strassel
and Tracey, 2016). For most of the world’s lan-
guages, the only readily available NER data is the
automatically annotated WikiANN dataset (Pan
et al., 2017), though this annotation paradigm intro-
duces data quality issues and limits its usefulness
for evaluation (Lignos et al., 2022).

To address this data gap, we propose Univer-
sal NER (UNER), an open community effort to

<figure><img src="image_3.png" /><figcaption>Figure 1: Parallel sentences annotated with person (PER) and location (LOC) named entities in English (EN), German (DE), Russian (RU), and Chinese (ZH).</figcaption></figure>

develop gold-standard named entity recognition
benchmarks across many languages. Each dataset
in Universal NER is annotated by primarily na-
tive speakers on the text of an existing Universal
Dependencies treebank (UD; Nivre et al., 2020).
Inspired by Universal Dependencies, the overarch-
ing philosophy of the UNER project is to provide a
shared, universal definition, tagset, and annotation
schema for NER that is broadly applicable across
languages (Figure 1).

The current version of Universal NER,
UNER v1, contains 19 datasets spanning 13
languages (Section 4). To establish performance
baselines on UNER, we finetune an XLM-R model
on various training configurations (Section 5)and
show that while NER transfer performance
between European languages is relatively strong,
there remains a gap when transferring to different
scripts or language typologies.

The goal of the UNER project is to facilitate
multilingual research on entity recognition by ad-
dressing the need in the multilingual NLP commu-
nity for standardized, cross-lingual, and manually
annotated NER data. With the release of UNER
v1, we plan to expand UNER to new languages
and datasets, and we welcome all new annotators
interested in developing the project.

<footnote n="1">¹https://www.universalner.org, UNER v1 available
at https://doi.org/10.7910/DVN/GQ8HDL</footnote>
## 2 Dataset Design Principles

Named entity recognition (NER) is the task of identifying text spans in a given context that uniquely refer to specific *named entities*. The task of NER has a long tradition (Grishman, 2019) and facilitates many downstream NLP applications, such as information retrieval (Khalid et al., 2008) and question answering (Mollá et al., 2006). Furthermore, successful NER tagging requires a model to reason about semantic and pragmatic world knowledge, which makes the task an informative evaluation setting for testing NLP model capabilities.

As with Universal Dependencies, the goal of Universal NER is to develop an annotation schema that can work in any language. Traditionally, the UD (Nivre et al., 2016) and UPOS (Petrov et al., 2012) projects have chosen what amounts to the intersection of tags across all language-specific tagsets, keeping the resultant tagset broad and simple. We follow a similar strategy, picking tags that broadly cover the space of proper nouns.

Universal NER's annotation schema emphasizes three coarse-grained entity types: Person (PER), Organization (ORG), and Location (LOC). We provide a short description and an example for each tag.

**PER** The PERSON tag includes names of people, real or fictional, but not nominals.

“Mr. Robinson PER smiled at the teacher.”

**ORG** The ORGANIZATION tag is used for named collections of people.

“The FDA ORG announced time travel pills tomorrow.”

**LOC** The LOCATION tag covers all types of named locations.

“I will arise and go now, and go to Innisfree LOC”

Figure 1 demonstrates how named entities and their corresponding annotations surface across languages. In some cases (such as in the English and German sentences), the surface forms of named entities are shared. However, often these forms vary—as in the Russian and Chinese examples—which makes entity identification and tagging more challenging, particularly in cross-lingual settings.

**Annotation Guidelines** In preparation for annotation, we developed extensive annotation guidelines,² using the NorNE project guidelines (Jørgensen et al., 2020) as a starting point. Along with

tag descriptions, our guidelines include many examples, as well as instructions for dealing with ambiguity and unclear constructions, such as email addresses, pet names, and typographical errors.

We expect that the guidelines will be further refined and updated as annotation proceeds. To manage this, we track version numbers and changelogs for different iterations of the guidelines. Each data release will include the corresponding annotation guidelines at the time of release.

## 3 Dataset Annotation Process

Having described the theoretical basis for the tagset, we now discuss the mechanics of annotation.

**Sourcing Data** We chose the Universal Dependency corpora as the default base texts for annotation. This jumpstarts the process: there is high coverage of languages, and the data is already collected, cleaned, tokenized, and permissively licensed. Further, by adding an additional annotation layer onto an already rich set of annotations, we not only support verification in our project (Section 4.3) but also enable multilingual research on the full pipeline of core NLP. Since UD is annotated at the word level, we follow a BIO annotation schema (specifically IOB2), where words forming the beginning (inside) part of an X entity (X ∈ {PER, LOC, ORG}) are annotated B-X (I-X, respectively), and all other words are given an O tag. For the sake of continuity, we preserve all tokenization from UD.

While UD is the default data source for UNER, we do not limit the project to UD corpora (particularly for languages not currently included in UD). The only criterion for inclusion in the UNER corpus is that the tagging schema matches the UNER guidelines. We are also open to converting existing NER efforts on UD treebanks to UNER. In this initial release, we include four datasets that are transferred from other manual annotation efforts on UD sources (for DA, HR, ARABIZI, and SR).

**Sourcing Annotators** For the initial UNER annotation effort, we recruited annotators from the multilingual NLP community through academic networks on social media. Annotators were organized via channels in a Slack workspace. Annotators of the datasets included in UNER thus far are unpaid volunteers. We expect that annotators are native speakers of their annotation language, or are highly proficient, but we did not issue any language tests. For the first release of UNER, the choice of

²http://www.universalner.org/guidelines/
the 13 dataset languages is solely dependent on the availability of annotators. As the project continues, we expect that additional languages and datasets will be added as annotators in more languages become available to contribute.

**Annotation Tool** We collect annotations for the UD treebanks using TALEN (Mayhew and Roth, 2018), a web-based tool for span-level sequence labeling.<fnref n="3" /> TALEN includes an optional feature that propagates annotations – if the user annotates “McLovin” in one section of the document, every other instance of “McLovin” in that document is annotated as well. This significantly speeds up annotation but risks over-annotation mistakes. For example, consider the token “US”, which may appear with different senses in contexts such as “The US economy…” or “THEY OFFERED TO BUY US LUNCH!”

**Secondary Annotators** In addition to collecting a complete set of annotations from a primary annotator for each dataset, we also gather secondary annotations from another annotator on (at least) a subset of the data in order to estimate inter-annotator agreement (Section 4.2). We aim for at least 5% coverage of each data split with these secondary annotations, although most datasets have significantly more (Table 2). When a document has multiple annotators, we include the labels from the annotator with the most entities annotated in that document in the final dataset. This means a dataset may have multiple annotators, but each document has exactly one. We retain annotator identities in the data files.

**Annotation Differences and Resolution** When annotators disagreed on annotation decisions or the inter-annotator agreement scores were low, we encouraged them to discuss the disagreements and decide if they were conflicting interpretations of the guidelines or fundamental disagreements. In the former case, annotators came to an agreement on guideline interpretations and updated annotations accordingly. In the latter, the annotations were kept as-is. Not every dataset had this resolution process.

The multilingual nature of this process also highlighted cross-language differences in named entities that affect NER annotation. For instance, most languages in UNER use capitalization as a marker of proper nouns and, therefore, named entities. However, Chinese does not include capitalization in its script, which makes identifying named

entities more difficult and time-consuming than in other languages, potentially leading to more annotation errors. Differences in annotating NER across languages also stem from divergent definitions of proper nouns (PROPN) by language and the effects of translation artifacts; these issues are discussed further in Sections 4.3 and 4.4, respectively.

**OTHER Tag** As a helpful check for annotators, we allow the option of annotating a fourth entity type, Other (OTH), which is not included in the final dataset. This had several purposes: to store annotations that behaved like mentions, but didn't conform to the guidelines of the other tags; to measure potential annotation disagreement on ambiguous cases; and to store an additional layer of annotation. Not all annotators used it, and those that did were sometimes inconsistent. In practice, OTH was most often applied to languages, nationalities, and brands. The OTH tag roughly corresponds to the MISC tag used in CoNLL 2003, which has been described as being “ill-defined” (Adelani et al., 2022).

**Dataset Transfer** Most of the included datasets are annotated from scratch using the annotation process detailed above, but a few (DA ddt, QAF arabizi, HR and SR set) are transferred from other sources. The Danish ddt annotations are derived from the News portion of the DaN+ dataset (Plank et al., 2020); this text corresponds to the Universal Dependencies ddt treebank. The Croatian hr annotations come from the hr500k dataset (Ljubešić et al., 2016), half of which, consisting of newspaper and various web texts, was used for producing the Croatian Universal Dependencies hr_set treebank (Agić and Ljubešić, 2015). The NArabizi arabizi dataset was annotated on UD data using a slightly different NER schema and then automatically converted to the UNER schema. The Serbian sr data come from the SETimes. SR dataset (Batanović et al., 2018), which was used in its fullness to produce the Serbian Universal Dependencies sr_set treebank (Samardžić et al., 2017). The original Croatian and Serbian NER annotations were annotated and curated in multiple iterations by various native speakers. However, the annotations added to the UNER dataset were slightly modified to conform to the UNER annotation guidelines; namely, while nationalities and similar groups are annotated as PER in the original dataset, in the UNER dataset such entities are omitted. Finally, we retain the original annotations from existing NER datasets in the “xner” label column.

<footnote n="3">³https://github.com/mayhewsw/talen-react</footnote>
<table><caption>Table 1: Universal NER has broad coverage of named entities in several languages and domains, adding annotations to the development, testing, and training sets from Universal Dependencies (Nivre et al., 2020).</caption><thead><tr><th rowspan="2">Lang.</th><th rowspan="2">Dataset</th><th colspan="4">Sentences</th><th colspan="4">Entities</th><th colspan="4">Tokens</th></tr><tr><th>Train</th><th>Dev</th><th>Test</th><th>All</th><th>Train</th><th>Dev</th><th>Test</th><th>All</th><th>Train</th><th>Dev</th><th>Test</th><th>All</th></tr></thead><tbody><tr><td>DA</td><td>ddt</td><td>4,383</td><td>564</td><td>565</td><td>5,512</td><td>3,022</td><td>379</td><td>446</td><td>3,847</td><td>80,378</td><td>10,332</td><td>10,023</td><td>100,733</td></tr><tr><td>EN</td><td>ewt</td><td>12,543</td><td>2,001</td><td>2,077</td><td>16,621</td><td>7,022</td><td>966</td><td>1,088</td><td>9,076</td><td>204,579</td><td>25,149</td><td>25,097</td><td>254,825</td></tr><tr><td>HR</td><td>set</td><td>6,914</td><td>960</td><td>1,136</td><td>9,010</td><td>8,261</td><td>1,218</td><td>1,403</td><td>10,882</td><td>152,857</td><td>22,292</td><td>24,260</td><td>199,409</td></tr><tr><td>PT</td><td>bosque</td><td>7,018</td><td>1,172</td><td>1,167</td><td>9,357</td><td>8,101</td><td>1,401</td><td>1,215</td><td>10,717</td><td>171,776</td><td>28,447</td><td>27,604</td><td>227,827</td></tr><tr><td>QAF</td><td>arabizi</td><td>1003</td><td>139</td><td>145</td><td>1287</td><td>1320</td><td>204</td><td>194</td><td>1718</td><td>15,522</td><td>2,124</td><td>2,118</td><td>19,764</td></tr><tr><td>SK</td><td>snk</td><td>8,483</td><td>1,060</td><td>1,061</td><td>10,604</td><td>2,707</td><td>636</td><td>915</td><td>4,258</td><td>80,628</td><td>12,733</td><td>12,736</td><td>106,097</td></tr><tr><td>SR</td><td>set</td><td>3,328</td><td>536</td><td>520</td><td>4,384</td><td>5,020</td><td>742</td><td>847</td><td>6,609</td><td>74,259</td><td>11,993</td><td>11,421</td><td>97,673</td></tr><tr><td>SV</td><td>talbanken</td><td>4,303</td><td>504</td><td>1,219</td><td>6,026</td><td>967</td><td>23</td><td>196</td><td>1,186</td><td>66,646</td><td>9,797</td><td>20,377</td><td>96,820</td></tr><tr><td>ZH</td><td>gsd</td><td>3,997</td><td>500</td><td>500</td><td>4,997</td><td>6,136</td><td>754</td><td>767</td><td>7,657</td><td>98,616</td><td>12,663</td><td>12,012</td><td>123,291</td></tr><tr><td></td><td>gsdsimp</td><td>3,997</td><td>500</td><td>500</td><td>4,997</td><td>6,118</td><td>753</td><td>763</td><td>7,634</td><td>98,616</td><td>12,663</td><td>12,012</td><td>123,291</td></tr><tr><td>DE</td><td>pud</td><td>-</td><td>-</td><td>1,000</td><td>1,000</td><td>-</td><td>-</td><td>1,039</td><td>1,039</td><td>-</td><td>-</td><td>21,331</td><td>21,331</td></tr><tr><td>EN</td><td>pud</td><td>-</td><td>-</td><td>1,000</td><td>1,000</td><td>-</td><td>-</td><td>1,038</td><td>1,038</td><td>-</td><td>-</td><td>21,176</td><td>21,176</td></tr><tr><td>PT</td><td>pud</td><td>-</td><td>-</td><td>1,000</td><td>1,000</td><td>-</td><td>-</td><td>1,099</td><td>1,099</td><td>-</td><td>-</td><td>23,407</td><td>23,407</td></tr><tr><td>RU</td><td>pud</td><td>-</td><td>-</td><td>1,000</td><td>1,000</td><td>-</td><td>-</td><td>1,036</td><td>1,036</td><td>-</td><td>-</td><td>19,355</td><td>19,355</td></tr><tr><td>SV</td><td>pud</td><td>-</td><td>-</td><td>1,000</td><td>1,000</td><td>-</td><td>-</td><td>1,029</td><td>1,029</td><td>-</td><td>-</td><td>19,076</td><td>19,076</td></tr><tr><td>ZH</td><td>pud</td><td>-</td><td>-</td><td>1,000</td><td>1,000</td><td>-</td><td>-</td><td>1,137</td><td>1,137</td><td>-</td><td>-</td><td>21,415</td><td>21,415</td></tr><tr><td>CEB</td><td>gja</td><td>-</td><td>-</td><td>188</td><td>188</td><td>-</td><td>-</td><td>49</td><td>49</td><td>-</td><td>-</td><td>1,295</td><td>1,295</td></tr><tr><td>TL</td><td>trg</td><td>-</td><td>-</td><td>128</td><td>128</td><td>-</td><td>-</td><td>92</td><td>92</td><td>-</td><td>-</td><td>734</td><td>734</td></tr><tr><td></td><td>ugnayan</td><td>-</td><td>-</td><td>94</td><td>94</td><td>-</td><td>-</td><td>61</td><td>61</td><td>-</td><td>-</td><td>1,097</td><td>1,097</td></tr></tbody></table>

# 4 Universal NER: Statistics and Analysis

This section presents an overview of the Universal NER (UNER) dataset. UNER v1 adds a NER annotation layer to 19 datasets (primarily treebanks from UD). It covers 13 geneologically and typologically diverse languages: Cebuano, Danish, German, English, Croatian, Narabizi, Portuguese, Russian, Slovak, Serbian, Swedish, Tagalog, and Chinese<fnref n="4" />. Overall, UNER v1 contains ten full datasets with training, development, and test splits over nine languages, three evaluation sets for lower-resource languages (TL and CEB), and a parallel evaluation benchmark spanning six languages.

## 4.1 Dataset Statistics

In Table 1, we report the number of sentences, tokens, and annotated entities for each dataset in UNER. The datasets in UNER cover a wide range of data quantities: some provide a limited amount of evaluation data for a commonly low-resourced language, whereas others annotate thousands of training and evaluation sentences.

The datasets in UNER also cover a diverse range of domains, spanning web sources such as social media to more traditional provenances like news text. Table 5 in the appendix presents the complete set of sources for the data and the distribution of NER tags in each dataset, along with references to

each original treebank paper. The variety in data sources leads to varied distributions of tags across datasets (Figure 2).

## 4.2 Inter-Annotator Agreement

We calculate inter-annotator agreement (IAA, Table 2) for each dataset in UNER that was annotated with the above process and for which we have secondary annotations. Table 2 reports agreement as per-label $F_1$ score, using one annotator as “reference,” and the other as “prediction.”

**ORG vs LOC Confusion** The agreement on ORG and LOC is generally lower than that on PER. The annotation guidelines allow certain named entities to take either the ORG or LOC tag based on context. In some cases, the context is underspecified, leading to ambiguity. For example, a restaurant is a LOC when you go there to eat, but it is an ORG when it hires a new chef. A city is a LOC when you move there, but it is an ORG when it levies taxes. Officially, it is the

<figure><img src="image_4.png" /><figcaption>Figure 2: Distribution of tags in different UNER training sets. zh_gdsimp has the same distribution as zh_gsd.</figcaption></figure>

<footnote n="4">⁴Languages sorted by their ISO 639-1/639-2 codes (International Organization for Standardization, 2002, 1998)</footnote>
<table><caption>Table 2: Inter-annotator agreement scores for the datasets annotated natively for the Universal NER project. We don't report IAA for the datasets adapted from other sources, or from zh_gdsimp, which has nearly identical annotations to zh_gsd. % Docs refers to the percentage of documents annotated by multiple annotators.</caption><thead><tr><th rowspan="2">Lang.</th><th rowspan="2">Dataset</th><th colspan="4">Train</th><th colspan="4">Dev</th><th colspan="4">Test</th></tr><tr><th>LOC</th><th>ORG</th><th>PER</th><th>% Docs</th><th>LOC</th><th>ORG</th><th>PER</th><th>% Docs</th><th>LOC</th><th>ORG</th><th>PER</th><th>% Docs</th></tr></thead><tbody><tr><td>DA</td><td>ddt</td><td>.875</td><td>.778</td><td>.959</td><td>100%</td><td>.917</td><td>.765</td><td>.934</td><td>100%</td><td>.882</td><td>.805</td><td>.975</td><td>100%</td></tr><tr><td>EN</td><td>ewt</td><td>.696</td><td>.533</td><td>.925</td><td>20%</td><td>.786</td><td>.640</td><td>.949</td><td>20%</td><td>.825</td><td>.869</td><td>.969</td><td>20%</td></tr><tr><td>PT</td><td>bosque</td><td>.928</td><td>.902</td><td>.974</td><td>11%</td><td>.850</td><td>.885</td><td>.980</td><td>25%</td><td>.955</td><td>.914</td><td>.975</td><td>23%</td></tr><tr><td>SK</td><td>snk</td><td>.840</td><td>.743</td><td>.900</td><td>100%</td><td>.801</td><td>.597</td><td>.770</td><td>100%</td><td>.837</td><td>.621</td><td>.823</td><td>100%</td></tr><tr><td>SV</td><td>talbanken</td><td>.857</td><td>.670</td><td>.913</td><td>100%</td><td>.800</td><td>.461</td><td>.888</td><td>100%</td><td>.937</td><td>.812</td><td>.871</td><td>100%</td></tr><tr><td>ZH</td><td>gsd</td><td>.800</td><td>.724</td><td>.917</td><td>14%</td><td>.795</td><td>.661</td><td>.956</td><td>100%</td><td>.860</td><td>.711</td><td>.944</td><td>23%</td></tr><tr><td>DE</td><td>pud</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>.709</td><td>.840</td><td>.812</td><td>6%</td></tr><tr><td>EN</td><td>pud</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1.00</td><td>.936</td><td>.966</td><td>6%</td></tr><tr><td>PT</td><td>pud</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>.903</td><td>.920</td><td>.985</td><td>14%</td></tr><tr><td>RU</td><td>pud</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>.719</td><td>.531</td><td>.891</td><td>100%</td></tr><tr><td>SV</td><td>pud</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>.865</td><td>.735</td><td>.944</td><td>100%</td></tr><tr><td>ZH</td><td>pud</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>.752</td><td>.776</td><td>.971</td><td>20%</td></tr><tr><td>CEB</td><td>gja</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>.769</td><td>1.00</td><td>.914</td><td>71%</td></tr><tr><td>TL</td><td>trg</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>.833</td><td>-</td><td>.957</td><td>100%</td></tr><tr><td></td><td>ugnayan</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>.913</td><td>-</td><td>-</td><td>100%</td></tr></tbody></table>

city government that levies taxes, but common usage allows, for example, “SpringfieldORG charges a brutal income tax.” CoNLL 2003 English also has this ambiguity, with many documents where city names, representing sports teams, are annotated as ORG. We find this ambiguity is particularly common in the en_ewt train and validation splits, primarily in documents in the reviews domain, which are short and very informal (e.g. “we love pamelas”).

## 4.3 Agreement with the PROPN POS Tag

The proper noun (PROPN) part-of-speech tag used in UD represents the subset of nouns that are used as the name of a specific person, place, or object (Nivre et al., 2020). We hypothesize that named entities as defined in UNER act roughly as a subset of these PROPN words or phrases, although not a strict subset due to divergent definitions. To test this, we calculate the precision of the UNER annotations against the UD PROPN tags (Table 3, F₁ scores reported in Table 4). Overall, precision is relatively high, with a mean precision of 0.761 across datasets. Lower precision is often due to multi-word names containing non-PROPN words (e.g., “Catherine the Great”). The differences in precision can also be due to language-specific PROPN annotation guidelines: for example, while the English PUD treebank tags the United States entity as “UnitedPROPN StatesPROPN”, Russian PUD tags it as “СоединенныхАДТ ШТАТОВНОН”.

<table><caption>Table 3: Comparing the overlap (Precision) between UNER annotations and UD PROPN tags.</caption><thead><tr><th>Lang.</th><th>Dataset</th><th>Train</th><th>Dev</th><th>Test</th></tr></thead><tbody><tr><td>DA</td><td>ddt</td><td>.709</td><td>.729</td><td>.722</td></tr><tr><td>EN</td><td>ewt</td><td>.890</td><td>.895</td><td>.892</td></tr><tr><td>HR</td><td>set</td><td>.683</td><td>.651</td><td>.671</td></tr><tr><td>PT</td><td>bosque</td><td>.864</td><td>.881</td><td>.844</td></tr><tr><td>QAF</td><td>arabizi</td><td>.952</td><td>.960</td><td>.985</td></tr><tr><td>SK</td><td>snk</td><td>.803</td><td>.783</td><td>.688</td></tr><tr><td>SR</td><td>set</td><td>.687</td><td>.631</td><td>.680</td></tr><tr><td>SV</td><td>talbanken</td><td>.766</td><td>.756</td><td>.842</td></tr><tr><td>ZH</td><td>gsd</td><td>.605</td><td>.624</td><td>.616</td></tr><tr><td>ZH</td><td>gsdsimp</td><td>.601</td><td>.604</td><td>.617</td></tr><tr><td>DE</td><td>pud</td><td>-</td><td>-</td><td>.712</td></tr><tr><td>EN</td><td>pud</td><td>-</td><td>-</td><td>.872</td></tr><tr><td>PT</td><td>pud</td><td>-</td><td>-</td><td>.749</td></tr><tr><td>RU</td><td>pud</td><td>-</td><td>-</td><td>.708</td></tr><tr><td>SV</td><td>pud</td><td>-</td><td>-</td><td>.810</td></tr><tr><td>ZH</td><td>pud</td><td>-</td><td>-</td><td>.634</td></tr><tr><td>CEB</td><td>gja</td><td>-</td><td>-</td><td>.980</td></tr><tr><td>TL</td><td>trg</td><td>-</td><td>-</td><td>.958</td></tr><tr><td>TL</td><td>ugnayan</td><td>-</td><td>-</td><td>.654</td></tr></tbody></table>

## 4.4 Cross-lingual Agreement in UNER

UNER contains sentence-aligned evaluation sets for six languages (German, English, Portuguese, Russian, Swedish, and Chinese) that are annotated on top of the Parallel Universal Dependencies treebanks (PUD; Zeman et al., 2017). Figure 3 summarizes the similarity of the NER annotations across these target languages in PUD.

We find that the overall distribution of tags is similar for the Western European languages (left
<figure><img src="image_5.png" /><figcaption>Figure 3: Cross-lingual comparison of NER Annotations on top of PUD treebanks. **Left:** Global distribution of tags for each PUD language. **Center:** Sentence-level agreement between languages for the number of entities. **Right:** Sentence-level agreement between languages for the identity of entities.</figcaption></figure>

panel): the English, German, and Swedish annotations contain very similar counts of LOC and PER entities, with slightly more variance in ORG tags. Portuguese has a similar distribution with slightly more LOC entities. However, the Russian and Chinese annotations contain differing distributions from both these languages and each other.

A similar trend occurs in the sentence-level pairwise agreement on entity counts and identities between languages (center). There is relatively high agreement on the number of entities between European languages, with Russian differing slightly more from English, German, Portuguese, and Swedish. However, the Chinese benchmark agrees less frequently: the Chinese annotations match other languages on the number of entities in 50.4% of sentences; the other languages have an average agreement of 71.7–75.6%. Pairwise agreement on the specific entities in a given sentence shows similar behavior, albeit with lower agreement overall (right).

Many of these annotation differences likely stem from the translation process. While the data is aligned at the sentence level, linguistic variation and translator decisions may cause an entity to be added to or removed from the sentence, or the concept may be expressed in a manner that no longer qualifies as a named entity under the annotation guidelines.<fnref n="5" /> While we cannot directly measure inter-annotator agreement across languages because of the above differences, some variation also undoubtedly stems from annotation differences and errors, just as these cause disagreement between annotators on the same benchmark.

In the case of Chinese and English, we manually audited the annotation discrepancies. The differences in the LOC and ORG tags mainly stem from the confusion outlined in Section 4.2. Additionally, we observed more than 30 instances that could be explained by language-specific morphological inflection rules. Specifically, country names are used directly to modify the following nouns in Chinese as opposed to English using the adjectival form.<fnref n="6" /> Finally, the increase in PER entities can be best explained by the style of Chinese writing, which tends to transliterate non-Chinese names into Chinese and append the Latin name in parentheses; in these cases, each instance of the name would be tagged as a separate PER entity.<fnref n="7" />

# 5 Baselines for UNER

This section establishes initial baselines on the datasets in UNER v1 and provides in-language and cross-lingual results with XLM-R<sub>Large</sub>.

## 5.1 Experiment Setup

We finetune XLM-R<sub>Large</sub> (560M parameters) (Conneau et al., 2020) on the UNER datasets in which train and dev sets are available,<fnref n="8" /> using a single NVIDIA GeForce RTX 3090 GPU. We also evaluate the performance of XLM-R<sub>Large</sub> jointly fine-tuned on all training sets (all) listed above. We use a learning rate of 3e-5 and batch size of 16, except for bosque, where we used a batch size of

<footnote n="5">⁵Consider the phrases: “奧巴馬對在北卡羅來納大學運動場上的群眾說道。” and “he told the crowd gathered on a sports field at the University of North Carolina.” In Chinese, *Obama* (奧巴馬) is referred to by name, whereas the English version uses a pronoun.</footnote>

<footnote n="6">⁶I.e., “韓國公司” ‘South Korean company’. The Chinese word “韓國” means the country ‘South Korea’, and in this case, directly modifies the noun “公司” ‘company’. This word was consequently labeled as LOC, whereas its English counterpart is 0.</footnote>

<footnote n="7">⁷An example is “聖羅斯季斯拉夫 (St. Rastislav)”, in which the English name is parenthesized and kept in the Chinese sentence, causing both names to be annotated.</footnote>

<footnote n="8">⁸ddt, ewt, set, bosque, arabizi, snk, set, talbanken, gsd, gsdsimp</footnote>
<figure><img src="image_6.png" /><figcaption>Figure 4: Heatmap of micro F₁ scores on test sets with different fine-tuned models. The y-axis indicates the dataset that the model is fine-tuned on, and the x-axis indicates the datasets that the models are evaluated on. **Left:** Model performance on datasets that contains the train, dev and test splits. The highlighted diagonal cells are the in-dataset results. **Center:** Model performance on the PUD datasets. **Right:** Model performance on all other datasets.</figcaption></figure>

8, and batch size of 4 in the cases of talbanken and all. All the code we used is adapted from the Huggingface transformers package (Wolf et al., 2020).

## 5.2 Results and Discussion

Figure 4 reports the micro F₁ scores on all test sets when XLM-R<sub>Large</sub> is finetuned on different languages. The in-language performance shown on the diagonal on the left of Figure 4 is almost always the highest among all test sets, with a few exceptions such as Simplified Chinese vs Traditional Chinese (ZH) and Croatian (HR) vs Serbian (SR). This most likely stems from the fact that both pairs are closely related languages.

We also observe that in most cases (i.e., between European languages), cross-lingual transfer performs well, achieving over .600 F₁. However, transfer results in strikingly low performance on all three Chinese datasets {gsd, gsdsimp, pud}, as well as on the Maghrebi-Arabic-French (QAF) dataset {arabizi}. The results on the Chinese datasets align with observations from previous work (Chen et al., 2023a; Wu et al., 2020a; Bao et al., 2019) that other languages do not transfer well to Chinese. Narabizi is a North-African Arabic dialect written in Latin script that often involves code-switching with French. The lack of similarities between this language and all other languages in our dataset might have resulted in poor transfer performance. Furthermore, Narabizi — along with Cebuano — are not included in the pretraining language

guages for XLM-R, which likely also affects their performance in this setting.

Table 6 (in the Appendix) shows the tag-level performance breakdown. For all languages, F₁ on ORG is always the lowest, and LOC is almost always the second lowest. This likely stems from the similarity between ORG and LOC entities discussed in Section 4.2, whereas the names of people are usually less ambiguous, resulting in the highest F₁ on PER for most datasets. Overall, the trained models finetuned on the UNER datasets exhibit promising results, and we leave further improvements on multi- and cross-lingual NER with these datasets to future work.

Finally, the performance of the model finetuned on all is included in Figure 5. Most all F₁ scores are similar to the F₁ scores from individual training sets or lead to a moderate decrease in performance; however, in some cross-lingual cases the joint training improves performance, such as on zh_pud which improved from .410 using a model finetuned on en_ewt to .860. Finetuning on a diverse multilingual dataset helps preserve and even improve the performance on benchmarks in diverse languages.

## 6 Related Work

**Adding a NER layer to UD** Some single-language efforts have added a manually annotated NER layer to emerging or existing UD data. Agić and Ljubešić (2014) annotated the SETimes.HR dataset with linguistic and NER information, be-
<figure><img src="image_7.png" /><figcaption>Figure 5: F₁ scores of each UNER test set after finetuning XLM-R<sub>Large</sub> on all training sets.</figcaption></figure>

coming the set_hr UD dataset later (Agić and Ljubešić, 2015). Plank (2019) added a layer of NER to the dev and test portions of the Danish UD treebank (DDT) for cross-lingual evaluation; Plank et al. (2020) fully annotated it with nested NER entities. Hvingelby et al. (2020) annotated the same Danish UD data with a flat annotation scheme.

Other languages have seen efforts in a similar spirit. Jørgensen et al. (2020) added a named entity annotation layer on top of the Norwegian Dependency Treebank, Luoma et al. (2020) built the Turku NER corpus, and Plank (2021) added a layer on top of English EWT. Recently, Muischnek and Müürisep (2023) introduced the largest publicly available Estonian NER dataset. Complementing these efforts, Riabi et al. (2023) added several annotation layers, including NER, to the NArabizi treebank (Seddah et al., 2020), a North-African Arabic dialect dataset written in Latin script with a high-level of language variability and code-switching.

**Multilingual NER resources** Several benchmark datasets for NER offer coverage for a variety of representative languages. Aside from well-known benchmarks such as CoNLL 2002/2003 (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003), other datasets were built to address a unique need, such as focusing on low-resource languages like LORELEI (Strassel and Tracey, 2016) or incorporating particularly challenging annotations, as seen in MultiCoNER (Malmasi et al., 2022a,b). MasakhaNER (Adelani et al., 2022) harnessed the *Masakhane* community to produce gold-standard annotations for ten African languages.

Other datasets for multilingual and non-English NER use a silver-standard annotation process (Nothman et al., 2013; Pan et al., 2017; Tedeschi et al., 2021). Nonetheless, CoNLL 2002/2003 re-

mains one of the main benchmarks in multilingual NER. A recent work, also called UNER (Alves et al., 2020), attempts to produce silver-standard corpora by propagating English annotations across parallel corpora but with no baseline evaluations. Lastly, another contemporary work called Universal NER (Zhou et al., 2023) bears no relation to our effort as it contains no annotation component.

**Modeling for multilingual NER** Several works have explored the task of NER outside of English. The earliest build language-independent methods (Cucerzan and Yarowsky, 1999; Lample et al., 2016, inter alia). Cross-lingual techniques have also emerged to transfer information between languages, especially from high- to low-resource languages (Ruder et al., 2019) or combining model and data transfer across languages (Wu et al., 2020b). Currently, the standard paradigm for multilingual NER involves finetuning or prompting multilingual language models (e.g., Wu and Dredze, 2020; Muennighoff et al., 2023). UNER supports these modeling efforts by providing gold-standard annotations across various languages.

**Community-driven annotation projects** The field of NLP has been shaped by community-driven annotation projects. One prime example is the Universal Dependencies (UD) project (Nivre et al., 2020), precipitated by the earlier introduction of the universal POS tagset (McDonald et al., 2013). Extensions and sister projects to UD have emerged (e.g., Savary et al., 2023; Kahane et al., 2021), to which UNER is now added. Another notable endeavor is UniMorph (Kirov et al., 2018; McCarthy et al., 2020), which covers 182 languages (Batsuren et al., 2022, 2021). The Masakhane Project has also produced several high-quality community efforts (Adelani et al., 2021, 2022; Dione et al., 2023b,a).

The UNER project follows the same community-driven approach by asking volunteers to contribute annotations for their respective languages.

## 7 Conclusion

We introduce Universal NER (UNER), a gold-standard data initiative covering 13 languages for named entity recognition (NER). The datasets included in UNER v1 cover a wide variety of domains and language families, and we establish initial performance metrics for these benchmarks. UNER opens several opportunities for research
in NER outside of English and for cross-lingual transfer; in particular, this project provides human-annotated and standardized evaluations for multi-lingual NER.

After releasing the current version of the UNER project, we plan to expand language coverage and diversity of this effort by both recruiting additional annotators and integrating existing NER datasets when possible. This will also allow us to obtain more robust agreement measures and verify the quality of existing annotations in UNER. In the longer term, our aims for Universal NER include rigorous quality checking of annotation results for robustness and further integration of finetuned models and data analysis tools into the project.

## Limitations

**Dataset Domains and Languages** The data included in UNER v1 covers a range of domains and languages, depending on the available annotators and datasets in UD (Appendix Table 5). The variance in domains and languages will generally affect the efficacy of cross-lingual learning and evaluation. However, we also provide a standardized, parallel evaluation set for a subset of the languages in UNER. Furthermore, we invite researchers who would like to see additional languages in UNER to join the annotation effort.

**Springboarding from Universal Dependencies**

Our preliminary criterion for languages and data to be included in the current version of UNER is that it should be already in the Universal Dependencies (UD) (de Marneffe et al., 2021). This is to ensure the quality of the underlying data and to facilitate research in conjunction with existing UD treebanks, which include part-of-speech tags, tokenization, lemmas, and glosses. However, future iterations of the UNER initiative are open to all languages, especially low-resource ones, regardless of whether they are present in UD.

**Number of Annotators** The UNER project relies on crowd-sourcing and community participation for annotation efforts. Thus, the languages included have varying numbers of annotators who have accepted the invitation to contribute. Nonetheless, as reported in Table 2, each language has at least two annotators for a subset of its documents and thus a corresponding measure of inter-annotator agreement.

## Ethics Statement

Our annotated data is built on top of Universal Dependencies, an already established data resource. Thus, we do not foresee any serious or harmful issues arising from its content. Interested volunteer annotators who were invited to the project have also been informed of the guidelines as discussed in Section 3 for annotating NER-ready datasets before starting with the process.

## Acknowledgments

This project could not have happened without the enthusiastic response and hard work of many annotators in the NLP community, and for that we are extremely grateful. Annotators additional to authors are: Elyanah Aco, Ekaterina Artemova, Vuk Batanović, Jay Rhald Caballes Padilla, Chunyuan Deng, Ivo-Pavao Jazbec, Juliane Karlsson, Jozef Kubík, Peter Krantz, Myron Darrel Montefalcon, Stefan Schweter, Sif Sonniks, Emil Stenström, Miriam Šuppová.

We would like to thank Joakim Nivre, Dan Zeeman, Matthew Honnibal, Željko Agić, Constantine Lignos, and Amir Zeldes for early discussion and helpful ideas at the very beginning of this project.

JMI is funded by National University Philippines and the UKRI Centre for Doctoral Training in Accountable, Responsible and Transparent AI [EP/S023437/1] of the University of Bath.

Arij Riabi is funded by the European Union's Horizon 2020 research and innovation programme under grant agreement No. 101021607.

Marek Šuppa was partially supported by the grant APVV-21-0114.

## References

David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Sham-suddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Kabore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh-Buabeng, Victoire Memdjokam Koagne, Allahsera Auguste Tapo, Tebogo Macucwa, Vukosi Marivate, Mboning Tchiaze Elvis, Tajuddeen Gwadabe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabende, Neo Lerato Mokono, Ignatius Ezeani, Chiamaka Chukwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme,
Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022. MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4488–4508, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constantine Liginos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, De gaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei. 2021. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics, 9:1116–1131.

Željko Agić and Nikola Ljubešić. 2014. The SE-Times.HR linguistically annotated corpus of Croatian. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1724–1727, Reykjavik, Iceland. European Language Resources Association (ELRA).

Željko Agić and Nikola Ljubešić. 2015. Universal Dependencies for Croatian (that work for Serbian, too). In The 5th Workshop on Balto-Slavic Natural Language Processing, pages 1–8, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA.

Diego Alves, Tin Kuculo, Gabriel Amaral, Gaurish Thakkar, and Marko Tadic. 2020. UNER: Universal Named-Entity Recognition Framework. arXiv preprint arXiv:2010.12406.

Angelina Aquino, Franz de Leon, and Mary Ann Bacolod. 2020. UD Tagalog-Ugnayan. https://github.com/UniversalDependencies/UD Tagalog-Ugnayan.

Glyd Aranes. 2022. The GJA Cebuano Treebank: Creating a Cebuano Universal Dependencies Treebank. Master's thesis, Itá-Suomen yliopisto.

Zuyi Bao, Rui Huang, Chen Li, and Kenny Zhu. 2019. Low-resource sequence labeling via unsupervised

multilingual contextualized representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1028–1039, Hong Kong, China. Association for Computational Linguistics.

Vuk Batanović, Nikola Ljubešić, and Tanja Samardžić. 2018. Setimes.SR—A Reference Training Corpus of Serbian. In Proceedings of the Conference on Language Technologies & Digital Humanities 2018 (JT-DH 2018), pages 11–17.

Khuyagbaatar Batsuren, Gábor Bella, and Fausto Giunchiglia. 2021. MorphyNet: a large multilingual database of derivational and inflectional morphology. In Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 39–48, Online. Association for Computational Linguistics.

Khuyagbaatar Batsuren, Omer Goldman, Salam Khalifa, Nizar Habash, Witold Kieraś, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yustinus Ghanggo Ate, Maria Ryskina, Sabrina Mielke, Elena Budianskaya, Charbel El-Khaissi, Tiago Pi mentel, Michael Gasser, William Abbott Lane, Mohit Raj, Matt Coler, Jaime Rafael Montoya Samame, Delio Siticonatzi Camaiteri, Esaú Zumaeta Rojas, Didier López Francis, Arturo Oncevay, Juan López Bautista, Gema Celeste Silva Villegas, Lucas Torroba Hennigen, Adam Ek, David Guriel, Peter Dirix, Jean-Philippe Bernardy, Andrew Scherbakov, Aziyana Bayyr-ool, Antonios Anastasopoulos, Roberto Zariquiey, Karina Sheifer, Sofya Ganieva, Hilaria Cruz, Ritván Karahóga, Stella Markantonatou, George Pavlidis, Matvey Plugaryov, Elena Klyachko, Ali Salehi, Candy Angulo, Jatayu Baxi, Andrew Krizhanovsky, Natalia Krizhanovskaya, Elizabeth Salesky, Clara Vania, Sardana Ivanova, Jennifer White, Rowan Hall Maudslay, Josef Valvoda, Ran Zmigrod, Paula Czarnowska, Irene Nikkarinen, Aelita Salchak, Brijesh Bhatt, Christopher Straughn, Zoey Liu, Jonathan North Washington, Yuval Pinter, Duygu Ataman, Marcin Wolinski, Totok Suhardijanto, Anna Yablonskaya, Niklas Stoehr, Hossep Dolatian, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Aryaman Arora, Richard J. Hatcher, Ritesh Kumar, Jeremiah Young, Daria Rodionova, Anastasia Yemelina, Taras Andrushko, Igor Marchenko, Polina Mashkovtseva, Alexandra Serova, Emily Prud’hommeaux, Maria Nepomniashchaya, Fausto Giunchiglia, Eleanor Chodroff, Mans Hulden, Miikka Silfverberg, Arya D. McCarthy, David Yarowsky, Ryan Cotterell, Reut Tsarfaty, and Ekaterina Vylomova. 2022. UniMorph 4.0: Universal Morphology. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 840–855, Marseille, France. European Language Resources Association.

Yang Chen, Chao Jiang, Alan Ritter, and Wei Xu. 2023a. Frustratingly easy label projection for cross-lingual
transfer. In *Findings of the Association for Computational Linguistics: ACL 2023*, pages 5775–5796, Toronto, Canada. Association for Computational Linguistics.

Yang Chen, Vedaant Shah, and Alan Ritter. 2023b. *Better Low-Resource Entity Recognition Through Translation and Annotation Fusion*. *arXiv preprint arXiv:2305.13582*.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. *Unsupervised cross-lingual representation learning at scale*. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 8440–8451, Online. Association for Computational Linguistics.

Silviu Cucerzan and David Yarowsky. 1999. *Language independent named entity recognition combining morphological and contextual evidence*. In *1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora*.

Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. *Universal Dependencies*. *Computational Linguistics*, 47(2):255–308.

Cheikh M. Bamba Dione, David Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukiibi, Blessing Sibanda, Bonaventure F. P. Dossou, Andiswa Bukula, Rooweither Mabuya, Allahsera Auguste Tapo, Edwin Munkoh-Buabeng, victoire Memdjokam Koagne, Fatoumata Ouoba Kabore, Amelia Taylor, Godson Kalipe, Tebogo Macucwa, Vukosi Marivate, Tajuddeen Gwadabe, Mboning Tchiaze Elvis, Ikechukwu Onyenwe, Grtien Atindogbe, Tolulope Adelani, Idris Akinade, Olanrewaju Samuel, Marien Nahimana, Théogène Musabeyezu, Emile Niyomutabazi, Ester Chimhenga, Kudzai Gotosa, Patrick Mizha, Apelete Agbolo, Seydou Traore, Chinedu Uchechukwu, Aliyu Yusuf, Muhammad Abdullahi, and Dietrich Klakow. 2023a. *Masakhapos: Part-of-speech tagging for typologically diverse african languages*.

Cheikh M. Bamba Dione, David Ifeoluwa Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukiibi, Blessing Sibanda, Bonaventure F. P. Dossou, Andiswa Bukula, Rooweither Mabuya, Allahsera Auguste Tapo, Edwin Munkoh-Buabeng, Victoire Memdjokam Koagne, Fatoumata Ouoba Kabore, Amelia Taylor, Godson Kalipe, Tebogo Macucwa, Vukosi Marivate, Tajuddeen Gwadabe,

Mboning Tchiaze Elvis, Ikechukwu Onyenwe, Grtien Atindogbe, Tolulope Adelani, Idris Akinade, Olanrewaju Samuel, Marien Nahimana, Théogène Musabeyezu, Emile Niyomutabazi, Ester Chimhenga, Kudzai Gotosa, Patrick Mizha, Apelete Agbolo, Seydou Traore, Chinedu Uchechukwu, Aliyu Yusuf, Muhammad Abdullahi, and Dietrich Klakow. 2023b. *MasakhaPOS: Part-of-speech tagging for typologically diverse african languages*. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 10883–10900, Toronto, Canada. Association for Computational Linguistics.

Ralph Grishman. 2019. *Twenty-five years of information extraction*. *Natural Language Engineering*, 25(6):677–692.

Rasmus Hvingelby, Amalie Brogaard Pauli, Maria Barrett, Christina Rosted, Lasse Malm Lidegaard, and Anders Søgaard. 2020. *DaNE: A named entity resource for Danish*. In *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pages 4597–4604, Marseille, France. European Language Resources Association.

International Organization for Standardization. 1998. *Codes for the representation of names of languages—Part 2: alpha-3 code*. Standard, International Organization for Standardization, Geneva, CH.

International Organization for Standardization. 2002. *Codes for the representation of names of languages—Part 1: Alpha-2 code*. Standard, International Organization for Standardization, Geneva, CH.

Anders Johannsen, Héctor Martínez Alonso, and Barbara Plank. 2015. *Universal Dependencies for Danish*. In *International Workshop on Treebanks and Linguistic Theories (TLT14)*, page 157.

Fredrik Jørgensen, Tobias Aasmoe, Anne-Stine Ruud Husevåg, Lilja Øvrelid, and Erik Velldal. 2020. *NorNE: Annotating named entities for Norwegian*. In *Proceedings of the Twelfth Language Resources and Evaluation Conference*, pages 4547–4556, Marseille, France. European Language Resources Association.

Sylvain Kahane, Bernard Caron, Emmett Strickland, and Kim Gerdes. 2021. *Annotation guidelines of UD and SUD treebanks for spoken corpora: A proposal*. In *Proceedings of the 20th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2021)*, pages 35–47, Sofia, Bulgaria. Association for Computational Linguistics.

Mahboob Alam Khalid, Valentin Jijkoun, and Maarten De Rijke. 2008. The impact of named entity normalization on information retrieval for question answering. In *European Conference on Information Retrieval*, pages 705–710. Springer.

Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sabrina J. Mielke, Arya McCarthy,
Sandra Kübler, David Yarowsky, Jason Eisner, and
Mans Hulden. 2018. UniMorph 2.0: Universal Mor-
phology. In Proceedings of the Eleventh Interna-
tional Conference on Language Resources and Eval-
uation (LREC 2018), Miyazaki, Japan. European Lan-
guage Resources Association (ELRA).

Guillaume Lample, Miguel Ballesteros, Sandeep Sub-
ramanian, Kazuya Kawakami, and Chris Dyer. 2016.
Neural architectures for named entity recognition.
In Proceedings of the 2016 Conference of the North
American Chapter of the Association for Computa-
tional Linguistics: Human Language Technologies,
pages 260–270, San Diego, California. Association
for Computational Linguistics.

Constantine Lignos, Nolan Holley, Chester Palen-
Michel, and Jonne Sälevä. 2022. Toward more mean-
ingful resources for lower-resourced languages. In
Findings of the Association for Computational Lin-
guistics: ACL 2022, pages 523–532, Dublin, Ireland.
Association for Computational Linguistics.

Nikola Ljubešić, Filip Klubička, Željko Agić, and Ivo-
Pavao Jazbec. 2016. New inflectional lexicons and
training corpora for improved morphosyntactic an-
notation of Croatian and Serbian. In Proceedings
of the Tenth International Conference on Language
Resources and Evaluation (LREC’16), pages 4264–
4270, Portorož, Slovenia. European Language Re-
sources Association (ELRA).

Jouni Luoma, Miika Oinonen, Maria Pyykönen,
Veronika Laippala, and Sampo Pyysalo. 2020. A
broad-coverage corpus for Finnish named entity
recognition. In Proceedings of the Twelfth Lan-
guage Resources and Evaluation Conference, pages
4615–4624, Marseille, France. European Language
Resources Association.

Tingting Ma, Qianhui Wu, Huiqiang Jiang, Börje F.
Karlsson, Tiejun Zhao, and Chin-Yew Lin. 2023. Co-
LaDa: A collaborative label denoising framework for
cross-lingual named entity recognition. In Proceed-
ings of the 61st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers).
Association for Computational Linguistics.

Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta
Kar, and Oleg Rokhlenko. 2022a. MultiCoNER: A
large-scale multilingual dataset for complex named
entity recognition. In Proceedings of the 29th Inter-
national Conference on Computational Linguistics,
pages 3798–3809, Gyeongju, Republic of Korea. In-
ternational Committee on Computational Linguistics.

Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022b. SemEval-2022 task 11: Multilingual complex named entity recognition (MultiCoNER). In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1412–1437, Seattle, United States. Association for Computational Linguistics.

Stephen Mayhew and Dan Roth. 2018. TALEN: Tool for annotation of low-resource ENtities. In Proceedings of ACL 2018, System Demonstrations, pages 80–86, Melbourne, Australia. Association for Computational Linguistics.

Arya D. McCarthy, Christo Kirov, Matteo Grella,
Amrit Nidhi, Patrick Xia, Kyle Gorman, Ekate-
rina Vylomova, Sabrina J. Mielke, Garrett Nico-
lai, Miikka Silfverberg, Timofey Arkhangelskiy, Na-
taly Krizhanovsky, Andrew Krizhanovsky, Elena
Klyachko, Alexey Sorokin, John Mansfield, Valts
Ernštreits, Yuval Pinter, Cassandra L. Jacobs, Ryan
Cotterell, Mans Hulden, and David Yarowsky. 2020.
UniMorph 3.0: Universal Morphology. In Proceed-
ings of the Twelfth Language Resources and Evalua-
tion Conference, pages 3922–3931, Marseille, France.
European Language Resources Association.

Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-
Brundage, Yoav Goldberg, Dipanjan Das, Kuzman
Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar
Täckström, Claudia Bedini, Núria Bertomeu Castelló,
and Jungmee Lee. 2013. Universal Dependency an-
notation for multilingual parsing. In Proceedings
of the 51st Annual Meeting of the Association for
Computational Linguistics (Volume 2: Short Papers),
pages 92–97, Sofia, Bulgaria. Association for Com-
putational Linguistics.

Diego Mollá, Menno van Zaanen, and Daniel Smith.
2006. Named entity recognition for question answer-
ing. In Proceedings of the Australasian Language
Technology Workshop 2006, pages 51–58, Sydney,
Australia.

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hai-ley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Al-banie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.

Kadri Muischnek and Kaili Müürisep. 2023. Named entity layer in Estonian UD treebanks. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 179–184, Tórshavn, Faroe Islands. University of Tartu Library.

Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin-
ter, Yoav Goldberg, Jan Hajič, Christopher D. Man-
ning, Ryan McDonald, Slav Petrov, Sampo Pyysalo,
Natalia Silveira, Reut Tsarfaty, and Daniel Zeman.
2016. Universal Dependencies v1: A multilingual
treebank collection. In Proceedings of the Tenth In-
ternational Conference on Language Resources and
Evaluation (LREC’16), pages 1659–1666, Portorož,
Slovenia. European Language Resources Association
(ELRA).
Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin-
ter, Jan Hajič, Christopher D. Manning, Sampo
Pyysalo, Sebastian Schuster, Francis Tyers, and
Daniel Zeman. 2020. Universal Dependencies v2: 
An evergrowing multilingual treebank collection. In
*Proceedings of the Twelfth Language Resources and
Evaluation Conference*, pages 4034–4043, Marseille,
France. European Language Resources Association.

Joel Nothman, Nicky Ringland, Will Radford, Tara Mur-
phy, and James R. Curran. 2013. Learning multilin-
gual named entity recognition from Wikipedia. *Arti-
ficial Intelligence*, 194:151–175.

Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Noth-
man, Kevin Knight, and Heng Ji. 2017. Cross-lingual
name tagging and linking for 282 languages. In *Pro-
ceedings of the 55th Annual Meeting of the Associa-
tion for Computational Linguistics (Volume 1: Long
Papers)*, pages 1946–1958, Vancouver, Canada. As-
sociation for Computational Linguistics.

Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In *Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)*, pages 2089–2096, Istanbul, Turkey. European Language Resources Association (ELRA).

Barbara Plank. 2019. Neural cross-lingual transfer and limited annotated data for named entity recognition in Danish. In *Proceedings of the 22nd Nordic Conference on Computational Linguistics*, pages 370–375, Turku, Finland. Linköping University Electronic Press.

Barbara Plank. 2021. Cross-lingual cross-domain nested named entity evaluation on English web texts. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021*, pages 1808–1815, Online. Association for Computational Linguistics.

Barbara Plank, Kristian Nørgaard Jensen, and Rob
van der Goot. 2020. DaN+: Danish nested named
entities and lexical normalization. In *Proceedings of
the 28th International Conference on Computational
Linguistics*, pages 6649–6662, Barcelona, Spain (On-
line). International Committee on Computational Lin-
guistics.

Peng Qi and Koichi Yasuoka. 2019. UD_Chinese-GSDSimp. https://github.com/UniversalDependencies/UD_Chinese-GSDSimp.

Alexandre Rademaker, Fabricio Chalub, Livy Real, Cláudia Freitas, Eckhard Bick, and Valeria de Paiva. 2017. Universal Dependencies for Portuguese. In *Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017)*, pages 197–206, Pisa, Italy. Linköping University Electronic Press.

Arij Riabi, Menel Mahamdi, and Djamé Seddah. 2023. Enriching the NArabizi treebank: A multifaceted approach to supporting an under-resourced language. In

*Proceedings of the 17th Linguistic Annotation Work-
shop (LAW-XVII)*, pages 266–278, Toronto, Canada.
Association for Computational Linguistics.

Sebastian Ruder, Anders Søgaard, and Ivan Vulić. 2019. Unsupervised cross-lingual representation learning. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts*, pages 31–38, Florence, Italy. Association for Computational Linguistics.

Tanja Samardžić, Mirjana Starović, Željko Agić, and Nikola Ljubešić. 2017. Universal Dependencies for Serbian in comparison with Croatian and other Slavic languages. In *Proceedings of the 6th Workshop on Balto-Slavic Natural Language Processing*, pages 39–44, Valencia, Spain. Association for Computational Linguistics.

Stephanie Samson and Cagrı Cöltekin. 2020. UD_Tagalog-TRG. https://github.com/UniversalDependencies/UD_Tagalog-TRG.

Agata Savary, Cherifa Ben Khelil, Carlos Ramisch, Voula Giouli, Verginica Barbu Mititelu, Najet Hadj Mohamed, Cvetana Krstev, Chaya Liebeskind, Hongzhi Xu, Sara Stymne, Tunga Güngör, Thomas Pickard, Bruno Guillaume, Eduard Bejček, Archna Bhatia, Marie Candito, Polona Gantar, Uxoa Iñurri-eta, Albert Gatt, Jolanta Kovalevskaite, Timm Lichte, Nikola Ljubešić, Johanna Monti, Carla Parra Escartín, Mehrnoush Shamsfard, Ivelina Stoyanova, Veronika Vincze, and Abigail Walsh. 2023. PARSEME corpus release 1.3. In *Proceedings of the 19th Workshop on Multiword Expressions (MWE 2023)*, pages 24–35, Dubrovnik, Croatia. Association for Computational Linguistics.

Djamé Seddah, Farah Essaidi, Amal Fethi, Matthieu
Futeral, Benjamin Muller, Pedro Javier Ortiz Suárez,
Benoît Sagot, and Abhishek Srivastava. 2020. Build-
ing a user-generated content North-African Arabizi
treebank: Tackling hell. In *Proceedings of the 58th
Annual Meeting of the Association for Computational
Linguistics*, pages 1139–1150, Online. Association
for Computational Linguistics.

Mo Shen, Ryan McDonald, Daniel Zeman, and Peng Qi. 2016. UD_Chinese-GSD. https://github.com/UniversalDependencies/UD_Chinese-GSD.

Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. 2014. A gold standard dependency corpus for English. In *Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)*, pages 2897–2904, Reykjavik, Iceland. European Language Resources Association (ELRA).

Stephanie Strassel and Jennifer Tracey. 2016. LORELEI language packs: Data, tools, and resources for technology development in low resource languages. In *Proceedings of the Tenth International Conference on Language Resources and Evaluation*
(LREC'16), pages 3273–3280, Portorož, Slovenia.
European Language Resources Association (ELRA).

Simone Tedeschi, Valentino Maiorca, Niccolò Campol-
ungo, Francesco Cecconi, and Roberto Navigli. 2021.
**WikiNEuRal: Combined neural and knowledge-
based silver data creation for multilingual NER. In
*Findings of the Association for Computational Lin-
guistics: EMNLP 2021*, pages 2521–2533, Punta
Cana, Dominican Republic. Association for Compu-
tational Linguistics.

Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In *COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)*.

Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In *Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003*, pages 142–147.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierre Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pages 38–45, Online. Association for Computational Linguistics.

Qianhui Wu, Zijia Lin, Börje Karlsson, Jian-Guang Lou, and Biqing Huang. 2020a. Single-/multi-source cross-lingual NER via teacher-student learning on unlabeled data in target language. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 6505–6514, Online. Association for Computational Linguistics.

Qianhui Wu, Zijia Lin, Börje F. Karlsson, Biqing Huang, and Jian-Guang Lou. 2020b. Unitrans : Unifying model transfer and data transfer for cross-lingual named entity recognition with unlabeled data. In *Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20*. International Joint Conferences on Artificial Intelligence Organization.

Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In *Proceedings of the 5th Workshop on Representation Learning for NLP*, pages 120–130, Online. Association for Computational Linguistics.

Daniel Zeman. 2017. Slovak dependency treebank in universal dependencies. *Journal of Linguistics/Jazykovedný casopis*, 68(2):385–395.

Daniel Zeman, Martin Popel, Milan Straka, Jan Hajič, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinková, Jan Hajič jr., Jaroslava Hlaváčová, Václava Kettnerová, Zdeňka Urešová, Jenna Kanerva, Stina Ojala, Anna Missilä, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kanayama, Valeria de Paiva, Kira Droganova, Héctor Martínez Alonso, Çağrı Çöltekin, Umut Sulubacak, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim Harris, Katrin Marheinecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpradit, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadová, Esha Banerjee, Ruli Manurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonça, Tatiana Lando, Rattima Nitisaroj, and Josie Li. 2017. CoNLL 2017 shared task: Multilingual parsing from raw text to Universal Dependencies. In *Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies*, pages 1–19, Vancouver, Canada. Association for Computational Linguistics.

Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2023. UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition. *arXiv preprint arXiv:2308.03279*.
## A Contributions

**Stephen Mayhew** conception, kickoff, all initial organization, recruitment, and annotation, development of annotation tool, manuscript writing.

**Terra Blevins** annotation, organization, PUD analysis scripts, core manuscript writing.

**Shuheng Liu** annotation, all baseline experiments and analysis.

**Marek Šuppa** annotation, PROPN analysis, paper writing, GPU resources.

**Hila Gonen** advising, organization of and feedback on manuscript.

**Joseph Marvin Imperial** facilitated annotations for Tagalog and Cebuano, additions to manuscript for TL/CEB results, limitations, ethics, and conclusion sections.

**Börje F. Karlsson** annotation, manuscript writing and editing, advising.

**Peiqin Lin** annotation, manuscript comments.

**Nikola Ljubešić** preparation and transfer of the HR SET and SR SET datasets, manuscript comments and edits.

**LJ Miranda** annotation, related work section, comments, edits.

**Barbara Plank** preparation and transfer of the DA DDT dataset, manuscript writing, comments and edits.

**Arij Riabi** preparation and transfer of North African Arabizi dataset, related work section, comments and edits.

**Yuval Pinter** advising, organization of and writing of manuscript.

## B Additional Dataset Details

In this section, we provide additional statistics and analysis of the datasets included in UNER v1. Table 5 documents the domains included in each dataset along with their distributions of NER tags, and Table 4 presents the F1 overlap score between named entities in UNER and PROPN tags in the underlying UD treebanks. We also report the full numerical results of our baseline experiments in Table 6.

<table><caption>Table 4: Comparing PROPN overlap (F1 scores).</caption><thead><tr><th>Lang.</th><th>Dataset</th><th>Train</th><th>Dev</th><th>Test</th></tr></thead><tbody><tr><td>DA</td><td>ddt</td><td>.824</td><td>.835</td><td>.836</td></tr><tr><td>EN</td><td>ewt</td><td>.813</td><td>.815</td><td>.817</td></tr><tr><td>HR</td><td>set</td><td>.810</td><td>.785</td><td>.799</td></tr><tr><td>PT</td><td>bosque</td><td>.844</td><td>.859</td><td>.856</td></tr><tr><td>QAF</td><td>arabizi</td><td>.942</td><td>.962</td><td>.971</td></tr><tr><td>SK</td><td>snk</td><td>.848</td><td>.783</td><td>.771</td></tr><tr><td>SR</td><td>set</td><td>.811</td><td>.769</td><td>.807</td></tr><tr><td>SV</td><td>talbanken</td><td>.830</td><td>.805</td><td>.839</td></tr><tr><td>ZH</td><td>gsd</td><td>.700</td><td>.696</td><td>.720</td></tr><tr><td>ZH</td><td>gsdsimp</td><td>.695</td><td>.695</td><td>.719</td></tr><tr><td>DE</td><td>pud</td><td>—</td><td>—</td><td>.785</td></tr><tr><td>EN</td><td>pud</td><td>—</td><td>—</td><td>.826</td></tr><tr><td>PT</td><td>pud</td><td>—</td><td>—</td><td>.805</td></tr><tr><td>RU</td><td>pud</td><td>—</td><td>—</td><td>.779</td></tr><tr><td>SV</td><td>pud</td><td>—</td><td>—</td><td>.877</td></tr><tr><td>ZH</td><td>pud</td><td>—</td><td>—</td><td>.708</td></tr><tr><td>CEB</td><td>gja</td><td>—</td><td>—</td><td>.926</td></tr><tr><td>TL</td><td>trg</td><td>—</td><td>—</td><td>.696</td></tr><tr><td>TL</td><td>ugnayan</td><td>—</td><td>—</td><td>.723</td></tr></tbody></table>

## C Dataset Licensing

The Universal Dependencies datasets are licensed under Creative Commons Attribution-ShareAlike (CC BY-SA)<fnref n="9" />. This license requires that “if you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.” Thus, we distribute all of our datasets under the same license.

<footnote n="9">⁹creativecommons.org/licenses/by-sa/4.0/</footnote>
<table><caption>Table 5: Domains and distribution of entity types for datasets in UNER. Domains are categorized for the underlying UD datasets at https://universaldependencies.org/.</caption><thead><tr><th rowspan="2">Data Source</th><th rowspan="2">Lang.</th><th rowspan="2">Dataset</th><th rowspan="2">Domains</th><th colspan="3">Entity Dist. (%)</th></tr><tr><th>LOC</th><th>ORG</th><th>PER</th></tr></thead><tbody><tr><td>Johannsen et al. (2015)</td><td>DA</td><td>ddt</td><td>fiction, news, nonfiction, spoken</td><td>28.0</td><td>30.8</td><td>41.2</td></tr><tr><td>Silveira et al. (2014)</td><td>EN</td><td>ewt</td><td>blog, email, reviews, social, web</td><td>37.8</td><td>21.8</td><td>40.4</td></tr><tr><td>Agić and Ljubešić (2015)</td><td>HR</td><td>set</td><td>news, web, wiki</td><td>37.4</td><td>33.0</td><td>29.6</td></tr><tr><td>Rademaker et al. (2017)</td><td>PT</td><td>bosque</td><td>news</td><td>29.5</td><td>33.9</td><td>36.6</td></tr><tr><td>Seddah et al. (2020)</td><td>QAF</td><td>arabizi</td><td>blog, web, social</td><td>57.5</td><td>27</td><td>15.4</td></tr><tr><td>Zeman (2017)</td><td>SK</td><td>snk</td><td>fiction, news, nonfiction</td><td>21.2</td><td>6.2</td><td>72.6</td></tr><tr><td>Samardžić et al. (2017)</td><td>SR</td><td>set</td><td>news</td><td>41.4</td><td>30.2</td><td>28.4</td></tr><tr><td>McDonald et al. (2013)</td><td>SV</td><td>talbanken</td><td>news, nonfiction</td><td>54.0</td><td>20.0</td><td>25.0</td></tr><tr><td>Shen et al. (2016)</td><td>ZH</td><td>gsd</td><td>wiki</td><td>48.1</td><td>17.9</td><td>34.0</td></tr><tr><td>Qi and Yasuoka (2019)</td><td></td><td>gsdsimp</td><td>wiki</td><td>48.0</td><td>18.0</td><td>34.0</td></tr><tr><td rowspan="6">Zeman et al. (2017)</td><td>DE</td><td>pud</td><td>news, wiki</td><td>41.3</td><td>18.5</td><td>40.2</td></tr><tr><td>EN</td><td>pud</td><td>news, wiki</td><td>39.5</td><td>21.9</td><td>38.6</td></tr><tr><td>PT</td><td>pud</td><td>news, wiki</td><td>44.4</td><td>18.0</td><td>37.6</td></tr><tr><td>RU</td><td>pud</td><td>news, wiki</td><td>33.4</td><td>26.6</td><td>40.0</td></tr><tr><td>SV</td><td>pud</td><td>news, wiki</td><td>43.0</td><td>15.7</td><td>41.3</td></tr><tr><td>ZH</td><td>pud</td><td>news, wiki</td><td>44.9</td><td>13.5</td><td>41.6</td></tr><tr><td>Aranes (2022)</td><td>CEB</td><td>gja</td><td>grammar examples</td><td>12.3</td><td>2.0</td><td>85.7</td></tr><tr><td>Samson and Cöltekin (2020)</td><td>TL</td><td>trg</td><td>grammar examples</td><td>10.9</td><td>0.0</td><td>89.1</td></tr><tr><td>Aquino et al. (2020)</td><td></td><td>ugnayan</td><td>fiction, nonfiction</td><td>47.5</td><td>0.0</td><td>52.5</td></tr></tbody></table>

<table><caption>Table 6: The full results of our baseline experiments from finetuning XLM-R<sub>Large</sub> on UNER. All scores are reported in micro-F<sub>1</sub>. ORG F<sub>1</sub> scores are not reported for the two TL datasets since there are no ORG entities labeled.</caption><thead><tr><th colspan="2">Source</th><th colspan="2">Target</th><th colspan="4">F<sub>1</sub></th></tr><tr><th>Lang.</th><th>Dataset</th><th>Lang.</th><th>Dataset</th><th>LOC</th><th>ORG</th><th>PER</th><th>Overall</th></tr></thead><tbody><tr><td>DA</td><td>ddt</td><td>DA</td><td>ddt</td><td>.879</td><td>.826</td><td>.924</td><td>.878</td></tr><tr><td>EN</td><td>ewt</td><td>EN</td><td>ewt</td><td>.871</td><td>.709</td><td>.950</td><td>.858</td></tr><tr><td>HR</td><td>set</td><td>HR</td><td>set</td><td>.977</td><td>.891</td><td>.970</td><td>.950</td></tr><tr><td>SK</td><td>snk</td><td>SK</td><td>snk</td><td>.846</td><td>.635</td><td>.882</td><td>.855</td></tr><tr><td>PT</td><td>bosque</td><td>PT</td><td>bosque</td><td>.882</td><td>.861</td><td>.966</td><td>.904</td></tr><tr><td>QAF</td><td>arabizi</td><td>QAF</td><td>arabizi</td><td>.780</td><td>.520</td><td>.717</td><td>.711</td></tr><tr><td>SR</td><td>set</td><td>SR</td><td>set</td><td>.981</td><td>.913</td><td>.983</td><td>.962</td></tr><tr><td>SV</td><td>talbanken</td><td>SV</td><td>talbanken</td><td>.904</td><td>.742</td><td>.928</td><td>.883</td></tr><tr><td>ZH</td><td>gsd</td><td>ZH</td><td>gsd</td><td>.906</td><td>.819</td><td>.922</td><td>.896</td></tr><tr><td>ZH</td><td>gsdsimp</td><td>ZH</td><td>gsdsimp</td><td>.906</td><td>.802</td><td>.925</td><td>.894</td></tr><tr><td rowspan="6">EN</td><td rowspan="6">ewt</td><td>DE</td><td>pud</td><td>.816</td><td>.603</td><td>.893</td><td>.814</td></tr><tr><td>EN</td><td>pud</td><td>.785</td><td>.593</td><td>.922</td><td>.805</td></tr><tr><td>PT</td><td>pud</td><td>.845</td><td>.698</td><td>.914</td><td>.848</td></tr><tr><td>RU</td><td>pud</td><td>.681</td><td>.451</td><td>.875</td><td>.715</td></tr><tr><td>SV</td><td>pud</td><td>.887</td><td>.655</td><td>.928</td><td>.872</td></tr><tr><td>ZH</td><td>pud</td><td>.465</td><td>.308</td><td>.389</td><td>.410</td></tr><tr><td rowspan="3">EN</td><td rowspan="3">ewt</td><td>CEB</td><td>gja</td><td>.556</td><td>.000</td><td>.842</td><td>.789</td></tr><tr><td>TL</td><td>trg</td><td>1.00</td><td>–</td><td>.923</td><td>.936</td></tr><tr><td>TL</td><td>ugnayan</td><td>.857</td><td>–</td><td>.000</td><td>.783</td></tr></tbody></table>