Do “English” Named Entity Recognizers Work Well on Global Englishes?
Authors: Alexander Shan
John Bauer
Riley Carlson
Christopher D Manning
Venue: EMNLP 2023 Findings (2023)
Keywords: NER
global english
TL;DR: Global English dataset for NER to accommodate English as spoken in regions other than US and Britain

Abstract: The vast majority of the popular English named entity recognition (NER) datasets contain American or British English data, despite the existence of many global varieties of English. As such, it is unclear whether they generalize for analyzing use of English globally. To test this, we build a newswire dataset, the Worldwide English NER Dataset, to analyze NER model performance on low-resource English variants from around the world. We test widely used NER toolkits and transformer models, including models using the pre-trained contextual models RoBERTa and ELECTRA, on three datasets: a commonly used British English newswire dataset, CoNLL 2003, a more American focused dataset OntoNotes, and our global dataset. All models trained on the CoNLL or OntoNotes datasets experienced significant performance drops---over 10 F1 in some cases---when tested on the Worldwide English dataset. Upon examination of region-specific errors, we observe the greatest performance drops for Oceania and Africa, while Asia and the Middle East had comparatively strong performance. Lastly, we find that a combined model trained on the Worldwide dataset and either CoNLL or OntoNotes lost only 1-2 F1 on both test sets.

Do "English" Named Entity Recognizers work well on Global Englishes?
Alexander Shan, John Bauer, Riley Carlson and Christopher D. Manning
Department of Computer Science
Stanford University
Stanford, CA 94305-9030, U.S.A.
{azshan, horatio, rileydc, maminggJestanfrr..cmm
Abstract
2003 (Tjong Kim Sang and De Meulder, 2003) and OntoNotes (Weischedel al., 2013), which focus on British and American English, with significant European Parliament coverage. Other recently cre- ated NER datasets study the medical domain, such as the n2c2 challenges (Henry et al., 2019), histor- ical English (Ehrmann et al., 2022), or music rec- ommendation terminology (Epure and Hennequin, 2023). still using American and British English, The lack of regional variety in these datasets sug- gests that models trained on these datasets might not accurately recognize entities from more global contexts. Furthermore, the lack of test data for other regions makes it difficult to even measure this phenomenon.
The vast majority of the popular English named entity recognition (NER) datasets con- tain American or British English data, despite the existence of many global varieties of En- glish. As such, it is unelear whether they gen- eralize for analyzing use of English globally. To test this, we build a newswire dataset, the Worldwide English NER Dataset, to analyze "low-resone new-resource"
NER model performance English variants from around the world. We test widely used NER toolkits and transformer models, including RoBERTa and ELECTRA. on three datasets: a commonly used British English newswire dataset, CoNLL 2003, more American-focusel dataset, OntoNotes, our global dataset. All models trained on the CoNLL or OntoNotes datasets experienced significant performance drops-over over 10% F1 some cases cases-when tested on the Worldwide English dataset. Upon examination of region- specific errors, we observe the greatest perfor- mance drops for Oceania and Africa, while Asia and the Middle East had comparatively strong performance. Lastly, we find that a com- bined model trained the Worldwide dataset and either CoNLL or OntoNotes lost only 1-2% Fl on both test sets.
In this work, we evaluate the performance of a variety of NER tools, including Flair and SpaCy on this dataset. We then retrain two commonly used NER models, those of Stanza (Qi al.. 2020) and CoreNLP (Manning et al., 2014), see whether their performance improves when the Worldwide English datasct is combined with their usual train- ing data. Using these toolkits allows for easy re- training while comparing the performance losses between models using categorical features, word embeddings, and transformers. When using Stanza, in order to compare the generalization of different language models, Large (Liu al., 2019), Electra-Large (Clark al., 2020), and just the CoNLL 2017 word vectors (Gin- ter et al., 2017).
1 Introduction
Most of English Named Entity Recognition (NER) uses American or British English data, with less at- tention paid low-resource English contexts. Mul- tiple problems may occur in low-resource NER settings; for example, named entities with region- specific meanings can be confused for common words. Indeed, the Japanese Diet a governmental body, but NER models focused on US and British English may incorrectly interpret this entity medical term.
retrain with each of Roberta-
We find that, as expected, there is a large dropoff in performance when the models are applied to regions not well covered by the training data in CoNLL or OntoNotes. This is true whether the model uses categorical features, word vectors, transformer language models. When considering specific regions around the world, we find that In- digenous Oceania and Africa had the worst per- formance due to a disproportionately large num- ber of unrecognized tokens. We believe these
Among many NER datasets released in recent years, the most widely used datasets are CoNLL
'A collection of NER references is available at https: // //github. waam//jaaddd/e/mneea recogiiittooooiiiiionioooooooooe datasstss

---

results reflect the impact of minimal discussions of the Global South in U.S. and U.K. newswire. Across all regions, models trained on CoNLL or OntoNotes struggled to accurately identify cultural and currency names. Furthermore, we find that re- training the models in question on the training split of the new dataset improves their performance on those regions, but has the opposite effect of hurting the performance on US and British English texts. When retrained on both an existing dataset and the new dataset the same time, however, the models maintain their previous performance on CoNLL or OntoNotes while drastically improving their perfor- mance on other worldwide regions. We make our new dataset public to better represent the diversity
3 Dataset Sourcing and Construction
3.1 Sourcing
The Worldwide NER dataset is composed of 1075 articles containing 674,000 tokens sourced from countries and regions across the globe where En- glish used. Articles were taken from a variety of press organizations per country, with organiza- tional diversity scaling with the number of texts from a given country. For subsequent testing rea- sons, we separate texts into regional buckets, parti- tioning the corpus into articles from Asia, Africa, Latin America, the Middle East, and Indigenous Commonwealth (indigenous Oceania and Canada). Appendix B contains a breakdown of the regions and their number of articles.
in written English around the world.
3.2 Labeling
2 Related Work
We partnered with data labeling platform Datasaur as well as an annotation workforce provided by third-party annotation service MLTwist. MLTwist contracted Ghana-based labeling service, Aya Data, for the annotations. The annotators were fluent English speakers and each labeling batch was annotated by one labeler before review from a member of MLTwist and our research team. When reviewing, MLTwist and the research team would sync to have a consensus on each annotation.
NER has been explored in non-English languages with great success, with one example being Stan- ford NLP's open-source Stanza NER model (Qi et al., 2020; Zhang et al., 2021), which covers many languages such Chinese and Arabic, along with including several other English medical models. There has also been research on low-resource for- cign languages with notable success using neural- based approaches (Cotterell and Duh. 2017; Ade- lani et al., 2022). However, low-resource contexts of English have not been recently examined. Louis et al. (2006) showed that Western- Englishh trained models performed poorly in South African contexts of English due to the presence of unknown words such as "Xabanisa", or alternative uses of words like "Peace", a common South African name. How- ever, their model was a simple Bayesian network, so their results are not indicative of how modern neural models might perform, especially with pre- training. Ghaddar et al. (2021) revealed that name- regularity bias still exists in neural models, even when contextual clues are present. For example, in the sentence "Obama is located in southwestern Fukui Prefecture.". state-of-the-art NER models la- beled "Obama" as a person rather than a location. Ghaddar et al. did not examine contexts of English, so further research remains important.
To assess our labeling quality, we use Cohen's 's Kappa scores reported by the Datasaur platform. The combined score when comparing all of the individual annotators with the review process de- scribed above is 77.47. While scales for Cohen's Kappa are heavily context-dependent, we believe this indicates a thorough labeling process.
Our Worldwide English dataset uses 9 classes: Date, Person, Location, Facility, Organization, Miscellaneous, Money, NORP (national, organiza- tional, religious, political identity), and Product. Appendix A contains full list of our dataset's la- bels, accompanied with definitions. We condensed the 18-class label framework of OntoNotes 5.0 into classes because we wanted to isolate the classes that would pose challenges across English contexts. Since classes like Percent, Ordinal, and Time are statically used across English contexts, we omitted most numeric and time expressions. Some classes appeared too infrequently to justify new class, like Work of Art and Law which were condensed into the MISC class, similar to CoNLLO3.
Outside the NER realm, Ziems et al. (2023) ex- plored dialectic differences in machine translation and question answering, but did not address NER.
'The dataset is available at stanfordnlp/en-wee worldwide www.diie-eewwwiree newswire
Additionally, the data labeling instructions were designed to align with that of the OntoNotes 5.0

---

Embedding
Train
CoNLL
All WW
Africa
Asia
Indigenous
Latin America
Middle
East
w2v char
CoNLL
91.02
77.35
77.19
79.68
69.80
77.95
75.20
Electra
CoNLL
93.18
82.94
83.35
84.92
76.90
82.51
81.10
Roberta
CoNLL
92.54
83.29
83.11
85.19
76.12
83.24
82.75
w2v char
Worldwide
70.67
87.19
86.39
89.71
82.60
87.17
85.28
Electra
Worldwide
75.06
90.11
89.66
92.20
84.08
89.93
89.27
Roberta
Worldwide
75.69
90.23
89.59
92.18
86.83
90.16
88.79
w2v char
Combined
90.72
86.48
85.59
89.15
81.15
86.45
84.70
Electra
Combined
92.82
89.99
89.32
92.39
84.07
89.55
89.13
Roberta
Combined
91.99
90.10
89.27
92.30
87.28
89.66
88.67
Table 1: Stanza entity F1 Scores on CoNLL and Worldwide test data when trained on CoNLL, Worldwide, or Combined training data. Combined training gives close to best results in all cases.
Entity
CoNLL
All WW
Africa
Asia
Indigenous
Latin America
Middle East
LOC
72.96
91.44
90.91
92.49
86.69
91.53
91.22
MISC
65.16
84.81
80.92
88.75
83.43
84.70
84.26
ORG 63.07
86.55
86.78
90.02
80.00
84.82
83.79
PER
94.88
96.02
96.08
96.65
95.60
96.30
94.45
Overall
75.69
90.23
89.59
92.18
86.83
90.16
88.79
Table 2: Stanza entity F1 scores when trained using Roberta Worldwide
dataset for consistency. The only exception was Date, where we did not label generic time expres- sions such as "the last three months", which receive a label in OntoNotes. Unlike OntoNotes, we use nested NER annotations when relevant, that phrase such "2022 FIFA World Cup" has the tire text labeled MISC represent the event, "2022" labeled Date, and "FIFA" labeled Organization.
(2013). For our own dataset, we perform a 70/10/20 train/dev/test split, randomized within each region using stratified sampling.
Experiments using Stanza
We first examine Stanza models (Qi et al., 2020) on CoNLL and our Worldwide English; we consider other NER systems and OntoNotes in section 7.
3.3 Preprocessing
4.1 Approach
To preprocess the data, we convert the annotations into BIOES format. BIOES marks tokens with their class and position a named entity span as one of beginning, intermediate, ending, or singular, with non-named-entity words labeled O.
We train and test Stanza's NER model with three different pre-training configurations (word2vec word embeddings and character model only, RoBERTa embeddings, ELECTRA embeddings). We train models on CoNLLO3 and the Worldwide English dataset separately. We also consider a com- bined model trained on both CoNLLO3 and our dataset. During testing, we compare the models for performance on both the CoNLLOS and World- wide datasets, evaluating the joint test set of all regions along with region-specifie results.
When comparing with CoNLL-based models, such as CoreNLP, Flair, and Stanza's class model, to maintain consistency when building combined models and comparing results, we collapsed the annotations for the Worldwide English data with 9 classes into 4 classes (person name, location, organization, and MISC). A complete explanation of the collapsing process is outlined in Appendix For CoNLLO33, we use the original version and its defined train, dev, and test sets. For OntoNotes, we use the train/dev/test split from Weischedel et al.
The OntoNotes model used in the following experiments not ideal for public re- lease, as it loses the granularity of the 18 classes provided by OntoNotes. Instead, we used the 9
Availability

---

class Worldwide data finetune the hidden rep- resentations of the 18 class OntoNotes model; the model available through Stanza.
performance to each other. As shown in Table 2, we find that the model trained on Worldwide performs well on PER, including on the original CoNLL03 dataset. The other categories, especially ORG and MISC, show larger degradation. Below, we discuss specific error cases from each region.
The source code for converting the raw dataset to regional and combined portions, training a Stanza model using multiple data sources, converting OntoNotes to 9 classes and our data to 4 classes, and reproducing the final combined model are avail- able in the latest Stanza release.
5 Analysis
To better understand model behavior in error cases, we examine the sentences with incorrect predictions, generalizing error patterns across all regions and examining region-specific challenges.
Hyperparameters We train using stochastic gra- dient descent (maximum gradient descent steps = 200,000, with early termination) and then score the model against the dev and test sets of CoNLLO3 and the Worldwide dataset. We use learning rate scheduler to decrease the learning rate after plateau, with the early termination condition of the learning rate reaching 0.0001. Since our goal was to examine the differences between models trained on the various datasets, and not to maxi- mize performance, we use the default hyperparam- eters provided by Stanza. Training took around 3 hours for each dataset on an Nvidia 3090, with the exception of the combined model (5 hours). Values for hyperparameters are shown in Appendix C
5.1 Universal error patterns
A large challenge for the model(s) trained on CoNLLO3 and tested on our Worldwide dataset was currencies. CoNLLO3 labels currencies as MISC, such as "AS", "USS", and "C$" for currencies; we find hundreds of instances where other currencies were incorrectly labeled "O" by the CoNLLO3- trained model. Upon inspection of CoNLLO3, we diagnose the error as the result of currencies such as the yen, zlotys, and lei incorrectly labeled "O" in training. Consequently, currencies our dataset such as the rupee, baht, colones, and riyal were often missed as MISC. Another error in many re- gions was the improper labeling of names. Take the following example for instance: President Yoon Suk yeol ordered on Monday the preparation of roadmap...
4.2 Overall Results
The results in Table for the Stanza model tested on CoNLL03 are consistent with other state-of-the art models tested on CoNLLO3 (Lim, 2023). As expected, the model's performance suffered when tested on the opposite dataset from which it was trained. While this effect is expected, the degree to which the model confuses tags is significant considering that it was supported with RoBERTa and word2vec, which included worldwide train- ing sources like Wikipedia. The word2vec-based model with transformer performs significantly worse, 5 F1, than its transformer counterparts, sug- gesting that the improved understanding of words from context and a larger pre-training set makes a difference. Additionally, the combined model per- forms well across both datasets; it is competitive with the solely Worldwide on the Worldwide English evaluation sets and out- performed the CoNLLO3-trainee model its re- spective evaluation sets. As for specific language regions, the CoNLLO3-trained model performed worst in indigenous language contexts, followed by African contexts. The other regions (Asia, Latin America, Middle East) respectively shared similar
Yoon Suk yeol is a full name, yet the model in- correctly labeled Yook and yeol as S-PER, leaving Suk as a tag instead of I-PER. This error is a consequence of the differences name structure across the world, as discussed in section 5.2.
5.2 Regional errors
Next, we analyze error patterns that frequently ap- peared in specific regions. Every region had some share of these errors, but highlight the regions where a type of error was overwhelmingly present.
trained model
Asia As described in section 5.1, Asia had the majority of person name errors, whether it be I- PER tags incorrectly labeled O, or E-PER tokens labeled O. This error is strongly correlated with the structure of names parts of Asia rather than the tokens being unrecognizable; names in CoNLLO3 frequently take on the structure of [first, last) and not first, middle, last]. However, in other English contexts, names with hyphens to join 3 or more
Phttps: git..b. io/stanza/

---

morphemes can be commonplace (e.g. Chen Mei hsiu). Hence, for the model, these names appear as multiple single names listed in sequential order, rather than a single named entity.
With Prime Minister Anthony Al- banese's attendance at Garma, a great sense of elation that maybe, perhaps, something might be about to change has taken hold.
Example:
Garma, a cultural festival for indigenous North Aus- tralians (MISC) was incorrectly labeled LOC. Con- textual clues signal that Garma is a location, given that Albanese attended "at" Garma. Furthermore, it typically shows up in the Wikipedia data used by Roberta and ELECTRA as a location. How- ever, with a stronger understanding of the word, would be clear that Garma is referred to as an event (MISC) in this sentence. Indeed, if Garma is replaced with Thanksgiving, an American tradi- tion, the model properly identifies the named entity. This example shows how unknown tokens cause models decide labels based potentially mis- leading contextual reasoning. The remedy to this issue is having stronger signals from word mean- ings, which achievable through greater inclusion of broadly sourced training data.
Africa African texts generally had below- average performance amongst the Worldwide cor- pus, but had disproportionately high error count when the gold tag was MISC and the predicted label was ORG. These errors were found in sen- tences where some tokens were unrecognized and lacked strong context clues. Consider the following example, for instance:
Directed and produced by Kwabena Gyansah { Azali ), High Currency also stars Brihanna Kinte Ghana Jollof ). Edward Kufuor ( Accra Medic ) Kweku Elliot To Have and To Hold) ,... Each bolded entity is Work of Art (properly MISC), but was labeled ORG. While High Cur- rency contextually seems like film production, the remaining named entities could be mistaken for groups. Some of the tokens are missing from training and pre-training, leaving the model to clas- sify unknown words amid unclear contextual clues, leading to these errors. The models also exhibit errors amid clear contexts, like in this case:
Latin America Our models had an average per- formance on the Latin America corpus compared to the other regions, but we observe some interesting error patterns. We once again see errors associated with unfamiliar tokens, such as the following: Example: Simón is a FARC ex combatant liv- ing in the Icononzo camp ETCR Antonio Nariño in the Andean region of Tolima " "I don 't want to live in fear for another fourr years, he said. Despite Andean (LOC) referring to the Andes Mountain Range, it was mistakenly labeled MISC, likely because it was misinterpreted as a NORP. That is, if one replaces Andean with a NORP, such as American, the sentence makes sense. Hence, since Andean was an unfamiliar word, the model tumndd to unreliable contextual clues for guidance, resulting in error. Next, we observe a case of name- recognition bias in a second example:
The la carte restaurant run by executive chef Ul- ric Denis is pricy, in a delightful, luxurious but somewhat impersonal surrounding and can offer some of the island's s best examples of modern cook- ing with true Seychellois taste.
Seychellois, a NORP (and therefore MISC in CoNLL parlance) describing the Seychelle peo- plc. is mistaken for an ORG. Despite the strong contextual clues hinting that the token is a MISC, the model is confused because Seychellois was an unknown word, missing during training. Hence, observe that the contextual signal the model was overpowered by the confusion induced by the pres- ence of this unknown token. As a general pattern, we find that when sentence context is uninforma- tive, unknown tokens are challenging for models, yet this is not solved with clear context either.
Example: Brazilian Amazon deforestation dou- bled from the 2009-2018 average, with 22 percent more foresorest lost in 2021 than the previous year. While the Brazilian Amazon is clearly a location, the model incorrectly predicted MISC. We find that the model has bias for Brazilian as NORP (MISC), inducing the erroncous prediction. This is unique error, as Brazilian appears in training and pre-training, but not with the same meaning has in this sentence. Hence, we observe that the lack of contextual diversity in training caused the sentence context signal to be overwhelmed by the
Indigenous The indigenous region, composed of Oceanic and Canadian native tribal news, posed the greatest challenge to the CoNLLO3-trainee trained model. We find that the large share of unknown tokens from these texts induced great confusion for the models. Similar to the African corpus, we observe unrecognizable tokens negatively impacting model performance regardless of contextual clues.

---

Train
Test
All
LOC
MISC
ORG
PER
CoNLL
CoNLL
89.27
91.41
81.60
85.63
94.07
CoNLL
Worldwide
70.47
79.65
59.87
58.75
76.25
Worldwide
CoNLL
63.67
71.10
50.03
38.28
83.28
Worldwide
Worldwide
82.65
84.92
76.12
75.78
90.23
Combined
CoNLL
88.86
91.10
77.75
85.36
94.83
Combined
Worldwide
83.04
85,95
76.36
75.45
90.63
Table 3: F1 Scores for CoreNL. by type
Train
All
Africa
Asia
Indigenous
Latin America
Middle East
CoNLL
70.47
72.13
72.67
60.87
71.47
66.88
Worldwide
82.65
82.12
85.06
76. 12
81.04
81.23
Combined
83.04
82.12
85.73
76.56
81.58
81.73
Table 4: Fl Scores for CoreNI. on Worldwide by region
model's learned meaning for Brazilian.
label and the collapsing of Location and GPE, we used CoNLL and OntoNotes as guidelines for the annotation team to follow.
Middle East The Middle Eastern corpus had comparatively strong performance among the Worldwide texts. Of course, there were several notable eтror patterns, such as improperly labeled person names and currency names. Additionally, find many error cases induced by unknown tokens.
Temporal Drift be temporal drift in annotations. We use a dataset similar to CoNLL to demonstrate that temporal drift alone does not account for the differences seen (Liu and Ritter, 2023). This dataset uses CoNLL style annotations, but with news articles from 2020. Liu and Ritter found that pretrained embeddings such Roberta do not degrade in performance when trained on CoNLL and tested on 2020 data.
concerning problem would
Example: Making the remarks in an interview with the state-owned broadcaster, TRT Haber, TRT Haber is Turkish news agency (ORG). However, it does not show up as an entity in either CoNLL or OntoNotes. Considering the CRF model, the word interview should be strong hint to the model. It occurs in the original datasets in contexts both PER and ORG, but in this case, the model incorrectly labels TRT Haber as PER.
Here, we instead use the 2020 data as additional training data, using Stanza with the Roberta embed- ding. We find that there is some improvement in the Worldwide test score from adding the 2020 data, but the regionally appropriate data from Worldwide has substantially more impact. Training CoNLL
CoNLL
WW
6 Alternate Explanations
92.54
83.29
Although our hypothesis is that regional variations are the largest contributor to performance gaps be- tween models, several other possibilities could be valid. We address some possible explanations here.
CoNLL + WW
91.99
90.10
CoNLL 2020
92.52
84.57
CoNLL + 2020 + WW
91.95
89.75
Additional Data As we saw, adding some amount of 2020 data improved the scores on the Worldwide test set, even when not regionally tar- geted. We perform an ablation study to show that having regionally appropriate training data is more relevant than simply adding more training data.
Dataset Domain One possible explanation of the scores would be different domains. However, CoNLL is trained entirely newswire, and the majority of OntoNotes also newswire. There- fore, aside from regional or temporal differences. the domains of the datasets should be very simi- lar. Differences in annotation style could also a culprit. However, except where noted for the Date
In particular, we choose one of the regions, Asia, and train Stanza with Roberta embeddings on in- creasingly large subsets of CoNLL and Worldwide

---

Model
CoNLL
Worldwide
Africa
Asia
Indigenous
Latin America
Middle East
ner-fast
91.45
78.70
79.70
80.27
72.43
79.37
75,43
ner
91.59
79.36
80.29
80.94
72.80 80.79
80.00
76.37
ner-large
91.71
85.81
86.40
86.93
85.31
85.13
Table 5: Entity F1 Scores for Flair by region
Test \ Train
OntoNotes Roberta
Worldwide Roberta
Combined Roberta
w2vec
Electra
w2vec
Electra
w2vec
Electra
OntoNotes
89.88
91.43 76.92
92.55 77.00
74.35 86.04
77.26 89.31
77.86
89.26
90.80
91.62
Worldwide
71.62
88.97
83.62
86.86
87.79
Table Entity
Scores for Stanza on OntoNotes
excluding the Asia portion of the training data. We find that although the Asia dataset is fraction of the entire dataset, training on just CoNLL and Asia is more accurate than the larger subsets. These results summarized in table 7.
7 Experiments using Other Models
7.1 CoreNLP on CoNLL
We used Stanford's CoreNLP package (Manning et al., 2014), an NLP software distribution written in Java before neural models became widespread, to test the effectiveness of its model on the world- wide NER data. Our results shown in table 3 for class-bases scores and in table 4 for scores by region.
Data Leakage Another explanation would be that of data leakage from a regional training sets to the test set, especially in the case of news story with worldwide coverage. For example, if an article from Latin America covered event concerning American presidential election in the training data, article from news source in Asia covering the same event might be randomly assigned the test set, giving an unfair advantage to a model trained on the Worldwide dataset. The data collection emphasized stories of regional in- terest, however, and so there should not be data leakage of international news stories.
CoreNLP includes model trained on CoNLL, with the caveat that it does not use B- and I- tags, but rather assigns labels either part of class or O for outside. This is rarely a practical issue in English, with exceptions such as lists without com- mas. The model is CRF which uses text features of the words and their neighbors. For example, the word "CoreNLP" would itself be a feature, along with the prefixes "C", "Co", and the suffixes "P". "LP". Similarly, "CoreNLP" and other categorical features of the word would be used for features for the neighboring words. The baseline CoNLL model included with CoreNLP has an Fl of 89, 22%2%%% on CONLLO3. (Finkel al., 2005)
Alternate Embeddings An embedding with more worldwide coverage might also show less degradation. For example, Multilingual Bert (De- vlin et al., 2018) and XLM-RoBERTT-Lange (Con- neau et al., 2019) both include multilingual text, perhaps including many of the terms within our Worldwide dataset. Instead, we find that they per- form no better than Roberta-Large Llectra-Large. Meanwhile, DeBERTaV3 (He al., 2023) is more accurate, but still exhibits degradation.
Testing the included model on the Worldwide dataset shows very large drop in performance, 70.47% F1. The largest drop was on Organization, which went from 85.63% 58.75% F1.
We retrained the model on the Worldwide dataset, achieving an F1 of 82.65%. The World- wide trained model also suffered a very large drop when tested on the CoNLL dataset, falling to 63.67% F1. Again, Organization was the largest drop, going from 75.78% to 38.28% Fl.
Emb
Training CoNLL
WW
DeBERTaV3
CoNLL
93.03
83.64
XLM-RoBERTa
CoNLL
92.54
82.53
mbert
CoNLL
92.09
81.44
DeBERTaV3
WW
75.17
90.20
XLM-RoBERTa
WW
75.46
90.25
A likely explanation for the large drop in perfor- mance is that categorical features can provide con-
mbert
WW
75.32
89.20

---

Training portions
Training tokens
Entity coverage
Asia Worldwide F1
CoNLL
203621
31.65
85.19
CoNLL + Latam
307537
35.49
87.75
CoNLL + Latam + Africa
409432
38.02
88.18
CoNLL + Latam + Africa + ME
468788
40.97
88.06
CoNLL + Latam + Africa + ME Ind.
521580
41.04
88.51
CoNLL + only Asia
351792
55.16
90.36
CoNLL + entire Worldwide
669751
56.90
92.30
Table 7: Test scores for Asia in Worldwide NER on progressively larger subsets of the data. (Ind. Indigenous)
text for common English words, such as in a sen- tence "(PERSON) works at (ORGANIZATION),, but the named entities themselves are completely different. For example, local organization "Nu- luujaat Land Guardians" starting with "Nuluujaat" does not resemble any organizations in CoNLL, and so the CoNLL trained model fails in the sen- tence "A recent legal move by the Naluujaat Land Guardians.... .o.to sue Baffinland Iron Mines In- stead, the capital letters are enough of a feature that the CoNLL trained model identifies it as a MISC.
7.2 Flair on CoNLL
Another widely used software package for NLP. especially for NER, is Flair (Akbik et al., 2019) 9).. The base English NER model for Flair makes ex- tensive use of character embeddings (Akbik et al., 2019), whereas the higher accuracy model uses transformer. Their models also use document-level features (Schweter and Akbik, 2020).
We performed experiments on the publicly avail- able Flair models, reported in table 5. We note that as there are different versions of CoNLL in exis- tence, it is not surprising that our results for Flair on CoNLL are not the same reported, such 91.71 instead of 94.09 for ner-large.
Person and Location, however, are much more robust when tested across datasets. For Person, es- pecially, even when the last names are different, many regions have similar first names. For exam- ple, Anne, Anthony, and Tom are common first names in the Indigenous set.
As expected, the transformer model performs much better than the basic models. In fact, the trans- former model performs significantly better than ei- ther Stanza with a transformer or CoreNLP. We still note large drop in performance on the Indige- text, further confirming that this is the most difficult region to process. Furthermore, the results of ner-large on Worldwide are not accurate Stanza when trained specifically on the Worldwide dataset, suggesting that there would be gains accuracy on the Worldwide dataset from retraining the Flair transformer model on combined dataset.
Finally, we retrained the model combination the CoNLL and Worldwide datasets. This model mostly combined the strengths the two previous models, achicving 0.8304 Fl on the Worldwide dataset and 0.8886 on CONLL. The number of fea- tures in the trained models increased from 658K for the CoNLL model and 1M for the Worldwide model to 1.4M for the combined model; this indi- cates there is very little overlap between the fea- tures for the two models, resulting in the model effectively learning the two datasets in parallel.
7.3 Stanza on OntoNotes
We also trained the Stanza model for OntoNotes. see Table Whereas for CoNLL, we condensed the Worldwide classes to directly compare World- wide and CoNLL, for OntoNotes, we condensed the OntoNotes classes to match the new class set. See Appendix E for complete explanation. Next, we retrain the Stanza model for OntoNotes and test Worldwide, then train model using the World- wide dataset and test against OntoNotes, and finally train combined model and test both datasets. The performance degradation observed for a
When compared on regional basis, we find that each model shows the same drop on Indigenous regions, with the original CoreNLP CoNLL model especially inaccurate the Worldwide Indigenous section. Considering there some Middle Eastern text in the original CoNLL,, the CoreNI. model also does surprisingly poorly on the Middle East region.

---

spaCy model
OntoNotes (collapsed)
Worldwide
Africa
Asia
Indigenous
Latin America
Middle
East
en wew.oorreeee___eeeee en ee_cor__web_tr
84.51
57.22 76.25
58.69
60.12 79.19
41.91 67.58
53,46
57.07
91.72
77.78
74.48
73.21
Table 8: Entity F1 Scores for spaCy OntoNotes and Worldwide
CoNLL model labeling the Worldwide dataset con- tinues when using an OntoNotes model on the Worldwide dataset. Under the experimental setup described above, the best performing model trained on OntoNotes, using Electra-Large as the bottom layer, scored 92.55 F1 on entities. On the World- wide dataset, though, it scored 77.00. Conversely, the Electra-Large model trained Worldwide scored 88.97 on the Worldwide test set, but 77.86 on the OntoNotes. As seen with both Stanza and CoreNI.P on CoNLL,, though, a model trained on both performed well on both, scoring 87.79 on the Worldwide test set and 91.62 on OntoNotes.
dation is more severe. The en_core ma__core_web_ web_wwebttt... trf model shows a smaller drop, although still performs comparatively worse on the Worldwide dataset than the Stanza models, which were retrained with the Worldwide training data. We conclude that the spaCy models are also affected by the regional vari- ations in named entities. SpaCy does relatively better in Africa and is most affected in the Indige- nous, Latin America, and Middle East contexts.
8 Conclusion
A challenging entity type for the OntoNotes models to classify was Facility. The Electra-Large model trained on OntoNotes guessed 30% of the Facility tokens to be Organization, for example, and 36% of the time when predicted Facility, the true label was Location. An example of this eтrоr is "Ras Dashen, the tallest mountain in Ethiopia", which the OntoNotes-trained Electra-Large model labels Facility, despite another mountain, Everest, being labeled Location OntoNotes.
We introduce an NER dataset composed of newswire from outside Britain and America, de- signed to act as an "adversarial" evaluation set for English NER models solely trained on American or British contexts. We find limitations in Stanza's NER when trained solely on CoNLL or OntoNotes and tested in global contexts of English (10 percent drop in F1 test scores). The model becomes most confused when contextual clues, which often dis- ambiguate named entities, are missing. However, context clues are not panacca- we also find that unknown tokens from the Worldwide English con- texts can cause confusion despite clear sentence context. These effects persist with the addition of pre-trained word embeddings from word2vec. RoBERTa, and ELECTRA, suggesting that large models are not catch-all solution the challenge of classifying regional contexts. With that said, we find that the American and British English-traiie -rained model performs better on global texts with the addi- tion of BERT embeddings than without them. One takeaway from this study is that one way to achieve stronger NER performance across diverse set of English contexts is to train on more diverse sets of English data. Indeed, we find that the combined model demonstrates strong performance on both datasets. Therefore, we propose that existing state- of-the-an art models would do well to combine their data with similarly sourced global English corpora.
Again, these scores will differ from those in the literature due to some of the classes left out. Fur- thermore, some performance loss in the combined dataset is from issues with the word "the" or simi- larly mundane differences in labeling schemes. In CoNLL and our dataset, "the" is rarely labeled as part of an entity, whereas OntoNotes frequently labels "the" entities such as "the White House".
7.4 SpaCy on OntoNotes
Another commonly used NLP framework is spaCy, which has both transformer and a non-transformer model (Honnibal et al., 2020). As these models are trained on OntoNotes, we follow the same class reduction as for Stanza.
The spaCy model which does not use transform- ers, coreeor_____e see__web__gg_ smm shows a very large drop in performance; see Table 8. This is especially true on the Indigenous portion of the datasct, as we have ob- served in other test settings, except here the degra-

---

Limitations
vestigate this question in other languages as well to broaden the ethical impact of our work beyond English.
First, the loss in performance appears to be related to incomplete vocabulary, which means that even if models are trained on increasingly diverse and large sets of data, there will still always be more named entities from novel language contexts that are miss- ing. Hence, observe that the difficulties with English NER contexts outside the US or the UK are rooted in the incompleteness of training data, which is an dufinitely-scalinng problem. However, we believe that given our findings, the inclusion of even a small dataset of diversely-sourced data is well worth the effort. Second, we admit the great amount of experimentation left to be completed. With larger datasets, expect model performance on this dataset improve dramatically compared to solely training on CoNLLO3, but we have not attempted to test models trained on larger Ameri- can or British English dominated datasets on the Worldwide dataset. We plan to conduct this re- search in coming months to understand if modern scaled models are fit to perform NER in global En- glish language contexts. Lastly, we cannot speak how this effect may manifest in other languages, but would be interesting how the concept we investigate applies languages with a vast dias- pora or high worldwide uptake. French, Spanish, Dutch, or any other language considered a primary language in several countries may have similar is- sues; indeed, there are often separate research tools for Portuguese from Portugal or Brazil because of exactly the sort of effect studied in this paper.
An important question when using labeling ser- vices is whether or not the annotation is ethically sourced. We note that our labeling workforce was composed of native English speakers from an over- seas country as an alternative to student labelers. To the best of our knowledge, with consultation with Datasaur and MLTwist, the annotator team was paid fair wage relative to the local market. Furthermore, we directly contacted Aya Data, the subcontractor in Ghana, and they responded that they enforce the following rules which they believe ensure the annotators are treated fairly:
Pay annotation employees average of >5x min wage in Ghana
Employ all staff full time, with pensions paid. This means job security and no project work/ zero hours contracts
Don't accept any explicit content moderation projects, anything that will expose employ- ees harmful content
Regular training in non-annotation skills, and structured career paths
We believe this indicates the team was treated fairly and the dataset can be considered ethically sourced.
Acknowledgements
Ethics Statement
Christopher Manning is CIFAR Fellow. We would like to thank the annotation team at Datasaur, MLTwist, and Aya Data for their hard work. We would like to thank Percy Liang Jeremy Cole. Shikhar Murty, and the Stanford NLP group for the helpful discussions we had of this paper. The anonymous reviewers each gave several valuable suggestions that strengthened the arguments made here. Finally, would like to thank our friends and family for their support of our work.
We believe that to confidently assert that NER is a task that is more-or-less solved in English, state- of-the-art models should be able to achieve strong performance in all English contexts, not just the most popular ones. After all, English a language spoken widely across the world, not only in the lands which it originated. Hence, we find that the ethical implications of our work are to bring stronger equity to the performance of English NER models. Additionally, since we find little related work in English NER, it is likely that NER in other languages has lack of research with this idea as well. For example, regional dialects or contexts of languages such as Mandarin Chinese may use named entities in different ways than we tradition- ally believe, or use entirely different named entities. We hope that this work may inspire others to in-

---

References
Recognition and Linking in Multilingual Historical Documents. In Proceedings of the Working Notes of CLEF 2022 Conference and Labs of the Evaluation Forum, volume 31 80. CEUR-WS.
David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani. Michael Beukman, Chester Palen- Michel, Constantine Lignos, Jesujoba Alabi, Sham- suddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor. Fatoumata Ka- bore. Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh- Buabeng, Victoire Memdjokam Koagne, Allah- sera Auguste Tapo. Tebogo Macuewa, Vukosi Mari- vate, Mhoning Tchiaze Elvis, Tajuddeen Gwad- abe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabende, Neo Lerato Mokono, Ig- natius Ezcani, Chiamaka Chukwancke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Abdulmumin, Odunayo Ogundepo. Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022. MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 4488 4508, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Elena Epure and Romain Hennequin. 2023. A human subject study of named entity recognition (NER) in conversational music recommendation queries. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics (EACL).
Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local informa- tion into information extraction systems by Gibbs sampling. In Proceedings of the 43rd Annual Meet- ing of the Association for Computational Linguistics (ACL pages 363-370, Ann Arbor, Michigan. As- sociation for Computational Linguisties.
Abhas Ghaddar, Philippe Langlais, Ahmad Rashid, and Mehdi Rezagholizadeh. 2021. Context-aware ad- versarial training for name regularity bias in named entity recognition. Transactions of the Association for Computational Linguistics, 9:586-604,
Filip Ginter, Jan Hajič, Juhani Luotolahti, Milan Straka, and Daniel Zeman. 2017. CONLI 2017 shared task- automatically annotated raw texts and word embed- dings. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL). Faculty of Mathematics and Physics, Charles Univer- sity.
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the- art NLP. In NAACI 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages
54-59.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre- training with ing.
Kevin Clark, Minh-Thang Luong, Quoe V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representa- tions.
embedding shar-
Sam Henry, Kevin Buchan, Michele Filannino. Amber Stubbs, and Ozlem Uzuner. 2019. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. Journal of the American Medical Informatics Association, 27(1):3-12 12.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02114
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spaCy: Industrial- strength Natural Language Processing in Python.
Ryan Cotterell and Kevin Duh. 2017. Low- resource named entity recognition with cross-lingual, character-level neural conditional random fields. In Proceedings of the Eighth International Joint Con- ference on Natural Languuge Processing (Volume 2: Short Papers), pages 91-96, Taipei, Taiwan, Asian Federation of Natural Language Processing.
David Lim. 2023. Dslim/bert-base-nne hugging face.
Shuheng Liu and Alan Ritter. 2023, Do CoNLL-2003 named entity taggers still work well in 2023? In Proceedings of the 61 Anmual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 8254-8277 Toronto, Canada. Association for Computational Linguisties.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing, CoRR, abs/1810.04805.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized BERT pretraining approach. CoRR, abs/1907.1t 1669. 1692.
Maud Ehrmann, Matteo Romanello, Sven Najem- Meyer, Antoine Doucet, and Simon Clematide. 2022. Extended overview of HIPE-2022: Named Entity

---

Anita Louis, Alta De Waal, and Cobus Venter. 2006. Named entity recognition in South African context. In Proceedings of the 2006 Annual Research Con- ference of the South African Institute of Computer Scientists and Information Technologists on IT Re- search in Developing Countries, SAICSIT '06, page 170-179, ZAF. South African Institute for Computer Scientists and Information Technologists.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc- Closky. 2014. The Stanford CoreNLP natural lan- guage processing toolkit. In Annual Meeting of the Association for Computational Linguistics.
Peng Qi, Yuhao Zhang. Yuhui Zhang, Jason Bolton, and Christopher Manning. 2020. Stanza: Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations.
Stefan Schweter and Alan Akbik. 2020. FLERT: Document-level features for named entity recogni-
tion.
Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Ni- anwen Xue. Ann Taylor, Jeff Kaufman, Michelle Franchini. Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013. OntoNotes Release
Yuhao Zhang, Yuhui Zhang, Peng Qi. Christopher Manning, and Curtis P Langlotz. 2021. Biomedical and clinical English model packages for the Stanza Python NLP library. Journal of American Medi- cal Informatics Association, 1899):1892-198
Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi- value; framework for cross dialectal English NLP.

---

A Named Entity Classes
Label
CoNLL
Description
Example
Person
PER
Names of humans, excluding deities (c.g. "God")
Ryan
Organization
ORG
Names of a group or collective body
Supreme Court
Location
LOC
A physical place
Paris
Facility
LOC
A place that is used for specific utility A national, organizational, religious, political identity
Brooklyn Bridge
NORP
MISC
Chinese
Money
MISC
The name of particular denomination of money
Yen
Product
MISC
The name of marketable item or brand Other named entities: events, works of art, etc Any time of day, month, or year Not a named entity
Ford F-150
Miscellaneous
MISC
Mona Lisa
Date
O
2/19/2023
O
water bottle
Table Named entity classes, with definitions and examples
B Dataset Statisties
Region
Arts
Tokens
Date
Fac
Loc
Misc
Money
NORP
Org
Per
Prod
Africa
246
153305
1445
349
3659
2268
541
1131
4692
4236
101
Asia
346
210295
2240
543
5604
2603
1166
1647
7245
6443
292
Indigenous
87
67065
489
150
1415
676
114
835
1410
1507
4
Latam
232
156162
1542
388
4773
1726
651
1065
4048
4045
70
Middle East
164
87984
517
134
2579
858
257
889
3006
3009
98
Table 10: Statistics by Region. Countries of Origin are further broken down
Arts
Tokens
Date
Fac
Loc
Misc
Money
NORP
Org
Per
Prod
Train
745
466130
4432
1112
12449
5637
2052
4035
14082
13416
496
Dev
100
64230
639
111
1705
863
207
403
2251
1894
31
Test
230
144451
1162
341
3876
1631
470
1129
4068
3930
38
Table 11: Statistics by Train/Dev/Test
C Stanza Hyperparameters
Hyperparameter
Value
Hyperparameter
Value
Optimizer
SGD
Word Embedding Size
100
Learning Rate
0.1
Character Embedding Size
1024
Learning Rate Decay
0.5
Gradient Clipping Max Norm
5.0
Early Termination
0.0001
Batch Size
32
LSTM Layers
Dropout
0.5
LSTM Hidden Layer Size
256
Word Dropout
0.01
Table 12: Hyperparameters used for training the Stanza NER models

---

D Condensing Worldwide Labels
Original label
Condensed label
Facility
Location
Work of Art
MISC
NORP
MISC
Currency
MISC
Product
MISC
Date
O
Table 13: Condensed Worldwide labels into CoNLL 4-class
An important remaining point of discrepancy between the datasets our inclusion of dates, while CoNLL03 labels dates as O, To maintain consistency with CoNLLO3, process out all Date tags.
E
Condensing OntoNotes Labels
OntoNotes label
Condensed label
GPE
Location
Work of Art
MISC
Law
MISC
Event
MISC
Language
MISC
Cardinal
Ordinal
O
Percent
O
Quantity
0
Time
0
Date
Table 14: Condensed OntoNotes labels into Worldwide class set
We note that labels such as Cardinal were ignored because they were not annotated in our Worldwide dataset. Additionally, due to differences in how Date is annotated in OntoNotes and our dataset, it is ignored well. In OntoNotes, unnamed lengths of time are labeled, whereas Worldwide does label such expressions.


Reviews:
[{"review_id": "ioVG3WV9Ha", "reviewer": "Reviewer_Mety", "summary": "", "questions": "Did the authors have a chance to experiment with multilingual models? (XLM-R or mBERT). Given that these models are trained on diverse multilingual datasets, they inherently possess a higher potential to mitigate challenges associated with English variants.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review; we were encouraged that you found the paper well written, providing a useful new resource with comprehensive evaluations. \n\nThe criticism of the experimental settings is a very good point.  To summarize our interpretation of the argument, the variation in regional English may not be the only cause of the discrepancy in the model scores when compared on the WorldWide dataset.  Other explanations would include different domains, annotation differences, or temporal drift.\n\nThe domain of the new dataset is newswire, as was the domain of both CoNLL and the majority of OntoNotes.  Because the original models for CoreNLP, Stanza, spAcy, and Flair were trained on one or the other dataset, those models should perform best on newswire data.  Therefore, the domain of the text should not explain the differences in performance we report.\n\nThe concern over temporal drift is entirely valid. Thank you for raising it; we should have discussed it.  We can refer to another recent work on modern NER taggers, \"Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?\" by Shuheng Liu and Alan Ritter.  In this work, they present a case that transformer models such as Roberta do not show much temporal drift.  On the other hand, in our work, we find that a Roberta model loses performance when trained on CoNLL and tested on WorldWide, whereas training on the combined data maintains a high level of performance.\n\nLiu and Ritter tests for temporal drift by creating a smaller gold labeled dataset of newswire from 2020 and using it as a test set for a variety of models.  As an additional experiment, we instead added that to the CoNLL training data to see if a model trained on CoNLL + 2020 newswire would perform better on the WorldWide dataset.  The results are\n\nRetraining Stanza with a Roberta transformer\n\n| Training | CoNLL result | WorldWide result |\n| :--- | :---: | :---: |\n| CoNLL | 92.54 | 83.29 |\n| CoNLL + WorldWide | 91.84 | 89.40 |\n| CoNLL + 2020 | 92.52 | 84.57 |\n| CoNLL + 2020 + WorldWide | 91.95 | 89.75 |\n\nThere is some gain from adding more recent training data, but not nearly as much as adding the entire WorldWide training set.  We believe this demonstrates that temporal drift alone does not explain much of the performance degradation we observed in our paper.  Iif this paper were to be accepted, we would be certain to add this result and the corresponding citation in order to address this question.\n\nThank you for the good suggestion on the ways we could extend the ablation study.  Since we have cross-sections of the training and test data by region, we could, for example, train on CoNLL plus a subset of the regions to see whether adding some of the regions improves scores on a final held out region's test set.  If it would address this concern, we can include these results as well in the final version of the paper.\n\n\nRetraining Stanza with a Roberta transformer\n\n| Training portions | Asia WorldWide result | Entity coverage |\n| :---  |  :---: | :---: |\n| CoNLL | 85.19 | 49.41 |\n| CoNLL + Latam | 87.75 | 56.75 |\n| CoNLL + Latam + Africa | 88.18 | 59.33 |\n| CoNLL + Latam + Africa + Middle East | 88.06 | 61.83 |\n| CoNLL + Latam + Africa + Middle East + Indigenous | 88.51 | 62.18 |\n| CoNLL + only Asia | 90.36 | 73.85 |\n| CoNLL + entire WorldWide | 92.18 | 75.90 |\n\nAgain we observe that adding some recent training data, starting with Latin America and continuing to add more, somewhat improves the results on the Asia test section of the dataset, going from 85.19 to 88.51, similar to adding the 2020 newswire from Liu and Ritter.  However, the largest gain is training on CoNLL plus the specific region in question, Asia, again indicating that simply having additional recent data is not as important as having localized data.\n\nThe question regarding possible data leakage is a good one.  In general, there may be some test set leakage within a region, as it is possible multiple news articles covered the same incident and that went unnoticed by the authors, as multiple people collected the articles to be labeled.  However, the rate of this should be very low and it is highly unlikely there is leakage from region to region. The reason for this is that we sought out local news rather than global news from each publication, to try to ensure that we were getting locally written text, rather than repackaged material from international newswires.\n\nThe specific question of whether xlm-roberta-large or mbert has less degradation is also an interesting topic to consider.  In initial experiments, we found that English Roberta and Electra-Large had better performance than the multilingual models, so we performed the final work using exclusively those models.  Running a subset of the experiments now, we find that\n\n\nStanza retraining\n\n| Transformer | Training Portion | CoNLL F1 | WorldWide F1 |\n| :--- | :---: | :---: | :---: |\n| xml-roberta-large | CoNLL | 92.54 | 82.53 |\n| mbert | CoNLL | 92.09 | 81.44 |\n| xml-roberta-large | WorldWide | 75.46 | 90.25 |\n| mbert | WorldWide | 75.32 | 89.20|\n\nThese numbers are all very similar to the numbers for Electra and Roberta.  A potential reason we did not see improved performance despite having a more international transformer model could be that although there is more international data in those transformers, it was not present in English contexts, so the prediction head on top of the transformer does not recognize the contextual embeddings of those words in English contexts.  Another possible explanation is that Roberta-Large already includes CC-News in its training data, which contains a wide variety of newswire, meaning that adding more non-English sources does not actually broaden the international applicability of the other transformers as much as expected. We will include this possible approach and results from it in a revised version of the paper.\n\nYour suggestions and questions for the authors have led to several experiments which we believe strengthen our case that it is regional variations causing low performance for models trained only on CoNLL Thank you again for your consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written, providing a clear motivation and view of the comparative landscape.\n2. The outcomes presented within the paper are indeed promising\n3. The authors provide a comprehensive ablation analysis showing the impact of English variants.\n4. Additionally, the authors have introduced novel datasets for experimentation, thereby enriching the research's empirical foundation.", "reasons_to_reject": "1. There appear to be potential issues within the experimental settings:\n\n1a. Unseen Entity Problems: The authors highlight a performance drop when training a NER model on CoNLL and subsequently testing on the Worldwide dataset. This discrepancy could potentially be attributed to the presence of unseen entities, given the substantial temporal gap between the formulation of CoNLL (1996-1997) and the Worldwide dataset (2020s). There might be a lot of unseen entities in the test set, and the authors might further delve into this phenomenon, exploring the possibility of numerous unseen entities in the train/test set.\n\n1b. Domain/Annotation Shift Problems: Building upon the observation in 1a, it's plausible that the performance drop is from a domain mismatch rather than the variations in English. Table 5 underscores this with OntoNotes, a dataset comprising diverse text genres (news, conversational telephone speech, weblogs, usenet newsgroups, broadcast, and talk shows). A notable contrast exists between the varied genres in OntoNotes and the Worldwide dataset, suggesting that the performance drop might be due to domain shift rather than English variants.  \n\n1c. Duplicate Problems (Data Leak): Considering the Worldwide datasets' compilation from global news, there's a potential for duplicate news items from distinct regions. For example, news about the US election might appear in Asia, Latin America, and Africa datasets. We might put the US news from Asia into the training set but split the US news from Latin America and Africa into the test set. Consequently, training a NER model on the Worldwide datasets could inadvertently result in improved performance within their dataset.\n\n2. The ablation studies shed some light on the significance of considering diverse English variants. However, the paper falls short in offering comprehensive guidance on effectively managing these variations, apart from the conventional combined strategy. It would benefit the authors to delve into a more detailed analysis, providing insights into potential strategies for addressing this challenge.", "questions_to_authors": "Did the authors have a chance to experiment with multilingual models? (XLM-R or mBERT). Given that these models are trained on diverse multilingual datasets, they inherently possess a higher potential to mitigate challenges associated with English variants.", "paper_topic_and_main_contributions": "The authors have presented the Worldwide English NER dataset, a newly introduced corpus, to assess NER model performance in the context of \"low-resource\" English variations across different global regions. The datasets consist of five main categories, such as Africa, Asia, Indigenous, Latin America, and Middle East. The experiment results demonstrated that current NER models cannot handle English variations, although using a transformer model.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gZlRDF2rD9", "reviewer": "Reviewer_nGXt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comment and for finding the work well-written with nice error analysis and utility for studying worldwide Englishes.\n\nThe annotation process was as follows.  First, the authors collected the articles from a variety of international websites and extracted the text.  We then sent this off to Datasaur and their partner, MLTwist, for annotation.  MLTwist's process was to have one annotator review each document, then have a final authority review the annotations and correct any discrepancies found.  The authors then reviewed the annotations ourselves, especially in cases where a model trained on the training set then reported errors on its own training set, and corrected any incorrect labels we found.  We can clarify this for the final version of the paper.\n\nMLTwist's annotation team was in Ghana, which may raise the ethical question of whether or not the work was exploitative or if the annotators were paid a fair wage.  To the best of our knowledge, the annotator team was paid a fair wage relative to the local market.  This issue was briefly discussed at the end of the Ethics Statement in the submitted paper.  To further answer this question, we followed up with the annotation team, and we heard directly from the subcontractor in Ghana that they enforce the following rules which they believe ensure the annotators are treated fairly:\n\n1) Pay annotation employees an average of >5x min wage in Ghana \n\n2) Employ all staff full time, with pensions paid. This means job security and no project work / zero hours contracts\n\n3) We don't accept any explicit content moderation projects, or anything that will expose our employees to harmful content\n\n4) Regular training in non-annotation skills, and structured career paths\n\nThe work creating the models should be very reproducible.  The conversion scripts to join our dataset with CoNLL, or OntoNotes with our dataset, will both be made public as part of the release of the dataset.  Furthermore, we used the default settings when retraining both CoreNLP and Stanza, so the models themselves can be recreated as needed.\n\nThank you for the suggestion to give more details on statistics of the dataset. We are happy to do this and include some below.\n\nStatistics for each region of the dataset\n\n| Region | Tokens | Date | Facility | Location | Misc | Money | NORP | Org | Person | Product |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Africa | 153305 | 1445 |349 |3659 |2268 |541 |1131 |4692 |4236 |101 |\n| Asia | 210295 | 2240 | 543 | 5604 | 2603 | 1166 |1647 |7245 |6443 |292 |\n| Indigenous | 67065  | 489 |150 |1415 |676 |114 |835 |1410 |1507 |4 |\n| Latam | 156162 | 1542 | 388 | 4773 | 1726 |651 |1065 |4048 |4045 | 70 |\n| Middle East | 87984 | 517 | 134 | 2579 | 858 | 257 |889 | 3006 | 3009 | 98|\n\nDivided by train/dev/test\n\n|  | Tokens | Date | Facility | Location | Misc | Money | NORP | Org | Person | Product |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Train | 466130 | 4432 | 1112 | 12449 | 5637 | 2052 | 4035 | 14082 | 13416 | 496 |\n| Dev | 64230 | 639 | 111 | 1705 | 863 | 207 | 403 | 2251 | 1894 | 31 |\n| Test | 144451 | 1162 | 341 | 3876 | 1631 | 470 | 1129 | 4068 | 3930 | 38 |\n\n\nThank you again for the valuable feedback regarding this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Important work on making available tools work on other English languages outside the US and the UK.\n- Nice error analysis and discussion of the results.", "reasons_to_reject": "- Details about the annotation process are lacking. \n- Details about the dataset are lacking.\n- No ethical discussions are mentioned about how the dataset has been collected.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a new dataset for English NER that focuses on English from other countries outside of the US and UK. The main claim is that the current NER datasets and models are UK- and US-centric, and do not perform equally good on English languages from other parts of the world. The authors create a training and test\ndataset built from newswire from outside the US and Europe containing 1075 articles comprising 674,000 tokens. These articles are from countries and regions across the world where English is used. The authors also divide the texts into regional buckets: Asia, Africa, Latin America, the Middle East, and Indigenous Commonwealth (indigenous Oceania and Canada). While the dataset is fairly described, very little statistical information and details about the sources are provided. \nThe dataset is annotated for NER using crowdsourcing. However, no inter annotator agreement is provided. \n\nOnce the dataset created, the authors move on to test a selection of NER toolkits and transformer models on CoNLL 2003 and OntoNotes5.0 with the new dataset. The tested toolkits are Stanza, CoreNLP, Flair, and Spacy. They show that the models trained on the CoNLL and OntoNotes datasets drop performance on the newly created dataset. They also observe the opposite, that models trained on the new dataset perform worse on CoNLL and OntoNotes. However, combining the datasets with the diverse English data gives better performance both for US and UK English and on English languages spoken elsewhere.    \n\nWhile the paper is well written and have sound arguments, some details are lacking about the annotation process and the dataset created. Since this is the main contribution, I see this as a serious flaw. Moreover, in the ethical section the authors do not discuss the ethical aspect of how they have collected their dataset (which is also lacking in the discussion in the paper).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1Schq1Rs0S", "reviewer": "Reviewer_cg6A", "summary": "", "questions": "[A] Can you elaborate on some of the missing details about the data annotation process? Specifically, how many annotators were there (per region)? Are the annotators native speakers of the dialect being annotated? Do you have any measures of inter-annotator agreement? \n\n[B] Line 156: I think these data statistics are missing -- could you include a pointer to them in the text? It would be helpful to know the breakdown across train/dev/test for the new dataset, as well as within each region.\n\n[C] Line 238-240: how were these hyperparameters chosen?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind comments emphasizing the value of the new resource introduced and the easy-to-read paper.\n\nIn the criticism section, you ask for more description of the dataset itself, as well as ask where the promised table on line 156 is.  Regrettably, it appears that it was omitted from the submitted draft of the paper.  We have recreated the statistics in our answer to Reviewer nGXt.\n\nAs per the previous response, we can also clarify the data annotation process.  The question of whether or not the annotators are from the regions in question is an interesting one.  As it turned out, the annotators were all either in Ghana or the US.   If we had sourced the work for each article to annotators from the country in question, that would have improved the overall quality of the annotations, but this would be impractical considering the variety of regions and countries we used.  Nevertheless, we think we have a dataset with high quality annotations.\n\nHyperparameters were chosen by hill climbing on the validation set of the CoNLL dataset, then reused for each training pass.  The purpose was not to find the absolute best parameters for each dataset, but to compare the scores with and without the regional data.  We can put a table of the hyperparameters in a final draft of the paper.\n\nThe work creating the models should be very reproducible.  The conversion scripts to join our dataset with CoNLL, or OntoNotes with our dataset, will both be made public as part of the release of the dataset.  Furthermore, we used the default settings when retraining both CoreNLP and Stanza, so the models themselves can be recreated as needed.\n\nThank you for the suggestion of including more quantitative analysis of the experiments.  We can certainly do that, such as by giving statistics on the types of errors per region.\n\nI would also say something about thanks for the suggestion of including more quantitative analysis of the experiments and say that we will try to do that, such as by giving the statistics on types of errors by region.\n\nThe question of inter-annotator agreement is a good one.  We present the following discussion of the annotation process, along with Inter Annotator Agreement and dataset transparency.\n\nWe labeled data using Datasaur’s labeling platform, from where we outsourced labeling management to MLTwist, an ML labeling firm. MLTwist used native English speakers, mostly in Ghana, who did not necessarily have dialect-specific knowledge for each language region we used. An overwhelming majority of the labelers were English speakers from Africa. As mentioned above, it would have been ideal to have representation from all of the regions we labeled, but that would not have been practical.\n\nWe collected inter-annotator agreement scores from Datasaur, the data labeling platform used to create our dataset. We note that the results found do not measure true IAA between two independent annotators, but rather an index for correction from reviewers (see process below). Datasaur computes inter annotator agreement scores using Cohen’s Kappa, while taking into account the possibility of chance agreement. \n\nOverall, we employed 7 labelers. Our labeling system structure was composed of having one annotator per file, with a team of 2–3 reviewers for each file. The annotator was instructed to label all of the named entities in the articles, before the reviewers (at least one representative of MLTwist and at least one paper author) would confirm or adjust the initial annotator’s labels. When reviewing, the MLTwist and Stanford reviewer would communicate to achieve a consensus decision which is why the Datasaur platform computes agreement between the whole reviewer team and the annotator, rather than comparing per reviewer. Hence, the inter annotator agreement score we compute is between the reviewer team (MLTwist + Stanford) and a single annotator. \n\nThe following table maps annotators (whose identities have been redacted for privacy) to their inter-annotator score. We computed an average inter-annotator score by averaging the score for each labeling batch. Finally, we also compute a joint inter-annotator agreement score (representing the average of all labelers) by averaging all inter-annotator scores.\n\nAverage inter-annotator agreement scores, using Cohen's Kappa.  Approximately 50 articles per batch\n\n\n| Annotator ID | Cohen's Kappa | Batches annotated |\n| :----: | :----: | :----: |\n| 1 | 73.95 | 8 |\n| 2 | 83.52 | 3 | \n| 3 | 68.43 | 3 |\n| 4 | 79.53 | 5 |\n| 5 | 77.74 | 1 |\n| 6 | 77.87 | 1 |\n| 7 | 90.57 | 2 |\n| Joint | 77.47 | 23 |\n\nUnlike many reported NLP inter-annotator agreement scores which are simply percentage agreement, we measure using Cohen’s Kappa. Furthermore, we find that the agreement scores between reviewers and the annotators were quite good. While scales for Cohen’s Kappa are arbitrary and heavily context-dependent, we feel that our results reflect a thorough labeling process. \n\nThank you for your consideration and helpful suggestions to improve our paper for the final revision. We will be sure to include additional explanations surrounding the annotation process in the final revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work points out an important flaw in the claim that English NLP is solved when this is not the case for non-standard dialects of English. It then documents the issue on a specific task (NER).\n- The paper also points to a generally applicable method for improving models on non-standard dialects -- being cognizant of these data variations and including them in your data annotation and training process.\n- They annotate and release a new NER dataset on lower-resource English language settings.\n- The paper is clearly written and easy to read.", "reasons_to_reject": "- A number of details about the data annotation process are not included in the paper (see Question A in the Questions to Authors)\n- All of the analysis presented in this work is qualitative (Section 5). While these insights are interesting, it would be helpful to provide some quantitative analysis of the data and model performance as well. For example, the paper could include statistics about the types of errors discussed in the text across the different regions in the dataset; another interesting analysis might be the overlap of named entities between the original NER corpora and the new dataset and how that related to performance, as well as the distribution of new/unknown entities across regions of the data.", "questions_to_authors": "[A] Can you elaborate on some of the missing details about the data annotation process? Specifically, how many annotators were there (per region)? Are the annotators native speakers of the dialect being annotated? Do you have any measures of inter-annotator agreement? \n\n[B] Line 156: I think these data statistics are missing -- could you include a pointer to them in the text? It would be helpful to know the breakdown across train/dev/test for the new dataset, as well as within each region.\n\n[C] Line 238-240: how were these hyperparameters chosen?", "paper_topic_and_main_contributions": "This paper asks whether Named Entity Recognition (NER) taggers trained on American/British English datasets for the task generalize to other dialects of English. To test this, it introduces a new NER dataset that contains texts from news sources from many different regions of the world where English is spoken. This dataset is then tested on many standard NER systems trained on one of two popular NER datasets (CONLL-03 and OntoNotes). The experiments show that models trained on these datasets perform much worse than models trained specifically for this setting, though the contextualized models generalize better than the non-contextualized ones. They also show that training the NER tagger with a combination of the original training data and the new data allows the models to generalize without losing performance on the original dataset.\n\nThe contributions of this paper are as follows:\n- Introduces a new NER resource covering many dialects of English that previously did not have any data for this task.\n- Demonstrates systems trained on existing NER datasets are not able to generalize to these new evaluation settings.\n- Show that incorporating data from these dialects into the training process for these systems improves generalization to these dialects while also achieving similar performance on the original dataset's evaluation set.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- It would be helpful to make some of the section titles more specific, particularly in the case of Sections 4 and 6, since they cover very similar topics.\n- Lines 269-275: A combined model is discussed here, but it isn't included in Table 1. It would be helpful to add it to the table or include a pointer in the text to the model you are discussing."}}]

Author rebuttal:

Meta-review: This paper introduces a new dataset for English NER that focuses on English varieties different from US and UK English. The authors then test various existing NER tools and models to evaluate their robustness to non-standard varieties.

The reviewers found that this paper addresses an important gap in current NLP that focuses on the varieties of English with the most (native) speakers. The error analysis was also found to be instructive. In contrast, the reviewers mentioned that some confounding factors (temporal drift etc.) were insufficiently taken into account, and that the paper lacked details about the annotation process, the statistics of the resulting dataset, and potential ethical issues in connection with the annotation. The authors addressed all these points in their rebuttals and promised to add them to the paper. As a result, all three reviewers were able to raise their scores.
Decision: Accept-Findings