A Multilingual Evaluation of NER Robustness to Adversarial Inputs
Year: 2023

Abstract:
Adversarial evaluations of language models typically focus on English alone. In this paper, we performed a multilingual evaluation of Named Entity Recognition (NER) in terms of its robustness to small perturbations in the input. Our results showed the NER models we explored across three languages (English, German and Hindi) are not very robust to such changes, as indicated by the fluctuations in the overall F1 score as well as in a more finegrained evaluation. With that knowledge, we further explored whether it is possible to improve the existing NER models using a part of the generated adversarial data sets as augmented training data to train a new NER model or as fine-tuning data to adapt an existing NER model. Our results showed that both these approaches improve performance on the original as well as adversarial test sets. While there is no significant difference between the two approaches for English, re-training is significantly better than fine-tuning for German and Hindi.

heading: 1 Introduction
text: NLP systems are traditionally evaluated and compared against a gold standard, which is generally immutable. Recent research has shown that even the NLP systems that perform well on the standard test set show a significant drop in performance even for small perturbations in the input test data, across a range of NLP tasks (Gardner et al., 2020). Although this strand of research covered many tasks, it has been exclusively focused on English, with a few exceptions (e.g., Shmidman et al. (2020), for Hebrew). Further, to our knowledge, the primary usage of such adversarial test sets have been in either evaluating NLP models or in usage as additional, augmented data to improve model robustness, without much focus on using the new
∗Work done during an internship at National Research Council, Canada
data for fine-tuning, instead of re-training. A better understanding of fine-tuning with adversarial test sets is important, and useful in most real-world scenarios, where we may not have access to the original training data while having access to the trained model itself.
Named Entity Recognition (NER) is among the most common NLP tasks both in research and in industry applications (Lorica and Nathan, 2021). Although much progress has been made on NER over the past decades, existing NER systems were also shown to be sensitive to small changes in input data in the past (Lin et al., 2021; Vajjala and Balasubramaniam, 2022). Table 1 shows an example of how predictions can change with minor changes in input, for one of the state of the art NER models1. Going by the original sentence, all three sentences should carry the LOC tag for the entity in the sentence. However, that is not the case, as the outputs shows. Clearly, small, and seemingly harmless changes are changing model predictions.
Even recent large language models such as ChatGPT struggle with sequence tagging tasks such as NER, across multiple languages(Qin et al., 2023; Lai et al., 2023; Wu et al., 2023), which clearly illustrates that NER is far from being considered solved. In this backdrop, considering the significance of NER in research and practical scenarios, a better understanding of how a model’s predictions
1https://huggingface.co/flair/ ner-english
40
change with slight changes in input becomes an important issue to address. Hence, we explore the following questions in this paper:
1. How does the performance of NER models across three languages change with small changes to the original input?
2. How does retraining an NER model with adversarial data augmentation compare with adversarial fine-tuning of NER across languages?
Our contributions are summarized as follows:
• We conducted the first comparative study of the robustness of NER models beyond English, covering three languages, in a space where all previous work focused on English alone.
• We report first results on the comparison between data augmentation and adversarial finetuning for NER, for all the three languages.
• We show how existing methods for data augmentation can be repurposed to develop language-agnostic methods to generate adversarial test sets for NER.
Starting with a conceptual background (Section 2), we describe our methods for adversarial dataset creation (Section 3) and the general experimental setup (Section 4) followed by a detailed discussion of our results (Section 5) and a summary (Section 6), focusing on the ethical impacts, limitations and broader impact towards the end.

heading: 2 Related Work
text: Evaluating using multiple datasets is one of the ways to assess the robustness and generalization capabilities of NLP models. Developing challenge sets, and generating adversarial datasets that can potentially cause a model to fail, are some possibilities in this direction (Isabelle et al., 2017; Ettinger et al., 2017; Glockner et al., 2018; Gardner et al., 2020). Adversarial data generation in NLP focuses on surfacelevel perturbations to the input text, proposing various means of insertion/deletion/swapping of words/characters/sentences (Jia and Liang, 2017; Gao et al., 2018; Ribeiro et al., 2018). Other approaches such as paraphrasing (Iyyer et al., 2018), generating semantically similar text using other
deep learning models (Zhao et al., 2018; Michel et al., 2019), using a human-in-the-loop (Wallace et al., 2019b) were also explored in the past. While many of the proposed methods are black-box approaches, assuming no knowledge about the NLP models themselves, some of the approaches are white box, with more access to the inner workings of a model (Liang et al., 2018; Blohm et al., 2018; Wallace et al., 2019a), and some models implement both (Li et al., 2019). We focus on one specific NLP task - NER, and only work on blackbox methods in this paper.
In terms of the strategies to protect models against adversarial attacks, the most common approach followed by past NLP research has been to incorporate adversarial data into the training process, through data augmentation, or using adversarial training as a regularization method (See Goyal et al. (2022) and Zhang et al. (2020) for a detailed overview). Using adversarial data for full re-training of a model (as is the case with data augmentation) assumes access to the original data and the model, which is not practical in many realworld scenarios. One possibility to explore in such cases is to test whether using adversarial data to fine-tune a trained NER model improves its robustness. We compare using the adversarial data (through data augmentation) for re-training an NER model versus using it only for fine-tuning a previously trained NER model in this paper.
Adversarial Testing and Data Augmentation in NER: Adversarial testing approaches for NER in the previous work was entirely done for English datasets, and primarily focused on methods replace entities in the original test set with new ones using gazetteers or other means (Agarwal et al., 2020; Vajjala and Balasubramaniam, 2022). Lin et al. (2021) used entity linking and masked language models coupled with an existing NER model to generate adversarial test sets for NER. More recently, Das and Paik (2022) used grammatical case information to generate adversarial test sets for NER. Simoncini and Spanakis (2021) proposed other ways to make small changes to the context around entities in a sentence, to generate adversarial test sets. Other related work (Mathew et al., 2019; Ding et al., 2020; Zhu et al., 2021) focused on data augmentation for NER, that require training of new, additional models. While we use our adversarial datasets for data augmentation too later in the paper, the novelty of the current research, compared to this existing body
of work focusing on NER, comes in two forms:
1. While all previous work exclusively focused on English NER so far, we perform experiments with three languages - English, German and Hindi.
2. The approaches we used are lightweight, language agnostic means to generated adversarial test sets for NER, which do not rely on the availability of additional tools like entity linkers, and do not also need any additional training to generate the datasets.

heading: 3 Adversarial Test Set Creation
text: Our adversarial dataset creation methods can be broadly classified into two approaches - replacing entities and changing contexts. All except one method work for all the three languages we tested, and can be easily expanded to add other languages. Relevant code and generated datasets are provided as supplementary material2.

heading: 3.1 Replacing Entities
text: We implemented two methods that replace the entity occurrences in the test set with another entity of the same category, keeping the rest of the sentence unchanged. Thus, they don’t change the grammatical structure of the sentence and tell us how much the NER systems learn beyond memorizing the entities.
Random Sampling (RS) All the entity occurrences of the same type are shuffled throughout the test set in this approach. This is a simple and easily portable method across languages, which could serve as a strong baseline.
Gazetteers (Faker) This approach replaces existing entities with new entities of the same type using an existing gazetteer. Faker3 is a python library that generates fake data for various application purposes, which supports multiple languages, and regions. We used it to replace the Person and Location entities in all the datasets, in all three languages we experimented with. Vajjala and Balasubramaniam (2022) used Faker for adversarial NER test sets, but they used only English (on OntoNotes dataset). We randomly choose among three locale
2https://dx.doi.org/10.6084/m9. figshare.22674079
3https://faker.readthedocs.io/
settings for English (USA, Canada, India) and German (Germany, Austria, and Switzerland) respectively for replacement. For Hindi, there was only one locale setting provided (HI-IN).

heading: 3.2 Changing the Context
text: These approaches deal with changing the context in which the entities occur in a sentence by making small changes to the tokens around it.
Masking (Mask) We leveraged transformer based pre-trained language models trained with a masked language modeling objective to change the context in the original test datasets. We masked up to three randomly chosen non-entity tokens per sentence, and used the language model to generate those tokens, thereby creating new sentences with the same entities, but slightly altered contexts. Since there are multilingual pre-trained language models available, this approach is applicable to multiple languages.
Paraphrasing (Para) The objective behind this approach is to alter the structure of the input sentences, while keeping the named entities intact. About 500 sentences were randomly chosen from the test set and fed to an online, subscription based english paraphraser, Quillbot4, which was shown to generate better paraphrases than other approaches such as back-translation or using GPT-3 in recent research (Shiri et al., 2022) and in our initial evaluations. The paraphrased output obtained for each sentence is then taken and the entity tokens from the original test set are mapped to the entity tokens of the paraphrased sentences with the respective entity tags, leaving the rest of the tokens as O. Limiting to 500 sentences is primarily due to the fact that Quillbot did not provide an API, and there is a limit to the amount of text one can paraphrase per request, even with subscription.
There are some challenges with this approach, though. While Quillbot’s paraphrases when we choose the "Fluency" setting are of good quality, and are always grammatically correct, they sometimes alter the entities themselves (e.g., United States can become U.S. in the paraphrased version) or change the original tokenization of the dataset. In such cases where an automatic mapping between the entity tags from the original sentence and tokens of the paraphrased sentence failed, we discarded the sentence from our test set. Note that this
4https://quillbot.com/
approach is compatible only with English and we aren’t aware of any reliable paraphrasers for other languages. To our knowledge, full paraphrasing wasn’t used for adversarial test sets in NER before. A total of 399 sentences from conll03-en, 373 sentences from mconer21-en, and 378 sentences from wnut17 were finally used as test sets.

heading: 3.3 Changing Entity + Context
text: Masking + Random Sampling (M+R) All the previous approaches focused on either trying to alter the context alone or replace the entities alone. This approach, does both by combining masking with random sampling. Since both these approaches are straightforward and can work across languages, they can be combined to create a new adversarial test set in all three languages. Table 2 shows one example of how the various approaches alter a single sentence from one of the English datasets. As with most data augmentation or adversarial generation approaches in NLP, some of the generated text may contain minor grammatical errors, as seen in in Mask and M+R settings. However, Compared to other common approaches such as those that involve insertion/deletion/swapping of words/characters or back-translation, the potential errors introduced by masking alone are minor. Further even the original datasets themselves contain sentences with such minor errors. So, we don’t foresee this affecting the main findings of the paper. A publicly available NER model 5 predicts correctly for all these sentences.

heading: 4 Experimental Setup
text: We experimented with three English, two German, and one Hindi NER datasets 6. The first half of our
5https://huggingface.co/flair/ ner-english
6While it is theoretically possible to add more languages to this paper, the choice of these languages is motivated by the
experiments focused on testing the NER models on adversarial test sets, following which we compared the effects of adversarial fine-tuning and data augmentation on the performance of the NER models. All the experiments were carried out on a high performing computing resource that ran an Nvidia A100 GPU with 32 GB RAM. Further details on the experimental setup are as follows.

heading: 4.1 Datasets
text: NER datasets from three popular shared tasks - conll03 (Tjong Kim Sang and De Meulder, 2003), multiconer21 (Malmasi et al., 2022) and wnut17 (Derczynski et al., 2017) were considered and their corresponding language subsets for English (conll03-en, mconer21-en, wnut17), German (conll03-de, mconer21-de) and Hindi (mconer21hi) were used. Conll03 datasets have a tag set with four entity types (PER, LOC, ORG and MISC), and multiconer21 and wnut17 share the tag set consisting of six entity types (PER, LOC, CORP, GRP, CREATIVE-WORK and PROD). While the sentences in conll03 came from news articles, multiconer21 was collected from three domains (wikipedia sentences, questions, search queries), and wnut17 consisted of sentences from social media sources such as twitter, youtube and reddit.7 We created five adversarial test sets for each of the three English datasets, four adversarial sets for each of the two German datasets and the Hindi dataset respectively, resulting in a total of 27 adversarial test sets covering three languages and six datasets.

heading: 4.2 NER Models
text: We used a combination of existing state of the art NER models (if available) and fine-tuning
authors’ familiarity with the languages, which is essential for qualitative analysis
7Some statistics about the datasets are shown in Appendix A.1.
a pre-trained language model for all the languages/datasets. They are explained below.
TNER and Fine-tuned BERT: TNER8 is a Python library to train transformer based language models for NER tasks (Ushio and CamachoCollados, 2021), which has pre-trained for mconer21-en, wnut17-en and conll03-en datasets. We use these models to perform our experiments for English NER. We will refer to this approach as tner. While TNER’s model hub had publicly available models for other languages as well, the performance of the models was lower compared to the state of the art, and hence, we fine-tuned the multilingual BERT (Devlin et al., 2019) model hosted on Huggingface9 for NER on German (conll03-de, mconer21-de) and Hindi (mconer21-hi) datasets. We will refer to this approach as mbertft.
We followed the same approach for the later retraining (i.e., training the NER model again using the original training data augmented with adversarial data) and fine-tuning (using adversarial data is used only to fine-tune the existing NER model) experiments in Section 5.4, and report the results with tner for English and mbertft for German and Hindi, as this setup gave the best results even in those experiments. For adversarial fine-tuning, tner and mbertft were fine-tuned for 4 epochs, learning rate was set to 0.0001 and the batch size was set to 16. For the Adversarial re-training however, the models were trained for 6 epochs since it is being trained from scratch, while the other hyper parameters were kept the same. The hyperparameter settings were from the original BERT paper (Devlin et al., 2019). We used 60% of the adversarial test set for data augmentation+training/fine-tuning and used the remaining 40% to test the approaches. For both adversarial fine-tuning and re-training, we report the average F1 score over 10 runs, with different random seeds, and compare them in terms of statistical significance using a paired t-test.
Stanza: Stanza (Qi et al., 2020) is a Python based NLP toolkit that hosts a few pre-trained NER models trained with a BiLSTM+CRF architecture. We evaluated Stanza’s pre-trained conll-en and conllde models using our generated adversarial test sets.
Flair: Flair is a popular NLP library that is widely used for performing NLP tasks (Akbik et al.,
8https://github.com/asahi417/tner 9https://huggingface.co/
bert-base-multilingual-uncased
2019). We evaluated the pre-trained NER models provided by Flair for conll-en and conll-de.
There are other NER models that offer slightly better performance than Flair/Stanza/BERT-fine tuning, and there are other large language models to explore, but we focused on publicly available/downloadable models for NER and and easily re-implementatble benchmarks (e.g., BERT finetuning) in this paper. It would be interesting to extend this to cover additional methods in future, but we limit to a smaller set of models to maintain a manageable number of experiments and do a meaningful analysis later.

heading: 4.3 Evaluation
text: Micro-F1 score from seqeval (Nakayama, 2018) was used as the evaluation metric to test the robustness of the NER models, as it is the most commonly reported measure for this task.
Nervaluate: Nervaluate10 is a Python library for performing a more fine-grained evaluation of NER, and is based on the metrics from a SemEval 2013 task (Segura-Bedmar et al., 2013). Apart from giving a single F1 score, it calculates the efficiency of the model using five error categories: correct, incorrect, partially correct, missing labels (an entity tagged as non-entity) and spurious labels(a nonentity tagged as entity). The error metrics are reported in four formats: strict (both entity span and entity type match), exact (entity span matches, irrespective of the type), partial (partial span match, irrespective of the type), and type (some overlap between gold annotation and system prediction). We used nervaluate to compare the performance of NER on the original and adversarial test sets, to understand what kind of errors affect their performance.

heading: 5 Results
text: Our experiments aimed at understanding the robustness of NER models to adversarial test sets and exploring whether adversarial data augmentation and fine tuning will help boost the performance on adversarial test sets. The results of these experiments are discussed below.

heading: 5.1 Adversarial Testing
text: Adversarial test sets were created by implementing the approaches mentioned in Section 3 and tested
10https://github.com/MantisAI/ nervaluate
on pre-trained NER models for all the three English datasets, and for conll03-de. As described earlier, we also fine-tuned a multilingual bert model for German and Hindi datasets. Tables 3, 4 and 5 summarize the performance of all the NER models we tested on, for English, German and Hindi respectively, in terms of the micro-F1 score.
For English NER, results from Table 3 show a clear drop in NER model performance for two out of the three datasets (conll03 and mconer21), with the largest drop seen in the test set obtained by paraphrasing using Quillbot. However, it is important to note that the size of the test set for paraphrasing is far smaller (399, 373 and 378 sentences respectively for conll-03, mconer21 and wnut17) than the original test set, as explained earlier in Section 4. So, the drop is not directly comparable with other test sets which are larger in size.
Apart from this, there are also differences among individual methods for all the datasets. For example, Faker dataset appears to have had a stronger
effect on all the three models trained on conll03-en dataset compared to multiconer-en dataset. Considering that there is generally similar drop across all three models of conll03-en, we would speculate that the difference in performance is due to the differences in the dataset composition and entity categories.
For wnut17, the drop is largest for M+R, followed by Mask. Considering that only those test sets involving masking resulted in a drop for this dataset, we could safely attribute this drop to the difference in the nature of the data that the masked language model was exposed to, compared to the very noisy social media data in wnut17. An interesting aspect of testing with wnut17 model is that the model’s performance was better on 3 out of 5 adversarial test sets, compared to the original test set. We believe it is important to note in this context that the NER model for wnut17 also has the lowest performance on the original test set among the three datasets and wnut17 is the noisiest data of them all (social media content). Considering that a model with a much lower overall F1 score, and trained on the most noisy dataset among the three, was still relatively more robust to three of the adversarial test sets, future research should perhaps also take a closer look at other, additional means of evaluating NER models instead of using the standard F1 score as the sole criterion to choose the best model, as was also suggested by other recent research on the topic. (Vajjala and Balasubramaniam, 2022).
For both German and Hindi NER (Table 4 and Table 5), we observe that the drop is the highest for masking+random sampling datasets in both the languages. Between the two German datasets, the drop in f1 scores for adversarial datasets appear to be much larger for mconer21 than conll03. A possible reason could be the poorer performance of the original mconer21 model itself.
While there are several other interesting results to compare and discuss across languages and datasets, overall, these results indicate that the NER models are not fully robust when tested against new test sets created with easily replicable, generally language-agnostic approaches. Their performance drop is larger when context altering approaches are employed. One question we asked ourselves at this point is: what exactly are the adversarial test sets changing in the model performance?

heading: 5.2 Fine-Grained Evaluation
text: We used nervaluate to understand what aspect of NER performance is mainly affected by the adversarial data. Since there are many models, train and test sets, we choose one train/test set and model combination for this analysis. Figure 1 shows the analysis for TNER’s pre-trained NER model trained on conll03-en dataset, as a comparison between the original test set and the M+R adversarial test set (Figures 2 and 3 in the appendix show the same analysis for German and Hindi respectively).
Apart from the overall decline in performance, a closer look at the ‘correct’ and ‘incorrect’ categories indicate that this NER model’s performance resulted in a larger drop for ‘organization’ entity type in English. This could be because of the ambiguity involved in the entity type itself, as not every ‘organization’ entity suits every context in which that entity type appears. Surprisingly, although the ‘misc’ category suffers from the same problem, we don’t see a large dip for that category.
Figure 1 also indicates that there are more missed entities in the adversarial test set compared to spurious labels (non-entities tagged as one of the entity types). We compared the "strict" versus "exact" evaluation schema in nervaluate, to understand whether this increase in missed entities is a result of getting the span right, but identifying the entity type wrong. This comparison showed that while there is a 9% drop in the overall F1 score between the original and the adversarial test sets with ‘strict’ evaluation, there is only a 3% drop in terms of identifying the entity spans correctly (More details in Table 9 in the Appendix. Tables 10 and 11 in the appendix show this comparison for German and Hindi datasets).

heading: 5.3 Qualitative Error Analysis
text: Apart from quantitative analysis, we also did some manual analysis to understand what kind of transformations led NER models to predict erroneous tags. Table 6 shows some of the correctly tagged examples taken from conll03-en test set and their adversarial counterparts with some tagging errors, using Stanza’s NER model.
When the entity ‘Nicole’ was replaced with the entity ’Major’ (Examples a and b), the first one resulted in the model missing the entity (Major not recognised as person), and the second example saw the replaced entity ‘Timor-Leste’ (another name for the country East Timor) being mis-identified as an ORG instead of a LOC. While the transformed sentence in c appears very different from its source, there is only one entity, and a human reader would not find it difficult to identify the entity. However, the model missed identifying it altogether. Finally, both the masked examples (d and e) made very minor changes to the original sentence. But the model predictions changed from ORG − > LOC for the entity ‘Victoria’ in a sentence, and LOC − > ORG for the entity ‘Indianapolis’ in the other. In the last two cases, it can be argued that the original labels are ambiguous themselves. While that is, indeed, the case, the issue we particularly highlight is the way model predictions changed because of textual changes that should not really cause label changes. Similar trends can be observed in German and Hindi as well (Examples for German and Hindi are in the appendix in Table 12 and Figure 4). While it is definitely possible to do further qualitative analysis, we would speculate that combining this kind of analysis with explainable NLP approaches may give a more complete picture in
the future on why NER model predictions fluctuate even for minimal perturbations in input text.

heading: 5.4 Adversarial Fine-Tuning
text: One approach to make models more robust to adversarial inputs is to include such data in building the model itself. We explored two methods in that direction: a) augmenting the training data with part of the adversarial dataset and re-training the NER model from scratch b) using adversarial data to finetune an existing NER model. The second method is especially useful in real-world scenarios where we have access to a trained model, but not to the original training data itself. We compared both the methods for all three languages, training one NER model per language (with conll-en, conll-de, and mconer-hi datasets respectively), and compared the performance with the M+R test set and the original test set in each case. As mentioned in Section 4, 60% of the adversarial test set was used for augmented re-training/fine-tuning and the remaining 40% was used as test data. Table 7 summarizes the results of this experiment.
While there are no significant differences between adversarial fine-tuning and re-training for English, and both give a 5% performance boost on adversarial test set without compromising on the original test set performance, for both German and Hindi, re-training was significantly better than finetuning with both test sets (p<0.001). A possible
reason for this difference could lie in the relatively superior performance of the original model itself. Re-training could be more useful when the performance of the original model is poor, as was the case for German and Hindi. Interestingly, just finetuning still improved performance on adversarial test sets by over 5% for all languages.
To connect these results back to our second research question (Section 1), considering that finetuning still resulted in better adversarial test set performance in all cases, and since that would not require access to the original training data itself, it could be a feasible, easily implementable approach
to improve the robustness of NER models without compromising on the original model performance. Re-training can be preferred when we have access to the original data and the model. Note that in both the cases, we are assuming no means to procure additional manually labeled training data, and the focus is on improving an NER model’s robustness to adversarial input without compromising on its performance on normal test data.

heading: 6 Conclusions and Discussion
text: We explored simple, language agnostic approaches to generate adversarial test sets for NER and demonstrated their generalizability by testing on six datasets covering three languages - English, German and Hindi. While exact results differ depending on language/datasets, our key findings from these experiments can be summarized as follows:
1. NER models for all three languages are sensitive to adversarial input.
2. Adversarial fine-tuning and re-training could improve the performance of NER models both on original and adversarial test sets, without requiring additional manual labeled data.
The proposed approaches and tested languages/models are by no means comprehensive, and extending this work to include more NER models, adding new languages, and developing new adversarial data generation methods for NER is an obvious next step, as the current results provide enough evidence on the sensitivity of state of the art NER models to adversarial inputs. The methods we employed for adversarial fine-tuning/re-training too are just a starting point towards exploring the use of adversarial data in building more robust NER systems. We only explored one paraphraser for this task. The usefulness of the recent generative language models for creating such test data can be an interesting next step in this direction.

heading: 7 Limitations
text: The adversarial test sets based on masked language models can introduce new noise into the sentence context, as there is no way to automatically ensure grammatical correctness. However, there were many cases where such introduction of noise did not affect the predictions, in all three languages. Further, adversarial datasets are expected to introduce such noise, as is seen in other research on
the topic for other tasks such as sentiment analysis, and the goal of such research is also to understand model robustness in the presence some noise. It is relevant to mention in this context that the NER datasets we considered already consist of other noise and ungrammatical examples such as score cards of sporting matches (conll03-en), social media content (wnut17) and fully lowercased sentences with weakly supervised annotations (mconer21). Further, masking does not alter the entities themselves, and only changes the non-entity tokens. So, the NER models still see the same entities. While there are no established means of quantifying the quality of adversarial datasets to our knowledge, exploring human-in-the-loop approaches to select appropriate examples to include in the final adversarial test set can be one way to address the issue.

heading: 8 Ethics and Impact Statement
text: The paper described the creation of several adversarial test sets for three languages. We used publicly available datasets for this purpose, and the research did not involve human participants. All the datasets we generated and the code to generate them are shared as supplementary material11, for replication and to further this line of research. Our goal in this paper was to study the sensitivity of state of the art NER systems to adversarial data, and suggest ways to overcome it. As such, the generated datasets are expected to be used only for that purpose, and the limitations of current approaches are discussed in the previous section. Apart from this, since the paper focuses on more foundational question of evaluating NER systems in general, we do not foresee any other potential risks involved with this research.
Broader Impact Considering the number of practical usecases of NER across industries, and the growth of multilingual NLP, NER evaluation beyond English is more important than ever before. In this paper, we explored a previously unexplored space for Named Entity Recognition, i.e., evaluating NER systems beyond English for their sensitivity to adversarial input, which will hopefully lead into better evaluation strategies when developing NER systems across languages in future.
11https://dx.doi.org/10.6084/m9. figshare.22674079

heading: Acknowledgements
text: We thank Justin Lee and Gabriel Bernier-Colborne for their feedback on an earlier draft, and Edwin Thomas for feedback on the final draft of the paper.

heading: A Appendix
text: A.1 Dataset Statistics:
We used the dev set from mconer21 to create adversarial test sets for all the three languages, considering the large size of its test set.
A.2 Detailed Evaluation

heading: Future Work
text: , evaluating NER systems beyond English for their sensitivity to adversarial input, which will hopefully lead into better evaluation strategies when developing NER systems across languages in future. 11https://dx. doi. org/10. 6084/m9. figshare. 22674079