Order-Agnostic Data Augmentation for Few-Shot Named Entity Recognition (2024)

Abstract: Data augmentation (DA) methods have been proven to be effective for pre-trained language models (PLMs) in low-resource settings, including few-shot named entity recognition (NER). However, existing NER DA techniques either perform rule-based manipulations on words that break the semantic coherence of the sentence, or exploit generative models for entity or context substitution, which requires a substantial amount of labeled data and contradicts the objective of operating in low-resource settings. In this work, we propose orderagnostic data augmentation (OADA), an alternative solution that exploits the often overlooked order-agnostic property in the training data construction phase of sequence-tosequence NER methods for data augmentation. To effectively utilize the augmented data without suffering from the one-to-many issue, where multiple augmented target sequences exist for one single sentence, we further propose the use of ordering instructions and an innovative OADA-XE loss. Specifically, by treating each permutation of entity types as an ordering instruction, we rearrange the entity set accordingly, ensuring a distinct input-output pair, while OADA-XE assigns loss based on the best match between the target sequence and model predictions. We conduct comprehensive experiments and analyses across three major NER benchmarks and can significantly enhance the few-shot capabilities of PLMs with OADA. Our code is available at https://github.com/Circle-Ming/OADA-NER.

heading: 1 Introduction
text: Named entity recognition (NER) (Tjong Kim Sang and De Meulder, 2003; Doddington et al., 2004) has been one of the most long-standing and fundamental tasks. However, the effectiveness of NER systems is often constrained by the need for substantial high-quality, annotated datasets, which are
∗Work done during the internship at DAMO Academy.
costly and labor-intensive to acquire, necessitating innovative approaches to data scarcity (Rijhwani et al., 2020; Yang and Katiyar, 2020).
By introducing more reasonable samples, data augmentation (DA) methods have been proven to be effective solutions in scenarios where labeled data is scarce (Şahin and Steedman, 2018; Kobayashi, 2018; Wei and Zou, 2019). Existing DA approaches for NER can be roughly divided into two categories: (1) rule-based manipulations and (2) text-to-text generations. Among them, rulebased manipulations utilize predefined rules for automatic modifications in texts, including word deletion, reordering, and substitution (Min et al., 2020). However, due to the discrete nature of natural language, these techniques struggle to maintain the semantic coherence. Conversely, text-to-text DA techniques such as DAGA (Ding et al., 2020), MELM (Zhou et al., 2022), and ENTDA (Hu et al., 2023) augment texts by substituting entities or their contextual elements using predictions from pretrained language models (PLMs). Nevertheless, to execute effective augmentations, these approaches require a substantial amount of labeled data to train the augmentation model for generating synthetic
7792
text, which poses a challenge in scenarios with scarcer labeled data, such as few-shot NER tasks, and violates the fundamental goal of operating in low-resource settings.
In this work, we propose an innovative OrderAgnostic Data Augmentation framework OADA as an alternative solution, and demonstrate that utilizing a fundamental yet often overlooked aspect (i.e., the inherent unordered nature of target entities, which we refer to as the “order-agnostic property”) of NER tasks can be extremely effective for data augmentation. To exploit this property for DA, we firstly notice the gap between the training data construction phase and prediction evaluation phase of traditional sequence-to-sequence (seq2seq) NER systems. As shown in Figure 1, given a source sentence “CNN’s David Ensor is reporting for us”, conventional seq2seq NER methods (Yan et al., 2021) will perceive entities with a fixed order and utilize only “[1, ORG], [1, 4, PER]” as the target sequence during training, while its equivalent “[1, 4, PER], [1, ORG]” is disregarded. However, when “[1, ORG], [1, 4, PER]” and “[1, 4, PER], [1, ORG]” are treated as the model predictions for evaluation, both of them will be recognized as correct generations. This fixed order assumption and the gap between training and evaluation phases cause the loss of many viable samples. Thus, in OADA, we seek to recognize the entities within a sentence as an unordered set, and treat different entity arrangements all as equivalent and also accurate generations. This perspective significantly broadens the range of acceptable target sequences for a given sentence, thereby introducing a novel and effective DA method. As shown in Figure 1, when incorporating the order-agnostic property into training data construction, both entity sequences are taken into consideration as reasonable target sequences.
In OADA, it is hypothesized that different entity arrangements provide equivalent information. Therefore, we further propose the use of ordering instructions and an innovative cross entropy (XE) loss OADA-XE to jointly fine-tune PLMs on these sequences together, without suffering from the oneto-many issue (Gu et al., 2018), where multiple possible entity sequences exist for the same sentence. For example, in Figure 1, if we make no distinction between “[1, ORG], [1, 4, PER]” and “[1, 4, PER], [1, ORG]” and pair them directly with the same input sentence, a PLM will have trouble deciding whether to generate “4” or “ORG” after “1”. To address this issue, we show that our proposed strate-
gies can effectively discriminate between different entity arrangements in two different aspects (i.e., inter-type and intra-type) as illustrated in Figure 2. Concretely, we first prioritize entity types as our primary factor of arranging entities. By treating each permutation of entity types like “LOC, ORG, MISC, PER” as an ordering instruction, we concatenate it with the input sentence and arrange the entity sequence following this instruction, ensuring a unique input-output pair. Second, within individual entity types, OADA-XE will align loss based on the best match between the target sequence and the model predictions. For example, in Figure 2, the first two predictions are both acceptable since it is possible to assign a match between them and the gold sequence, while the third prediction is wrong as it contradicts the ordering instruction and unruly rearranges the entities.
In summary, our work presents several key contributions, including: (1) To the best of our knowledge, we for the first time investigate to perform DA from the order-agnostic perspective. (2) We propose a novel DA framework OADA for fewshot NER, which enables jointly fine-tuning PLMs on various entity arrangements together by utilizing the ordering instructions and an innovative OADAXE loss. (3) To demonstrate the effectiveness and generalization ability of OADA, we conduct comprehensive experiments and extensive analysis on three datasets, including one for nested NER, with five representative PLMs (e.g., BERT (Devlin et al., 2019), BART (Lewis et al., 2020), Flan-T5 (Chung et al., 2022), LLaMA2 (Touvron et al., 2023) and ChatGPT (OpenAI, 2022)), and show notable improvements over established baselines.

heading: 2 Related Work

heading: 2.1 Data Augmentation for NER
text: As shown in Table 1, we compare OADA with two main categories of existing DA methods for NER.
Rule-based Manipulation Rule-based DA methods are primarily performing token-level manipulations, including synonym substitution (Wei and Zou, 2019; Cai et al., 2020), word deletion (Kobayashi, 2018) and reordering (Min et al., 2020). By utilizing their predefined rules, these methods can generate large amounts of synthetic texts efficiently. However, they might suffer seriously from the token-label misalignment issue (Zhou et al., 2022), where an entity token might be replaced with alternatives that mismatch its original label. In addition, due to the discrete nature of natural language, token-level manipulations will introduce incoherent replacement to the text.
To avoid the token-label misalignment issue, Dai and Adel (2020) proposed to randomly replace the whole entity mentions with alternative entities of the same type. Despite the effectiveness on flat NER tasks, its application to complex NER subtasks like nested NER remains challenging and they still inevitably introduce incoherent substitution (Hu et al., 2023). In this work, we make augmentations to only target entity sequences while preserving the input sentences intact, ensuring the semantic coherence of the texts.
Text-to-Text Generation Text-to-text generation based augmentations in NER tasks are mostly inspired by back-translation (Sennrich et al., 2016; Fadaee et al., 2017; Dong et al., 2017; Hou et al., 2018; Xia et al., 2019), which aims to transfer texts between languages while preserving their original meaning. When applied to token-level NER tasks, approaches like DAGA (Ding et al., 2020) and MELM (Zhou et al., 2022) explored to fine-tune PLMs using linearized sequences that merge token labels with tokens, and create augmented texts on corrupted sentences. However, these methods face challenges in handling nested entities, limiting their applicability across varied NER tasks. The more recent ENTDA (Hu et al., 2023) offers a context replacement strategy that adapts well to various NER tasks. Nevertheless, a common limitation among all these text-to-text generation methods is their reliance on a substantial amount of labeled data for training augmentation models, which contradicts the objective of low-resource settings.
On the contrary, by leveraging the order-agnostic property shared across NER tasks, our approach OADA possesses the strengths of both rule-based and text-to-text DA methods and can effectively augment data without the need for additional model
training, while introducing no change to the sentence thereby maintaining the semantic coherence.

heading: 2.2 Few-Shot NER
text: Few-shot NER is a challenging task of identifying entities using only a small number of labeled examples (Wiseman and Stratos, 2019; Yang and Katiyar, 2020; Ding et al., 2021). Recent advancements in this field fall into mainly two categories: metric-based and prompt-based methods. This categorization also aligns with the two diverse few-shot NER settings. Metric-based methods aim to identify entities by learning a feature space and classifying test samples using nearest class prototypes or neighbor samples (Snell et al., 2017; Fritzler et al., 2018; Yang and Katiyar, 2020; Das et al., 2022; Zhang et al., 2023). However, most of these studies assume a rich-resource source domain, which is in contrast to the real world application scenarios that only very limited labeled data is available.
Following Gao et al. (2021), the more practical few-shot setting makes minimal assumptions about available resources and only provides few samples each class for training. By leveraging linguistic prompts to adapt PLMs to NER tasks, prompt-based methods (Cui et al., 2021; Ma et al., 2022; Lee et al., 2022; Shen et al., 2023; Xu et al., 2023) demonstrate impressive performance in fewshot scenarios, which can be further enhanced with more substantial data. In this work, we follow the more challenging few-shot setting where only few samples are provided for each entity type, and implement OADA over prompt-based methods, with a comprehensive comparison in Section 4.3.
Additionally, there has recently been a remarkable development in large language models (LLMs) such as GPT series (Brown et al., 2020; OpenAI, 2022), which show impressive capabilities in few-shot prompting and in-context learning (ICL) (Jimenez Gutierrez et al., 2022; Chen et al., 2023). Thus, in this work, we specifically examine OADA’s generalization ability on LLMs and include the results in Table 4.

heading: 3 Order-Agnostic Data Augmentation

heading: 3.1 Formulation
text: The NER tasks aim at detecting all the spans that can represent entities within a given sentence X . The N entities in sentence X form the corresponding entity set E = {y1, y2, ..., yN}. In OADA, we view the entities as the basic units and only
perform entity-level rearrangement, preserving the integrity of individual entities. An entity yi can be represented as a tuple yi = (si, ti), where si, ti represent the entity span and type of yi respectively. The generation procedure can be formulated as:
L1 = − N∑
i=1
logP (yi|X,Y<i). (1)

heading: 3.2 Augmenting Data via Entity Rearranging
text: In OADA, we define that two entity sequences are equivalent if and only if they possess same entity sets, thus two equivalent sequences can vary greatly in how they arrange sequence components (i.e., entities). For the given sentence X , a target sequence Y i is defined as a specific arrangement of entities from the set E, such as Y i = [yN−1, yN , ..., y1]. We further define the arrangement space as O = {O1, ..., OI}, which encompasses all possible arrangements of entities from E. According to our definition, Y i can be uniquely determined when provided with the corresponding Oi. This implies that the cardinality of the set O, denoted as I = |O|, directly controls the size of our augmented texts on E.
By randomly shuffling and rearranging the entity set E, we will acquire N ! different arrangements in O, which are computationally infeasible to be included into training. Moreover, it is hard to simultaneously model the relations between X and a set of target sequences like Y i via standard XE (Shao et al., 2019). To tackle this one-to-many issue, we
introduce the ordering instructions, which complements the construction of input-output pairs and precisely controls the quantity of O.

heading: 3.3 Constructing Unique Input-Output Pairs
text: In OADA, we firstly separate the one-to-many mapping into multiple one-to-one mappings in the intertype aspect, and ensures unique input-output pairs.
Instead of augmenting data with all possible N ! arrangements, we propose an alternative strategy: prioritizing entity types as the primary factor of arranging entities.1 Concretely, consider a certain dataset with a set of entity types T = {t1, t2, ..., tl} such as {LOC, ORG, MISC, PER}. By arranging entities based on their types, we can define Oi ∈ O as a random permutation p of entity types from T (e.g., [PER, LOC, ORG, MISC]). In accordance with the entity type arrangement p, entities in E are firstly grouped by their types and be subsequently arranged into Yp by different groups. For example, in Figure 2, entities in type PER and entity “(MIT, ORG)” are firstly organized into two different groups. And these two groups are then arranged into different entity rearrangements following distinct ordering instructions.
To maintain unique one-to-one mappings between X and a set of its corresponding target sequences like Yp, we view p as the unique ordering instruction and concatenate X with p as [p;X]. In the real generation procedure of a PLM, p will indi-
1We discuss alternative rearranging factors in Appendix A.
cate to the model which entity type to focus on at a certain generation step. In this way, we address the one-to-many issue in the inter-type aspect, and the complexity of arrangement space O experiences a significant reduction from O(N !) to at most O(l!).
For example, in Figure 2, when paired with the ordering instruction “[PER, LOC, ORG, MISC]”, the entity set can be uniquely rearranged into the target sequence “[(He, PER), (a professor of physics at MIT, PER), (MIT, ORG)]”. Thus, the third prediction “[(He, PER), (MIT, ORG), (a professor of physics at MIT, PER)]” will be judged as a wrong prediction.

heading: 3.4 Calibrating Predictions with OADA-XE
text: In this section, we introduce how to alleviate the one-to-many issue within a certain entity type, namely intra-type issue. This issue will be raised since we can not discriminate the entities of a same type with only ordering instructions and standard XE. As an example, in Figure 2, the entities in the first prediction “[(a professor of physics at MIT, PER), (He, PER), (MIT, ORG)]” are precisely following the given ordering instruction, but will be penalized when performing standard XE, since the first two PER entities are not positionally aligned with those in the target entity sequence. Unlike inter-type mapping where entities possess different types and can be divided by their types, the only difference between these intra-type entities is their absolute position. Thus, the one-to-many mapping between these intra-type entities arises.
To mitigate this issue, we utilize a novel XE loss for OADA as inspired by Du et al. (2021). For example, in Figure 3, standard XE loss requires a strict per-position match between target entities and model predictions, thus will heavily penalize the predicted sequence “[(a professor of physics at MIT, PER), (He, PER)]”, although it is equivalent to the target entity sequence from our perspective.
We define the OADA-XE objective as finding the best ordering Oi ∈ O to minimize the XE loss:
LOADA-XE = argmin Oi∈O
( − logP (Oi|X) ) . (2)
It is important to note that, unlike the token-level comparison in Du et al. (2021), our work will calibrate at the entity-level (i.e., span and type) as shown in Figure 3. If a best match can be found between the model prediction and the target entity sequence, this prediction will be regarded as a correct prediction. Furthermore, since a large portion of rearrangements in O are invalid and log loss is sensitive to invalid or noisy sequences which can cause large changes in model behavior (Kang and Hashimoto, 2020), we start the training with XE loss to make sure the model can effectively deal with the large search space of orderings O, and apply OADA-XE loss during training with an annealing schedule (Ma et al., 2018; Clark et al., 2019; Du et al., 2021) to gradually teach the model to alleviate the per-position penalty:
L = (1− τ) ∗ L1 + τ ∗ LOADA-XE, (3) where τ is linearly increased from 0 to 1 throughout training. After equipped with OADA-XE, we can successfully prevent the intra-type issue during calibrating the predictions.

heading: 4 Experiments
text: Datasets We conduct comprehensive experiments on two flat NER datasets and one nested NER dataset in several few-shot settings. For flat NER datasets, we choose CoNLL-2003 (Tjong Kim Sang and De Meulder, 2003) and MITMovie (Liu et al., 2013) from two different domains. For MIT-Movie, we randomly select 15% samples from its training set as the development set. For nested NER, we conduct experiments on ACE2005 (Doddington et al., 2004), using the same data split as Lu and Roth (2015).

heading: 4.1 Comparison with Different DA Methods
text: We firstly compare OADA with other NER DA methods introduced in Section 2.1. To make fair comparison, we follow the low-resource setting in ENTDA where 10% training data is available, and the results are reported in Table 2. Among them, DAGA and MELM can not address nested entities and we only include their results on CoNLL2003. For these text-to-text methods, we follow their settings and augment the training set by 3x. Since our method does not contribute directly to the diversity of the data, we study the impact of the number of permutations used for each entity set in Section 5.2 and utilize 20 random rearrangements for ACE-2005 and all (4! = 24) for CoNLL-2003 throughout our paper. From the results, we can observe that OADA can be applied to various NER subtasks and achieves the greatest improvement compared with other NER DA methods. Besides, our approach also eliminates the need for training any additional models unlike text-to-text methods. In addition, it is important to note that the entities in ENTDA are still maintaining a fixed order. This implies that their augmented data can be further enhanced with OADA as demonstrated by the results of “+ENTDA+OADA”, which also shows the generalization ability of our method.

heading: 4.2 Experimental Settings
text: As introduced in Section 2.2, in this work, we follow Ma et al. (2022) where only K samples of each entity type are provided. We conduct experiments in K = {5, 10, 20, 50} settings for supervised fine-
tuning, and K = {1, 2, 3} for ICL with LLMs. For all the settings, we adopt the same sampling strategy as Yang and Katiyar (2020) and report the mean performance over three splits. To demonstrate that OADA can be uniformly applied to different models and even other few-shot methods, we implement OADA over BERT, PromptNER (Shen et al., 2023), BART, BART-NER (Yan et al., 2021)) for supervised fine-tuning, and perform ICL over Flan-T5-XXL, LLaMA2-13B-Chat and ChatGPT.2
We compare OADA with several strong and competitive few-shot methods: Template-NER (Cui et al., 2021), BART-NER, SEE-Few (Yang et al., 2022), Ent-LM (Ma et al., 2022), FIT (Xu et al., 2023) and PromptNER. Please refer to Appendix B for a detailed introduction.

heading: 4.3 Main Results
text: Table 3 shows the results of our proposed OADA compared with these baselines. Based on the results, we have the following observations: (1) OADA consistently improves the performance of both discriminative and generative PLMs. Despite the varying baseline performances of models like BERT and BART, OADA achieves notable improvements across all few-shot settings. Particularly in 5-shot settings, OADA enhances BERT’s F1 score by 15.29 and 14.07 on CoNLL2003 and ACE-2005, and those of BART by 11.88 and 15.36, respectively. (2) OADA can be generally applied to various tagging schemes, further advancing the capabilities of existing few-shot NER methods. To further demonstrate the generalization ability of our approach, we also apply OADA to previous SOTA few-shot NER methods with different tagging schemes. For instance, while BART-NER utilizes start and end indexes to represent entity spans which differs significantly from its pretraining corpus, OADA effectively adapts to this scheme. The enhanced performance of methods like PromptNER and BART-NER with OADA underscores its generalization potential across diverse NER applications. (3) Among the promptbased methods, OADA shows to be the most efficient and effective. Typical prompt-based methods Template-NER and FIT suffer from slow inference due to span enumeration, while our inference speed is 20.17x and 15.90x faster compared to them. PromptNER, while efficient in inference, requires
2We implement OADA over BERT based on the seq2seq LM architecture of UNILM (Dong et al., 2019) and also compare with BERT of sequence-tagging.
lengthy sequences combining multiple templates with the input sentence, leading to increased memory demand. Our approach OADA, by leveraging the order-agnostic property, not only streamlines this process but also achieves superior performance across different PLMs compared to these existing methods. A more detailed analysis to the computational efficiency is included in Appendix C.

heading: 5 Analysis

heading: 5.1 Ablation Study
text: We conduct ablation experiments on fine-tuning BART to analyze the contributions of individual components of OADA. Results in Figure 4 show that, while directly incorporating augmented data can already improve the performance, addressing one-to-many issue with ordering instructions and OADA-XE can further boost the effect by a large margin, which demonstrates the effectiveness of
each component of OADA. Results of ablation experiments to BERT are included in Appendix B.

heading: 5.2 Analysis of Permutations on T
text: As introduced in Section 3.3, we choose to rearrange the entity sequences given the permutations of the entity types and reduce the arrangement space to at most l!, where l is the number of entity types. However, for the datasets like ACE-2005 and MIT-Movie with 7 and 12 entity types separately, acquiring l! entity rearrangements is still impractical. Thus, we conduct experiments to investigate the effect of the number of permutations. From Figure 5, we can observe that most improvement comes from the first 20 permutations, and subsequent improvement is marginal with a great increase in training time. In real practice, we randomly select 20 rearrangements for ACE-2005 and MIT-Movie and all (4! = 24) for CoNLL-2003.

heading: 5.3 Entity Recall in Different Positions
text: We examine the recall performances of entities at different positions within a sentence. The detailed results are illustrated in Figure 6. For vanilla BART on flat CoNLL-2003 5-shot, entities appearing later are more likely to be recalled. Conversely, in nested ACE-2005, entities in the middle exhibit notably lower recall probabilities. We hypothesize that this pattern arises because, for flat NER, the dependencies among entities are less pronounced. While for nested NER, where latter entities may encompass the former entities, leading to a cascading effect where errors in earlier entities adversely impact the recall of subsequent ones. With OADA, we disrupt the arrangement of entities, so that subsequent entities are possible to appear before their preceding entities, thereby reducing the dependency of an entity’s recall on previous entities. We further include some related case studies in Appendix D.

heading: 5.4 ICL of LLMs with OADA
text: Recently, there has been a remarkable development of LLMs. However, the huge number of parameters, coupled with their significant demand for computational resources, make ICL a more practical approach. To verify our applicability, we conduct experiments over distinct LLMs: Flan-T5-XXL,
LLaMA2-13B-Chat and ChatGPT (i.e., gpt-3.5turbo). The results are shown in Table 4. From the table, we can see that, although the results of LLMs are relatively higher than fine-tuning smallscale PLMs, OADA can be still valid and improve their performance further, which demonstrates the efficacy of OADA in utilizing ICL with LLMs. We include the details of how we perform ICL with LLMs in Appendix E.

heading: 5.5 The Impact of Model Scales
text: We study how sensitive ICL with OADA is to the model scales as shown in Figure 7. Although we can observe significant performance improvements as the scale of LLaMA model increases, our method is consistently effective on these models with different scales. As shown in the results of LLaMA-70B, OADA can still be valid even when its model scale is much larger than other versions, and improves its base performance on CoNLL-2003 by more than 2.00 F1 scores. Besides, we can also observe from the results on CoNLL-2003 that, the larger base LLaMA models we use, the improvement coming from OADA appears to be slowly declining. We assume that this trend comes from the saturation of the demonstration used by ICL. Since the instruction tuning process of LLMs will incorporate data from NER
tasks (Longpre et al., 2023), they show superiority performance on CoNLL-2003. To demonstrate this, we further employing OADA over LLaMA on ACE-2005 which LLMs will be less familiar with (Zhang et al., 2024). And the results indicate that our method shows good robustness in this scenario where more demonstrations can be helpful for LLMs to understand the tasks, rather than including extra but redundant demonstrations.

heading: 6 Conclusions
text: In this paper, we propose a novel data augmentation method OADA by leveraging the often-overlooked order-agnostic property of NER. Furthermore, to jointly utilize the augmented data together without suffering from the one-to-many issue, we introduce the use of ordering instructions and an innovative OADA-XE loss, tackling the issue in inter-type and intra-type aspects separately. Experiments on three major NER benchmarks, and extensive analyses demonstrate the effectiveness of OADA.

heading: Acknowledgements
text: This work was substantially supported by DAMO Academy through DAMO Academy Research Intern Program.
Limitations
Despite the effectiveness of OADA, there are still some potential directions worth exploring and we leave as future work.
Reordering Factors In our work, to reduce the huge search space of O and provide a clear distinguishing criteria, we choose entity types as the basic reordering factor, and also discuss other potential candidates in Appendix A. From our analyses, we know that the original “left-to-right” order achieves the worst performance among them, which also demonstrates our claim that the strict order assumption does not need to be maintained. There are still other reordering factors worth discovering, which may further improve the effect. Maybe our current choice (i.e., entity types) is not the optimal solution, but our work also provides enough clues for subsequent work based on this, and the design we proposed to solve the one-tomany mapping problem will still have enough application scenarios in future work.
More Diverse Inference Strategies and Best Order During the training stage of OADA, we un-
lock the capability of generative models to perform diverse inference. As introduced in Section 3.3, we will construct a unique ordering instruction for each specific permutation of entity types to tackle inter-type mapping problem. Thus, for each input sentence, we will use |O| different ordering instructions, each indicating a specific generation priority. Besides, there are already some works (Mitchell et al., 2022; Wang et al., 2023) demonstrating that performing consistency-check will largely improve the performance of ICL with LLMs. Thus, we also conduct experiments on majority-voting based inference strategy and the results are shown in Appendix F. From the results, we found that the performance of applying majority-voting is not significantly better than the inference guided by the best ordering instruction. Furthermore, we also conduct some studies to select the best order and include them in Appendix G. From the results, we also can not observe significant improvement by selecting the “top-k” order. In our work, we consistently adopt all entity arrangements for fine-tuning and the original order for decoding, as reported in the main results of OADA. We believe that if there is a proper algorithm that can select entities inside all target sequences generated with different instructions, the performance will be further improved.

heading: A Analysis of Rearranging Factors
text: In this paper, we propose to leverage the orderagnostic property for data augmentation. Unlike traditional NER systems that perceive entities as sequences with a fixed left-to-right order, we rearrange the entity set following a certain permutation of entity types. Another natural choice for rearrangement is based on the positions of entities, for example, specifying the 3rd entity in a left-to-right order to be generated at the beginning. Besides, some research (Jie and Lu, 2019; Yu et al., 2020) has demonstrate the effectiveness of dependency relationships in modeling the relations between the entities. Thus, we compare these four rearranging factors as shown in Figure 8. Since there is only one unique target sequence following a specific order (i.e., left-toright or dependency relation), we only augment data with factors as positional information and entity types. Our findings indicate that rearranging entities based on types provides a clearer distinction among them, leading to enhanced performance. Furthermore, the proposed OADA-XE, addresses the one-to-many issue prevalent in both augmenting scenarios, resulting in improved effect. This evidence strongly supports our claim that adopting an order-agnostic approach, especially when combined with entity type-based rearrangement, significantly benefits NER systems.

heading: B Supplementary Results of Fine-Tuning
text: We compare OADA with several strong and competitive few-shot methods: TemplateNER (Cui et al., 2021), the very first promptbased method for few-shot NER, enumerates all spans within a sentence for entity typing, with extremely high time complexity. BARTNER formulates the NER task as an entity span sequence generation task, which can be directly applied to various NER subtasks. SEE-Few (Yang et al., 2022) reformulates span classification as a textual entailment task and leverages both the contextual clues and entity type information. Ent-LM (Ma et al., 2022) proposes a template-free prompt tuning method and induces the language mod-
els to predict label words at entity positions, while Ent-LM+struct leverages the viterbi algorithm to further boost the performance. FIT (Xu et al., 2023) is a focusing, bridging and prompting framework specially for fewshot nested NER, which also suffers from the huge time complexity because of enumeration. PromptNER unifies entity locating and entity typing in one prompt and reduplicates this prompt for many times in the input sequence, which represent different entities.
In Section 5.1, we provide the results of ablation study on fine-tuning BART. We also conduct ablation study over BERT and show the effectiveness of each component and the results are illustrated in Figure 9.

heading: C Computational Efficiency
text: As we show in Section 4.3, OADA can be applied to different methods, including various tagging schemes. In addition, as a DA method, we introduce more reasonable data into training, resulting in the inevitable increase of training time. During inference, since we perform standard decoding, we share the similar time complexity with the backbones. Thus, to provide the detailed analyses to the computational efficiency of OADA, we provide the performance comparison in terms of training time, memory against backbone models as shown in Table 5 (with a single NVIDIA V100).
The increase in time is acceptable because the data size is enhanced by 24 times. Meanwhile, with OADA-XE, training time and memory have increased slightly, but it also brings considerable performance improvements. Moreover, as discussed in Section 4.3, PromptNER will combine the duplication of a template with the input sentence, leading to increased memory demand as shown
in the table. By comparing the results of BERT-NER+OADA w/o X and PromptNER, our method can enhance the performance of BART-NER by a large margin and even make up for the gap in their performance, with an acceptable time increase.

heading: D Case Study
text: In this section, we delve into case studies, as detailed in Table 6, and hope to provide some clues for understanding why OADA can help improve the performance of backbone PLMs. As discussed in Section 5.3, we found that without OADA, latter entities exhibit notably lower recall probabilities than former entities, leading to a cascading error effect.
In the first example sentence, vanilla BART will make a mistake and predict “staff” as an “ORG” entity, thus all the following entities will be disregarded. Conversely, our OaDAenhanced approach, despite missing an internal entity, does not perpetuate this error. It successfully identifies the complete entity “staff sergeant tom ridge”, demonstrating a more robust performance in nested NER scenarios. In another example “here is cnn ’ s candy crowley with some war stories .”, vanilla BART fails to recognize “cnn ’ s candy crowley” as a singular entity due to its earlier misclassification of “cnn”. Our approach, however, correctly identifies this compound entity, showcasing its superior entity recognition capabilities. Finally, in the sentence “he ’ s a professor of psychiatry at nyu , chairman of the forensic
panel .”, vanilla BART even overlooks the entity “nyu” due to its previous error, while our method accurately identifies all entities within the sentence. Table 6 summarizes these examples, providing a clear comparison between the performance of vanilla BART and our OADAenhanced model.
E In-Context Learning with OADA
In this section, we discuss how we perform ICL with OADA.
E.1 Input-Output Template for ICL
We provide the details of our prompts.
Instruction Prompt
Instruction: please extract entities and their types from the input sentence, all entity types are in options.
Option Prompt
Option: T .
Ordering Instruction Prompt
Following the order: p.
T and p are the set of entity types and a permutation of T . One example is provided in Figure 10.

heading: F Majority-Voting Inference
text: In Table 7, we compare 3 different ordering instructions for inference: 1. random, for this setting, we run experiments with 10 different random sampled instructions and report the best and worst performance. 2. originalorder, we utilize “following the order: from
left to right”. 3. majority-voting, we perform majority-voting on the generated entity sequences guided by 10 randomly sampled instructions. If an entity appears in more than 5 sequences, it will be considered in the final entity set. From the result, we see that performing majority-voting will not increase the performance by a large margin, and is not efficient. Thus, in our main results, we only adopt the original order for inference, and leave this as future work.

heading: G Best Order Selection
text: In our work, we regard OADA as a data augmentation method and propose the use of ordering instructions and OADA-XE to address the one-to-many issue. The basic assumption of utilizing all target sequences jointly is that they are equivalent to each other. In this section, we further discuss whether there could be a standing-out order that improves the performance of PLMs trained with the rearrangements of its guidance. Given a specific permutation of entity types, the output logits are utilized to compute the entropy on the target sequence arranged with this permutation.
For each sentence, its target sequence can be rearranged by l! times, where l represents the number of entity types. Thus, we can select top-k different permutations based on the average entropy of all instances constructed by a certain permutation. We show the performance of training PLMs on these selected permutations in Figure 11. From the results, we can see that top-k can outperform randomk in nearly every setting. However, selecting the top-k permutations can not get comparable with utilizing more permutations, and they already calculated all the entropy loss based on the entity sequences in all orders, which means selecting top-k permutations will not bring a more efficient training or achieve better performance. Thus, we still adopt all (4! = 24 for CoNLL-2003, 20 for MIT-Movie and ACE2005) permutations for training, rather than selecting the best orders.

heading: Future Work
text: Thus, in our main results, we only adopt the original order for inference, and leave this as future work