ACLM: A Selective-Denoising based Generative Data Augmentation Approach for Low-Resource Complex NER
Year: 2023

Abstract:
Complex Named Entity Recognition (NER) is the task of detecting linguistically complex named entities in low-context text. In this paper, we present ACLM (Attention-map aware keyword selection for Conditional Language Model fine-tuning), a novel data augmentation approach, based on conditional generation, to address the data scarcity problem in lowresource complex NER. ACLM alleviates the context-entity mismatch issue, a problem existing NER data augmentation techniques suffer from and often generates incoherent augmentations by placing complex named entities in the wrong context. ACLM builds on BART and is optimized on a novel text reconstruction or denoising task we use selective masking (aided by attention maps) to retain the named entities and certain keywords in the input sentence that provide contextually relevant additional knowledge or hints about the named entities. Compared with other data augmentation strategies, ACLM can generate more diverse and coherent augmentations preserving the true word sense of complex entities in the sentence. We demonstrate the effectiveness of ACLM both qualitatively and quantitatively on monolingual, crosslingual, and multilingual complex NER across various low-resource settings. ACLM outperforms all our neural baselines by a significant margin (1%-36%). In addition, we demonstrate the application of ACLM to other domains that suffer from data scarcity (e.g., biomedical). In practice, ACLM generates more effective and factual augmentations for these domains than prior methods.

heading: 1 Introduction
text: Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP) that aims to detect various types of named entities (NEs) from text. Recently, there has been
1Code: https://github.com/Sreyan88/ACLM ∗These authors contributed equally to this work.
considerable progress in NER using neural learning methods that achieve state-of-the-art (SOTA) performance (Wang et al., 2021; Zhou and Chen, 2021) on well-known benchmark datasets, including CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003) and OntoNotes (Schwartz et al., 2012). However, these datasets are designed to evaluate the performance on detecting “relatively easy” NEs like proper names (e.g., people such as “Barack Obama,” locations such as “New York,” or organizations such as “IBM”) in well-formed, contextrich text that comes from news articles (Augenstein et al., 2017). On the other hand, complex NER benchmarks like MultiCoNER (Malmasi et al., 2022) present several contemporary challenges in NER, including short low-context texts with emerging and semantically ambiguous complex entities (e.g., movie names in online comments) that reduce the performance of SOTA methods previously evaluated only on the existing NER benchmark datasets. Our experiments reveal that the performance of the current SOTA NER method (Zhou and Chen, 2021) (previously evaluated only on the CoNLL 2003 dataset) drops by 23% when evaluated on MultiCoNER and 31.8% when evaluated on a low-resource setting with just 500 training samples (more details in Table 8). Thus, we emphasize that research on building systems that can effectively detect complex NEs in the text is currently understudied in the field of NLP.
In the past, researchers have made several attempts at building supervised approaches to detect complex and compositional noun phrase entities in sentences (Doddington et al., 2004; Biggio et al., 2010; Magnolini et al., 2019). However, the scarcity of annotated training data for building effective systems has always been a challenge. Data augmentation has been shown to be an effective solution for low-resource NER (Ding et al., 2020; Liu et al., 2021; Zhou et al., 2022). In practice, though these systems perform well and generate
104
coherent augmentations on common NER benchmark datasets with easy proper noun NEs, they fail to be effective for complex NER, often generating incoherent augmentations. We first argue that certain types of complex NEs follow specific linguistic patterns and appear only in specific contexts (examples in Appendix 4), and augmentations that do not follow these patterns impede a NER model from learning such patterns effectively. This sometimes also leads to augmentations with context-entity mismatch, further hurting the learning process. For e.g., unlike proper names, substituting complex NEs from other sentences in the corpus or replacing them with synonyms (Dai and Adel, 2020a) often leads to augmentations where the NE does not fit into the new context (e.g., swapping proper names across sentences might still keep the sentence coherent but swapping the name of a book with a movie (both creative work entity) or the name of a football team with a political party (both group entity) makes it incoherent). Fine-tuning pretrained language models (PLMs), similar to priorwork (Ding et al., 2020; Liu et al., 2021; Zhou et al., 2022), fail to generate new context around complex NEs or completely new NEs with the desired linguistic patterns due to low-context sentences and the lack of existing knowledge of such linguistically complex NEs (examples in Fig. 3). This leads to in-coherent augmentations and poses a severe problem in knowledge-intensive tasks like biomedical NER, where non-factual augmentations severely hurt learning. Our experiments also reveal that introducing new context patterns around NEs proves to be a more effective data augmentation technique for complex NER than diversifying NEs (ACLM vs. MELM in Table 1).
Main Results: To overcome the aforesaid problems, we formulate data augmentation as a conditional generation task and propose ACLM, a conditional text generation model that generates augmentation samples by introducing new and diverse context patterns around a NE. ACLM builds on BART (Lewis et al., 2020) and is fine-tuned on a modification of the text reconstruction from corrupted text task, a common denoising-based PLM pre-training objective. In contrast to other PLM pretraining strategies, which randomly mask a portion of the text for corruption, our modified objective is based on selective masking, wherein we mask all other words in the sentence except the NEs and a small percentage of keywords related to the NEs.
We refer to this corrupted sentence as a template, and it serves as input to the model for both the training and generation phases. These keywords are other non-NE tokens in the sentence that provide contextually relevant additional knowledge or hints to BART about the complex NEs without the need of retrieving knowledge from any external sources. We select these keywords using attention maps obtained from a transformer model fine-tuned on the NER task, and they help the PLM overcome the problem where it might not possess enough knowledge about a semantically ambiguous complex NE (example in Fig. 3). Training ACLM on this modified objective allows us to generate diverse, coherent, factual, and high-quality augmentations given templates. We also propose mixner, a novel algorithm that mixes two templates during the augmentation generation phase and boosts the diversity of augmentations. Our primary contributions are as follows:
• We propose ACLM, a novel data augmentation framework specially designed for lowresource complex NER. Compared with previous methods in the literature, ACLM effectively alleviates the context-entity mismatch problem by preserving the true sense of semantically ambiguous NEs in augmentations. Additionally, to accompany ACLM, we propose mixner, which boosts the diversity of ACLM generations.
• We qualitatively and quantitively show the benefits of ACLM for monolingual, crosslingual, and multilingual complex NER across various low-resource settings on the MultiCoNER dataset. Our proposed ACLM outperforms all other baselines in literature by a significant margin (1%-36%) and generates more diverse, coherent, and high-quality augmentations compared to them.
• We perform extensive experiments to study the application of ACLM in three other domains, including science and medicine. ACLM outperforms all our baselines in these domains (absolute gains in the range of 1%- 11%) and generates more factual augmentations.

heading: 2 Background and Related Work
text: Complex NER Background: Complex NER is a relatively understudied task in the field of NLP.
Building on insights from Augenstein et al. (2017), we discuss key reasons behind high performance on common NER benchmark datasets and try to understand why modern SOTA NER algorithms do not work well on complex NER benchmarks: (1) Context: Most of the common benchmark datasets are curated from articles in the news domain. This gives them several advantages, including rich context and surface features like proper punctuation and capitalized nouns, all of which are major drivers of success in these datasets (Mayhew et al., 2019). In contrast, for entity recognition beyond news text, like search queries or voice commands, the context is less informative and lacks surface features (Guo et al., 2009; Carmel et al., 2014); (2) Entity Complexity: Data from news articles contain proper names or “easy” entities with simple syntactic structures, thus allowing pretrained models to perform well due to their existing knowledge of such entities. On the other hand, complex NEs like movie names are syntactically ambiguous and linguistically complex and which makes Complex NER a difficult task (Ashwini and Choi, 2014). Examples of such entities include noun phrases (e.g., Eternal Sunshine of the Spotless Mind), gerunds (e.g., Saving Private Ryan), infinitives (e.g., To Kill a Mockingbird), or full clauses (e.g., Mr. Smith Goes to Washington); (3) Entity Overlap: Models trained on these common benchmark datasets suffer from memorization effects due to the large overlap of entities between the train and test sets. Unseen and emerging entities pose a huge challenge to complex NER (BernierColborne and Langlais, 2020).
Complex NER: Prior work has mostly focused on solving the entity complexity problem by learning to detect complex nominal entities in sentences (Magnolini et al., 2019; Meng et al., 2021; Fetahu et al., 2022; Chen et al., 2022). Researchers have often explored integrating external knowledge in the form of gazetteers for this task. Gazetteers have also proven to be effective for low-resource NER (Rijhwani et al., 2020). GemNet (Meng et al., 2021), the current SOTA system for complex NER, conditionally combines the contextual and gazetteer features using a Mixture-of-Experts (MoE) gating mechanism. However, gazetteers are difficult to build and maintain and prove to be ineffective for complex NER due to their limited entity coverage and the nature of unseen and emerging entities in complex NER.
Data Augmentation for Low-Resource NER: Data Augmentation to handle data scarcity for lowresource NLP is a well-studied problem in the literature and is built on word-level modifications, including simple synonym replacement strategies (Wei and Zou, 2019), or more sophisticated learning techniques like LSTM-based language models (Kobayashi, 2018), Masked Language Modeling (MLM) using PLMs (Kumar et al., 2020), auto-regressive PLMs (Kumar et al., 2020), or constituent-based tagging schemes (Zhou et al., 2019). However, most of these methods, though effective for classification tasks, suffer from tokenlabel misalignment when applied to token-level tasks such as NER and might require complex preprocessing steps (Bari et al., 2020; Zhong and Cambria, 2021). One of the first works to explore effective data augmentation for NER replaces NEs with existing NEs of the same type or replaces tokens in the sentence with one of their synonyms retrieved from WordNet (Dai and Adel, 2020b). Following this, many neural learning systems were proposed that either modify the Masked Language Modelling (MLM) training objective using PLMs (Zhou et al., 2022; Liu et al.) or use generative language modeling with LSTM LMs (Ding et al., 2020) or mBART (Liu et al., 2021), to produce entirely new sentences from scratch. However, all these systems were designed for low-resource NER on common benchmark datasets and failed to generate effective augmentations for low-resource complex NER with semantically ambiguous and complex entities.

heading: 3 Methodology
text: In this section, we give an overview of our approach. Fig. 1 represents the entire workflow of our ACLM data augmentation framework. A sentence is first passed through a fine-tuned XLMRoBERTa fine-tuned on only gold data to generate the attention map for each token in the sentence. This attention map is then used to selectively mask the sentence and create a template. This template is then used as an input to optimize the model on the text reconstruction objective for fine-tuning ACLM: the model is asked to reconstruct the entire original sentence from only the content in the template. While generating augmentations, ACLM follows the same template generation process in addition to adding two templates through mixner, which we discuss in detail in Section 3.3.

heading: 3.1 Template Creation
text: To corrupt a sentence and create a template, we follow a 4-step process described below:
1. Keyword Selection: For each sentence in our training corpus, we first obtain a set of non-NE tokens in the sentence that are most attended to by its NEs. We call these tokens keywords. For our research, we consider a non-NE token as a keyword if the NEs in the sentence contextually depend on them the most. We measure contextual dependency between NE and non-NE tokens using attention scores from attention maps extracted from a transformer-based NER model fine-tuned only on gold data. We hypothesize that attention heads in a transformer when fine-tuned for NER, formulated as a token-level tagging task, tend to pay the highest attention to the most contextually relevant tokens around it. Thus, formally put, consider a sentence with a total of T tokens comprised of tother non-NE and tentity NE tokens. Our primary aim is to find the top p% of tother tokens, which we call keywords. To calculate the total attention score that each token in the sentence assigns to each other token, we sum up the attention scores across each of the heads in the transformer network and across the last a layers (a = 4 in our case). Different heads in different layers tend to capture different properties of language, and taking the average attention scores across the last 4 layers ensures that diverse linguistic relations are taken into account while choosing the keywords (e.g., syntactic, semantic, etc.). This also makes the keyword selection process more robust, as in low-resource conditions the attention maps may be noisy, and the NEs might not be focusing on the right context always. Additionally, the choice of just the last four layers is inspired by the fact that the lower layers have very broad attention and spend at most 10% of their attention mass on a single token (Clark et al., 2019). Note tentity might be comprised of (1) multiple contiguous tokens forming an individual NE and (2) multiple such individual NEs. To handle the first case, inspired from Clark et al. (2019), we sum up the attention scores over all the individual tokens in the NE. For the second case, we find tattn for each individual NE and take a set union of tokens in these tattn. Thus, as an extra pre-processing step, to improve robustness, we also ignore punctuations, stop words, and other NEs from the top p% of tother tokens to obtain our final keywords. We provide examples of templates in Appendix C. 2. Selective Masking: After selecting the top p% of tother tokens in the sentence as keywords, we now have K non-NE keyword tokens and E entity tokens. To create the template, we now substitute each non-NE token not belonging to the K with the mask token and remove contiguous mask tokens.
3. Labeled Sequence Linearization: After we have our initial template, inspired by Zhou et al. (2022), we perform labeled sequence linearization to explicitly take label information into consideration during fine-tuning and augmentation generation. Similar to Zhou et al. (2022), as shown in Figure 1, we add label tokens before and after each entity token and treat them as the normal context in the sentence. Additionally, these label tokens before and after each NE provide boundary supervision for NEs with multiple tokens.
4. Dynamic Masking: Post labeled sequence linearization, our template goes through further masking wherein we dynamically mask a small portion of the K keywords during each iteration of training and generation. To be precise, we first sample a dynamic masking rate ε from a Gaussian distribution N (µ, σ2), where the Gaussian variance σ is set to 1/K. Next, we randomly sample tokens from the K keywords in the sentence according to the masking rate ε and replace this with mask tokens, followed by removing consecutive mask tokens. At every round of generation, dynamic masking helps boost 1) context diversity by conditioning ACLM generation on different templates with a different set of keywords and 2) length diversity by asking ACLM to infill a different number of mask tokens.

heading: 3.2 Fine-tuning ACLM
text: As discussed earlier, ACLM is fine-tuned on a novel text reconstruction from corrupted text task wherein the created templates serve as our corrupted text and ACLM learns to recover the original text from the template. Text reconstruction from the corrupted text is a common denoising objective that PLMs like BART and BERT are pre-trained on. For this work, we use it as our fine-tuning objective and differ from other existing pre-training objectives by our selective masking strategy for creating templates.

heading: 3.3 Data Generation
text: Post fine-tuning on the text reconstruction task, we utilize ACLM to generate synthetic data for data augmentation. For each sentence in the training dataset, we apply steps 1-4 in the Template Creation pipeline for R rounds to randomly corrupt the sentence and obtain a template which is then passed through the fine-tuned ACLM model to generate a total of R× augmented training samples. Additionally, to boost diversity, during auto-regressive
generation, we randomly sample the next word from the top-k most probable words and choose the most probable sequence with beam search.
mixner: During the R rounds of augmentation on our training dataset, we propose the use of mixner, a novel template mixing algorithm that helps ACLM generate diverse sentences with new context and multiple NEs in the sentence. More specifically, given the template for any arbitrary sentence a in the training set in step 3 of the template creation process, we retrieve the template for another sentence b that is semantically similar to a and join both the templates before passing on the template to step 4. We show examples of sentences generated with mixner in Fig. 3 and Section D.1. Note that we apply mixner only in the generation step and not during fine-tuning.
As mentioned earlier, to retrieve b from the training set, we randomly sample a sentence from the top-k sentences with the highest semantic similarity to a. To calculate semantic similarity between each sentence in the training set, we first take the embedding e for each sentence from a multi-lingual Sentence-BERT (Reimers and Gurevych, 2019) and then calculate semantic similarity by:
sim(ei, ej) = ei ⋅ ej∥ei∥∥ej∥ (1) where sim(. ) is the cosine similarity between two embeddings , and i, j ∈ N where i ≠ j, and N
is the size if the training set. Additionally, we don’t apply mixner on all rounds R but sample a probability γ from a Gaussian distribution N (µ, σ2) and only apply mixner if γ crosses a set threshold β.

heading: 3.3.1 Post-Processing
text: As a post-processing step, we remove augmentations similar to the original sentence and also the extra label tokens added in the labeled sequence linearization step. Finally, we concatenate the augmented data with the original data to fine-tune our NER model.

heading: 4 Experiments and Results

heading: 4.1 Dataset
text: All our experiments were conducted on the MultiCoNER dataset (Malmasi et al., 2022), a large multilingual dataset for complex NER. MultiCoNER covers 3 domains, including Wiki sentences, questions, and search queries, across 11 distinct languages. The dataset represents contemporary challenges in NER discussed in Section 2 and is labeled with six distinct types of entities: person, location, corporation, groups (political party names such as indian national congress), product (consumer products such as apple iPhone 6), and creative work (movie/song/book titles such as on the beach). We conduct experiments on a set of 10 languages L where L = {English (En), Bengali (Bn, Hindi (Hi), German (De), Spanish (Es), Korean (Ko), Dutch (Nl), Russian (Ru), Turkish (Tr), Chinese (Zh)}. Language-wise dataset statistics can be found in Table 12. We would also like to highlight that the number of sentences in MultiCoNER test sets ranges from 133,119 - 217,887, which is much higher than test sets of other existing NER datasets. For more details on the dataset, we refer our readers to Malmasi et al. (2022). For monolingual and cross-lingual low-resource experiments, we perform iterative stratified sampling over all the sentences by using the entity classes in a sample as its target label across four low-resource settings (100, 200, 500, and 1000). We downsample the development set accordingly. For multi-lingual experiments, we combine all the data sampled for our monolingual settings. We evaluate all our systems and baselines on the original MultiCoNER test sets. We report micro-averaged F1 scores averaged across 3 runs for 3 different random seeds.
Algorithm 1 ACLM: Our proposed augmentation framework Given training set Dtrain, and PLM L Dmasked ← ∅,Daug ← ∅ for {X,Y } ∈ Dtrain do ▷ Training Loop
tother, tentity ← X K ← Top p% ofATTNMAP(tother) ▷ Keyword Selection X̃ ← GENTEMPLATE(X, {tother} − {K}) ▷ Selective Masking X̃ ← LINEARIZE(X̃, Y ) ▷ Labeled Sequence Linearization Dmasked ← Dmasked ∪ {X̃}
end for for {X,Y } ∈ Dmasked do
X̃ ← DYNAMICMASK(X,η) ▷ Dynamic Masking Lfinetune ← FINETUNE(L, X̃) ▷ Fine-tune ACLM end for for {X,Y } ∈ Dtrain do ▷ Generation Loop
repeat R times: X̃ ← GENTEMPLATE(X, {tother} − {K}) ▷ Selective masking X̃ ← LINEARIZE(X̃, Y ) ▷ Labeled Sequence Linearization X̃ ← DYNAMICMASK(X̃, µ) ▷ Dynamic Masking Xaug ← GENAUG(Lfinetune(X̃)), if γ < β Xaugmix ← MIXNER(Lfinetune(X̃)), if γ > β Daug ← Daug ∪ {Xaug} ∪ {Xaugmix} end for Daug ← POSTPROCESS(Daug) ▷ Post-processing return Dtrain ∪ Daug

heading: 4.2 Experimental Setup
text: ACLM. We use mBart-50-large (Tang et al., 2020) with a condition generation head to fine-tune ACLM. We fine-tune ACLM for 10 epochs using Adam optimizer (Kingma and Ba, 2014) with a learning rate of 1e−5 and a batch size of 32. NER. We use XLM-RoBERTa-large with a linear head as our NER model. Though the field of NER has grown enormously, in this paper, we adhere to the simplest formulation and treat the task as a token-level classification task with a BIO tagging scheme. We use the Adam optimizer to optimize our model, set the learning rate to 1e−2, and train with a batch size of 16. The NER model is trained for 100 epochs, and the model with the best performance on the dev set is used for testing. Hyper-parameter Tuning. For template creation during fine-tuning and generation, we set the selection rate p and the Gaussian µ to be 0.3 and 0.5, respectively. The number of augmentation rounds R is set as 5. For mixner we set Gaussian µ and β to be 0.5 and 0.7, respectively. All hyper-parameters are tuned on the development set with grid search. More details can be found in Appendix A.

heading: 4.3 Baselines
text: To prove the effectiveness of our proposed ACLM, we compare it with several strong NER augmentation baselines in the literature. In this sub-section, we briefly describe each of these baselines. All baselines were run for R rounds. Gold-Only. The NER model is trained using only gold data from the MultiCoNER dataset without any augmentation.
Label-wise token replacement (LwTR).(Dai and Adel, 2020b) A token in a sentence is replaced with another token with the same label; the token is randomly selected from the training set.
DAGA.(Ding et al., 2020) Data Augmentation with a Generation Approach (DAGA) proposes to train a one-layer LSTM-based recurrent neural network language model (RNNLM) by maximizing the probability for the next token prediction with linearized sentences. During generation, they use random sampling to generate entirely new sentences with only the [BOS] token fed to the model. MulDA.(Liu et al., 2021) Multilingual Data Augmentation Framework (MulDA) builds on DAGA and trains a pre-trained mBART model on next token prediction with linearized sentences for generation-based multilingual data augmentation. For a fair comparison, we replace mBART in MulDA with mBART-50.
MELM.(Zhou et al., 2022) Masked Entity Language Modeling (MELM) proposes fine-tuning a transformer-encoder-based PLM on linearized labeled sequences using masked language modeling. MELM outperforms all other baselines and priorart on low-resource settings on the CoNLL 2003 NER dataset across four languages in mono-lingual, cross-lingual, and multi-lingual settings.
ACLM random. We train and infer ACLM with templates created with randomly sampled keywords instead of taking keywords with high attention scores. This baseline proves the effectiveness of our keyword selection algorithm which provides NEs in the template with rich context. ACLM only entity. We train and infer ACLM with templates created with only linearized entities and no keywords. This baseline proves the effectiveness of additional context in our templates.

heading: 4.4 Experimental Results
text: Monolingual Complex NER. Table 1 compares the performance of all our baselines with ACLM on the MultiCoNER test sets under various lowresource settings for 10 languages. As clearly evident, ACLM outperforms all our baselines in all settings by consistently achieving the best results in all individual languages. Moreover, ACLM improves over our neural baselines (MELM and DAGA) by a significant margin (absolute gains in the range of 1.5% - 22% across individual languages). Although LwTR performs better than ACLM in rare instances, we emphasize that (1) LwTR generates nonsensical, incoherent augmentations, (discussed further in Section D.1) and (2) Based on a learningbased paradigm, ACLM shows bigger margins to LwTR at slightly higher gold training samples (200
and 500) which we acknowledge is a reasonable size in real-world conditions.
Cross-lingual Complex NER. We also study the cross-lingual transferability of a NER model trained on a combination of gold and generated augmentations. Thus, we evaluated a model, trained on En, on 4 other languages, including Hi, Bn, De, and Zh in a zero-shot setting. ACLM outperforms our neural baselines by a significant margin (absolute gains in the range of 1% - 21%). None of these systems perform well in cross-lingual transfer to Zh which was also observed by (Hu et al., 2021). Multi-lingual Complex NER. Table 2 compares the performance of all our baselines with ACLM on the MultiCoNER test sets under various multilingual low-resource settings. As clearly evident, ACLM outperforms all our baselines by a significant margin (absolute gains in the range of 1%-21% across individual languages). All our baselines, including our Gold-Only baseline, also perform better than their monolingual counterparts which demonstrates the effectiveness of multi-lingual finetuning for low-resource complex NER.

heading: 5 Further Analysis

heading: 5.1 Generation Quality
text: Quantitative Analysis. Table 3 compares augmentations from various systems on the quantitative measures of perplexity and diversity. Perplexity (Jelinek et al., 1977) is a common measure of text fluency, and we measure it using GPT2 (Radford et al., 2019). We calculate 3 types of diversity metrics: for Diversity-E and Diversity-N, we calculate the average percentage of new NE and non-NE words in the generated samples compared with the original samples, respectively. For Diversity-L, we
calculate the average absolute difference between the number of tokens in generated samples and the original samples. ACLM achieves the lowest perplexity and highest non-NE and length diversity compared with other baselines. NE diversity in ACLM is achieved with mixner where ACLM fairs well compared to MELM which just replaces NEs. LwTR achieves the highest perplexity, thereby reaffirming that it generates incoherent augmentations. Qualtitative Analysis. Fig. 3 illustrates the superiority of augmentations generated by ACLM when compared with our other baselines. As clearly evident, while MELM generates just minor changes in NEs, augmentations produced by LwTR often tend to be nonsensical and incoherent. On the other hand, ACLM generates meaningful and diverse sentences around NEs, which is further boosted with mixner. We provide examples in Appendix D.1.

heading: 5.2 Application to other domains
text: To evaluate the transferability of ACLM to other domains, we evaluate ACLM on 4 more datasets beyond MultiCoNER. These datasets include CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003) (news), BC2GM (Smith et al., 2008) (bio-medical), NCBI Disease (Doğan et al., 2014) (bio-medical) and TDMSci (Hou et al., 2021) (science). Table 4 compares our baselines with ACLM across 2 lowresource settings on all 4 datasets. ACLM outperforms all our baselines on all settings except LwTR on CoNLL 2003. This occurs because LwTR generates a large variety of effective augmentations with NE replacement on easy entities in CoNLL 2003. The results demonstrate the effectiveness of ACLM over diverse domains, including domains with an acute scarcity of data (bio-medical). Additionally, we also emphasize that ACLM produces more factual augmentations and, unlike our other baselines, avoids context-entity mismatch, which makes the NER model store wrong knowledge in
data-sensitive domains. We show samples of generated augmentations in Fig. 3 and Appendix D.1.

heading: 6 Conclusion
text: In this paper, we propose ACLM, a novel data augmentation framework for low-resource complex NER. ACLM is fine-tuned on a novel text reconstruction task and is able to generate diverse augmentations while preserving the NEs in the sentence and their original word sense. ACLM effectively alleviates the context-entity mismatch problem and generates diverse, coherent, and highquality augmentations that prove to be extremely effective for low-resource complex NER. Additionally, we also show that ACLM can be used as an effective data augmentation technique for low-resource NER in the domains of medicine and science due to its ability to generate extremely reliable augmentations.
Limitations
We list down some potential limitations of ACLM: 1) PLMs are restricted by their knowledge to generate entirely new complex entities due to their syntactically ambiguous nature. Adding to this, substituting complex NEs in existing sentences leads to context-entity mismatch. Thus, as part of future work, we would like to explore if integrating external knowledge into ACLM can help generate sentences with new complex entities in diverse contexts. 2) We do not conduct experiments in the language Farsi from the MultiCoNER dataset as neither mBart-50-large nor XLM-RoBERTa-large was pre-trained on this language. 3) The use of mBart-50-large for generation also restricts ACLM from being transferred to code-switched settings, and we would like to explore this as part of future work.

heading: A Hyperparameter Tuning
text: Augmentation rounds R: Augmenting the training dataset with several augmentation rounds R proves effective until a saturation point is reached. Continuing to add more augmented data to the gold dataset starts introducing noise to the combined data. Additionally, with an increase in R, the chances of auto-regressive generation with top-k sampling generating similar sentences increase. To determine the optimal value of R, we experiment on 2 low-resource settings on the English sub-set of MultCoNER and report the micro F1 results on the test-set for R ranging from 1 to 7. All other hyperparameters are kept constant. As shown in Table 5, R = 5 gives us the best test-set performance, and the performance decreases after 5 rounds. Attention layers a for Keyword Selection: Selecting the right keywords for creating a template
is integral to the success of ACLM. A clear example of this can be seen in Table 1, where ACLM outperforms ACLM random (which chooses random tokens as keywords for template creation) by a significant margin. Transformer encoders consist of multiple layers, and each layer consists of multiple attention heads. While all heads in the same layer tend to behave similarly, different layers generally encode different semantic and syntactic information (Clark et al., 2019). Thus we experiment with different values of α, or different combinations of transformer encoder layers which are used for calculating the attention scores for keyword selection. As mentioned in Section 3.1, by default, we average attention scores across all tokens, all heads, and the last α layers. For all our low-resource experiments, we use attention maps from a 24-layer XLMRoBERTa-large fine-tuned on the low-resource gold dataset for that particular setting. Table 7 compares the performance of 3 settings of α on 2 low-resource settings on the English sub-set of MultCoNER: 1. Only last layer 2. Last 4 layers. 3. All 24 layers. As clearly evident, though setting 2 achieves the best performance, the difference in performance among different values of α is not too high. As part of future work, we would like to explore better ways to search for the optimal α.

heading: B Additional Results
text: Current state of state-of-the-art: Most current state-of-the-art systems are built and evaluated on common NER benchmarks like CoNLL 2003 and OntoNotes v5.0. As discussed in Section 2, these benchmarks do not represent contemporary challenges in NER and contain sentences with easy entities and rich context. Table 8 compares the performance of a simple XLM-R (Conneau et al., 2019), and Co-regularized LUKE (Zhou and Chen,
2021) (SOTA NER system) on 2 common NER and 1 complex NER benchmarks in both low- and high-resource settings. As we can clearly see, both systems achieve remarkable performance on both CoNLL 2003 and OntoNotes v5.0 but struggle on MultiCoNER. Additionally, the gap widens in lowresource settings.
Training on the entire dataset: Beyond just evaluating ACLM performance on low-resource settings, we also compare ACLM with all our baselines on the entire MultiCoNER dataset (each language split contains ≈ 15300 sentences). Similar to low-resource settings, ACLM outperforms all our baselines across all languages and achieves an absolute average gain of 1.58% over our best baseline.
Entity-wise Performance Analysis: Previous to MultiCoNER, common benchmark datasets like CoNLL 2003 had only “easy entities” like names of Persons, Locations, and Organizations. The MultiCoNER dataset has 3 additional types of NEs, namely Products (PROD), Groups (GRP), and Creative Work (CW). These entities are syntactically ambiguous, which makes it challenging to recognize them based on their context. The top system from WNUT 2017 achieved 8% recall for creative work entities. Table 10 compares the entity-wise performance of ACLM with our various baselines on two low-resource settings on the MultiCoNER dataset. All results are averaged across all 10 lan-
guages. ACLM outperforms all our baselines on all individual entities, including PROD, GRP, and CW, which re-affirms ACLM’s ability to generate effective augmentation for complex NER.
Length-wise Performance Analysis: As mentioned in Section 2, low-context is a major problem in complex NER, and an effective complex NER system should be able to detect NEs in sentences with both low and high context (by context we refer to the number of words around the NEs in the sentence). By the nature of its fine-tuning pipeline, ACLM is able to generate augmentations of variable length, and our dynamic masking step further boosts the length diversity of generated augmentations. Adding to this, we acknowledge that effective augmentations for syntactically complex entity types should enable a model to learn to detect these entities in even low-context. Table 11 compares the entity-wise performance of ACLM with our various baselines on two low-resource settings on the MultiCoNER dataset. All results are averaged across all 10 languages. ACLM outperforms all our baselines across all length settings, which re-affirms ACLM’s ability to generate effective augmentation for complex NER. To be specific, ACLM improves over our best baseline by 8.8% and 7.4% for 200 and 3.2% and 6.7% for 500 for low- and high-context sentences, respectively.

heading: C Templates and Attention Maps
text: Creating templates with keywords that effectively provides the PLM with additional knowledge about the NEs in the sentence is an integral part of ACLM. Fig. 11, 12, 13, 14, 15 shows examples of templates created for our sentences in MultiCoNER English subset, Spanish subset, Hindi subset NCBI Disease and TDMSci datasets, respectively. Additionally, we provide examples of attention maps used to create templates in Fig. 16f.

heading: D Qualitative Analysis of Augmentations
text: D.1 Augmentation Examples
MultiCoNER Dataset: We provide additional examples of augmentations generated by ACLM and all our baselines in Fig. 9 and Fig. 10 for Hindi and English subsets of MultiCoNER dataset respectively. Extra Datasets: Fig 5, 6, 7 and 8 illustrate augmetation examples for CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003) (news), BC2GM (Smith et al., 2008) (bio-medical), NCBI Disease (Doğan et al., 2014) (bio-medical) and TDMSci (Hou et al., 2021) (science) datasets respectively. Except for on CoNLL 2003 datasets, both our baselines, LwTR and MELM, generate incoherent and unreliable training samples for the other 2 datasets. We only compare ACLM with LwTR and MELM as these methods don’t generate augmentations from scratch and modify existing sentences. We define unreliable sentences as sentences generated with an entity-context mismatch (eg. a NE describing a disease prone to cows is placed in the context of humans or vice-versa). Generating unreliable augmentations prove fatal in data-sensitive
domains like bio-medical as it may make the model store wrongful knowledge. Our detailed analysis of generated augmentations shows that: (1) LwTR is prone to generating such incoherent sentences because it randomly samples entities from the corpus with the same tag for replacement. (2) MELM on the other hand, fine-tuned on a transformerencoder-based PLM, gets to see the entire context of the sentence for generating a new NE. However, it does not learn to focus on particular keywords and tends to generate a new NE based on the broader context of the sentence (e.g., it does not learn to differentiate between human and cow diseases and generates a new NE based on the broader context of the sentence). (3) ACLM generates highly reliable samples by conditioning on templates with keywords related to the NE. We illustrate examples of such templates in Fig. 14 and 15.

heading: E Additional Details
text: Model Parameters: XLM-RoBERTa-large has ≈ 355M parameters with 24-layers of encoder, 1027- hidden-state, 4096 feed-forward hidden-state and 16-heads. mBART-50-large ≈ has 680M parameters with 12 layers of encoder, 12 layers of decoder, 1024-hidden-state, and 16-heads.
Compute Infrastructure: All our experiments are conducted on a single NVIDIA A100 GPU. An entire ACLM training pipeline takes ≈ 40 minutes.
Dataset Details: We use 5 datasets in total for our experiments: MultiCoNER 2 (Malmasi et al.,
2https://registry.opendata.aws/multiconer/
2022) (CC BY 4.0 licensed), CoNLL 2003 3 (Tjong Kim Sang and De Meulder, 2003) (Apache License 2.0), BC2GM 4 (Smith et al., 2008) (MIT License), NCBI Disease 5 (Doğan et al., 2014) (Apache License 2.0) and TDMSci 6 (Hou et al., 2021) (Apache License 2.0). All the datasets are available to use for research purposes, and for our work, we use all these datasets intended for their original purpose, i.e., NER. MultiCoNER has data in 11 languages, including code-mixed and multilingual subsets. We experiment with 10 monolingual subsets discussed in Section 4.1 with appropriate reason for not experimenting on Farsi in our Limitations Section. According to the original papers of all 5 datasets used in the research, none of them contains any information that names or uniquely identifies individual people or offensive content.
Data statistics (train/test/dev splits): Detailed dataset statistics for MultiCoNER, CoNLL 2003, BC2GM, NCBI Disease and TDMSci can be found in Table 12 (language codes in Table 13), 14, 16, 17 and 15 respectively.
Implementation Software and Packages: We implement all our models in PyTorch 7 and use the HuggingFace 8 implementations of mBART50 and XLM-RoBERTA (base and large). We use the FLAIR toolkit (Akbik et al., 2019) to fine-tune all
3https://huggingface.co/datasets/conll2003 4https://github.com/spyysalo/bc2gm-corpus 5https://huggingface.co/datasets/ncbidisease 6https://github.com/IBM/science-result-extractor 7https://pytorch.org/ 8https://huggingface.co/
our NER models. Potential Risks: Conditional Language Models used for Natural Language Generation often tend to hallucinate (Ji et al., 2022) and potentially generate nonsensical, unfaithful or harmful sentences to the provided source input that it is conditioned on.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations Section (After conclusion and before citations).
3 A2. Did you discuss any potential risks of your work? Section E.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1 and Abstract.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.1.
3 B1. Did you cite the creators of artifacts you used? Citations.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section E.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section E.
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section E.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section E.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section E.
C 3 Did you run computational experiments? Section E.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section E.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4.2. Section A.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.1.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section E.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: As part of future work, we would like to explore better ways to search for the optimal α