Entity-to-Text based Data Augmentation for various Named Entity Recognition Tasks
Year: 2023

Abstract:
Data augmentation techniques have been used to alleviate the problem of scarce labeled data in various NER tasks (flat, nested, and discontinuous NER tasks). Existing augmentation techniques either manipulate the words in the original text that break the semantic coherence of the text, or exploit generative models that ignore preserving entities in the original text, which impedes the use of augmentation techniques on nested and discontinuous NER tasks. In this work, we propose a novel Entity-toText based data augmentation technique named ENTDA to add, delete, replace or swap entities in the entity list of the original texts, and adopt these augmented entity lists to generate semantically coherent and entity preserving texts for various NER tasks. Furthermore, we introduce a diversity beam search to increase the diversity during the text generation process. Experiments on thirteen NER datasets across three tasks (flat, nested, and discontinuous NER tasks) and two settings (full data and low resource settings) show that ENTDA could bring more performance improvements compared to the baseline augmentation techniques.

heading: 1 Introduction
text: Recent neural networks show decent performance when a large amount of training data is available. However, these manually labeled data are laborintensive to obtain. Data augmentation techniques (Shorten and Khoshgoftaar, 2019) expand the training set by generating synthetic data to improve the generalization and scalability of deep neural networks, and are widely used in NLP (Feng et al., 2021; Li et al., 2022a). One successful attempt for data augmentation in NLP is manipulating a few words in the original text, such as word swapping (Şahin and Steedman, 2018; Min et al., 2020) and random deletion (Kobayashi, 2018; Wei and
∗*Work done during an internship at Alibaba DAMO Academy.
The cancer patient has constant stomach discomfort and pain.
cancer patient, stomach discomfort, stomach pain
Aug. Text
Aug. Entity
People with cancer may experience stomach discomfort and pain.Ori. Text
Replace Tokens Shuffle Segments
DAGA MELM
｛ ｛[cancer, cancer patient] Discontinuous
cancer, stomach discomfort, stomach painOri. Entity
Nested [stomach discomfort, stomach pain]
People with <B-DISORDER> disease <B-DISORDER> may experience stomach discomfort and pain.
People cancer may with experience and pain stomach discomfort.
People with cancer like event stomach inconvenience and pain.
｛Unable to mark with flat <B> and <I>.
9072
tities are masked during the generation process, so that the diversity of generated texts is also limited. For example, as shown in Figure 1, rule-based models replace tokens or shuffle segments, such as “with” and “cancer may” are shuffled, which makes the augmented text no longer semantically coherent, and even modifies the semantic consistency of the text to affect the prediction of entity labels. The Text-to-Text based generative models cannot leverage flat ⟨B − Type⟩ and ⟨I − Type⟩ labels to mark the “stomach” token in the discontinuous entities: “stomach discomfort” and “stomach pain”, thus limiting the application of this method to nested and discontinuous NER tasks.
To maintain text semantic coherence during augmentation and preserve entities for various NER tasks, in this work, we propose a novel Entity-toText instead of Text-to-Text based data augmentation approach named ENTDA. As illustrated in Figure 2, we first obtain the entity list [EU, German, British] in the original text, and then add, delete, swap, and replace the entity in the entity list to obtain the augmented entity list, e.g. [EU, German, British, Spanish]. We investigate that leveraging the rule-based methods to modify the entities in the entity list could generate more combinatorial entity lists without introducing grammatical errors. Then we adopt a conditional language model to generate the semantically coherent augmented text based on the augmented entity list. Thanks to the augmented entity list (including flat, nested, and discontinuous entities) we have already obtained, we can mark these preserved entities in the augmented text as shown in Figure 4. Since the augmented entity list provide the similar entity information in the text augmented by the language model, which may leads to insufficient diversity of text generation. Therefore, we propose a diversity beam search method for generative models to enhance text diversity. Overall, the main contributions of this work are as follows:
• To the best of our knowledge, we propose the first Entity-to-Text based data augmentation technique ENTDA. ENTDA leverages the pretrained large language model with semantic coherence and entity preserving to generate the augmented text, which could be used to benefit for all NER tasks (flat, nested, and discontinuous NER tasks). • We propose the diversity beam search strategy for ENTDA to increase the diversity of the
augmented text during generation process. • We show that ENTDA outperforms strong
data augmentation baselines across three NER tasks and two settings (full data and low resource settings).

heading: 2 Related Work

heading: 2.1 Various NER Tasks
text: Named Entity Recognition (NER) is a pivotal task in IE which aims at locating and classifying named entities from texts into the predefined types such as PERSON, LOCATION, etc. (Chiu and Nichols, 2016; Xu et al., 2017; Yu et al., 2020). In addition to flat NER task (Sang and De Meulder, 2003), Kim et al. (2003) proposed nested NER task in the molecular biology domain. For example, in the text: Alpha B2 proteins bound the PEBP2 site, the entity PEBP2 belongs to the type PROTEIN and PEBP2 site belongs to DNA.
Furthermore, some entities recognized in the text could be discontinuous (Mowery et al., 2013, 2014; Karimi et al., 2015). For example, in the text: I experienced severe pain in my left shoulder and neck, the entities pain in shoulder and pain in neck contain non-adjacent mentions. Some previous works proposed the unified frameworks which are capable of handling both three NER tasks (Li et al., 2020; Yan et al., 2021; Li et al., 2021). However, there is no unified data augmentation method designed for all three NER tasks due to the complexity of entity overlap. In this work, we try to bridge this gap and propose the first generative augmentation approach ENTDA that can be used to generate augmented data for all NER tasks (flat, nested, and discontinuous NER tasks).

heading: 2.2 Data Augmentation for NLP and NER
text: As shown in Table 1, we compare ENTDA with rule-based and traditional generative techniques, and present the comparison results below.
The cancer patient has constant stomach discomfort and pain.
0 1 2 3 4 5 6 7 8 9
cancer patient, stomach discomfort, stomach pain
Three NER subtasks:
Text
Entity
Flat: stomach discomfort [5,6,DISORDER] Nested: cancer, cancer patient [1,1,DISORDER] [1,2,PERSON] Discontinuous: stomach pain [5,5,8,8,DISORDER]
British
farmer
market
-1.1-0.5=-1.6
-1.1
-1.1-0.8=-1.9
German
farmer
market
-1.8-0.4=-2.2
-1.8
-1.8-0.9=-2.7
British
farmer
market
-1.1-0.5-1=-2.6
-1.1
-1.1-0.8-2=-3.9
German
farmer
market
-1.8-0.4-1=-3.2
-1.8
-1.8-0.9-2=-4.7
Diversity Beam Search DecodingBeam Search Decoding
Rule-based Augmentation Various rule-based augmentations for NLP tasks such as word replacement (Zha g et al., 2015; C et al., 2020), random deletion (Kobayashi, 2018; Wei and Zou, 2019), and word swapping (Şahin and Steedman, 2018; Min et al., 2020) manipulate the words in the original texts to generate synthetic texts. However, these manipulated tokens could not maintain the original labels since the change of syntax and semantics.
Dai and Adel (2020) proposes a replacement augmentation method to decide whether the selected token should be replaced by a binomial distribution, and if so, then the token will be replaced by another token with the same label. Furthermore, the similar approaches could be extended from token-level to mention-level. However, these methods still inevitably introduce incoherent replacement. In this work, we try to introduce the Entity-to-Text based augmentation approach to improve the coherence of the augmented texts.
Generative Augmentation Classic generative augmentations for NLP tasks such as back translation, which could be used to train a question answering model (Yu et al., 2018) or transfer texts from a high-resource language to a low-resource language (Hou et al., 2018; Xia et al., 2019). Anaby-Tavor et al. (2020); Kumar et al. (2020) adopt language model which is conditioned on sentence-level tags to modify original data for classification tasks exclusively. To utilize generative augmentation on more fine-grained and fragile token-level NER tasks, Ding et al. (2020) treats the NER labeling task as a text tagging task and requires generative models to annotate entities during generation. Zhou et al. (2022) builds the pretrained masked language models on corrupted train-
ing sentences and focuses on entity replacement. However, these methods rely on the Text-to-Text based generative models which cannot tag a token with nested labels during generation. In this work, we adopt the Entity-to-Text based generative model to tackle all NER tasks and bootstrap the diversity of the model with diversity beam search.

heading: 3 General NER Task Formulation
text: Considering that ENTDA has sufficient augmentation ability on flat, nested and discontinuous NER, we first formulate the general NER task framework as follows. Given an input text X = [x1, x2, ..., xn] of length n and the entity type set T , the output is an entity list E = [e1, e2, ..., em, ..., el] of l entities, where em = [sm1, dm1, ..., smj , dmj , tm]. The s, d are the start and end indexes of a space in the text X . The j indicates that the entity consists of j spans. The tm is an entity type in the entity type set T . For example, the discontinuous entity stomach pain in the text: “The cancer patient has constant stomach discomfort and pain” will be represented as em = [5, 5, 8, 8, DISORDER].

heading: 4 Proposed Method
text: The proposed Entity-to-Text based data augmentation approach ENTDA consists of three modules: Entity List Augmentation, Entity-to-Text Generation, and Augmented Text Exploitation. Now we give the details of the three modules.

heading: 4.1 Entity List Augmentation
text: Entity List Augmentation aims to adopt four rulebased methods: Add, Delete, Replace, and Swap to modify the entities in the entity list obtained from the original sentences. Now, we give the
details of four operations on the original entity list E = [e1, e2, ..., em, ..., el] as follows:~i1 Add. We first randomly select an entity em
from the entity list E. Then we search for other entities in the training set and add e ′ m with the same entity type as em to the original entity list: E = [e1, e2, ..., em, e
′ m, ..., el].~i2 Delete. We randomly select an entity em
from the original entity list E and delete it as E = [e1, e2, ..., em−1, em+1, ..., el].~i3 Replace. We first randomly select an entity em from the original entity list E. Similar to 1 , we search e′m with the same entity type to
replace em as E = [e1, e2, ..., e ′ m, ..., el].~i4 Swap. We randomly select two entities
em, e ′ m in the original entity list E and swap their positions as E = [e1, e2, ..., e ′ m, ..., em, ..., el].

heading: 4.2 Entity-to-Text Generation
text: After we obtain the augmented entity lists, the Entity-to-Text Generation module aims to generate the text for each entity list. Since the augmented entity list provide the similar entity information for augmented text, so we propose a diversity beam search method to increase text diversity.
Compared to traditional generation models that rely on greedy decoding (Chickering, 2002) and choosing the highest-probability logit at every generation step, we adopt a diversity beam search decoding strategy. More specifically, we first inject the entity types into the augmented entity list E = [[t1], e1, [/t1], ..., [tm], em, [/tm], ..., [tl], el, [/tl]] as the input sequence, which should provide sufficient type guidance for the generation model, then we adopt T5 (Raffel et al., 2020) as the generation model. We first fine-tune T5 on the original Entity-to-Text data and then adopt T5 (θ) to estimate the conditional probability distribution over all tokens in the dictionary V at time step t as:
θ (yt) = log Pr (yt | yt−1, . . . , y1, E) . (1)
where yt is the tth output token y in texts. We simplify the sum of log-probabilities (Eq. 1) of all previous tokens decoded Θ(y[t]) as:
Θ ( y[t] ) = ∑
τ∈[t] θ (yτ ) , (2)
1.EU bans German beef from British market. 2.A British farmer accused the EU of failing to protect his sheep from German imports. ……..
EU, German, British
Add
Swap
Delete Replace EU, German, British, Spanish EU, German
EU, German, Spanish
British, EU, German
Entity to Sequence m odel
1.EU’s veterinary committee ruled that German beef, British lamb and Spanish sheep should be banned. 2.EU bans German, British and Spanish wheat from sale. …….. 1.EU’s German wing says it has received a warning. 2.EU had already imposed a ban on German exports. …….. 1.EU’s veterinary committee ruled that German beef was to eat after being imported from Spanish. 2.European Union bans German, Spanish beef imports.
……..
Diversity Beam Search
Se qu
en ce to
En tit
y m
od el
Consistency Restriction
The cancer patient has constant stomach discomfort and pain. 0 1 2 3 4 5 6 7 8 9
cancer patient, stomach discomfort, stomach pain
Three NER subtasks:
Sequence Entity
Flat NER subtask: [5,6,DISORDER] Nested NER subtask: [1,1,DISORDER] [1,2,PERSON] Discontinuous NER subtask: [5,6,8,9,DISORDER]
where y[t] is the token list consisting of [y1, y2, ..., yt]. Therefore, our decoding problem is transformed into the task of finding the text that could maximize Θ(y). The classical approximate decoding method is the beam search (Wiseman and Rush, 2016), which stores top beam width B candidate tokens at time step t. Specifically, beam search selects the B most likely tokens from the set:
Yt = Y[t−1] × V, (3)
where Y[t−1] = { y1,[t−1], . . . ,yB,[t−1] } and V is the dictionary. However, traditional beam search keeps a small proportion of candidates in the search space and generates the texts with minor perturbations (Huang, 2008), which impedes the diversity of generated texts. Inspired by Vijayakumar et al. (2016), we introduce an objective to increase the dissimilarities between candidate texts and finalize the Eq. 2 as diversity beam search decoding:
Θ̂ ( y[t] ) = ∑
τ∈[t] (θ (yτ )− γkτ ), (4)
where γ is a hyperparameter and represents the punishment degree. kτ denotes the ranking of the current tokens among candidates. In practice, it’s a penalty text of beam width: [1, 2, ..., B] which punishes bottom ranked tokens among candidates and thus generates tokens from diverse previous tokens. For a better understanding, we give an example about the text with beam search decoding and diversity beam search decoding in Figure 3.
The traditional greedy decoding chooses the highest-probability logit at every generation step and results in British farmer. Compared to the diversity beam search decoding method, the beam search decoding method maintains a small proportion of candidates in the search space without
1.EU bans German beef from British market. 2.A British farmer accused the EU of failing to protect his sheep from German imports. ……..
EU, German, British Add
Swap
Delete Replace EU, German, British, Spanish EU, German EU, German, Spanish
British, EU, German
Entity-to-Text m odel
1.EU’s veterinary committee ruled that German beef, British lamb and Spanish sheep should be banned. 2.EU bans German, British and Spanish wheat from sale. …….. 1.EU’s German wing says it has received a warning. 2.EU had already imposed a ban on German exports. ……..
1.EU’s veterinary committee ruled that German beef was to eat after being imported from Spanish. 2.United Nations bans German, Spanish beef imports.
……..
Diversity Beam Search
Te xt
-t oEn tit y (N ER ) m od el
The cancer patient has constant stomach discomfort and pain. 0 1 2 3 4 5 6 7 8 9
cancer patient, stomach discomfort, stomach pain
Three NER subtasks:
Text
Entity
Flat: stomach discomfort [5,6,DISORDER] Nested: cancer, cancer patient [1,1,DISORDER] [1,2,PERSON] Discontinuous: stomach pain [5,5,8,8,DISORDER]
British
farmer
market
-1.1-0.5=-1.6
-1.1
-1.1-0.8=-1.9
German
farmer
market
-1.8-0.4=-2.2
-1.8
-1.8-0.9=-2.7
British
farmer
market
-1.1-0.5-1=-2.6
-1.1
-1.1-0.8-2=-3.9
German
farmer
market
-1.8-0.4-1=-3.2
-1.8
-1.8-0.9-2=-4.7
Diversity Beam Search DecodingBeam Search DecodingFigure 4: Details of marking the texts with the augmented entity lists for three NER tasks.
the introduction of a penalty text of beam width [1, 2, ..., B]. This additional objective increases the dissimilarities between candidate texts and thus generates tokens from diverse previous tokens. For example, British farmer and German farmer are generated instead of British farmer and British market, which brings the diversity token German. Likewise, the diversity token market will also be considered in the subsequent generation. Overall, at each time step t:
Y[t] = argmax y1,[t],...,yB,[t]∈Yt
∑ b∈[B] Θ̂ ( yb,[t] ) . (5)
This process will generate the most likely texts that are selected by ranked the B beams based on the diversity beam search decoding.

heading: 4.3 Augmented Text Exploitation
text: To utilize these augmented Entity-to-Text data, we need to mark the texts with the augmented entity lists. As illustrated in Figure 2, we first automatically judge whether the entities match the tokens in the texts and remove these noisy texts of mismatched entities. For example, EU is generated as United Nations and this generated text is automatically deleted. Then as illustrated in Figure 4, we provide the details of the text marking process:
(1) If the entity is flat, we obtain the start and end position indexes through the exact match between entity and text.
(2) If the entity is nested, we first store all the overlapping entity mentions belonging to the same nested entity and match these mentions with text to obtain start and end position indexes.
(3) If the entity is discontinuous, we match the entity mentions which belong to the same discontinuous entity with text to obtain start and end position indexes.
Note that the process of text marking is done automatically based on the above three situations. After we obtain these augmented data with marked
flat, nested, and discontinuous entities, we naturally formulate the texts as input to NER tasks.

heading: 5 Experiments and Analyses
text: We conduct extensive experiments on thirteen NER datasets across three tasks (flat, nested, and discontinuous NER) and two settings (full data and low resourc NER) to show the effectiv ness of ENTDA on NER, and give a detailed analysis.

heading: 5.1 Backbone Models
text: We adopt two SOTA backbone models which could solve all three NER tasks: 1) The unified Seq2Seq framework (Yan et al., 2021) formulates three NER tasks as an entity span text generation task without the special design of the tagging schema to enumerate spans. 2) The unified Word-Word framework (Li et al., 2022b) models the neighboring relations between entity words as a 2D grid and then adopts multigranularity 2D convolutions for refining the grid representations.
These two backbone models are leveraged to solve the general NER tasks illustrated in Section 3 and demonstrate the effectiveness of ENTDA.

heading: 5.2 Datasets
text: To demonstrate that ENTDA could be used in various NER tasks and backbone models, we follow Yan et al. (2021); Li et al. (2022b) and adopt the same datasets (split) as follows:
1) Flat NER Datasets: We adopt the CoNLL2003 (Sang and De Meulder, 2003) and OntoNotes (Pradhan et al., 2013) datasets. For OntoNotes, we evaluate in the English corpus with the same setting as Yan et al. (2021).
2) Nested NER Datasets: We adopt the ACE 2004 (Doddington et al., 2004), ACE 2005 (Christopher Walker and Maeda., 2005) and GENIA (Kim et al., 2003) datasets. Following Yan et al. (2021), we split the ACE 2004/ACE 2005 into train/dev/test sets by 80%/10%/10% and GENIA into 81%/9%/10% respectively.
3) Discontinuous NER Datasets We adopt the CADEC (Karimi et al., 2015), ShARe13 (Mowery et al., 2013) and ShARe14 (Mowery et al., 2014) datasets from biomedical domain. Following Yan et al. (2021), we split the CADEC into train/dev/test sets by 70%/15%/15% and use 10% training set as the development set for ShARe13/ShARe14.
We show the detailed statistics and entity types of the datasets in Appendix A.

heading: 5.3 Baseline Augmentation Methods
text: Unlike sentence-level classification tasks, NER is a fine-grained token-level task, so we adopt six entity-level data augmentation baselines, which are designed for various NER tasks.
The four rule-based baseline augmentation techniques: (1) Label-wise token replacement (Dai and Adel, 2020) utilizes a binomial distribution to decide whether each token should be replaced, and then replaces the chosen token with another token that has the same entity type. (2) Synonym replacement (Dai and Adel, 2020) replaces the chosen token with the synonym retrieved from WordNet. (3) Mention replacement (Dai and Adel, 2020) replaces the chosen entity with another entity, which has the same entity type. (4) Shuffle within segments (Dai and Adel, 2020) splits the sentences into segments based on whether they come from the same entity type, and uses a binomial distribution to decide whether to shuffle tokens within the same segment. The two generative baseline augmentation techniques are: (5) DAGA (Ding et al., 2020) treats the NER labeling task as a text tagging task
and annotates entities with generative models during generation. (6) MELM (Zhou et al., 2022) generates augmented data with diverse entities, which is built upon pre-trained masked language models. MELM is further finetuned on corrupted training sentences with only entity tokens being randomly masked to focus on entity replacement.
We present another model: ENTDA (All), which adopts four entity list operations simultaneously to generate augmented texts. Note that we focus on entity-level NER augmentation tasks, so to the best of our knowledge, we have employed all entity-level augmentation techniques.

heading: 5.4 Experiment Settings
text: For ENTDA, we fine-tune the T5-Base (Raffel et al., 2020) with the initial parameters on the Entity-to-Text data of the training set and utilize the default tokenizer with max-length as 512 to preprocess the data. We use AdamW (Loshchilov and Hutter, 2018) with 5e−5 learning rate to optimize the cross entropy loss. The batch size is set to 5 and the number of training epoch is set to 3. During diversity beam search decoding, we set γ as 10 and beam width B as 3, which means that each entity set will generate three texts.
ENTDA and all baselines augment the training set by 3x for a fair comparison. For example, the number of texts in the training set is 100, we generate 300 texts and add them to the training set. We replace the language model in MELM (Zhou et al., 2022) with XLM-RoBERTa-large (355M) (Conneau et al., 2020), and we use T5-Base (220M) with fewer parameters for comparison.

heading: 5.5 Results and Analyses
text: Table 2 shows the average F1 results on three runs. All backbone NER models gain F1 performance improvements from the augmented data when compared with the models that only use original training data, demonstrating the effectiveness of data augmentation approaches in the various NER tasks. Surprisingly, ENTDA (None) outperforms the baseline methods by 0.11% F1 performance among the backbone models, which shows that the generative models using a diversity beam search have sufficient capacity to generate high-quality augmented data.
More specifically, for flat NER datasets, MELM is considered as the previous SOTA data augmentation approach. The proposed ENTDA (All) on average achieves 0.23% higher in F1 among flat NER datasets and two backbone models. For nested and discontinuous NER datasets, the label-wise token replacement method achieves the best performance among baselines. ENTDA (All) achieve an average 0.78% F1 boost among nested and discontinuous NER datasets, which demonstrates that leveraging generative model to augment semantically coherent texts is effective.
Among all NER datasets, ENTDA is undoubtedly capable of achieving state-of-the-art results (with student’s T test p < 0.05). Except ENTDA (All), ENTDA (Add) achieves the largest F1 performance gains of 0.99% and 0.76% on the unified Seq2Seq and Word-Word frameworks, respectively. We attribute this delightful improvement of the “Add” operation to the additionally introduced knowledge: we add the entity from the training set with the same entity type. Ablation Study
In Table 2, we remove the entity list augmentation module (ENTDA(None)), or change the diversity beam search to the traditional beam search (ENTDA(All) w/o Diver.). We can conclude that entity list augmentation and diversity beam search modules bring an average F1 improvement of
0.56% and 0.38% on the eight datasets. Using the entity list augmentation module can give a richer entity combination, which brings more improvement. Adopting the diversity beam search brings more diverse texts and gains greater improvements. Handling Low Resource NER Scenarios
We further introduce an extreme yet practical scenario: only limited labeled data is available. This low resource NER scenario demonstrates that our ENTDA approach bootstraps the generalization ability of the NER model and is a quite appealing approach for data-oriented applications in the real-world. In practice, we randomly choose 10% training data from CoNLL2003/ACE2005/CADEC to represent the three NER tasks. Note that the fine-tuning of T5-large and our four operations on the entity list are also done on 10% training data.
From Table 3, compared to training directly on the 10% training set, leveraging the augmented data achieves the performance improvement in F1. We also observe that ENTDA approach obtains the most competitive F1 performance improvement when compared with baseline data augmentation approaches. More specifically, ENTDA (All) achieve an average 2.97% F1 boost among three backbone models, which means ENTDA obtains more performance gains under the low resource scenario than in the full data scenario. Especially for the most challenging discontinuous dataset CADEC, ENTDA (All) obtains the largest F1 performance gain of 4.22%. Surprisingly, on
10% CoNLL2003, ENTDA (All) has only a 2.94% decrease in F1 performance compared to using the full training data, but ENTDA (All) saves 10x the annotated data, which shows that adopting ENTDA is quite appealing for real-world applications. Tackling Real Low Resource NER Tasks
We adopt real low resource NER datasets (Liu et al., 2021) from Wikipedia which contains politics, natural science, music, literature and artificial intelligence domains with only 100 or 200 labeled texts in the training set. ENTDA and baseline data augmentation approaches still augment the training set by 3x. From Table 4, we are delighted to observe ENTDA could quickly learn from the extremely limited Entity-to-Text data and bring 3.45% F1 performance gains over various domains. Compared with baseline augmentation methods, ENTDA generates more diverse texts and undoubtedly gains greater advantages. Various Augmentation Multiples Performance
We further vary the multiples of augmented data from 2x to 10x the training set to study the influence of data augmentation approaches for the NER backbone models under low resource scenarios. We choose different low resource datasets
and three representative augmentation approaches (Mention replacement, MELM, and ENTDA (All)), then represent the results in Figure 5.
We could observe that the unified Seq2Seq framework has more performance gains with everincreasing augmented data. ENTDA (All) consistently achieves better F1 performance, with a clear margin, compared to baseline augmentation approaches under various augmentation multiples. Especially for Music, ENTDA (All) brings an incredible 4.01% improvement in F1 performance with only 300 augmented data. Semantic Coherence Analysis
Compared with baseline augmentation approaches, ENTDA conditionally generates texts with the diversity beam search decoding, which provides more coherent texts. We analyze the coherence through perplexity based on a large Transformer language model: GPT-2 (Radford et al., 2019). From Table 5, ENTDA obtains the lowest perplexity. Although DAGA and MELM are also based on generative models, the texts are not natural enough since only partial text is replaced. Diversity Evaluation
We measure the diversity of augmented sentences through automatic and manual metrics. For automatic metric, we introduce the Type-Token Ratio (TTR) (Tweedie and Baayen, 1998) to evaluate the ratio of the number of different words to the total number for each original text. Higher TTR (%) indicates more diversity in sentences. Besides that, we ask 5 annotators to give a score for the degree of diversity of the 200 generated texts, with
score range of 1~5. According to the annotation guideline in Appendix D, a higher score indicates the method can generate more diverse texts.
We present the average scores on the datasets in Table 6. ENTDA could obtain 7.8% TTR and 1.4 diversity performance boost in average compared to MELM.

heading: 6 Case Study
text: We show eight approaches to obtain augmented data for the AI domain in Table 7. Compared with baseline augmentation methods, ENTDA introduces a knowledge expansion and conditionally generates texts based on the diversity beam search, which provides more coherent and diverse texts. For example, The Mention Replacement approach replaces the entities unsupervised learning with heterodyning, which ignores the semantics of the context and makes an ungrammatical replacement, resulting in incoherent and un-
reasonable texts. For the DAGA approach, it simply stacks three entities: unsupervised learning, principal component analysis, cluster analysis in the text, which could not provide knowledge expansions to the NER models.

heading: 7 Conclusions and Future Work
text: In this paper, we propose an Entity-to-Text based data augmentation approach ENTDA for NER tasks. Compared with traditional rule-based augmentation methods that break semantic coherence, or use Text-to-Text based augmentation methods that cannot be used on nested and discontinuous NER tasks, our method can generate semantically coherent texts for all NER tasks, and use the diversity beam search to improve the diversity of augmented texts. Experiments on thirteen public real-world datasets, and coherence and diversity analysis show the effectiveness of ENTDA. Moreover, we can also apply the method of data augmentation to low-resource relation extraction (Hu et al., 2020, 2021b,a; Liu et al., 2022b; Hu et al., 2023), natural language inference (Li et al., 2023, 2022c), semantic parsing (Liu et al., 2022a, 2023), and other NLP application tasks, thus realizing knowledge enhancement based on data augmentation approach.

heading: 8 Limitations
text: We discuss the limitations of our method from three perspectives.
First, our method is based on pre-trained language models, so compared to rule-based data augmentation methods (synonym replacement, shuffle within segments, etc.), our method requires higher time complexity.
Second, the entity matching process (Section 4.3) will discard sentences which cannot match entities in the entity list, which will affect the utilization of data.
Third, our data augmentation method based on the pre-trained language models, whose generalization ability is limited since the augmented knowledge comes from the pre-trained language models. However, the knowledge in pre-trained language models is limited and not domain-specific. How to improve the generalization ability of the data augmentation methods is a future research work.

heading: 9 Acknowledgement
text: We thank the reviewers for their valuable comments. Yong Jiang and Lijie Wen are the corresponding authors. Xuming Hu, Aiwei Liu and Lijie Wen were partially supported by the National Key Research and Development Program of China (No. 2019YFB1704003), the National Nature Science Foundation of China (No. 62021002), Tsinghua BNRist and Beijing Key Laboratory of Industrial Bigdata System and Application. Philip S. Yu was partially supported by the NSF under grants III-1763325, III-1909323, III-2106758, SaTC-1930941.

heading: A Dataset Statistics
text: we show the detailed statistics of the datasets in Table 8. We further give details on entity types for thirteen datasets in Table 9.

heading: B Entity Addition and Replacement Strategy
text: ENTDA add and replace the entity in the training set that has the same entity type. This strategy can provide the knowledge expansion during the generation, which is an appealing property when the hand-craft knowledge base is difficult to construct for augmentation approaches.
If we directly replace the entities in the text with other entities of the same type, this is equivalent to the baseline: Mention Replacement. From Table 2, 3 and 4, we could observe that compared to ENTDA (Replace), the improvement of F1 performance is greatly reduced. The main reason is that context-free entities are replaced, resulting in obscure and unreasonable texts. For example, “EU’s German wing says it has received a warning.” may be changed to “EU’s World War Two wing says it has received a warning.” since the two entities share the same type: MISC.

heading: C Hyperparameter Analysis
text: We study the hyperparameter γ in the diversity beam search, which represents the degree of probability penalty in the decoding process and determines the diversity of sentences. Modifying γ allows us to control the diversity of the texts. We vary the γ from 1 to 100 and represent the F1 results using the unified Seq2Seq framework and ENTDA (All) in Table 10. With no more than 1% F1 fluctuating results among three datasets, ENTDA appears robust to the choice of γ.

heading: D Annotation Guideline
text: Each annotator needs to carefully read each augmented text, compare it with the original text, and give a score according to the following criteria. Note that all augmented texts for a dataset are given an average score.
• Score:1. The augmented texts under the same original text are almost the same.
• Score:2. The augmented texts under the same original text are slightly different, with serious grammatical errors.
• Score:3. The augmented texts under the same original text are slightly different, and there are almost no grammatical errors.
• Score:4. The augmented texts under the same original text are diverse, with serious grammatical errors.
• Score:5. The augmented texts under the same original text are diverse, and there are almost no grammatical errors.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 7

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract, Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4, Section 5, Appendix A, Appendix B

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4, Section 5, Appendix A, Appendix B

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4, Section 5, Appendix A, Appendix B

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4, Section 5, Appendix A, Appendix B

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 4, Section 5

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 5.4, Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5.2, Appendix A
C 3 Did you run computational experiments? Section 5, Appendix D

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5.4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5.4, Appendix D

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5.4, Section 5.5, Appendix D
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5.5, Appendix F

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 5.5, Appendix F

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 5.5, Appendix F
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section 5.5

heading: Future Work
text: How to improve the generalization ability of the data augmentation methods is a future research work