ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptions (2024)

Abstract: We present ABEX, a novel and effective generative data augmentation methodology for low-resource Natural Language Understanding (NLU) tasks. ABEX is based on ABstractand-EXpand, a novel paradigm for generating diverse forms of an input document – we first convert a document into its concise, abstract description and then generate new documents based on expanding the resultant abstraction. To learn the task of expanding abstract descriptions, we first train BART on a largescale synthetic dataset with abstract-document pairs. Next, to generate abstract descriptions for a document, we propose a simple, controllable, and training-free method based on editing AMR graphs. ABEX brings the best of both worlds: by expanding from abstract representations, it preserves the original semantic properties of the documents, like style and meaning, thereby maintaining alignment with the original label and data distribution. At the same time, the fundamental process of elaborating on abstract descriptions facilitates diverse generations. We demonstrate the effectiveness of ABEX on 4 NLU tasks spanning 12 datasets and 4 low-resource settings. ABEX outperforms all our baselines qualitatively with improvements of 0.04% 38.8%. Qualitatively, ABEX outperforms all prior methods from literature in terms of context and length diversity 1.

heading: 1 Introduction
text: Improving the performance of deep learning models on downstream Natural Language Understanding (NLU) tasks requires sufficient good-quality training data. However, data annotation is an expensive, time-consuming, and noisy task (Abad and Moschitti, 2016). Data augmentation has proven to be an effective approach for overcoming the data
1Code and data: https://github.com/Sreyan88/ABEX ∗Equal Technical Contribution.
scarcity issue in low-resource NLU tasks with limited training samples (Chen et al., 2023). The two major categories of study in data augmentation include online data augmentation by interpolation in the latent space (Guo et al., 2019; Ng et al., 2020a; Sun et al., 2020; Kumar et al., 2020; Guo, 2020; Sawhney et al., 2021) and offline data augmentation that expands an existing small-scale dataset by generating additional synthetic data (Wei and Zou, 2019; Kumar et al., 2020; Zhou et al., 2021; Kim et al., 2022; Guo et al., 2022). Owing to advancements in generative models that facilitate the creation of high-quality synthetic data, the latter is gaining traction (Yu et al., 2023).
However, generative data augmentation faces
726
two major challenges: diversity in generated augmentations (Geiping et al., 2023) and consistency with the underlying data distribution (Chen et al., 2023). It is crucial to strike a balance between these two aspects, as overemphasizing one at the expense of the other can lead to poor downstream performance. Current augmentation methods based on text-infilling (Ghosh et al., 2023c; Guo et al., 2022; Wang et al., 2022), where the primary task is to generate a new sentence constrained with keywords, are prone to replicate biases and overfit specific linguistic patterns in the low-resource training data, thereby hurting diversity. Additionally, we show that keyword-constrained free-form generation is unable to maintain the core semantic properties of the document, like style, which proves to be critical for specific tasks (e.g., question style document for intent classification. See example in Table 3). Diversity also proves to be an issue with token-level editing methods (Wei and Zou, 2019; Shou et al., 2022) that rarely introduce novel entities or contexts and often randomly edits important tokens. Finally, prompt-based methods that employ Large Language Models (LLMs) require well-curated attributes selected from the data to control the distribution of the generated data (Yoo et al., 2021; Sahu et al., 2023; Yu et al., 2023).
Main Contributions. In this paper, we propose ABEX, a novel data augmentation methodology based on a novel paradigm - Abstract-and-Expand. We first convert an input document into a concise, abstract description of itself and then generate augmentations by expanding the resultant abstraction. The task emulates human language perception and processing: the abstraction phase mirrors how humans distill core ideas from text, focusing on essential meanings, while the expansion phase reflects human creativity in generating varied narratives from a single abstract concept, akin to human extrapolation of ideas into diverse discussions. Our proposed Abstract-and-Expand task, which differs from all tasks proposed in prior art, generates augmentations that are both more consistent and diverse. To learn the task of expanding abstract descriptions, we first synthesize a large-scale synthetic dataset by prompting LLMs and then train an Encoder-Decoder Pre-trained Language Model (BART (Lewis et al., 2019)) on the dataset. Next, we propose a simple and controllable algorithm to generate abstract descriptions for training instances in any given downstream low-
resource dataset. Our proposed algorithm leverages AMR-to-Text and Text-to-AMR and generates abstract descriptions by editing Abstract Meaning Representation (AMR) graphs (Banarescu et al., 2013). Inspired by the success of mixup in data augmentation (Zhang et al., 2018), we also optionally mix AMR graphs of two sentences to boost the diversity of abstract descriptions. Finally, we synthesize diverse augmentations using the fine-tuned model and synthesized abstract descriptions. To summarize, our main contributions are:
1. We propose ABEX, a novel and effective generative data augmentation methodology for low-resource NLP. We employ a novel Abstract-and-Expand task and fine-tune an Enc-Dec PLM to learn the task. ABEX differs from all prior work in its motivation and methodology and closely mimics the human perception and processing of language.
2. We propose a simple, controllable, and training-free method for generating abstract descriptions of source documents from downstream NLU datasets. Our proposed methodology provides explicit control in the documentto-abstract generation process and overcomes the contained generation issue that LLMs face in abstract generation.
3. To evaluate the efficacy of ABEX augmentations, we experiment on 12 datasets across 4 NLU tasks under 4 low-resource settings and show that ABEX outperforms most prior works quantitatively by 0.04% - 38.8%. Additionally, generations by ABEX are superior to prior work in terms of context, token (including entity), and length diversity.
4. We also contribute the large-scale synthetic dataset with ≈0.2 million abstract-expansion pairs to promote further research in this space.

heading: 2 Background and Related Work
text: Definition of abstract description. An abstract description is a concise summary of a text, distilling it to its key concepts and themes while omitting non-essential details, effectively retaining the text’s core message. Examples can be seen in Table 13. Difference between an abstract description and an (abstract) summary. A summary provides a concise overview of the main points or themes of a text, maintaining the original structure and
727
order of ideas. In contrast, an abstract description distills the essence or core concept of the text, often rephrasing or reorganizing the content to capture its fundamental meaning in a more generalized form. In the case of summary generation, while including entities and primary events in the text is incentivized, abstract descriptions should only describe the broad semantic meaning of the text. Contrasting examples are in Tables 13 and 14. Background on AMR graphs. An AMR graph (Banarescu et al., 2013) is a linguistic representation of a sentence that captures the meaning of a document in a structured manner. Formally put, an AMR graph can be represented as G = (V , E), where each vertex V represents a concept, and each edge E represents a relationship between concepts. Generative Data Augmentation for NLP. Generative data augmentation for low-resource NLP can be broken down into 4 main categories: (1) Textinfilling: Given a source text, the task is to corrupt parts of the text and infill the corrupted parts using a Pre-trained Language Model (PLM). The task is generally completed by conditioning the corrupted text (also framed as keyword conditioning by some prior work) to an auto-regressive model (Zhou et al., 2021; Guo et al., 2022; Ghosh et al., 2023c,a,b). The parts of the input text to be corrupted are either chosen randomly (Kumar et al., 2020) or algorithmically (Guo et al., 2022; Ghosh et al., 2023c). (2) Text-editing: Given a source sentence, the task is to edit parts of the sentence (Wei and Zou, 2019; Shou et al., 2022). (3) Prompting: The task is to prompt LMs to generate novel training sentences (Ye et al., 2022; Sahu et al., 2023). The prompt may be further conditioned on attributes extracted from the training data, exemplars, or constraints extracted from the training data. (4) Style conversion: The task is to rephrase or change the style of the source sentence (Chen et al., 2022; Sharma et al., 2022). Chen et al. (2023) perform a large-scale evaluation comparing several augmentation methods.

heading: 3 Methodology
text: Overview. Fig. 1 illustrates the entire workflow of generating augmentations with ABEX. The workflow has 2 major steps: (1) We first learn the task of expanding abstract descriptions by fine-tuning BART on a large-scale synthetic dataset. To accomplish this, we first synthesize a dataset Dab, with abstract-document pairs (xabi ,y ab i ) by prompting LLMs on a large unlabeled dataset Du. (2)
We then generate synthetic augmentations for a downstream NLU dataset Ddown with documentlabel pairs (xdowni ,y down i ) by first converting the documents into abstract descriptions and then employing the fine-tuned BART to generate multiple diverse expansions. Directly prompting LLMs for abstraction and expansion affects controllability, and we also show that it underperforms ABEX.

heading: 3.1 Learning to Expand Abstract Descriptions
text: In this subsection, we provide an overview of the upper half in Fig. 1. We describe how we synthesize the synthetic dataset Dab and fine-tune BART on this dataset to obtain a model capable of expanding abstract descriptions.
(1) Generating a synthetic dataset (Dab). Due to the lack of open-source datasets available for the task, we generate high-quality synthetic data for learning this task by prompting LLMs. We prompt an LLM with documents from Du and ask it to generate an abstract description of them. However, the primary challenge in the proposed generation process is the choice of seed unlabeled datasets. Largescale open-source datasets consist of long documents, in contrast to the nature of instances in the majority of downstream fine-tuning datasets that are made of much shorter documents. Mismatch in the length of training and inference datasets have been shown to degrade performance in various tasks in prior art (Rogers et al., 2021; Ghosh et al., 2023a). The other alternative is to select individual sentences from these long documents. However, this creates an informativeness mismatch as individual and context-less sentences from these documents are rarely self-contained, unlike sentences in downstream datasets. Thus, to overcome these issues, we follow a two-step prompting strategy: (i) We first generate summaries of the original long documents in Du (ii) We then generate abstract descriptions of each summary. We denote our final synthetic dataset by Dab, and Dab is made of abstract-document pairs (a,d) where a is the final output of the LLM from step (ii) and d is the output from step (i). An example can be seen in Fig. 1, and more examples are available in Tables 13 and 14. We employ LLaMA-2 13B (Touvron et al., 2023) for this task and generate ≈0.2 million abstractdocument pairs for fine-tuning. Prompts are listed in Appendix B.
(2) Fine-tuning BART on Dab. After generating paired data, we fine-tune BART on Dab to learn
728

heading: Abstract Description
text: the task of expanding abstract descriptions. The abstract a and the document d serve as the input and target, respectively.

heading: 3.2 Data Augmentation using ABEX
text: This section provides an overview of the lower half in Fig. 1. The primary aim is to generate multiple diverse augmentations of every source document in the downstream task dataset Ddown, which can then be added to Ddown to improve downstream task performance. We first generate abstract descriptions for each instance in Ddown in a controlled manner using our proposed method (described next), followed by employing fine-tuned BART from step (1) to generate multiple expansions of the abstractions. These expansions then act as augmentations.
3.2.1 Controllable Generation of Abstract descriptions for Ddown
Primary Motivation. The most straightforward method to generate abstract descriptions for each instance xdowni in Ddown would have been to employ an LLM with the same prompt discussed in Section 3.1. However, there are 2 major challenges with this approach: (1) Maintaining Label Consistency. A key requirement of effective augmentations is that they maintain label consistency with the underlying Gold-only training instance. For example, a synthetic augmentation of an instance from a sequence classification dataset with a label: positive sentiment should also be of positive sentiment. Prior data augmentation methods based on text-infilling usually retain target-related information (TRI) (or phrases relevant to the label) in the corrupted sentence, followed by infilling text around the TRI to generate augmentations (Guo et al., 2022; Ghosh et al., 2023a,c). Inspired by this, our primary motive is to generate an abstract description of xdowni that retains the TRI corresponding to its label ydowni . Doing this would also ensure that the expansion (or augmentations) would be label-consistent. Accomplishing this using the prompting method discussed in Section 3.1 would require the LLM to be effective at constrained generation. Recent studies, such as the work by Lu et al. (2023) and Sun et al. (2023), suggest that while constrained generation can make prompts more complex, it may also present challenges for LLMs in consistently adhering to the constraints mentioned in prompts. (2) Controlling the degree of abstraction. The degree of abstraction for generating abstract descriptions affects the final augmentations in terms of diversity and label consistency. These factors,
729
in turn, affect downstream performance, and the optimal degree of abstraction varies from task to task. Similar to the above, controlling the degree of abstractions proves to be difficult for LLMs. Additionally, the nature of TRIs differs from task to task, which increases the complexity of the prompts significantly.
Proposed Solution. To overcome the controlled generation bottleneck in LLMs, we propose a simple yet controllable and effective method for generating abstract descriptions. Based on AMR editing, our proposed method is training-free and essentially performs text-editing, so there is no need to learn a model for every dataset. Additionally, it is flexible and can easily cater to a wide range of tasks without significant algorithmic changes.
(1) Text - to - AMR. Our first step is to convert a document into its AMR graph. To perform this step, we employ text-to-AMR AMR-BART (Bai et al., 2022), which is built on BART and trained to generate AMR graphs from text.
(2) Editing the AMR. Following the definition of abstract descriptions and AMRs in Section 2, editing AMR graphs provides a feasible way to generate an abstract description by deleting nodes corresponding to specific, non-central details and keeping the ones that capture the meaning and essence. The editing operations are designed such that the edited AMR graph, once converted back to text, results in an abstract description of the original document. We first linearize the AMR graph generated in Step 1 into a sequence (Bai et al., 2022) to achieve this. However, before editing, we want to ensure we retain the original TRI for the document in the AMR. Thus, inspired by Ghosh et al. (2023a) and Guo et al. (2022), we first extract top-k keywords in the document by measuring the similarity between n-grams from the document and the document label. Once extracted, we ensure these keywords are not edited in the AMR. Note that TRI extraction differs from task to task, and we request that our readers refer to our code for more details.
Next, we perform multiple rounds of deletion operation on the AMR graph. First, we remove certain pre-defined types of attributes from the AMR. Some examples of these types are : value, : wiki, : mod and : quant. We list all such attributes that serve as our candidates for the deletion operation in Appendix F.1. After attribute deletion, we then delete sub-graphs in the AMR graph. A sub-graph can be seen as a broader conceptual unit
describing a specific idea entailed to a concept or entity. Deleting a sub-graph leads to a higher level of abstraction, thereby leading to more diverse sentences (ablation in A.1). We select our candidate subgraphs for deletion based on a metric we define as the depth-ratio. To calculate the depth ratio, we calculate the ratio of the depth of the sub-graph to the entire graph. We define depth as measuring the distance between the root node and the farthest leaf node. Specifically, it captures the vertical span and the nesting level within an AMR graph. We select a sub-graph as an eligible candidate for deletion only if its depth ratio is less than a given threshold α. The maintenance of a depth ratio enables us to regulate the size of the removed graph, thereby determining the level of abstraction. We then sample a deletion rate ε from a Gaussian distribution N (µ, σ2) and dynamically delete ε% sub-graphs among eligible candidates.
(3) Mixing AMR graphs of 2 documents. Mixing samples in the training data to generate new data with concepts from both samples has been a successful augmentation approach across modalities (Zhang et al., 2018; Sahu et al., 2023). The method, also commonly known as mixup, improves the diversity of generated data through semantic interpolation, which in turn leads to more generalized models. To perform mixup in the ABEX framework, we can generate abstract descriptions with mixed concepts from a pair of training instances and then employ B for diverse expansions. Formally, let xdowni be the source document and x down k be another retrieved sentence that is semantically similar to in. We retrieve xdownk using cosine similarity with SentenceBERT (Reimers and Gurevych, 2019). After editing the AMR graphs, Gi and Gk, of documents xdowni and x down k respectively, we first extract all their possible sub-graphs from both AMR graphs. Each sub-graph intuitively represents an individual concept in an AMR graph. We denote the set of sub-graphs as Si and Sk, where Si = {si0, · · · , sin} and n is the total number of sub-graphs (similar for Sk). We now calculate the sub-graph similarity between each pair of subgraphs in Si and Sk and append the top-k subgraphs in Sk to their most similar to sub-graphs Si. To calculate sub-graph similarity, we employ SMATCH++ (Opitz, 2023) at the sub-graph level (details on SMATCH++ in Appendix F.2). The resultant AMR graph Ĝin is then used in Step 4. For generating R× augmentations of xdowni , we do not
730
apply this step on all rounds R but sample a probability γ from a Gaussian distribution N (µ, σ2) and only apply this if γ crosses a set threshold β. (4) AMR - to - Text. To convert the edited graph back to text, we employ AMR-to-text AMRBART (Bai et al., 2022). For our experiments, we employ pre-trained checkpoints provided by the authors in their code release.

heading: 3.2.2 Augmentation Generation
text: Optional Fine-tuning on Ddown. We optionally fine-tune the fine-tuned BART (from the 1st step) on the low-resource downstream dataset for domain adaptation. To obtain abstract-document pairs for this step, we employ the methodology defined in Section 3.2.1 to generate abstracts for each document in the downstream dataset but skip Step (3) (note that mixing AMR graphs of 2 sentences in Step (3) voids the relationship of the abstract with the original document). Generation. After optional fine-tuning, we feed the generated abstracts from Ddown to the finetuned BART capable of expanding abstract descriptions and generating diverse expansions that serve as augmentations. To boost diversity, during autoregressive generation, we perform random multinomial sampling and sample the next word from the top-k most probable words and choose the most probable sequence with beam search. For generating R× synthetic data, we repeat this process for R rounds and add the synthetic augmentations with the Gold-only data for training the downstream NLU model. Note that post fine-tuning BART on Dab, ABEX can be considered as a training-free data augmentation method, i.e., ABEX does not require fine-tuning for specific downstream datasets. Fine-tuning on Ddown is optional, and generating abstracts only requires pre-trained models.

heading: 4 Experimental Setup

heading: 4.1 Tasks and Datasets
text: Upstream Fine-tuning Dataset. For learning to expand abstract descriptions, we employ Dab which consists of 0.2 million unique abstractdocument pairs. Downstream Fine-tuning Datasets. To evaluate the efficacy of ABEX augmentations on downstream low-resource NLU tasks, we are largely inspired by the evaluation setup followed by a wealth of prior work in data augmentation (Sahu et al., 2023; Wang et al., 2022; Guo et al., 2022; Ye et al.,
2022). We additionally evaluate ABEX on the NER task, which prior work does not. Specifically, we evaluate 12 challenging datasets across 4 NLU tasks under 4 low-resource settings as follows:
For Sequence Classification (SC) task, we employ Huffpost (Misra and Grover, 2021) (news category classification), IMDB (Maas et al., 2011) and Yahoo!(Zhang et al., 2015) (answer topic classification), and ATIS (Coucke et al., 2018) and Massive (FitzGerald et al., 2022) (intent classification).
For NER, we employ ConLL-2003 (Tjong Kim Sang and De Meulder, 2003), OntoNotes5.0 (Pradhan et al., 2013) and MultiCoNER (Malmasi et al., 2022) datasets, where all have a common set of tags and some unique tags.
For the Question Answering (QA), we employ SQuAD (Rajpurkar et al., 2016) and NewsQA (Trischler et al., 2017).
For the Sentence Similarity (SS), we employ MRPC (Dolan and Brockett, 2005) and the Quora Question Pairs (QQP) dataset.
Finally, to show that ABEX does not replicate spurious correlations from the training data in the generated augmentations, we employ SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018). These two datasets are known to have spurious correlations. We evaluate on the hard subsets of the test set in a setting similar to Wu et al. (2022). Appendix D provides more details and statistics about these datasets.

heading: 4.2 Hyper-parameters
text: We employ BARTlarge for learning to expand abstract descriptions. Our choice is motivated by the popularity of BARTlarge in data augmentation literature (Ghosh et al., 2023a,c; Wang et al., 2022). We train it 15 epochs using Adam optimizer with a fixed learning rate of 5.6e−5. For downstream NLU fine-tuning, we employ BERTbase-cased (Chalkidis* et al., 2023). We finetune for 100 epochs with a batch size of 4,8 for 100 and 200 splits and 16 for 500 and 1000 splits. For SC and QA, we use Adam optimizer with a fixed learning rate of 1e−5. For NER, we employ FLAIR (Akbik et al., 2019) with a starting lr of 1e−5 and constant decay. For AMR editing, we set µ, σ2, and α to be 0.5, 0.1, and 0.35, respectively. For AMR mixing, we set µ, σ2, and β to be 0.5, 0.1, and 0.6, respectively. Appendix A provides hyperparameter tuning experiments. For low-resource experiments, we perform iterative stratified sampling over the dataset across four low-resource set-
731
tings: 100, 200, 500, and 1000. We generate R=5 augmentations for all baselines and ABEX for all our experiments. We downsample the development set accordingly. We report the micro-average F1 score averaged across 3 runs for 3 random seeds. We provide model results on hyper-parameter tuning in Appendix A.

heading: 4.3 Baselines
text: Gold-only. Gold-only refers to training our model only on the low-resource gold training data. SC Baselines. For SC, we compare ABEX with text editing baselines: EDA (Wei and Zou, 2019), AEDA (Karimi et al., 2021), and AMRDA (Shou et al., 2022), learning-based infilling baselines: SSMBA (Ng et al., 2020b), GENIUS(ft version from the original paper) (Guo et al., 2022), PromDA (Wang et al., 2022), LLM-based prompting baselines: ZeroGen (Ye et al., 2022), GPT3Mix (Yoo et al., 2021) and rephrasing baselines: BackTrans (Yu et al., 2018). IC Baselines. For SC’s IC task subset, we add PromptMix (Sahu et al., 2023) as another LLMbased prompting baseline. NER Baselines. For NER, we compare with LwTR (Dai and Adel, 2020), DAGA (Ding et al., 2020),
MulDA (Liu et al., 2021), MELM (Zhou et al., 2021) and PromDA (Wang et al., 2022).
QA Baselines. For QA, we compare it with ZeroGen, BackTrans, GENIUS, EDA, and AEDA. For SS, we use BackTrans, EDA, AEDA, SSMBA, and AMR-DA.
Additional Details. For all LLM-based baselines (ZeroGen, GPT3Mix, and PromptMix), we employ LLaMa-13B for a fair comparison. Additionally, for all baselines, we generate 5 synthetic augmentations for a fair comparison. The working of all baselines is detailed in Appendix E. In all our result tables, ABEX refers to a model trained on synthetic data with optional fine-tuning after training. Finally, we also employ LLaMA-213B as a baseline, where we prompt the LLM to first abstract and then expand. For abstraction, we employ the same prompt in Section 3.1. For expansion, we provide the prompt in Appendix B.
Ablations. As ABEX ablations, we compare our model with ABEX-stage-2, which does include the fine-tuning on Dab, ABEX-stage-1, which does not include optional fine-tuning on Ddown and ABEXAbs, which does not include the expansion stage and only trains on abstracts as augmentations.
732

heading: 5 Results and Analysis
text: Quantitative Results. Table 2 compares ABEX on the SC task with our baselines. ABEX outperforms all our baselines by 0.04% - 29.12% except on IMDB on the 1000 low-resource setting, where the downstream model overfits the train distribution post data augmentation. Table 5 compares ABEX on the NER task where ABEX outperforms all our baselines by 0.33% - 36.82%. Table 3 compares ABEX on the SS task where ABEX outperforms most of our baselines by 0.48% - 11.22%. Finally, Table 4 compares performance on the QA task, where ABEX outperforms all our baselines by 4.05% - 38.8%. Text-editing baselines like EDA and LwTR are most competitive to ABEX, while generative ones like DAGA and GENIUS lag behind by considerable margins. Robustness against Spurious Correlations. Data augmentation methods often amplify spurious correlations in the training set (Evuru et al., 2024). ABEX strikes a better balance between consistency and diversity, which would prove to be beneficial in OOD scenarios. Table 6 further compares ABEX performance on SNLI and MNLI with spurious correlations.
Qualitative Results. Table 7 compares the generation quality of ABEX with all our baselines (averaged baseline-wise across all tasks and splits) on the measures of perplexity (Jelinek et al., 1977),
diversity (average percentage of new tokens introduced in R augmentations relative to the total tokens in the original document) and length diversity (average absolute difference in length of source and R augmentations). ABEX outperforms all our baselines in all settings.
Figure 2 compares ABEX augmentations with our baselines on MultiCoNER (Malmasi et al., 2022), a dataset with relatively complex semantics. We define Coherence as the quality of the generated augmentation to be linguistically coherent. We define Label Consistency as the quality of the generated augmentation to maintain the same label as the original sample from which the augmentation was generated. Finally, we define Context Diversity as the quality of the generated augmentation to generate a context around the TRI that is diverse and unique compared to the original document. For all 3 criteria, we provide a red cross if it doesn’t meet them and a green tick if it does. ABEX consistently generates augmentations that are coherent, diverse, and label-consistent. The augmentations demonstrate significantly higher degrees of context,
733
entity, and length diversity. Additional examples can be found in Fig. 3, 4, and 6, where we also demonstrate that ABEX maintains key syntactic features of the document, such as its style. This is particularly beneficial for tasks like IC, where other methods often alter the style from a question to a statement, negatively impacting performance.

heading: 6 Conclusion
text: This paper proposes ABEX, a novel data augmentation framework based on a novel paradigm – Abstract-and-Expand. Abstract-and-Expand involves first abstracting a given document and then expanding it. To achieve this, we fine-tune BART on a large-scale synthetic dataset to learn expanding abstract descriptions and then propose a controllable and training-free method to generate abstract descriptions for downstream dataset documents by editing AMR graphs. ABEX outperforms all our baselines, quantitatively and qualitatively, on various downstream datasets and tasks.

heading: Limitations and Future Work
text: In this section, we list down some potential limitations of ABEX:
1. Sentences generated by ABEX may lack factuality. Though factuality is not a requirement
for generated synthetic data that serve as augmentations, and most data augmentation methods from literature don’t guarantee (Ghosh et al., 2023a), we would like to explore ways to overcome this in future work by methods like knowledge-graph grounded decoding.
2. Due to its propensity for creating augmentations that are not factually accurate, ABEX is unsuitable for generative tasks such as instruction tuning or generative question answering. Generative natural language understanding (NLU) tasks acquire new knowledge during training, and the introduction of non-factual augmentations by ABEX could negatively impact this knowledge acquisition. The core mechanism of ABEX involves introducing additional augmentations centered around Targeted Reference Information (TRI), which is beneficial primarily for discriminative tasks like sequence classification, named entity recognition (NER), question answering (QA), and others. This is because the model in these tasks focuses on identifying patterns in the data rather than acquiring new information. The introduction of varied contexts by ABEX enhances the model’s ability to learn these discriminative patterns more efficiently and adapt to new, unseen data distributions. Con-
734
sequently, in alignment with previous methodologies, our evaluation of ABEX is limited to discriminative NLU tasks, excluding generative tasks.
3. ABEX depends on pre-trained AMR-to-Text and Text-to-AMR models for controllable abstract generation. However, AMR parsing is not a solved problem; these models often make errors. Therefore, as part of future work, we would like to explore better methods for controllable abstract generation.

heading: A Hyper-parameter Tuning

heading: A.1 Effect of µ on the diversity of generations
text: Table 8 compares the performance and the diversity of augmentations generated by ABEX at different values of µ. The parameter µ plays a crucial role
in controlling the deletion rate ε during the editing of the AMR graph. By increasing the mean of the Gaussian distribution, we observe a corresponding increase in the average deletion rate, leading to a higher level of abstraction. Consequently, this strategy enhances the performance and diversity of generated augmentations, reaching a peak value before exhibiting a decline.

heading: A.2 Effect of augmentation rounds R
text: Table 9 compares the performance of ABEX at different values of R. Augmenting the training dataset with several augmentation rounds R proves effective until the model overfits to the training data. The observation is similar to prior work in data augmentation for NLU tasks (Zhou et al., 2021; Ghosh et al., 2023c).

heading: A.3 Effect of α
text: Table 10 compares the performance of ABEX at different values of α. While a lower α leads to deleting smaller sub-graphs which would effectively decrease abstraction, a higher α leads to deleting bigger sub-graphs and thus higher abstraction. Similar to our finding in Section A.1, training and inferring with highly abstract sentences leads the model to generate sentences that do not match the underlying data distribution and, thus, sub-optimal performance.

heading: A.4 Effect of β
text: Table 11 compares the performance of ABEX augmentations at different values of β. A lower β leads
738
to less diverse sentences (as a result of lesser augmentations generated using mixed abstracts), and a higher β leads to more diverse sentences (as a result of more sentences generated using mixed abstracts). While token diversity in augmentations improves performance, too much might lead to sub-optimal performance.

heading: B Prompts
text: Document - to - Summary For summarizing a document from Du with LLaMA-2, we use the following prompt: Write me a summary of the article in one line. Don’t include entities; write the summary just describing key events and concepts in the article. Here is the article:. Summary - to - Abstract For generating an abstract from the summary of a document in Du with LLaMA-2 we use the following prompt: I will provide you with a small document. You need to return a short and abstract description of it. Don’t mention named entities, and just describe the key message of the document in a few words. Here are some examples: Input 1: Shatrughan Sinha, a Congress candidate and actor-politician, will run against Union Law Minister Ravi Shankar Prasad, a BJP candidate, in the Patna Sahib seat. Sinha has dismissed BJP’s claim that the seat is their stronghold and has expressed his confidence in winning the election. He has also criticized the BJP’s decision to field Prasad, a four-term Rajya Sabha member, in the seat. Sinha has served two terms in the Rajya Sabha and has been a member of the union council of ministers. He has also defended his record, citing his spending of 106% of his MPLAD fund, which is available on the net. Output 1: A political competition between two candidates from major parties for a significant electoral seat, involving critique of the opposition’s choice and defense of personal achievements. Input 2: Said Baalbaki, a Palestinian artist, has curated an exhibition featuring 50 of Abbo’s sketches, etchings, and objects, along with texts from Baalbaki’s personal collection, showcasing the elusive sculptor’s work and life. Output 2: An exhibition curated by an artist, displaying sketches, etchings, and objects from a lesser-known sculptor, accompanied by per-
sonal texts, highlighting the sculptor’s work and life. Here is the input document:. The exemplars are human written. Abstract - to - Expansion (for LLaMA-13B baseline) I will provide you with an abstract version of a document. You need to understand the abstract and return an expanded version of the document from the abstract. The expansion can be diverse and can add new context and entities. However, it should follow the following constraints while expanding: 1) It should be semantically similar to the abstract, i.e., retain the key points and the message in the abstract. 2) It should retain the following keywords or phrases: [TRI extracted from Section 3.1] 3) The generated sentence should be of the label [Ground-truth document label]. Here is an example of a sentence from the label [Randomly retrieved sentence with the same label]. Here are some examples: [2 Human written exemplars of the process]

heading: C Algorithm
text: We show the Algorithm for ABEX in Algorithm 1.

heading: D Dataset Details

heading: D.1 Classification
text: HuffPost. The HuffPost dataset (Misra and Grover, 2021) is a popular multiclass classification dataset in NLP. It is a collection of news articles from the HuffPost website, covering a wide range of topics, including politics, business, entertainment, and more. For multiclass classification, the HuffPost dataset is labeled with a diverse set of categories and for our experiments, we take sentences from five categories, including politics, sports, entertainment, tech, and business. Dataset statistics can be found in Table 12. Yahoo. The Yahoo Answers topic classification dataset (Zhang et al., 2015) is a widely used dataset for multi-class text classification tasks. It is derived from the Yahoo Answers community-driven question-answering platform, where users ask questions on various topics, and community members provide answers. The dataset contains a large number of question-and-answer pairs covering a wide range of categories or topics. Each question in the dataset is associated with one primary category. The primary categories span diverse subjects, including Society & Culture, Science & Mathematics, Health, Education & Reference, Computers
739
& Internet, Sports, Business & Finance, Entertainment & Music, Family & Relationships, Politics & Government, Travel, Cars & Transportation, Food & Drink, Games & Recreation, Home & Garden, Local Businesses, News & Events, Pets, Beauty & Style and Pregnancy & Parenting. Dataset statistics can be found in Table 12.
Algorithm 1 ABEX: Our proposed augmentation framework ABEX Pre-training Given an instruction-tuned LLM, unlabelled dataset Du, and pre-trained BART Synthesize Dab with abstract-document pairs by prompting the LLM on Du Train BART on Du Data Augmentation with pre-trained BART Given training set Ddown, and pre-trained BART on Du Dab ← ∅,Daug ← ∅ for {X,Y } ∈ Dtrain do ▷Training Loop
tamr ← TEXTTOAMR(X) t ′ amr ← FILTERATTR(tamr) ▷Remove Attributes t ′ amr ← DELETESUBTREE(t ′ amr), if depth-ratio < α X̃ ← AMRTOTEXT(t′amr) Dabstract ← Dabstract ∪ {X̃, Y }
end for for {X̃, Y } ∈ Dabstract do
BARTfinetune ← FINETUNE(BART, X̃) ▷Fine-tune BART end for for {X,Y } ∈ Ddown do ▷Generation Loop
repeatR times: tamr ← TEXTTOAMR(X) t ′ amr ← FILTERATTR(tamr) ▷Remove Attributes t ′ amr ← DELETESUBTREE(t ′ amr), if depth-ratio < α X ′ ← SIMILAR(X) ▷Semantically similar sentence ST ← SUBTREEPAIRS(X,X′ ) ∀ (x1, x2) ∈ ST, tsim ← ARGMAX(SMATCH++(x1, x2)) t ′ mix = t ′ amr + tsim ▷Append similar subtree X̃ ← AMRTOTEXT(t′amr) X̃mix ← AMRTOTEXT(t ′ mix) Xaug ← BARTfinetune(X̃), if γ < β Xmix ← BARTfinetune(X̃mix), if γ > β Daug ← Daug ∪ {Xaug, Y } ∪ {Xmix, Y } end for Daug ← POSTPROCESS(Daug) ▷Post-processing return Dtrain ∪ Daug

heading: D.2 Named Entity Recognition
text: CoNLL-2003. The CoNLL-2003 dataset (Tjong Kim Sang and De Meulder, 2003) is a widely used benchmark dataset for Named Entity Recognition
(NER) tasks in NLP. It was created for the Conference on Computational Natural Language Learning (CoNLL) shared task in 2003. The dataset consists of news articles from the Reuters Corpus, a collection of English news articles. It is annotated with four named entities: person, organization, location, and miscellaneous entities (such as dates and percentages). The annotations indicate the boundaries of the named entities within the text. Dataset statistics can be found in Table 12.
MultiCoNER. MultiCoNER (Malmasi et al., 2022) is large multilingual dataset for complex NER. MultiCoNER covers 3 domains, including Wiki sentences, questions, and search queries, across 11 distinct languages. The dataset represents contemporary challenges in NER and is labeled with six distinct types of entities: person, location, corporation, groups (political party names such as indian national congress), product (consumer products such as apple iPhone 6), and creative work (movie/song/book titles such as on the beach). Dataset statistics can be found in Table 12.
Ontonotes 5.0. Ontonotes 5.0 Pradhan et al. (2013) is a widely used dataset in the field of Natural Language Processing (NLP) and specifically for Named Entity Recognition (NER) tasks. It is a large-scale corpus that provides annotations for a variety of linguistic phenomena, including named entities, across multiple languages. The dataset contains a diverse range of text genres, including news articles, conversational data, and web data, making it suitable for training and evaluating NER models in different domains. It covers three languages: English, Chinese, and Arabic. The dataset is annotated with 11 categories: Person, Organization, Location, Date, Time, Money, Percent, Quantity, Ordinal and Miscellaneous. Dataset statistics can be found in Table 12.
740
D.3 Intent Classification ATIS. The ATIS (Airline Travel Information System) dataset2 is a widely used benchmark dataset for intent classification in the field of NLU. It was developed to address understanding user intents in the context of airline travel information. The dataset consists of queries or utterances that users might input when interacting with a flight reservation system. Each query is labeled with an intent representing the user’s intention or purpose behind the query. The dataset is labeled with intents that are: Flight-Booking, Flight-Status, Flight-Information, Ground-Service, Airfare, Airport-Information, Travel-Preferences, Flight-Cancellation, and None/No-Intent. Dataset statistics can be found in Table 12. MASSIVE. The MASSIVE (Multilingual Amazon Slu resource package for Slot-filling) FitzGerald et al. (2022) dataset is a widely used benchmark dataset for intent classification in the field of NLU. It contains 1M realistic, parallel, labeled virtual assistant utterances spanning 51 languages, 18 domains, 60 intents, and 55 slots. The dataset is labeled with intents some of which are: Alarm set, Play music, Audio volume mute, Weather query, Takeaway order and General joke etc. Dataset statistics can be found in Table 12.

heading: D.4 Sentence Similarity
text: MRPC. The Microsoft Research Paraphrase Corpus (MRPC) dataset (Dolan and Brockett, 2005) is a benchmark for paraphrase identification and semantic similarity tasks. It was developed by Microsoft Research to support research in natural language processing (NLP) and machine learning. The MRPC dataset consists of pairs of sentences manually annotated as either paraphrases (sentences with similar meanings) or non-paraphrases (sentences with different meanings). The sentences cover various domains and topics, including news, fiction, and general web data. Dataset statistics can be found in Table 12. QQP. The Quora Question Pairs (QQP) dataset3 is a widely used benchmark dataset in the field of natural language processing (NLP). It was created by Quora, a popular question-and-answer platform, and released for research. The QQP dataset consists of pairs of questions collected from the Quora
2https://github.com/howl-anderson/ATIS_ dataset/tree/master
3https://quoradata.quora.com/ First-Quora-Dataset-Release-Question-Pairs
platform. Each question pair is labeled as duplicate or non-duplicate, indicating whether the two questions have the same meaning. The dataset contains many question pairs covering diverse topics, allowing for the exploration of semantic similarity and question-matching tasks. Dataset statistics can be found in Table 12.

heading: D.5 Question Answering
text: SQUAD. The SQUAD (Stanford Question Answering Dataset) (Rajpurkar et al., 2016) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. Dataset statistics can be found in Table 12. NEWSQA. NewsQA (News Question Answering) (Trischler et al., 2017) is a challenging machine comprehension dataset of over 100,000 humangenerated question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10,000 news articles from CNN, with answers consisting of spans of text from the corresponding articles. Dataset statistics can be found in Table 12.

heading: D.6 Bias Testing
text: SNLI. The SNLI (Stanford Natural Language Inference) (Bowman et al., 2015) corpus is a collection of 570k human-written English sentence pairs manually labeled for balanced classification with the labels entailment, contradiction, and neutral, supporting the task of natural language inference (NLI), also known as recognizing textual entailment (RTE). Dataset statistics can be found in Table 12. MNLI. The MNLI (Multi-Genre Natural Language Inference) (Williams et al., 2018) corpus is a crowdsourced collection of 433k sentence pairs annotated with textual entailment information. The corpus covers a range of genres of spoken and written text, and supports a distinctive cross-genre generalization evaluation. Dataset statistics can be found in Table 12.

heading: E Baseline Details
text: SSMBA. SSMBA (Ng et al., 2020b) generates synthetic training examples by using a pair of corruption and reconstruction functions to move randomly on a data manifold.
741
AEDA. AEDA (Karimi et al., 2021) is similar to EDA but only employs random insertion of punctuation marks in the original text to generate synthetic augmentations. GENIUS. GENIUS (Guo et al., 2022), pre-trains and optionally fine-tunes BART (Lewis et al., 2019) on a denoising objective using sketches generated with an extreme masking algorithm. The extreme masking algorithm just preserves keywords in a sentence and masks everything else. MELM. MELM (Zhou et al., 2021), which stands for Masked Entity Language Modeling, suggests the fine-tuning of a transformer-encoder-based PLM on linearized labeled sequences through masked language modeling. In low-resource scenarios, MELM surpasses all other baselines and prior techniques on the CoNLL 2003 NER dataset across four languages, including mono-lingual, cross-lingual, and multi-lingual settings. DAGA. DAGA (Ding et al., 2020), short for Data Augmentation with a Generation Approach, suggests the training of a one-layer LSTM-based recurrent neural network language model (RNNLM) by maximizing the probability of predicting the next token using linearized sentences. For sentence generation, they employ random sampling to create entirely new sentences, with the model being fed only the [BOS] token. LwTR. LwTR (Dai and Adel, 2020) replaces a token in a sentence with another token of the same label; the token is randomly selected from the training set. PromDA. PromDA (Wang et al., 2022) proposes a data augmentation framework based on T5 that trains soft prompts using a novel keyword-tosentence algorithm. AMR-DA. AMR-DA (Shou et al., 2022) converts a sample document from a dataset to an AMR graph, modifies the graph according to various data augmentation policies, and then generates augmentations from graphs. The method combines both sentence-level techniques like back translation and token-level techniques like EDA. PromptMix. PromptMix (Sahu et al., 2023) PromptMix prompts instruction-tuned LLMs to generate augmentations for text classification tasks that are close to the class boundary. ZeroGen. ZeroGen (Ye et al., 2022), similar to PromptMix, generates data using LLMs but in a zero-shot manner without any Gold-only data. It prompts pre-trained LLMs (not instruction finetuned) for data synthesis. Baselines not considered. We do not consider more recent baselines provided by Cai et al. (2023), Hu et al. (2023) and Rahamim et al. (2023) as the code for the same was not available at the time of writing the paper. Additionally, we do not consider Zhou et al. (2022) as label flipping is not applicable for our paper for all tasks considered, and Chen et al. (2022) as style transfer is better suited for cross-domain tasks and applying it to single domain tasks is not trivial. Finally, we do not consider Yu et al. (2023) as it requires manual human intervention for attribute extraction for a dataset.

heading: F Additional Details

heading: F.1 AMR Attributes
text: In Section 3.2.1, we describe the removal of a predefined set of attributes from the AMR graph. These sentence-specific attributes are deemed nonessential to the underlying semantics of the sentence and are thus removed. The targeted attributes for removal include: :mod, :wiki, :quant, :value and :op. This process ensures that the resulting AMR graph primarily captures the essential semantic information relevant to the sentence, improving the clarity and conciseness of the abstract description.

heading: F.2 Similar Sentence Retrieval
text: We employ semantic retrieval to mix AMR graphs of 2 semantically similar sentences and generate a single abstract description covering the contents of both sentences. Note that the retrieval uses the original sentence, not the AMR graph of the sentence. Specifically, we calculate the cosine similarity sim(.) between embeddings e(a) and e(b) as follows: sim(a, b) = e(a) · e(b) ∥e(a)∥ ∥e(b)∥ (1) where e(.) is a sentence-encoder (SentenceBERT in our case) and a, and b are text sentences. We take b as the corpus sentence with the highest cosine similarity to a.

heading: F.3 SMATCH++
text: SMATCH (Semantic Matching of Nodes Anchored on Trees) is a graph-matching algorithm designed to evaluate the semantic similarity between structured data, such as parse trees or semantic graphs. It is commonly used in NLP and information retrieval
742
tasks. The SMATCH algorithm considers two input graphs and measures their similarity based on the common structure and semantic alignment between nodes. It operates by recursively matching nodes in a top-down manner, considering both the nodes’ syntactic relationships and semantic properties. The key idea behind SMATCH is to find the best alignment between nodes of the two input graphs, aiming to maximize the matching score while minimizing structural and semantic inconsistencies. It assigns similarity scores to matched nodes based on their attribute values and relationships and calculates the overall graph similarity as the weighted average of node similarity scores.
The output of the SMATCH algorithm is a similarity score that quantifies the semantic similarity between the two input graphs. Higher scores indicate greater similarity, while lower scores indicate dissimilarity.
SMATCH aims to measure the structural similarity of graphs via the number of triples shared by GA and GB. To obtain a meaningful score, it leverages an alignment map: vars(a) ↔ vars(b) that tells it how to map a variable in the first MR to a variable in the second MR. In this alignment, at maximum, every variable from a can have one partner in b (and vice versa). Let an application of a map to a graph a be denoted as amap := {tmap ; t ∈ a}, where tmap of a triple t = <x, :rel, y> is set to tmap = <map(x), :rel, map(y)> for binary triples, and tmap = <map(x), :rel, c> for unary triples. Under any alignment map, we can calculate an overlap score f . In original smatch, f is the size of the triple overlap of a and b:
f(a, b,map) = |amap ∩ b|. (2)
, The primary aim is to find F as follows:
F = max map f(a, b,map), (3)
Finding a maximizer map⋆ lies at the heart of SMATCH. For now, we assume that we have map⋆ at our disposal. Therefore, we can calculate precision (P ) and recall (R):
P = |a|−1F, R = |b|−1F, (4)
to obtain a final F1 evaluation score: 2PR/(P+R). With such a score, we can assess the similarity of MRs, and compare and select parsing systems.
SMATCH++ (Opitz, 2023) improves over SMATCH by proposing a standardized and extended metric calculation of fine-grained sub-graph meaning aspects, making it more suitable for our task. Specifically, they show the feasibility of optimal alignment in a standard evaluation setup and develop a lossless graph compression method that shrinks the search space and significantly increases efficiency. We request our readers to refer to the original paper for more details.

heading: G Extra Details
text: Model Parameters: BARTlarge ≈ has 680M parameters with 12 layers of encoder, 12 layers of decoder, 1024-hidden-state, and 16-heads. BERTbase has ≈ 110M 12-layers of encoder, 768-hiddenstate, 2048 feed-forward hidden-state, and 8-heads.
Compute Infrastructure: All our experiments are conducted on a single NVIDIA A100 GPU. An entire ABEX training pipeline takes ≈ 2 hours. Implementation Software and Packages: We implement all our models in PyTorch 4 and use the HuggingFace 5 implementations of BERTbase and BARTlarge.
We also use the following repositories for running the baselines: BackTrans (Yu et al., 2018), EDA6(Wei and Zou, 2019), AEDA7 (Karimi et al., 2021), AMR-DA8 (Shou et al., 2022), SSMBA9 (Ng et al., 2020b), GENIUS(-ft)10 (Guo et al., 2022), PromDA11 (Wang et al., 2022), PromptMix12 (Sahu et al., 2023), ZeroGen13 (Ye et al., 2022), GPT3Mix14 (Yoo et al., 2021), LwTR15 (Dai and Adel, 2020), DAGA16 (Ding et al., 2020)(Ding et al., 2020) and MELM17 (Zhou et al., 2021). All the baseline repositories are covered under the MIT License.
We use the following datasets to evaluate: Huff-
4https://pytorch.org/ 5https://huggingface.co/ 6https://github.com/jasonwei20/eda_nlp 7https://github.com/akkarimi/aeda_nlp 8https://github.com/zzshou/amr-data-augmentation 9https://github.com/nng555/ssmba
10https://github.com/beyondguo/genius 11https://github.com/GaryYufei/PromDA 12https://github.com/servicenow/promptmix-emnlp-2023 13https://github.com/jiacheng-ye/ZeroGen 14https://github.com/naver-ai/hypermix 15https://github.com/boschresearch/data-augmentation-
coling2020 16https://github.com/ntunlp/daga 17https://github.com/randyzhouran/melm
743
post18 (Misra and Grover, 2021), Yahoo19 (Zhang et al., 2015), IMDB20 (Maas et al., 2011), Massive21 (FitzGerald et al., 2022), ATIS22 (Coucke et al., 2018), ConLL-200323 (Tjong Kim Sang and De Meulder, 2003), OntoNotes-5.024 (Pradhan et al., 2013), MultiCoNER25(Malmasi et al., 2022), MRPC26(Dolan and Brockett, 2005) and the Quora Question Pairs (QQP) 27, SQuAD28 (Rajpurkar et al., 2016), NewsQA29 (Trischler et al., 2017), SNLI30 (Bowman et al., 2015) and MNLI31 (Williams et al., 2018). All the datasets have been released under various licenses for research purposes. Potential Risks: Generative models learn from vast amounts of textual data, including biased or prejudiced content present on the internet. As a result, there is a risk of bias amplification, where the models unintentionally perpetuate or reinforce existing biases. Also, generative models can generate highly coherent and contextually plausible text, raising concerns regarding the potential for generating misinformation or disinformation.

heading: H Augmentation Examples
text: Figure 3, Figure 4 and Figure 6 compare augmentations generated by ABEX with all our baselines. The figures show generations from the ATIS (Microsoft, 2023), Yahoo (Zhang et al., 2015) and MRPC (Dolan and Brockett, 2005) datasets. In addition, we assess the augmentations on their coherence, ability to include diverse contexts and maintain label consistency. Notably, all baselines demonstrate the ability to generate augmentations with label consistency. However, they fall short of introducing new contextual information within the sentences. Conversely, augmentations generated
18https://www.kaggle.com/datasets/rmisra/news-categorydataset
19https://huggingface.co/datasets/yahoo_answers_topics 20https://ai.stanford.edu/ amaas/data/sentiment/ 21https://huggingface.co/datasets/AmazonScience/massive/viewer/en-
US 22https://github.com/howl-anderson/ATIS_dataset 23https://huggingface.co/datasets/conll2003 24https://catalog.ldc.upenn.edu/LDC2013T19 25https://registry.opendata.aws/multiconer/ 26https://www.microsoft.com/enus/download/details.aspx?id=52398 27https://quoradata.quora.com/First-Quora-DatasetRelease-Question-Pairs 28https://rajpurkar.github.io/SQuAD-explorer 29https://www.microsoft.com/enus/research/project/newsqa-dataset/download/ 30https://nlp.stanford.edu/projects/snli/ 31https://cims.nyu.edu/ sbowman/multinli/
by AMR-DA and Backtrans. consistently exhibit coherence, while those produced by AEDA and SSMBA often lack coherence. The generations from ABEX excel in all three evaluated areas.
744
747
748

heading: Future Work
text: , 2023a), we would like to explore ways to overcome this in future work by methods like knowledge-graph grounded decoding. 2. Due to its propensity for creating augmentations that are not factually accurate, ABEX is unsuitable for generative tasks such as instruction tuning or generative question answering. Generative natural language understanding (NLU) tasks acquire new knowledge during training, and the introduction of non-factual augmentations by ABEX could negatively impact this knowledge acquisition. The core mechanism of ABEX involves introducing additional augmentations centered around Targeted Reference Information (TRI), which is beneficial primarily for discriminative tasks like sequence classification, named entity recognition (NER), question answering (QA), and others. This is because the model in these tasks focuses on identifying patterns in the data rather than acquiring new information. The introduction of varied contexts by ABEX enhances the model’s ability to learn these discriminative patterns more efficiently and adapt to new, unseen data distributions. Con-
734
sequently, in alignment with previous methodologies, our evaluation of ABEX is limited to discriminative NLU tasks, excluding generative tasks. 3. ABEX depends on pre-trained AMR-to-Text and Text-to-AMR models for controllable abstract generation. However, AMR parsing is not a solved problem; these models often make errors. Therefore, as part of future work, we would like to explore better methods for controllable abstract generation