Controllable Data Augmentation for Few-Shot Text Mining with Chain-of-Thought Attribute Manipulation (2024)

Abstract: Prompting large language models (LLMs) for data augmentation has recently become a common practice in few-shot NLP tasks. In this paper, we propose Chain-of-Thought Attribute Manipulation (CoTAM), a novel approach that generates new data from existing examples by only tweaking in the user-provided, taskspecific attribute, e.g., sentiment polarity or topic in movie reviews. Instead of conventional latent representation controlling, we leverage the chain-of-thought prompting to directly edit the text in three steps, (1) attribute decomposition, (2) manipulation proposal, and (3) sentence reconstruction. Extensive results on various tasks, such as text (pair) classification, aspect-based sentiment analysis, and conditional text generation, verify the superiority of CoTAM over other LLM-based augmentation methods with the same number of training examples for both fine-tuning and in-context learning. Remarkably, the 2D visualization of the augmented dataset using principal component analysis revealed a human-recognizable decision boundary that is likely hinted by the attribute manipulation, demonstrating the potential of our proposed approach.

heading: 1 Introduction
text: Prompting large language models (LLMs) for data augmentation has recently become a common practice in few-shot natural language processing (NLP) tasks. Existing methods (Yoo et al., 2021; Sahu et al., 2022b; Dai et al., 2023; Lin et al., 2023) typically first generate new task-specific data with LLMs hinted by few-shot demonstrations and then fine-tune a (small) pre-trained language model with the augmented dataset for better performance. The same augmented data can be also incorporated into in-context learning (ICL) (Li et al., 2023; Dong et al., 2023). However, these augmentation methods usually prompt LLMs to generate new examples wildly without proper control, which hinders
∗Corresponding author.
the informativeness of generated data and might induce spurious correlation. As shown in Figure 1 (left), the generated data without control has no clear pattern and could even possibly mislead the fine-tuning or ICL under few-shot supervision.
In this paper, we propose a controllable data augmentation for few-shot text mining. The general idea is to generate new data from existing examples by only tweaking the user-provided, task-specific attribute, e.g., sentiment polarity or topic in movie reviews. Intuitively, as shown in Figure 1, one can expect that this approach can efficiently find the decision boundary because we (1) directly manipulate along the direction of task-specific attributes and (2) maintain the rest of the attributes as before.
Different from the existing controllable generation works in computer vision (Shen et al., 2020; Shen and Zhou, 2021) and natural language processing (Kruengkrai, 2019a; Zhou et al., 2022), where attributes are manipulated in the latent space of the encoder before reconstructing new instances, we leverage the chain-of-thought (CoT) prompting (Wei et al., 2022c) to directly edit the text using LLMs in three steps, (1) attribute decomposition, (2) manipulation proposal, and (3) sentence
1
reconstruction. Specifically, we start with the userprovided, task-specific attributes, and then prompt LLMs to decompose each individual text example into other orthogonal attributes. Compared with a pre-defined attribute set per dataset, we believe that such dynamically constructed, per-example sets of attributes can better capture the uniqueness of every piece of text. Second, we instruct LLMs to propose a plan to manipulate the values of the task-specific attributes while maintaining the other attribute values the same. Finally, we prompt the LLMs to reconstruct the sentence based on the manipulation proposal. All these steps are written in a single prompt and fed to the LLM at once. Furthermore, using LLMs benefits the interpretability of our framework where attributes are completely transparent to users.
We conduct extensive experiments to evaluate CoTAM and baselines using a series of few-shot classification tasks with very different classification targets, aspect-based sentiment analysis, and conditional text generation for more complex attribute manipulation. For a fair comparison, all compared methods utilize the same LLMs and generate the same amount of data. We assess the quality of generated data by looking at (1) the performance of trained small language models via fine-tuning or tuning-free methods on the augmented data and (2) the ICL performance of LLMs using the augmented data as demonstrations. Extensive experimental results including label-scarce and out-of-domain scenarios demonstrate the advantage of proposed controllable data augmentation over conventional methods. The ablation study further reveals the necessity of attribute manipulation comparing to directly flipping the labels. Finally, we present PCA analysis on the embeddings of generated augmentations that visually illustrates the effectiveness of method.
Our contributions are three-fold:
• We propose a novel controllable data augmentation approach CoTAM based on chain-ofthoughts prompting using LLMs, which directly edits the text examples in an interpretable way instead of tweak latent representation vectors. • We conduct experiments on a wide spectrum of tasks and datasets, demonstrating the effectiveness of the augmented data by CoTAM in both fine-tuning and in-context learning. • Our detailed analyses, especially the humanrecognizable decision boundaries revealed by the
2D visualization of the augmented dataset using principle component analysis, demonstrate the significant potential of our proposed attribute manipulation approach. Reproducibility. We will open-source the code. 1

heading: 2 Problem Formulation
text: We aim to generate more efficient training data using controllable augmentation on a few-shot dataset D focusing on a target attribute Y (e.g., the classification objective) with N possible values {y1, y2, · · · , yN} (i.e., N -way). For each possible attribute value yi, the dataset D provides K examples (i.e., K-shot) of texts with the value. We here showcase two mainstream few-shot learning schemes as the basis to discuss the augmentation: • In-context Learning (ICL) is a scheme for
LLMs, which takes a few examples of sentences with their target attribute values (i.e., a series of (X, yi)) as the context to handle new inputs. With these demonstrations, the LLM is expected to understand the underlying mapping and then predict the label of new inputs. • Fine-tuning generally trains smaller models with the (limited) labeled data. The model has a text embedder E and a classifier C. A text x from the dataset D will be represented as a dense vector E(x), which is learned to encode the attributes of x, including the target attribute Y and other attributes. The classifier C further processes the vector E(x) and outputs a distribution over y1, y2, · · · , yN , indicating the probability of each Y value in x. Ideally, our controllable augmentation shall supply efficient demonstrations and training data under the ICL and fine-tuning settings, respectively.

heading: 3 Our CoTAM Framework
text: To boost the performance of few-shot methods, we suppose a scenario, shown in Figure 2, to augment examples that well improve the task awareness of the inference models. For a given sample x with target attribute value y from D, we will manipulate its attribute value to y′ that y ̸= y′ to form a build a new sentence x′. We set two requirements for the manipulation: 1) Significant Manipulation on the target attribute Y , which means the manipulated result x′ should be viewed with yj by oracle like humans. 2) Minor Manipulation on all other attributes Z , which indicates x and x′ to share a
1Code: https://github.com/anonymous_repo
similar value zk for all Z ∈ Z . To meet the two requirements above will ensure x and x′ only differ in attribute Y , making them an efficient pair for learning on the dataset D. Take fine-tuning as an example, the loss L(X, yi) + L(X ′, yj) will be attributed to the only different attribute Y , thus let each annotation by humans efficiently reflect the target attribute with its augmentations.
Based on our desiderata above, we propose CoTAM that benefits from the strong text manipulation capability of LLMs (OpenAI, 2023) with its workflow demonstrated in Figure 2. To be more specific, we first create chain-of-thought (CoT) queries to decompose the input texts into many attributes, which approximates the latent space. We aim to get rid of human labor to propose other possible attributes for efficiency. Moreover, in some cases, even human experts cannot give you a complete list of other attributes among all the possible texts. Finding a shared and fixed set of attributes for various kinds of texts is hard since different sentences rarely share a common set of applicable attributes. Encouraged by Wang et al. (2023), we instruct LLMs to propose a dynamic attribute set for each input text, which are customized among inputs dependent on which attributes are applicable. The CoT then switches the value of the target attribute to other possible values in the task and prompts the LLM to reconstruct the manipulated sentence. Finally, the LLM is guided to compose such a sentence to finalize the manipulation.
Different from the existing controllable generation works in computer vision (Shen et al., 2020; Shen and Zhou, 2021) and natural language processing (Kruengkrai, 2019a; Zhou et al., 2022), where attributes are manipulated in the latent space of the encoder before reconstructing new instances, our CoTAM is proposed to directly edit the text using LLMs.

heading: 3.1 Step 1: Attribute Decomposition
text: Following the macro-level design of CoTAM, the first step in the CoT is to decompose the sentence into various attributes. The LLM takes the sentence and a human-annotated attribute-value pair as the input and then propose other attributes and their values.
For example, The sentence “While some actors perform well, the movie is dull overall” with be processed with its known attribute-value yi, here is “Sentiment: Negative”. The LLM then proposes a set of other applicable attribute-values Ẑ = LLMAD(X, yi) ⊂ Z like “Topic: Moive”, “Structure: Shift” as in Figure 2, which is a subset of Z but is generally detailed enough to approximate the irrelevant attributes. The value of the known attribute is then flipped to another one given by the user like “Sentiment: Positive”, which is then combined with other LLM-proposed attributevalues for the next step.

heading: 3.2 Step 2: Manipulation Proposal
text: In the second step, we will instruct the LLM to propose the methodology to reconstruct a sentence with the switched attribute and others from the decomposition step. This step is incorporated as understanding how to achieve the goal, which is important to the CoT inference (Wei et al., 2022c). In this step, the LLM takes all elements in the manipulation as the input and proposes an instruction I = LLMMP(X, yi, yj , Ẑ) for LLM to execute in the next step. A proposed manipulation is shown as in Figure 2, the LLM suggest several instructions to complete the manipulation.

heading: 3.3 Step 3: Sentence Reconstruction
text: This step simply asks the LLM to follow its proposed manipulation instruction I to reconstruct the sentence and output a label-flipped one as X ′ = LLMSR(X, I). As in Figure 2, the LLM follows the self-generated instructions to edit the input sentence to generate our desired X ′ that has
significant different in Y (sentiment polarity) and minor difference in Ẑ (proposed other attributes).

heading: 4 Experiments
text: In this section, we evaluate different LLM-based augmentation methods on a series of classification tasks, with different target attributes. We incorporate comprehensive ways of utilizing augmentations with different classification techniques, such as fine-tuning, in-context learning and inference with sentence embedding. We further evaluate the augmentation ability of methods on more complex tasks like aspect-based sentiment analysis and conditional text generation.

heading: 4.1 Datasets
text: We verify the advantage of CoTAM on text classification and other tasks using 6 classification datasets, including SST-2 (sentiment polarity) (Socher et al., 2013), TweetEmo (fine-grained sentiment) (Barbieri et al., 2020), AG-NEWS (topic) (Zhang et al., 2015), MNLI (natural language inference) (Williams et al., 2018), MRPC (semantic textual similarity) (Dolan and Brockett, 2005), and CSQA (multiple choice question answering) (Talmor et al., 2019). MNLI includes matched (MNLIm) and mismatched (MNLImm) datasets for evaluation. To further test the ability of CoTAM on attributes other than classification targets, we include a manipulation on aspect-based sentiment analysis (ABSA) and conditional text generation tasks. For ABSA datasets, we include Restaurant and Laptop from SemEval2014 (Pontiki et al., 2014). For conditional text generation, we include CommonGen (Lin et al., 2020). We report the results on 1000 samples for ICL from the mixture of validation and test datasets due to cost issues. For other setups, we report results on the validation dataset when the test dataset is not publicly available considering the efficiency to get multirun results. The statistics of datasets are presented
in Appendix A. We present some examples of attribute names in Table 1.

heading: 4.2 Compared Methods
text: CoT Data Augmentation (CoTDA) is an augmentation variant of our method that applies a similar CoT for conventional augmentation. Instead of directly asking for augmentation, we let the LLM follow our proposed CoT and propose a methodology to write a sentence with the same attributes as the input sentence. CoTDA is the main baseline for comparison to explore the importance of attribute switching in our CoTAM. For each seed data, we augment it for N-1 times with 0.1 temperature, where N refers to the number of classes in the dataset. Thus, CoTDA generates the same number of new data as CoTAM to achieve a fair comparison.
FlipDA (Zhou et al., 2022) is a traditional labelswitched augmentation method based on conditional generation by a fully-tuned T5 (Raffel et al., 2020). Specifically, the sentence is combined with the switched label as the input to T5. Then, some spans in the sentence are randomly masked and recovered by T5 conditioning on the new label to switch the semantics of the sentence. As the original FlipDA requires a large supervised dataset that is inapplicable to few-shot learning, we build an LLM-based FlipDA (FlipDA++) baseline by sending span replacement instructions to LLMs.
Human/LLM Annotation directly using the texts labeled by humans or LLMs. For human annotation, we include the K-shot (Base) and NKshot (Extra Annotation) setups. K-shot represents the baseline before integrating the data generated from LLMs. NK-shot has the number of training data after augmentation with human annotation, thus we expect it to be a upper bound of augmentation methods. Whereas, we will see CoTAM able to outperform this upper bound, which can be attributed to higher data quality resulting from attribute manipulation. NK-shot LLM annotation2 (Pseudo Label) represents a simple baseline that is generally applied when much unlabeled in-domain data is available.
Comparison Fairness We select GPT-4 (OpenAI, 2023) as the LLM to construct the dataset. The temperature of GPT-4 to set to 0 towards high
2K-shot data are used for in-context inference.
quality and reproducibility. We apply each augmentation method to a fixed subset of each dataset to create a small subset from which we sample training data. For a fair comparison, this subset is also used in other baselines for data generation. By default, we set K to 10 for fine-tuning and 3 to ICL. All reported results are the average over 10 runs (except for ICL due to expense) to eliminate the bias.
All the prompts in our experiments are presented in Appendix C for better reproducibility.3

heading: 4.3 Classification Result
text: Fine-tuning A simple way to evaluate the data quality is to tune a model on it and then check its performance. We select RoBERTa-Large (Liu et al., 2019) as the learner on different datasets. With the validation dataset unavailable, we train the model for 32 epochs4 and then evaluate it.
3To further increase reproducibility, we also include results on open-sourced LLMs in Appendix D
4Except 8 epochs for MRPC, on which the model is more likely to overfit.
As presented in Table 2, our CoTAM achieves the best fine-tuning results on all 7 tasks in comparison with other LLM-based data generation methods. On most tasks, the two label-switching methods (FlipDA and CoTAM) outperform other methods, which indicates using the LLM to switch labels creates more efficient data. On label switching, attribute manipulation shows superiority over simple span replacement as our CoTAM performs better than FlipDA on all tasks. The prominent performance of CoTAM also verifies the capability of LLMs to manipulate complex attributes which might refer to premises or questions.
On 6 out of 7 tasks, our CoTAM breaks the supposed upper boundary of (N-way) NK-shot with extra human annotations. This indicates that carefully crafted data from LLMs have the potential to train better models than ones trained on the same number of human annotations. Also, aur CoTAM is verified to be such a method that improves the data efficiency by attribute manipulation.
In-context Learning The performances of ICLbased inference with different augmentation methods are demonstrated in Table 2. Our CoTAM show superior ability on providing LLMs with few-shot examples for inference, thus broadening the application of our method. The only fail case for CoTAM is the out-of-domain MNLI, where few-shot examples do not benefit the inference. Still, among all augmentation scenarios, our CoTAM performs the best for this evaluation.
Inference w/ Sentence Embedding In the field of few-shot text classification, text embedding has
proven to be a powerful tool for improving performance and efficiency (Muennighoff et al., 2023). This section is dedicated to exploring instancebased techniques designed explicitly for text classification with text embedding models.
In instance-based inference, a text embedding model converts the input sentence into a representation. The label of this representation is then determined based on its proximity to annotated sentence representations. We utilized two tuning-free algorithms in our experiments—Nearest Centroid (NC) (Manning et al., 2008) and K-Nearest Neighbors (KNN)—and applied them to three different text classification datasets. NC assigns a label to an input sentence depending on how close it is to centroids, defined as the average representation of sentences sharing the same label. In contrast, KNN labels the input sentence according to the most common label amongst its nearest K neighbors. We set K to 5 in our experiments. We harness the Simple Contrastive Sentence Embedding (SimCSE) model (Gao et al., 2021), with RoBERTa-Large as the backbone model5, to encode the texts.
Table 3 showcases the performance of different data generation methods when used with instancebased algorithms. In contrast to methods that generate new texts (such as FlipDA and CoTDA), our proposed method, referred to as CoTAM hereafter, exhibits superior performance in most configurations. This implies that data created by CoTAM also benefits from improved distributions in the latent space of text embedding models. On the AG-NEWS dataset, instance-based algorithms show a preference for in-domain annotations, whether made by humans or Large Language Models (LLMs). This highlights the importance of using in-domain texts when employing these algorithms for certain tasks.

heading: 4.4 Aspect-based Sentiment Analysis
text: Here we further expand the utility of CoTAM to a more complex scenario to manipulate multiple span representations. We experiment on aspect-based sentiment analysis (ABSA), which aims to extract spans targeted by sentiment (aspects) in a statement and corresponding polarities. For instance, the aspect extracted from “The food is good.” will be “positive aspect: food”.
For attribute manipulation on ABSA, we view the aspects as the ABSA attributes like “positive aspect: food”. We query the LLMs to decompose texts into ABSA and other attributes. The polarities of ABSA attributes are then randomly switched and used for the reconstruction. The reconstructed data are merged into the initial dataset as the augmentation.
We use the SemEval2014 ABSA dataset which has two subsets: restaurant and laptop and three sentiment polarities: positive, negative, and neutral6. We set the shot number (K) to 10 and generate 2 times for each instance (N = 3), which is the maximal manipulation time for an instance with only one aspect. The results on ABSA are presented in Table 4, our CoTAM successfully outperforms other LLM-based augmentation methods, which confirms that CoTAM is applicable to more complex scenarios than single sentence attribute manipulation.

heading: 4.5 Conditional Text Generation
text: We run experiments on the CommonGen dataset to apply our CoTAM to conditional text generation. CommonGen targets to generate a sentence that contains a set of keywords. For instance, with input words: “ski, mountain, skier”, the output can be “Skier skis down the mountain” We use CoTAM
5huggingface.co/princeton-nlp/sup-simcse-roberta-large 6We remove the conflict polarity because of its sparsity in
the dataset.
to manipulate the data by switching the group of keywords to another one (proposed by the LLM) while keeping other attributes unchanged.
According to the metrics shown in Table 5, we can view CoTAM holds its advantage over other LLM-based augmentation methods. Thus, we conclude that CoTAM is not only limited to classification tasks but also has the potential for information extraction and natural language generation.

heading: 5 Further Analysis

heading: 5.1 Workflow Demonstration
text: In Figure 3, we demonstrate the workflow of the dynamic attribute decomposition mechanism. We include the attributes that most commonly appear in the manipulation according to the statistics in Appendix E. In the workflow, our CoTAM decomposes sentences into applicable attributes and reconstructs while maintaining these attributes. For instance, tone (X) is more applicable to the first sentence due to its subjectivity and comparison (X) is more applicable to the last sentence since only it involves comparison. These attributes comprehend the unchanged parts of texts to guide the reconstruction during the manipulation. Subsequently, the reconstruction switch the targeted label (sentiment (X) in the case) with minor change to other attributes.

heading: 5.2 Ablation Study
text: We launch an ablation study to verify the importance of each thought in the CoT. We also explore the effect of different LLMs. We thus change the LLM in our experiments to GPT-3.5-turbo. The experiments show that the GPT-4 leads to significantly better fine-tuning results. Also, this gap can be narrowed down by text embedding models on text classification.
The outcomes of our ablation study are detailed in Table 6. In this study, we found that eliminating each “thought” from our CoT resulted in a decline in performance. Interestingly, the “what” (decomposition) thought proved more critical than the “how” (methodology) thought, accentuating the predominance of attribute proposal over auxiliary methodology proposal. The CoT is necessary for label switching as the removal of it leads to significant performance degradation. In comparison between LLMs, GPT-4 outperforms GPT-3.5-turbo, indicating that CoTAM favors larger LLM with better language capability, especially on more complex tasks like MNLI. Finally, we compare the performance of between CoTAM with a fixed attribute pool (FAP) and with a dynamic attribute pool in our experiments. The result shows the advantage to remove the type limitation of attribute the LLM decomposes into.

heading: 5.3 Visualization of Attribute Manipulation
text: In an attempt to confirm our hypothesis that LLM is adjusting a single feature while keeping other attributes constant, we illustrate data pair representations from CoTAM in Figure 4. We use principal component analysis (PCA) (F.R.S., 1901) to take the high-dimensional (1024-dimensional) text representations from SimCSE and simplify them into a 2-dimensional space for ease of visualization.
The diagram distinctly demarcates between positive and negative representations, which underscores the value of our method in fine-tuning and instance-based inference. Additionally, the direction of representation switching is largely consistent, providing further evidence that LLMs have the ability to tweak one attribute while keeping others stable. This consistency in the direction of the switch hints at the predictability and control we have exercised over LLM behavior for targeted feature manipulation. In comparison to CoTDA, our CoTAM depicts a clearer boundary, thus enabling more efficient data learning than traditional data augmentation.

heading: 5.4 Data Scale Analysis
text: In this section, we analyze how the number of initial data affects the performance of our CoTAM. Thus, we sample 3000 more instances from SST2 to scale up the sampling pool. As presented in Figure 5, CoTAM is able to break the NK-Shot boundary with few examples (K ≤ 64) for finetuning. With text representation models, CoTAM shows a significant advantage on very few examples (K ≤ 4) and converges to a similar performance with human annotation. Though fine-tuning on more human annotation leads to higher performance than CoTAM, the in-domain performance improvement might be a result of overfitting to the domain. Thus, we further evaluate CoTAM and NK-Shot on custom review, an out-of-domain dataset with the same labels as SST-2. On custom
review, CoTAM shows a consistent advantage with different data numbers. Thus, we conclude our CoTAM is more robust to domain mismatching than direct tuning.

heading: 5.5 Case Study
text: Figure 6 specifies the real attribute manipulation process in our experiments. For better depiction, we simplify the response by only presenting the attributes proposed by the LLMs.
In the SST-2 example, other attributes include labels in a different categorization (Topic: Movie Review), actor entities (Actor: Ford, Neeson), and overall style (Opinion: Overall). These attributes are well preserved in the reconstruction, which contributes to a strong contrast in the task target and consequently improves the data efficiency.
Moving on to the MNLI example, the sentence primarily breaks down into different semantic elements. When these elements are reconstructed, they follow a logical sequence that differs from the original sentence. Thus data from CoTAM reinforces the learner’s comprehension of textual logic which is crucial for tackling MNLI.

heading: 6 Related Work
text: Attribute Manipulation aims to control certain attributes of the data. A general application of attribute manipulation is to change the visual attributes in facial images (Shen et al., 2020; Shen and Zhou, 2021). Image manipulation generally involves the transformation of image representations (Perarnau et al., 2016; Xiao et al., 2018; Shen et al., 2020) in the latent space. In natural language processing, the closest topic to attribute manipulation is data flipping (Kruengkrai, 2019b; Zhou et al., 2022), which replaces key spans in the text to switch its label. Obviously, many textual attributes like topics cannot be manipulated by span replacement. Thus, we choose to adapt the LLM to manipulate a latent space approximated by a series of attributes proposed by the LLM. Controllable Generation is another close topic to our CoTAM. These methods typically generate texts from a continuous latent space discretely by controlling certain dimensions (Bowman et al., 2016; Hu et al., 2017; Yang and Klein, 2021). The controllable generator is trained by maximizing a variational lower bound on the data log-likelihood under the generative model with a KL divergence loss (Hu et al., 2017). The limitation of the cur-
rent controllable generation is no explicit control of other dimensions to maintain them the same. Our method addresses this issue by completely decomposing the input text into multiple labels with LLMs and then reconstructing it with switched attributes. Large Language Models are large-scale models trained on a massive number of texts (Brown et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022) that have been shown to have emerging capabilities (Wei et al., 2022b). One of these capabilities is learning from few-shot demonstrations, which is often referred to as in-context learning (Dong et al., 2022). However, these demonstrations must be concatenated into contexts during inference time, increasing the computational costs and carbon footprints. Another important capability is to follow instructions for zero-shot task transferrability (Wei et al., 2022a). Following this idea, ChatGPT (Ouyang et al., 2022; OpenAI, 2023) was trained with human feedback and reinforcement learning. Our work benefits from these instruction-tuned models to generate attributes and reconstruct sentences. Data Augmentation is widely employed in lowresource scenarios to mitigate model overfitting. It is usually conducted in a label-preserving manner where only minor perturbations are added (Wei and Zou, 2019; Fadaee et al., 2017). Recently, a line of research propose to use LLMs for data augmentation. Specifically, they use few-shot data as demonstrations and prompt LLMs to generate new data (Yoo et al., 2021; Sahu et al., 2022a). They claim that the LLM is able to mix few-shot data and synthesize similar ones. Lin et al., 2023 further propose to use Pointwise V-information to filter unhelpful data from generations. Most recently Dai et al., 2023; Whitehouse et al., 2023 propose to generate data using ChatGPT and GPT-4 and observe performance improvement. Finally Cheng et al., 2023 use GPT-3 generated data to improve sentence embedding via contrastive learning. Our work aims at improving LLM-based data augmentation via attribute manipulation.

heading: 7 Conclusion
text: The study introduces a novel method, Chain-ofThought Attribute Manipulation (CoTAM), which uses manipulated data from Large Language Models (LLMs) for few-shot learning. Our CoTAM creates label-switched data by modifying task-specific attributes and reconstructing new sentences. Our testing validated the effectiveness of CoTAM over other LLM-based text generation techniques. The results also showcase the potential for LLM-guided learning with less supervision. Future work will aim to adapt the attribute manipulation technique for smaller language models, increasing its accessibility. This would reduce reliance on the resource-intensive processes inherent to large language models, improving efficiency. Limitation Despite the significant advancements in few-shot learning and attribute manipulation reported in this paper, our proposed CoTAM does come with certain limitations. Firstly, our approach leverages a chain-of-thoughts decomposition and reconstruction procedure which, while yielding improved data efficiency and model performance, tends to result in a decrease in the overall generation efficiency compared to traditional methods. This may affect the method’s scalability, particularly in scenarios requiring rapid data generation. Secondly, the current implementation of CoTAM is primarily con-
fined to attribute-related tasks, limiting its scope of application. While this constraint is a direct result of our method’s design focused on manipulating task-specific attributes, we acknowledge that extending CoTAM’s applicability to a broader set of tasks could significantly increase its utility. Our future work will thus aim to address this limitation. Lastly, it should be noted that the effectiveness of CoTAM is fundamentally dependent on the abilities of the underlying Large Language Models. As a consequence, the limitations inherent in these LLMs, such as biases in their training data or limitations in their understanding of nuanced contexts, could potentially impact the performance of CoTAM. It is thus crucial to continually improve and refine the LLMs used in our method to ensure the accuracy and robustness of the generated data.
Ethical Consideration
Our work instructs large language models to generate efficient training data, which generally does not raise ethical concerns.

heading: Acknowledgments
text: This work is supported by the National Science Foundation under grants CCF-1955457 and CCF2220892. This work is also sponsored in part by NSF CAREER Award 2239440, NSF Proto-OKN Award 2333790, as well as generous gifts from Google, Adobe, and Teradata.

heading: A Dataset Statistics
text: The statistics of the dataset used in the experiments are presented in Table 7. The numbers of test instances in matched and mismatched are both 9.8K.

heading: B Attribute Names
text: The attribute names of the dataset used in the experiments are presented in Table 8.

heading: C Prompts
text: The prompts used in the experiments are presented in Table 11.

heading: D Results on Open-sourced LLMs
text: To improve the reproducibility of our results, we present results on open-sourced LLMs as presented in Table 10. The performance of different augmentation methods on the open-sourced LLM is generally consistent with our main experiments.
E Attribute Statistics
In this section, we further explore the dynamic attribute decomposition mechanism in CoTAM. For 1315 instances from SST-2, there are 4513 decomposed attributes (3.43 per instance) and 2409 different ones. The distribution is in a long-tail pattern with 2124 attributes only appearing once. We show the statistics the most frequent 10 attributes from the decomposition in Table 7. We can observe a semantic diversity among the attributes, which verifies the ability of LLMs to comprehend the features of different inputs. As the most popular attribute subject (X) only appear in about 20%, there is no dominant attribute in the decomposition, which shows the flexibility of LLM-driven feature analysis. We also provide a quantitative comparison with a fixed feature pool in the ablation study.
Clustering Analysis We also present an analysis of the semantic distribution of the attributes. We use the sentence encoder to encode the attribute names and run K-Means (K = 10) clustering, which is presented in Figure 8. We find the attributes that are the closest to the cluster centers and get 10 attributes:
• an implied evaluation or opinion
• a passive tone
• subject (X)
• subjectivity
• action (X)
• it mentions specific elements of a film (X)
• brevity: short phrase
• focus (X)
• evocative
• comparison (X)
which shows the diversity of the discovered attributes and improves our understanding of the attribute distribution in the pipeline.

heading: F Pipeline Comparison
text: We use a simple case study to compare different text augmentation methods. We can see CoTDA is not very different from conventional data augmentation. While FlipDA++ successfully flips the selected attribute, it fails to preserve the attribute "contrast: between actors’ performance and overall movie experience". The data generated from our proposed CoTAM best satisfies our demand to switch selected attributes and keep others constant.

heading: Future Work
text: Future work will aim to adapt the attribute manipulation technique for smaller language models, increasing its accessibility. This would reduce reliance on the resource-intensive processes inherent to large language models, improving efficiency. Limitation Despite the significant advancements in few-shot learning and attribute manipulation reported in this paper, our proposed CoTAM does come with certain limitations. Firstly, our approach leverages a chain-of-thoughts decomposition and reconstruction procedure which, while yielding improved data efficiency and model performance, tends to result in a decrease in the overall generation efficiency compared to traditional methods. This may affect the method’s scalability, particularly in scenarios requiring rapid data generation. Secondly, the current implementation of CoTAM is primarily con-
fined to attribute-related tasks, limiting its scope of application. While this constraint is a direct result of our method’s design focused on manipulating task-specific attributes, we acknowledge that extending CoTAM’s applicability to a broader set of tasks could significantly increase its utility. Our future work will thus aim to address this limitation. Lastly, it should be noted that the effectiveness of CoTAM is fundamentally dependent on the abilities of the underlying Large Language Models. As a consequence, the limitations inherent in these LLMs, such as biases in their training data or limitations in their understanding of nuanced contexts, could potentially impact the performance of CoTAM. It is thus crucial to continually improve and refine the LLMs used in our method to ensure the accuracy and robustness of the generated data. Ethical Consideration
Our work instructs large language models to generate efficient training data, which generally does not raise ethical concerns