Text Augmentation Using Dataset Reconstruction for Low-Resource Classification
Year: 2023

Abstract:
In the deployment of real-world text classification models, label scarcity is a common problem. As the number of classes increases, this problem becomes even more complex. One way to address this problem is by applying text augmentation methods. One of the more prominent methods involves using the text-generation capabilities of language models. We propose Text AUgmentation by Dataset Reconstruction (TAU-DR), a novel method of data augmentation for text classification. We conduct experiments on several multi-class datasets, showing that our approach improves the current state-of-the-art techniques for data augmentation.

heading: 1 Introduction
text: The deployment of deep learning models in the real-world requires an abundance of labels. However, labeled data is often difficult and expensive to obtain, especially when the models are deployed in highly specialized domains. Therefore, in this paper, we focus on data augmentation for text classification in low-resource environments.
Text classification (Sebastiani, 2002) is fundamental to machine learning and natural language processing. It includes various tasks, such as intent classification (Kumar et al., 2019; Rabinovich et al., 2022), which is a vital component of many automated chatbot platforms (Collinaszy et al., 2017); sentiment analysis (Tang et al., 2015); topic classification (Tong and Koller, 2001; Shnarch et al., 2022); and relation classification (Giridhara et al., 2019). The design and development of such AI applications may begin with a dataset containing only a limited amount of data.
To improve the performance of downstream models in such low-resource settings, a data augmentation mechanism is often implemented (Wong
∗The work was completed during an internship at IBM Research.
et al., 2016). To achieve this, new data are synthesized from existing training data. It has been demonstrated that the use of such mechanisms can significantly improve the performance of various neural network models. For computer vision and speech recognition, a number of well-established methods are available for synthesizing labeled data and enhancing classification accuracy. Some of the basic methods, which are also class preserving, include transformations such as cropping, padding, flipping, and shifting along time and space dimensions (Cui et al., 2015; Krizhevsky et al., 2017).
However, the application of simple transformation for textual data augmentation is more challenging, since simple transformations often invalidate and distort the text, thereby producing grammatically and semantically incorrect texts that are different from the actual text distribution. Consequently, rule-based data augmentation methods for texts typically involve replacing one word with a synonym, deleting a word, or changing a word (Wei and Zou, 2019; Dai and Adel, 2020).
Recent advances in text generation models (Radford et al., 2018) facilitate an innovative approach for handling scarce data situations. In an effort to reduce the cost of obtaining labeled in-domain data, Wang et al. (2021) use the self-training framework to generate pseudo-labeled training data from unlabeled in-domain data. Xu et al. (2021) have recently demonstrated the difficulty in extracting such domain-specific unlabeled data from general corpora.
A number of existing works (Ding et al., 2020; Anaby-Tavor et al., 2020; Yang et al., 2020) have overcome this difficulty by using the generation capabilities of pre-trained language models.
In this paper, we follow the latter paradigm and propose Text Augmentation by Dataset Reconstruction (TAU-DR), a novel text augmentation algorithm that generates new sentences based on the reconstruction of the original sentences from the
7389
hidden representations of a pre-trained classifier. TAU-DR utilizes frozen auto-regressive language models by soft-prompt tuning, using a relatively small number of trainable parameters compared to the language model and unlike most existing methods that rely on language models, it does not require an additional pertaining phase. During training, we extract the hidden representation from the pre-trained classifier and use a Multi-Layer Perceptron (MLP) to turn the hidden representation into a soft-prompt. The soft-prompt is then fed into the frozen language mode.
Our approach is motivated by the observation that if the pre-trained classifier is trained from a language model (i.e. BERT), then the hidden representation is a contextual embedding of the original sentence. Thus, the soft-prompt will also summarize contextual information from a small neighborhood of the hidden representation, giving the frozen language model additional information for enriching the original dataset.
By using this training approach and manipulating the trained prompts, we are able to generate novel sentences with their corresponding pseudolabels. Then, as in previous works (Anaby-Tavor et al., 2020; Wang et al., 2022), we apply a filtering mechanism and filter out low-quality sentences.
We conduct experiments on four multi-class datasets: TREC, ATIS, Banking77, and T-Bot (in various low-resource settings) and show that our approach consistently outperforms the current stateof-the-art approaches. We also conduct several experiments measuring the quality of the generated sentences1.
Our contributions are two-fold, and can be summarized below:
• We propose a novel approach for data augmentation using dataset reconstruction. We demonstrate that our method achieves stateof-the-art performance on several text classification datasets.
• We suggest two novel filtering approaches for better exploitation of the generated sentences - one approach for cases where the evaluation set is available, and another approach for cases where such datasets are absent.
The remainder of the paper is organized as follows: Section 2 introduces the problem framework
1Our implementation will be released after the anonymity period.
and relevant studies. In Section 3, we present TAU-DR and our approach. In Section 4, we conduct the experiments. Section 5 concludes the paper and includes a discussion of future work.

heading: 2 Problem Setup and Related Work
text: In this section, we introduce the data augmentation setting in a low-resource text classification. Let Xtrain = {(xi, yi)}Ni=1 be a text classification dataset with L classes, where we denote xi to be the example and yi to be its corresponding label. We assume that for each class, we havem examples where m is a relatively low number. As in previous works (e.g., Anaby-Tavor et al. 2020; Wang et al. 2022), we assume the existence of a validation set Xval and a test set Xtest.2
Our goal is to create an augmented dataset Xgen by using Xtrain so that by training a classifier on the union of the generated and the original dataset Xtrain ∪ Xgen, we improve the performance of the same classifier trained on Xtrain. The performance of each classifier is measured on Xtest.
The task of text augmentation is relatively challenging, since even small modifications can change the meaning and label of the text. By carefully setting up a rule-based approach, one can deal with this challenge. This was tried by Wei and Zou (2019), who proposed Easy Data Augmentation (EDA), which utilizes simple predefined rules to edit, remove, and substitute portions of the text while maintaining its meaning. Dai and Adel (2020) suggested a rule- based augmentation method named SDANER, tailored for named entity recognition.
A different line of research, which is the prominent approach, uses pre-trained language models. Wu et al. (2019) proposed Conditional BERT (CBERT) for contextual data augmentation. Given a sentence and its label, words in the sentence are masked randomly. The label is then used as a context to predict substitute words while keeping the original sentence in the same class.
Anaby-Tavor et al. (2020) introduced Language Model Based Data Augmentation (LAMBADA), which is also a conditional generation-based data augmentation. LAMBADA fine-tunes an entire language model, GPT2, by concatenating all of the sentences together with their corresponding
2Because this assumption does not hold in some real-world scenarios, in Section 4.5 we abandon that assumption and discuss the no-validation case.
labels, thereby creating additional textual data on which the language model can be fine-tuned. Due to the noisiness of the generation process, a filtering process is used to ensure that only high-quality sentences remain. The filtering process consists of a classifier that was trained on the original dataset by taking those sentences with the top-K softmax scores.
Wang et al. (2022) recently suggested PromDA. This approach first trains an entire pre-trained language model on the task of converting keywords to sentences from a general corpus. Then, using RAKE (Rose et al., 2010), keywords are extracted from the original dataset. By concatenating these keywords to a learned prefix, the language model from the previous step is used to reconstruct the original sentence. Then, the same filtering process as in LAMBADA is used, with the exception that all sentences for which the original classifier agrees with the pseudo-label are taken.

heading: 2.1 Soft-Prompts
text: TAU-DR, as will be discussed in the next section, exploits the language-generation capabilities of language models by using soft-prompts, one of the dominant approaches for parameter-efficient tuning. Prompt-based learning was introduced by Brown et al. (2020). Their study demonstrated that a large language model can be adapted for downstream tasks by carefully constructing prompts (i.e., textual instructions). A method proposed by Gao et al. (2020) for simplifying the construction process involves expanding prompts by using pretrained language models. Each downstream task requires manual construction of discrete prompts. The construction of discrete prompts is still an independent process that is difficult to optimize together with downstream tasks.
A study by Lester et al. (2021); Li and Liang (2021) suggests using soft-prompts. Soft-prompts do not represent actual words, as opposed to hard prompts, and can be incorporated into frozen pretrained language models. As demonstrated by Li and Liang (2021), pre-trained language models (PLMs) with soft-prompts provide better performance in low-resource settings, and enable end-toend optimization of downstream tasks.
,

heading: 3 Text Augmentation by Dataset Reconstruction (TAU-DR)
text: In this section, we introduce Text AUgmentation by Dataset Reconstruction (TAU-DR), our novel text augmentation algorithm.
Algorithm 1 Text Augmentation by Dataset Reconstruction (TAU-DR) Require: Training dataset Xtrain, pre-trained clas-
sifier Cbase, pre-trained language model LM %% training phase
1: while training steps not done do 2: for (x, y) in Xtrain do 3: Extract h from Cbase 4: P ← MLP (h) % transform the hid-
den representation into soft-prompt. 5: x̂ ← LM(P ) % predict a sentence
using the soft-prompt 6: θMLP ← θMLP −∇θMLPLlm(x, x̂) 7: end for 8: end while %% generation phase 9: Xintra ← GENintra(LM,MLP,Xtrain)
10: Xinter ← GENinter(LM,MLP,Xtrain) 11: Xgen ← Xintra ∪ Xinter %% filtration phase 12: Xgen ← Filtration(Xtrain,Xval,Xgen)
TAU-DR consists of three stages: training, generation, and filtration, as described below.

heading: 3.1 Training
text: We now describe the training phase in TAU-DR as shown in Algorithm 1. Given an example x from the original dataset, we extract its hidden representation, h, from the pre-trained classifier, which we denote by Cbase (line 3). For instance, if Cbase is a BERT classifier, it can be the [CLS] token representation in the last layer. The next step in line 4 is to apply a multi-layer perceptron (MLP ) with parameters θMLP , and turn the hidden representation, h, into a prompt of length n denoted as P . P is then fed into the frozen language model LM (line 5). The training objective of the language model is to reconstruct the original sentence using only the hidden representation. The training step is illustrated in Figure 1.

heading: 3.2 Generation
text: To generate new sentences that will challenge the classifier and ultimately improve its accuracy, we
perturb the learned soft prompts. We suggest two novel strategies to provide new soft prompts for the frozen language model3.
Intra-class generation The motivation behind the following approach is that by combining soft prompts from the same class, we will be able to lexically and semantically enrich the class itself. The method can be described as follows: We select two sentences, x1, x2 from the same class, and then extract their corresponding hidden representation, h1, h2, using the pre-trained classifier Cbase. Using the trained MLP , we transform them into their corresponding soft prompts, P1 and P2. Then, by averaging the two prompts, we achieve a new aggregated soft prompt Pagg. The latter is passed into the language model. The pseudo-label for the generated sentences is set as class x1. This is illustrated in Figure 2.
Inter-class generation With inter-class generation, we help the classifier to better distinguish between the different classes. This is done by generating sentences using soft-prompts, which are created by combining soft-prompts from two different classes. First, we randomly sample two sentences from two different classes, x1, x2, and then, as detailed above, extract their soft-prompts denoted as P1 and P2, respectively. We then aggregate the two prompts by taking their weighted mean, Pagg = wP1 + (1 − w)P2, where 0 < w < 1 is sampled uniformly. In this case, we set the pseudolabel as the label of the closest prompt as illustrated in Figure 3.

heading: 3.3 Dynamic Consistency Filtering
text: By generating new sentences for our classifier, we risk the creation of low-quality data. This can hap-
3The closest work to this approach is the work of Asai et al. (2022), suggesting the aggregation of prompts for multitask generalization.
pen if we set an incorrect pseudo-label or if the language model generates out-of-domain examples. Therefore, it is common to apply a consistency filtering mechanism (Anaby-Tavor et al., 2020; Wang et al., 2022).
The consistency filtering suggested by AnabyTavor et al. (2020) used the pre-trained classifier and considered the top-K sentences (ordered by their softmax scores). Wang et al. (2022) also used the trained classifier. However, instead of using the top-K approach, they kept all the generated sentences for which the classifier agrees with the pseudo-label.
Clearly, the chosen filtration method has a large effect on the final classifier, as it controls the data quality of the final trained classifier. The top-K approach might be too conservative, keeping a large safety margin, which results in filtering out most of the generated instances. On the other hand, keeping all the instances on which the classifier agrees with the pseudo-label might include many noisy-label sentences, resulting in a degraded classifier.
We now present Dynamic Consistency Filtering - our filtering approach for a case where an evaluation set exists. In Section 4.5, we discuss the no- evaluation case. Our method relies upon the evaluation dataset to approximate the optimal portion of the generated instances to include in the augmented dataset. We do so by training k classifiers, one of which trained on a different quantile of the generated instances, ordered by their softmax scores (received from the pre-trained classifier Cbase). After training the k instances, we choose the best preforming classifier using the evaluation dataset.
It is important to note that there is a possibility of applying the filtering mechanism in a recursive manner, for example, training a classifier on the filtered data and running that classifier on the original
generated dataset, with the hope of improving the filtering of the instances. This way, one can further improve the performance of the final classifier, as discussed by Anaby-Tavor et al. (2020).

heading: 3.4 Training and Generating in Low-Resource Setting
text: In a preliminary study we conducted, we investigated how over-training affects the quality of the generated text in terms of diversity and distance from the original train distribution. To show the effect of over-training, we apply LAMBADA (Anaby-Tavor et al., 2020) on several internal lowresource multi-class datasets. We trained LAMBADA for 2500 steps and generated augmented sentences every 100 steps. We then evaluated the quality of the generated sentences as a function of the trained steps by using distributional measures: Precision and Recall (Sajjadi et al., 2018) summarized as F1, DC (Naeem et al., 2020) and
MAUVE (Pillutla et al., 2021) 4. We can observe, on Figure 4, that DC, Precision and Recall and MAUVE converges to 1. This suggest that without any control measures in place, the distribution of the generated text quickly converges into the training distribution. This is not a desired property since our goal is to generate texts which will expand the support of the training distribution. It is interesting to note that the nature of the results remains the same, even when soft-prompt tuning is applied.
Therefore, to address the above, we deploy two heuristics. The first heuristic is to increase the number of training samples. We do so by using the EDA rule-based simple augmentation method discussed earlier (Wei and Zou, 2019). Please note that in this enrichment we do not consider the pseudo-labels, since our goal is to provide more reference points for the MLP training. Moreover, we checkpoint the MLP several times during training, and generate sentences from the different checkpoints.

heading: 4 Experiments

heading: 4.1 Setup
text: We conduct experiments on four multi-class classification datasets (described in the next subsection). Each benchmark dataset is split into 80% train ,10% evaluation and 10% test. We then take the train dataset and sample K examples for each class where classes without K examples are removed, resulting in a shot-K dataset. In our experiments, we chooseK ∈ (5, 10). As a base classifier, we choose the BERT-base model5, as in the study of Anaby-Tavor et al. (2020); Wang et al. (2021).
4The measures are introduced on Section 4.4. 5https://huggingface.co/bert-base-uncased
The same set of hyperparameters is used for the training of Cbase, for training without the original data, and for training with the generated data. The performance of Cbase is evaluated during training using Xval.
We compare TAU-DR to the methods discussed in Section 4.1: The rule-based data augmentation methods EDA (Wei and Zou, 2019); CBERT (Wu et al., 2019), LAMBADA (Anaby-Tavor et al., 2020), and PromDA (Wang et al., 2022) which is implemented with a T5-large model (700M parameters). All hyperparameters used for these methods are those recommended by the authors. We repeat the experiments five times and report the averaged accuracy for each shot-k dataset.
For TAU-DR we used the T5-large model for all of our experiments. This model was fine tuned an additional 100k steps on the C4 dataset using the regular LM loss, to achieve better adaptivity to soft prompt tuning (Lester et al., 2021)6. We choose MLP with 2 hidden layers and a ReLU activation. The prompt-length is set as 10 in all of our experiments. TAU-DR was trained for 100 epochs. We checkpointed the model every 20 epochs, resulting in 5 checkpoints. The pre-trained classifier Cbase used in our method is the same classifier discussed above. For the dynamic filtering, we use 10 classifiers with the same configuration as the pre-trained classifier, where each classifier is trained on a different portion of the generated dataset ordered by the softmax score of Cbase. The experimental results are shown in Table 1 for shot-5 and shot-10 for the different multi-class benchmarks.

heading: 4.2 Datasets
text: All datasets used are classification datasets, with different numbers of classes and across several do-
6https://huggingface.co/google/t5-large-lm-adapt
mains, three of which are available in the public domain.
Airline Travel Information Systems (ATIS, Hemphill et al. 1990): The ATIS dataset provides a large set of queries about flight information along with the intent, the subject of the various questions.
Text Retrieval Conference (TREC, Hovy et al. 2001): TREC is a question classification dataset that consists of a variety of questions from different areas and their intent.
Banking77 (Casanueva et al., 2020): The Banking77 dataset offers questions from singledomain banking, annotated with their labels.
Teleco-Bot (T-Bot): An internal intent classification dataset, includes data used for the training of chatbots used by telco companies for customer support.
The datasets used are summarized in Table 2

heading: 4.3 Main results
text: First, we can observe that the addition of the generated data from TAU-DR to the classification models significantly improves the performance of Cbase and outperforms the existing method. Overall, the EDA rule-based approach does not lead to a significant improvement over Cbase on the more challenging datasets Banking77 and T-Bot on both shot-5 and shot-10. On the other hand, the
language-model-based approaches, i.e., C-BERT, LAMBADA, PromDA and TAU-DR outperform the rule-based approach. PromDA can provide better results than LAMBADA on the ATIS and TREC datasets. However, with the exception of Banking77 (shot-5) it fails when considering domainspecific datasets with a larger number of classes, such as Banking77 and T-Bot.
On T-Bot and Banking77 in the shot-10 setting none of the methods expect TAU-DR where able to give a statistically significant improvement over Cbase.
The accuracy improvements of TAU-DR over Cbase on the ATIS dataset are approximately 20% for both shot-5 and shot-10. For TREC the improvement rate is 29% for the shot-5 setting and 9% for the shot-10 setting. For Banking77, the average improvement rate is 4.5% and for the challenging T-Bot dataset the average improvement rate is 5%.

heading: 4.4 Estimating the Generation Quality
text: We now turn to estimate the quality of text generated by the different methods. We use the following measures:
• Recall and Precision (Sajjadi et al., 2018): Given two distributions P,Q, this measure compares their "precision", or how much of Q can be generated by a "part" of P , while "recall" measures how much of P can be generated by a "part" of Q. Recall and Precision are summarized as F1.
• Complexity (Kour et al., 2021): Quantifies how difficult observations are, given their true class label and how they will challenge the classifier. The measure can be used to automatically determine a baseline performance threshold..
• MAUVE (Pillutla et al., 2021): This metric measures the gap between two text distributions by calculating the area under the information divergence curve.
A recent study (Kour et al., 2022) compared several statistical and distributional measures. The different measures were compared over several desired criteria. In their experiments, MAUVE turned out to be the most robust performance measure for text generation quality.
In this set of experiments, we took the generated text and compared it to the test set, which represents the actual text distribution. A desired property of the augmented texts is that their distribution will expand the intersection between the support of the train distribution with the test distribution. Thus, we can compare the generation quality of the different methods by looking on how close they are to the test distribution.
We report the average results on Table 3. The implementation details for this experiment are detailed on Appendix B. The measures of the text generated by TAU-DR are superior to 2 out of 3 in all configurations. Showing that we can generate text that is close to the actual distribution of the data. In addition, by looking at the Atis dataset, we observe that we were able to produce more challenging and complex sentences for the classifier.
It has not been explored if or how these measures relate to the classifier’s performance. Nevertheless, these measures can provide some insight into how well a model can reproduce the test distribution.

heading: 4.5 Dynamic Consistency Filtering with no Evaluation
text: Our suggested dynamic filtering method is shown to be effective in filtering out low-quality generated data. However, the existence of such datasets is not obvious in real-world scenarios.
In this subsection, we suggest an approach for filtering the generated data without relying on the existence of an evaluation dataset. The method can be described as follows: As in the Dynamic Consistency Filtering approach, for each class we order the generated examples according to their softmax scores obtained from the pre-trained classifier Cbase. We then filter out all instances on which the classifier disagrees with the pseudo-label. Then we train k classifiers on a different quantile of the ordered data (i.e., for k = 5, we train the i-th classifier i = 1, ..., 5, on the i/5 quantile). We then use the obtained classifiers to filter out the generated instances based on the majority vote of the classifiers, we denote this approach as TAU-DRmaj . As shown in Table 4, with the exception ATIS (shot-5) and Banking77 (shot-5), TAU-DRmaj also outperforms the benchmark methods and on average only slightly degrades the performance of TAU-DR.

heading: 5 Conclusion and Future Work
text: In this paper, we present TAU-DR, a novel textaugmentation method for low-resource classification using dataset reconstruction. We test our method on four multi-class classification datasets in various few-shot scenarios and show that our approach outperforms the state-of-the-art approaches. In the future, we plan to explore the learned prompt space and check how it can be used for generating helpful sentences. In our preliminary experiment, we found that the averages of the prompts were concentrated in a narrow cone. This concentration hinders the exploitation of the geometry in the learned prompt space. The above observation is aligned with other findings regarding the anisotropy of the word embedding space in pre-trained language models (Li et al., 2020; Ethayarajh, 2019). Finally, we wish to explore if and how additional information (e.g, in-domain textual-data) might improve the performance of text augmentation methods on highly specialized domains.
Limitations
To address the low-resource data in the training of TAU-DR, we apply two heuristics, dataset enrichment and generation from different checkpoints. Despite being effective, they require additional
computational time that might be challenging in applications with low-computational resources. A possible approach to reduce the computational time might be to average the checkpoints. We believe that this might lead to competitive results, with a significant reduction in computational time, since checkpoint averaging proved to be an effective approach in low-resource settings. Another limitation is when the original dataset is in a highlyspecialized domain that might contain domainspecific phrases that were most likely not included in the pre-training data of the language model. The results obtained by existing data augmentation approaches will most likely exhibit only marginal improvement.
Ethics Statement
Text generation by nature entails a number of ethical considerations when considering possible applications. The main failure is when the model generates text with undesirable properties (bias etc.) for training the classifier but these properties are not present in the original training data.
Because our model converges and learns to generate data close to the underlying source material, the above considerations, in our approach, are negligible. As a result, the generated text may be harm-
ful if users of such models are unaware that such issues appear on their training data or if they fail to consider them, e.g., by selecting and evaluating data more carefully.

heading: A Ablation
text: In our ablation studies, we evaluated the independent effect of 5 different components on our method: enrichment, intra-class, inter-class, checkpointing and dynamic filtering. In this ablation study we want to emphasize the contribution of each module for the success of our method. Results are summarized in Table 5. During the MLP training we used dataset enrichment in order to add more reference points. As we can observe from the results this enrichment is an important aspect of our method as our method without dataset enrichment results in an average degradation of 4.5 accuracy points. In addition, we evaluated the effect of each generation method we proposed – intra-class and inter-class. The intra-class generation is meant to enrich the number of examples in a given class, whereas inter-class is meant to highlight the difference between different classes. We can see that both generation methods are vital components of our method, with degradation of 2 and 3.25 accuracy points when not using intra-class or inter-class, respectively.
Moreover, we determined the efficacy of the checkpointing paradigm. We utilized checkpointing to overcome the over training affects as discussed on Section 3. Based on the results, we can see that the checkpointing paradigm plays an important role in the method’s success. Generating
sentences using only the last checkpoint results in degradation of 3.75 accuracy points. The last ablation conducted is to evaluate the performance of the dynamic filtering method. As discussed earlier in all the generation methods this plays a vital component in keeping high-quality instances. On the ablation experiment we kept all the sentences on which Cbase agrees with the pseudo-label. Not surprisingly this approach caused a major decrease in the accuracy with an average of 6.5 accuracy points.

heading: B Additional implementation details
text: The optimizer used for training the MLP is AdamW, we tested the following learning rate {1e− 3, 1e− 2, 1e− 4} and a 1e− 2 weight decay. We experimented with the following batch sizes {16, 32, 64}. he size of the hidden layer is set as dim(h) ∗ n/2, where n is the prefix-length. The MLP architecture was not optimized during our experiments. We experiment also with prefix-lengths of 5, 10, 15, 20. These different prefix lengths have a negligible impact, since we used a medium-sized model. This aligns with the findings of Lester et al. (2021) . We used an internal multi-class dataset which was not reported in the main paper to search for the best training configuration. The classifier was trained for 5000 steps with 8 batch size with AdamW optimizer and 1e− 5 learning rate We run all experiments on a single NVIDIA A100 GPU. For the generation phase, we used the nucleus sampling (Holtzman et al., 2019) with k = 100, p = 0.95 both for the intra- and inter-generation approaches.
To calculate Precision and Recall, MAUVE and Complexity we sampled 1000 instances and compared against 1000 sentences in the generated set. We repeated this process 10 times for every one of the 5 splits for each dataset.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section number 5

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section number 5

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 4

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B and Section 4

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 and Appendix B

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we plan to explore the learned prompt space and check how it can be used for generating helpful sentences. In our preliminary experiment, we found that the averages of the prompts were concentrated in a narrow cone. This concentration hinders the exploitation of the geometry in the learned prompt space. The above observation is aligned with other findings regarding the anisotropy of the word embedding space in pre-trained language models (Li et al. , 2020; Ethayarajh, 2019). Finally, we wish to explore if and how additional information (e. g, in-domain textual-data) might improve the performance of text augmentation methods on highly specialized domains. Limitations
To address the low-resource data in the training of TAU-DR, we apply two heuristics, dataset enrichment and generation from different checkpoints. Despite being effective, they require additional
computational time that might be challenging in applications with low-computational resources. A possible approach to reduce the computational time might be to average the checkpoints. We believe that this might lead to competitive results, with a significant reduction in computational time, since checkpoint averaging proved to be an effective approach in low-resource settings. Another limitation is when the original dataset is in a highlyspecialized domain that might contain domainspecific phrases that were most likely not included in the pre-training data of the language model. The results obtained by existing data augmentation approaches will most likely exhibit only marginal improvement