Venue: NAACL 2022

# Learning to Generate Examples for Semantic Processing Tasks

Danilo Croce

Dept. of Enterprise Engineering

University of Rome, Tor Vergata

Roma, Italy

croce@info.uniroma2.it &Simone Filice

Amazon

Tel Aviv, Israel

filicesf@amazon.com Giuseppe Castellucci

Amazon

Seattle, USA

giusecas@amazon.com Roberto Basili

Dept. of Enterprise Engineering

University of Rome, Tor Vergata

Roma, Italy

basili@info.uniroma2.it

###### Abstract

Even if recent Transformer-based architectures, such as BERT, achieved impressive results in semantic processing tasks, their fine-tuning stage still requires large scale training resources. Usually, Data Augmentation (DA) techniques can help to deal with low resource settings. In Text Classification tasks, the objective of DA is the generation of well-formed sentences that (_i_) represent the desired task category and (_ii_) are novel with respect to existing sentences. In this paper, we propose a neural approach to automatically learn to generate new examples using a pre-trained sequence-to-sequence model. We first learn a task-oriented similarity function that we use to pair similar examples. Then, we use these example pairs to train a model to generate examples. Experiments in low resource settings show that augmenting the training material with the proposed strategy systematically improves the results on text classification and natural language inference tasks by up to 10% accuracy, outperforming existing DA approaches.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Deep Learning models achieve state-of-the-art results in many domains, including Computer Vision and Natural Language Processing (NLP). Training these large models typically requires many examples, whose collection and annotation can be costly and time-consuming. Data augmentation (DA) has proven an efficient way to acquire more training samples without incurring in the prohibitive annotation cost in a variety of fields, including computer vision Perez and Wang (2017) and speech recognition Rebai et al. (2017). In NLP, some DA techniques have been proposed too, as surveyed in Feng et al. (2021): common approaches create synthetic data by manipulating real examples, using Text Editing Wei and Zou (2019) or BackTranslation Sennrich et al. (2015); the resulting examples are automatically labeled by inheriting the class of the original example they were generated from.

Unfortunately, when using recent pre-trained language models, such as BERT Devlin et al. (2018) or RoBERTa Liu et al. (2019), the effectiveness of DA methods is extremely limited, and sometimes they can even hurt the results Longpre et al. (2020). A possible explanation for this inconficacy is that these DA techniques introduce lexical and structural variability that accurate language models directly induce during pre-training. The usefulness of synthetic examples is strictly related to their diversity from the original training data. At the same time, diverging too much from the initial data might increase the risk of introducing noisy annotations, i.e., synthetic data not reflecting the class of the original examples they were generated from.

To directly tackle the trade-off between diversity and label consistency, in this paper we propose DATS1Data Augmentation based on Task-specific Similarity, a novel data-augmentation technique for text classification based on Natural Language Generation (NLG) models. Starting from a reduced set of annotated examples, we first learn a task-oriented similarity function that we use to automatically create pairs of similar examples. Then, we use these pairs to train a generative model to generate an example similar to the input one. Finally, we employ this model to generate new synthetic examples and augment the training data. We show that pairing examples with respect to their task

oriented similarity is striking in order to allow the generative model to automatically understand the lexical and structural variations that can be applied to an instance without changing its label. Experimental results on four different text classification datasets demonstrate that DATS achieves better results than several existing DA solutions and that it systematically improves NLU models based on state-of-the-art pre-trained language models. In the remaining, Section 2 summarizes related works, Section 3 describes our method and Section 4 provides the experimental evaluation.

## 2 Related Work

Most of the existing approaches for DA perform some token-level manipulations on individual sentences Kolomiyets et al. (2011); Kobayashi (2018). For instance, Easy Data Augmentation (EDA) Wei and Zou (2019) applies simple operations including synonym replacement and random swap. These operations are also performed in Ren et al. (2021) where the framework named Text AutoAugment (TAA) uses Bayesian Optimization algorithm to automatically search for the best manipulation policy. On the contrary, Wu et al. (2018) uses mask random tokens and use BERT to generate substituting words. Similarly, Kumar et al. (2020) use transformer-based models to apply token-level or span-level text content manipulation.

Other attempts operate at the entire sentence-level by paraphrasing the original text using back-translation (BT) Edunov et al. (2018); Shleifer (2019); however, BT tends to skew towards high-frequency words which not only causes redundancy but also leads to lexical shrinkage in the augmented data Liu et al. (2020). More recent approaches for data augmentation use generative models to create more diverse synthetic data. Anaby-Tavor et al. (2019) fine-tuned a GPT-2 model to generate text given a target class, and use a text classifier to filter out those synthetic examples whose predicted class does not match the target class. In our work, we show that by conditioning a target model not only on the class labels, but also on representative examples, it is possible to better control the diversity-consistency trade-off.

## 3 Learning to generate examples

Recent advances in NLG Vaswani et al. (2017) demonstrated that sequence-to-sequence (seq2seq) models can generate natural sounding and meaningful text given a prompt. As shown in Keskar et al. (2019), the prompt can include style or content-related information that can help control the generation process.

In our work, we capitalize these techniques to augment a dataset \({\cal D}\) for the training of a text classifier \({\cal C}\). Specifically, our goal is to fine-tune a pre-trained seq2seq model \({\cal M}(*)=e\) that, given a prompt \(*\), generates novel examples not in \({\cal D}\). The first application of this idea was investigated in Anaby-Tavor et al. (2019), where the authors trained a seq2seq model \({\cal M}(c)=s_{o}\) that generates examples \(s_{o}\) of a given class \(c\). While this approach provides interesting results, it must be said that the variability of the generated examples can be quite low: by conditioning \({\cal M}\) only on \(c\), it tends to produce the most frequent syntactic and lexical patterns associated with \(c\), neglecting other modes this can exhibit. The key challenge here is to ensure diversity while preserving consistency, i.e., generating diverse examples that are valid representatives of the desired class.

We propose to train a seq2seq model \({\cal M}\) that is conditioned not only on the class \(c\), but also on an example \(s_{i}\) of that class, i.e., such that \({\cal M}(c,s_{i})=s_{0}\). The model is thus expected to synthesize a new example \(s_{0}\), which is consistent with the input class \(c\) and is also "inspired" by an existing example \(s_{i}\). The problem now is how to build the dataset to train \({\cal M}\). Pairing two random examples \(s_{i}\) and \(s_{o}\) belonging to the same class might be a viable solution. However, in classification tasks, examples are not necessarily similar, and coupling together radically different examples risks destabilizing the training of \({\cal M}\). Ideally, we would like to identify different modes in the same class and pair only examples belonging to the same mode. An alternative way to reach the same outcome is the adoption of a semantic similarity function that can be used to select linguistically related example pairs. Several unsupervised metrics exist Croce et al. (2011); Cer et al. (2018); Poerner et al. (2020), however, they are inherently task-independent and not adequate in capturing task-specific similarities: two sentences such as "_The movie is good_" and "_The movie is not good_" (or even "_The movie is awful_") should not be paired together, when dealing with sentiment classification, while they are good candidate pairs in the training of a topic classifier.

To define a task-oriented similarity measure, we leverage the text classifier \({\cal C}\) we want to improve 

via data augmentation. Independently on the underlying neural architecture, a neural classifier has an encoder \({\cal E}\) that projects an example \(s\in{\cal D}\) in a \(d\)-dimensional space, i.e., \({\cal E}(s)=\vec{s}\in{\mathbb{R}^{d}}\). In these spaces, simple linear classifiers (i.e., the output layers) identify the sub-spaces reflecting the target classes Goodfellow et al. (2016); Goldberg (2016). As a consequence, \({\cal E}\) is expected to project examples in sub-spaces representative for individual classes. Our task-aware similarity measure is the cosine similarity operating on these representations. For instance, after training a BERT model on the question classification task Li and Roth (2006), the encoding [CLS] of the entire question _"Who developed the vaccination against polio?"_ allows retrieving other questions corresponding to the most similar [CLS] embeddings, such as _"Who invented the Moog Synthesizer?"_: they are clearly not paraphrases, and their purely lexical similarity is quite low, but they show a similar pattern characterizing the question class [HUM] (i.e., human).

```
Input:A dataset \({\cal D}\), number of folds \(f\), input pairs per example n , number of generated samples per input l , Output:An augmented dataset \({\cal D}_{synst}\)
1\(D_{synst}=\emptyset\)
2C=train_classifier(\({\cal D}\))
3\({\cal E}=\text{get\_encoder}({\cal C})\)
4\({\cal D}_{1r}=\text{get\_encoder}({\cal C})\)
5\({\cal D}_{ir}=\mathcal{O}_{lr}({\cal D}_{ir},{\cal D}_{N})=\text{split\_ inf\_folds}({\cal D},f)\)
6for\(i=0\)do
7\({\cal D}_{tr}=\bigcup_{Vj\neq j}{\cal D}_{j}\)
8if Split \({\cal D}\equiv A\setminus{\cal S}\) from kMeans fold schema */
9\({\cal D}_{k}={\cal D}_{l}\)
10\(T=\emptyset\)
11\(r\)Initializing the training set \({\cal T}\) for the seq seq2seq model */
12foreach\(e_{i}\in{\cal D}_{rt}\)
13 end if
14
15 end for
16\({\cal M}=\textsc{train\_seg2seq}({\cal P})\)
17foreach\(e_{g}\in{\cal D}_{te}\)do
18\(c=\text{GET\_class}(e_{g})\)
19\({\cal D}_{pr}^{c}=\textsc{get\_encoder}({\cal C})\)
20\(M\leftarrow\textsc{test\_training}(e_{i}\in{\cal D}_{pr}^{c},e_{i})\)
21\(Q_{synst}=D_{synst}\cup{\cal M}^{c}(e_{g})\)
22
23 end while
24return\(D_{synt}\)
```

**Algorithm 1****Pseudo-code of DATS.**

To generate an augmented dataset for a generic text classification task, we propose DATS (Data Augmentation based on Task-specific Similarity) that is described in Algorithm 1. First, we train a classifier \({\cal C}\) on \({\cal D}\). We use the resulting encoder \({\cal E}\) to project each training example \(e\) into the task-specific vector space, obtaining the embedding \(\vec{e}\). Then, we split the training data \({\cal D}\) into \({\cal D}_{tr}\) and \({\cal D}_{te}\). Each example \(e_{i}\in{\cal D}_{tr}\) is paired with the \(n\) examples \(e_{j}\in{\cal D}_{tr}\) of the same class having the highest cosine similarity computed on their corresponding embeddings vectors \({\cal E}(e_{i})\) and \({\cal E}(e_{j})\). The resulting pairs are expected to lie in the same subspace and share some task-oriented linguistic relatedness. We use these example pairs to fine-tune a seq2seq model to solve the task \({\cal M}(c_{e_{i}},e_{i})=e_{j}\), where \(c_{e_{i}}\) is the category of \(e_{i}\). Finally, examples in \({\cal D}_{te}\) are provided in input to \({\cal M}\) to generate the new synthetic dataset \({\cal D}_{synt}\). By applying multiple splits of \({\cal D}\) in a cross-fold scheme, we can use each training example to condition the model and generate new synthetic instances. As in almost all existing formulations, a generator can be used to generate a set of \(l\) variants for each input \((c,e)\), for simplicity referred as \({\cal M}^{l}(c,e)\). In particular, techniques such as _nucleus-sampling_Holtzman et al. (2020) enable the generation of large sets of variants, generally characterized by a good diversity.

It is worth noting that no assumption is applied when selecting \({\cal C}\) or \({\cal M}\). Even though in the experimental evaluation we consider only specific architectures (namely BERT and BART), this methodology can be applied to a wider plethora of models. Moreover, there is no restriction on the classification task type: the experimental section shows that DATS is applicable to classification tasks operating on both individual sentences and text pairs, e.g., in natural language inference Bowman et al. (2015): given text pairs \(s_{i,1}\) and \(s_{i,2}\), it is sufficient to extend the above process defining \({\cal M}\) differently, i.e., \({\cal M}(c_{(s_{i,1},s_{i,2})},s_{i,1},s_{i,2})=(s_{j,1},s_{j,2})\).

## 4 Experimental Evaluation

We test our approach on four tasks: 50-class Question Classification (QC) over the TREC dataset Li and Roth (2006); 5-class Sentiment Classification (SA) over the SST dataset Socher et al. (2013); 7-class Intent Classification (IC) over the SNIPS dataset Coucke et al. (2018); sentence-pair classification for Natural Language Inference (NLI) over the 3-category SNLI dataset Bowman et al. (2015). Details about the datasets are in Appendix A.1.

#### Baselines.

We compare our approach with multiple baselines and simplified versions of DATS for ablation study: (_i_) Easy Data Augmentation (EDA) Wei and Zou (2019); (_ii_) Back-Translation (BT) Sennrich et al. (2015) in an English-German-English setting2; (_iii_) Random Pairing (RP) - DATS 

without task-oriented similarity functions where we created the training input-output pairs for \({\cal M}\) by selecting two random examples of the same class from the training set; (_iv_) Only Class (OC) - DATS where the prompt is only the category name (i.e., no representative example), similar to Anaby-Tavor et al. (2019).

**Experimental Setting.** For QC, SA and IC, we report the Accuracy when using \(q\)=10, \(q\)=50 or \(q\)=100 average examples per class3. For the NLI task, since it is more challenging, we report the Accuracy also for \(q\)=500 and \(q\)=1000 average examples per category. We also report the performance of each model when using the entire (F) training set4. We use the bert-base-uncased model from the Huggingface library Wolf et al. (2019) as the classifier \({\cal C}\), and bart-base Lewis et al. (2020) as the NLG model \({\cal M}\). Both are trained for 10 epochs with early stopping (patience=3) and learning rate \(5e^{-5}\). We adopt nucleous sampling5Holtzman et al. (2020) with \(p\)=0.90. We repeat the experiments 5 times with different seeds and we report the average classification accuracy.

Footnote 3: To maintain the original class distribution we randomly sample \(10c\), \(50c\) or \(100c\) from the original training set, with \(c\) being the number of classes. We ensure that at least one example per class is sampled.

Footnote 4: We omit results on the full dataset for NLI (made of more than \(500,000\) examples) for which DA is not needed.

Footnote 5: In preliminary evaluations, we tested alternative decoding strategies, e.g., standard beam search and k-sampling. Overall, nucleous sampling was superior in terms of diversity and consistency, in line with the literature Holtzman et al. (2020).

**Results.** Table 1 shows the results of a BERT-based model without DA (NoDA), the DA baselines and our approach (DATS). We perform hyper-parameters tuning6 on the development set of each task w.r.t. the number of similar examples \(n\) and the number of generated examples \(l\). Task-agnostic DA approaches, like BT or EDA, seem slightly beneficial when using a transformer based classifier \({\cal C}\), as also reported in Longpre et al. (2020). In some cases, they significantly hurt accuracy.

Footnote 6: An analysis of the hyper-parameters is in Appendix A.4

For instance, in QC or IC when \(q\)=10, EDA Accuracy is lower than NoDA by about 1 and 6 points, respectively. The same is for BT, where the drop is even higher, i.e., 8 and 18 points. Using the category information in the generation process, i.e., RP and OC, provides variable results, with minor improvements only in few specific settings (e.g., RP on SA). Instead, DATS improves accuracy for almost all \(q\) and tasks and such improvements are often statistically significant. When \(q\)=10, the improvement ranges between 1 and 10 points. In the few cases where a minor drop is observed, this is never statistically significant. By comparing DATS with RP it is clear that the task-specific similarity measure is striking to learn a good data generator7. DATS works well also on sentence-pair tasks, like NLI. However, given the complexity of this type of tasks, it requires more training examples (e.g., \(q\)>100) to show a positive impact. Furthermore, DATS outperforms other competitive methods 8. With \(q\)=10 Kumar et al. (2020) report 67.30% accuracy in QC and 87.24% in IC, which is 1.74% and 4.5% lower than DATS. In the same setting, the gap with Wu et al. (2018) is even larger, i.e., 4.71% and 5.95%. Moreover, Ren et al. (2021) report 52.55% accuracy in SA using the entire training set, and DATS outperforms it by 1.71%.

Footnote 7: We tried to substitute the task-oriented similarity with a cosine similarity on USE embeddings Cer et al. (2018) - a general-purpose sentence encoder. Preliminary results on QC were in between DATS and RP, demonstrating that a task-oriented similarity is beneficial.

Footnote 8: Due to lack of budget, we did not run experiments on these models. The reported comparisons are based on the results reported by the authors.

**Qualitative Analysis.** To better understand the advantages provided by DATS in the consistency-diversity trade-off, we reporti a qualitative evalua

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|c|} \hline \(q\) & **NoDA** & **EDA** & **BT** & **RP** & **OC** & **DATS** \\ \hline \multicolumn{7}{|c|}{**Question Classification (QC)**} \\ \cline{2-7}
**10** & 66.39 & 65.32 & 58.76\({}^{*}\) & 66.72 & 62.64 & **69.04** \\ \hline
**50** & 90.64 & 90.48 & 89.84 & 90.28 & 90.48 & **90.68** \\ \hline
**100** & 91.60 & 92.04 & 91.48 & 91.08 & 91.44 & **92.28** \\ \hline
**F** & 91.60 & 91.32 & 91.48 & 90.72 & 91.36 & **92.28** \\ \hline \multicolumn{7}{|c|}{** Sentiment Analysis (SA)**} \\ \hline
**10** & 26.64 & 27.95 & 27.50 & **29.82** & 27.68 & 28.14 \\ \hline
**50** & 40.06 & 40.89 & 39.90 & 40.18 & 40.20 & **43.48** \\ \hline
**100** & 42.58 & 44.38 & 43.79 & 43.16 & 44.26 & **46.30*** \\ \hline
**F** & 53.10 & 53.29 & 53.40 & 52.56 & 52.76 & **54.26** \\ \hline \multicolumn{7}{|c|}{**Intent Classification (IC)**} \\ \hline
**10** & 81.92 & 75.94\({}^{*}\) & 63.71\({}^{*}\) & 81.92 & 70.06\({}^{*}\) & **91.74*** \\ \hline
**50** & 95.18 & 95.14 & 94.69 & 95.42 & 95.28 & **95.62** \\ \hline
**100** & 95.38 & 95.54 & 95.57 & 95.54 & **95.78** & 95.62 \\ \hline
**F** & 97.32 & 97.80 & 97.42 & 97.96 & 98.00\({}^{*}\) & **98.12*** \\ \hline \multicolumn{7}{|c|}{**Natural Language Inference (NLI)**} \\ \hline
**10** & **40.67** & 39.42 & 36.07\({}^{*}\) & 35.78\({}^{*}\) & 36.30\({}^{*}\) & 38.74 \\ \hline
**100** & **57.08** & **58.92** & 56.23 & 50.64\({}^{*}\) & 49.60\({}^{*}\) & 58.36 \\ \hline
**500** & 71.48 & 72.23 & 73.23\({}^{*}\) & 71.04 & 63.78\({}^{*}\) & **73.66*** \\ \hline
**1k** & 75.10 & 76.26 & 76.60\({}^{*}\) & 75.86 & 75.48 & **76.86*** \\ \hline \end{tabular}
\end{table}
Table 1: Accuracy of DA approaches: the best results are in bold while the results higher than NoDA are underlined. The \({}^{*}\) symbol indicates statistically significant differences (p<0.05) with respect to NoDA according to an unpaired T-test.

 

tion of the generated sentences for the SA task.

In particular, Table 2 reports some of the SA examples generated by different models on classes 5 and 1 when using \(q=10\) average training examples per class. Exception for a single case "(_It is a masterpiece, brilliantly directed, and incredibly well done._)" that is not a negative sentiment example as it was supposed to be), the examples generated by DATS (with \(n=1\) and \(l=1\)) are all label-consistent. However, the diversity introduced by DATS is impressive: all the examples are very different from the input example used to condition the NLG model while generating them, and therefore can be very useful when augmenting the training data of NLU models. On the opposite, BT introduces very minor modifications to the input text, resulting in a significant lower diversity A larger and systematic qualitative analysis on the quality and diversity of the generated material is in Appendix A.2 and A.3.

## 5 Conclusions

This paper proposes DATS, a data augmentation method based on Natural Language Generation (NLG) models. A generative model is fine-tuned to produce examples similar to the input ones. The training input-output pairs are selected according to a task-oriented similarity function. This pairing allows the NLG model to learn the lexical and structural variations that can be applied to an instance without changing its label. The experimental results suggest that the generated sentences are diverse and label consistent, and can improve state-of-the-art text classifiers, outperforming existing DA methods. In the future, we plan to apply DATS to further tasks (e.g., Question Answering) and neural architectures.

## Acknowledgments

We would like to thank the "Istituto di Analisi dei Sistemi ed Informatica - Antonio Ruberti" (IASI) for supporting the experimentations through access to dedicated computing resources.

## References

* Anaby-Tavor et al. [2019] Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2019. Not enough data? deep learning to the rescue! _CoRR_, abs/1911.03118.
* Bowman et al. [2015] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
* Cer et al. [2017] Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua,

\begin{table}
\begin{tabular}{p{14.2pt} p{142.3pt} p{142.3pt}} \hline
**cat.** & \multicolumn{2}{c}{**Original Data**} \\ \hline  & _A colorful , vibrant introduction to a universal human impulse , lushly photographed and bautifully recorded ._ & _This is a real film._ & _A colourful, lively introduction to a universal human impulse, lavishly photographed and bautifully captured._ \\ \cline{2-3}  & \multirow{4}{*}{\begin{tabular}{} \end{tabular} } & _A generous , inspiring film that unfolds with grace and humor and gradually becomes a testament to faith._ & _Is in the midst of a monumental advantage, and I can’t wait to see what the next one looks like._ & _A generous, inspiring film that unfolds with grace and humor and gradually becomes a witness of faith._ \\ \cline{2-3}  & \multirow{4}{*}{\begin{tabular}{} \end{tabular} } & _A solid and refined piece or understanding imposed with passion and attitude ._ & It ’s a film that develops with conviction and a deep, passionate heart._ & _A solid and refined piece of film steeped in passion and attitude._ \\ \cline{2-3}  & \multirow{4}{*}{\begin{tabular}{} \end{tabular} } & _Enormously enjoyable , high-adrenaline documentary_ & _It is a remarkable film..._ & _Enormously entertaining, adrenalinea-pumping documentary._ \\ \cline{2-3}  & \multirow{4}{*}{\begin{tabular}{} \end{tabular} } & _- spy action flick with Antonio Banderas and Lucy Liu never comes together_ & _'It just felt... too heavy... for me have it in my trunk._” & _- Espionage action with Antonio Banderas and Lucy Liu never comes together._ \\ \cline{2-3}  & \multirow{4}{*}{\begin{tabular}{} \end{tabular} } & _If you value your time and money , find an escape clause and avoid seeing this trite , predictable rehash ._ & Isni’t the best spy film ever made._ & _If you value your time and money, find an exit clause and avoid this mundane, predictable repetition._ \\ \cline{2-3}  & \multirow{4}{*}{\begin{tabular}{} \end{tabular} } & _No amount of burning , blasting , stabbing , and shooting can hide a weak script ._ & _It is a masterpiece, brilliantly directed, and incredibly well done._ & _No amount of firing, blasting, stabbing and shooting can hide a weak script._ \\ \cline{2-3}  & \multirow{4}{*}{
\begin{tabular}{} \end{tabular} } & _This Tuxedo ... should have been sent back to the tailor for some major alterations ._ & _This is a film that I was unable to stomach... and very much enjoyed... until the last 30 minutes.... when I finally got close to the film... and it turned into a garbage dump._ \\ \cline{2-3}  & & & \\ \cline{2-3}  & & & \\ \cline{2-3}  & & & \\ \cline{2-3}  & & & \\ \cline{2-3}  & & & \\ \hline \end{tabular}
\end{table}
Table 2: A qualitative analysis of the diversity in the Sentiment Analysis task over SST-5.

 

Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder.
* Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Theodore Bluche, Alexandre Caulier, David Leroy, Clement Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Mael Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.
* Croce et al. (2011) Danilo Croce, Alessandro Moschitti, and Roberto Basili. 2011. Structured lexical similarity via convolution kernels on dependency trees. In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_, pages 1034-1046, Edinburgh, Scotland, UK. Association for Computational Linguistics.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. _CoRR_, abs/1810.04805.
* Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
* Feng et al. (2021) Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard H.ovy. 2021. A survey of data augmentation approaches for NLP. _CoRR_, abs/2105.03075.
* Goldberg (2016) Yoav Goldberg. 2016. A primer on neural network models for natural language processing. _J. Artif. Intell. Res._, 57:345-420.
* Goodfellow et al. (2016) Ian J. Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. _Deep Learning_. MIT Press, Cambridge, MA, USA. [http://www.deeplearningbook.org](http://www.deeplearningbook.org).
* Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration.
* Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. _CoRR_, abs/1909.05858.
* Kobayashi (2018) Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. _CoRR_, abs/1805.06201.
* Kolomiyets et al. (2011) Oleksandr Kolomiyets, Steven Bethard, and Marie-Francine Moens. 2011. Model-portability experiments for textual temporal analysis. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 271-276, Portland, Oregon, USA. Association for Computational Linguistics.
* Kumar et al. (2020) Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained transformer models. In _Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems_, pages 18-26, Suzhou, China. Association for Computational Linguistics.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Li and Roth (2006) Xin Li and Dan Roth. 2006. Learning question classifiers: the role of semantic information. _Natural Language Engineering_, 12(3):229-249.
* Liu et al. (2020) Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. 2020. Data boost: Text data augmentation through reinforcement learning guided conditional generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9031-9041, Online. Association for Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. _CoRR_, abs/1907.11692.
* Longpre et al. (2020) Shayne Longpre, Yu Wang, and Christopher DuBois. 2020. How effective is task-agnostic data augmentation for pretrained transformers? _CoRR_, abs/2010.01764.
* Perez and Wang (2017) Luis Perez and Jason Wang. 2017. The effectiveness of data augmentation in image classification using deep learning. _CoRR_, abs/1712.04621.
* Poerner et al. (2020) Nina Poerner, Ulli Waltinger, and Hinrich Schutze. 2020. Sentence meta-embeddings for unsupervised semantic textual similarity. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7027-7034, Online. Association for Computational Linguistics.
* Rebai et al. (2017) Ilyes Rebai, Yessine BenAyed, Walid Mahdi, and Jean-Pierre Lorte. 2017. Improving speech recognition using data augmentation and acoustic model fusion. _Procedia Computer Science_, 112:316-322. Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 21st International Conference, KES-20176-8 September 2017, Marseille, France.
* Ren et al. (2021) Shuhuai Ren, Jinchao Zhang, Lei Li, Xu Sun, and Jie Zhou. 2021. Text AutoAugment: Learning compositional augmentation policy for text classification. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 

9029-9043, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. _CoRR_, abs/1511.06709.
* Shleifer (2019) Sam Shleifer. 2019. Low resource text classification with ulmfit and backtranslation. _CoRR_, abs/1903.09244.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Wei and Zou (2019) Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. _CoRR_, abs/1901.11196.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. _ArXiv_, abs/1910.03771.
* Wu et al. (2018) Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2018. Conditional BERT contextual augmentation. _CoRR_, abs/1812.06705.

 

Appendix

### Task Description

In this section we report details and statistics of the dataset we adopted in the experimental section.

Trec.The TREC (Li and Roth, 2006) dataset contains 4,907, 545 and 500 examples for training, development and test, respectively. We adopted the fine-grained version of the dataset that contains 50 categories.

Sst5.The Sentiment Analysis Treebank dataset (Socher et al., 2013) consists of 8,544, 1,101 and 2,210 examples for training, development and test, respectively. The dataset is characterized by 5 categories for sentiment, i.e., 1 (Very Negative) 2 (Negative), 3 (Neutral), 4 (Positive) and 5 (Very Positive) and it contains movie review sentences.

Snips.The SNIPS dataset (Coucke et al., 2018) consists of natural language commands for a voice assistant. The commands are classified into 7 categories, i.e., RateBook, BookRestaurant, AddToPlaylist, PlayMusic, GetWeather, SearchScreeningEvent, SearchCreativeWork. The dataset consists of 13,084, 700, 700 examples for training, development and test, respectively.

Snli.The SNLI dataset (Bowman et al., 2015) consists of examples of pairs for the Natural Language Inference task. There are 3 categories: _entailment_, _neutral_ and _contradiction_. The dataset consists of 550,152, 10,000 and 10,000 for training, development and test, respectively.

### Qualitative Analysis

This section reports additional examples generated by DATS. In particular, we report the examples generated with \(n=2\) and \(l=10\). Notice that, the generated examples are not in the original datasets. It is worth noting that some examples may result odd, such as a question like "_Name the American president born in 2005?_" but we only expected that they are linguistically sound and consistent with the corresponding class. In general, the number \(n\) strongly affects the example novelty: high values of \(n\) make the generator observe multiple times the same text as the target sentence, making the generation process more conservative. As a result, when using \(n>20\) the generator tends to produce the same examples from the original dataset.

Question classification.Here is a list of 20 questions generated by DATS for different classes. Please refer to the original dataset description for the details about the specific classes.

* ABBR:exp_What does D-DAY stand for?_
* DESC:def_What does the term "Italian Renaissance" mean?_
* DESC:desc_What process takes place after a hydrogen release?_
* DESC:reason_Why do people go to the bathroom at night?_
* ENTY:animal_What animal was the first domesticated creature in the world?_
* ENTY:color_What color is the cross on the French flag?_
* ENTY:event_What happened on February 27, 1991?_
* ENTY:food_What cereal is "sweet, soft and orange"?_
* ENTY:plant_What tree has the longest trunk?_
* ENTY:subst_What are diamond rings made of?_
* ENTY:veh_What was the name of the U.S. Navy gunboat used by Dwight D. Eisenhower?_
* HUM:gr_Name the two major companies in the energy industry._
* HUM:ind_What's the name of the author of "Harry Potter"?_
* LOC:city_What's the name of the largest city in Germany?_
* LOC:country_What country in 1991 recorded the largest number of cocaine seizures?_
* NUM:count_How many hundred ships sank in the Norwegian Sea in 1923?_
* NUM:date_What is the date of the first inauguration of President Nixon?_ 

* NUM:money _What is the cost of university admission to Stanford?_
* NUM:period _How long does it take to clean up a cache?_
* NUM:weight _What is the maximum weight for a healthy adult?_

Although the above examples seem coherent, we can argue if they can be really new, i.e., useful for the task according to a Data Augmentation process. We thus focused on the specific class ENTY:lang that is underrepresented in the original dataset, with only the six original examples reported below:

* _What is the name of a language spoken by the Sioux?_
* _What is the only modern language that capitalizes its singular first-person pronoun?_
* _What is one of the languages spoken by the Sioux called?_
* _What's the official language of Algeria?_
* _What are the two languages of Malta?_
* _What is the main language of Sao Paulo , Brazil?_

Below some of the synthetic examples generated by DATS with \(q\)=10. It is worth noting that language model introduces some expressions like _sub-dialect_ that are novel with respect to the original training material.

* _What is the language of Switzerland?_
* _What is the oldest language in the Americas?_
* _What's the language spoken by the Kootenai people?_
* _What is the sub-dialect of English?_

Conversely, if we only use the category name as a prompt (i.e., the OC model), the generated questions will have a lower quality. For instance, with \(q=10\) and \(l=10\), the OC model produces questions like:

1. _What is the name of the island of Cote D 'Azubis?_
2. _What English language does the language speak?_
3. _What languages of English and French are spoken as well as Arabic?_
4. _What what languages would French be spoken?_

They have several issues, including label-inconsistency (question 1), malformed syntax (question 4) and unclear semantics (question 2-3).

Similarly, the questions generated by EDA seems of lower quality. For example, with \(q=10\), the EDA model produces the questions:

1. _What is the name of a language spoken by?_
2. _What is the name of a language spoken by the Sioux?_
3. _What is language mostly spoken in Brazil?_
4. _What language is mostly spoken in Brazil?_
5. _Name a gaelic language?_
6. _What is one of the spoken communication spoken by the Sioux called?_
7. _What is one of the languages spoken by the Sioux called?_
8. _What's the most commonly verbalise language in Belgium?_
9. _What's the most commonly spoken language in Belgium?_

Similarly, to the OC case, these questions have several problems, including malformed syntax (question 1, 3, 8), incompleteness (question 1).

**Sentiment Analysis.** Below is a list of statements generated by DATS in the movie review domain. To simplify the qualitative analysis, we report only Negative (class 1 in the dataset), Neutral (class 3) and Positive (class 5) examples. In general, the method generates sound judgments, with a high syntactic variability even though a quite limited lexical variability, i.e., a manual inspection suggested 

that most of the adjectives used to express judgments are reused in the generated examples. Sentences in general are less complex than the original ones. This is reflected by generally shorter generated sentences, i.e., with an average length of 16 words per review against the original 20 words per review.

Negative reviews:

* It's almost too derivative to stand on its own as a stark portrait of desperation and violence.
* A mediocre work of storytelling, lacking the slightest bit of wit or charm.
* A self-conscious, incoherent, self-interested fable.

Neutral reviews:

* It could be a lot worse if it were, well, more adventurous.
* If Tuck had ever made a movie about a vampire, it probably would look a lot like this one.
* only it takes a whole lot more to feel good about.

Positive reviews:

* A gripping and wildly unpredictable comedy.
* It's lovely, funny, different, odd.
* A compelling example of why animation is a part of the human spirit.

The sentences generated using only the category as a prompt (i.e., OC model) have a lower quality. For instance, with the same \(q\)=10 the OC model sentences look like:

Negative reviews:

* Why, a good movie, an interesting and absorbing film
* The, says a filmmaker and the film 's creator.
* The its full potential.

Neutral reviews:

* No at its most extreme.
* This and an epic comedy of a documentary.
* So a little razzle dazzle, but not at all riveting,.

Positive reviews:

* No the greatest of the time
* This the first movie to touch a dark heart
* No in the form of a poem and a movie itself

Similarly, the sentences generated by EDA contain different issues, which can be explained by the substitution and word swapping strategies adopted by this method. For example,

* This taxedo should have been sent back to the tailor for john major some major alterations
* If you value your time and money find an escape clause rehash avoid seeing this trite predictable and
* Spy litigate flick with Antonio Banderas and Lucy Liu never comes together

Neutral reviews:

* It would work a better as much one hour tv documentary
* The fast runner transports the viewer into an unusual space
* Often overwrought and at times positively irritating the film turns into an engrossing thriller almost in spite of 

Positive reviews:

* _Documentary enjoyable high adrenaline enormously_
* _Quite simply a joy to watch follow and especially to listen to_
* _A solid and refined piece of moviemaking imbued with passion and attitude_

**Intent Classification.** Below is a list of examples generated by DATS of each intent in the SNIPS dataset. Please refer to the original dataset description for the details about the specific classes. In general, the examples are different from the original ones introducing variability mostly on the involved named entities, e.g., proper nouns of the music authors or places. In general, the syntactic complexity is the same as the original material. Sometimes, odd dates are introduced (here 2037 as a date for a reservation).

* AddToPlaylist _add this track to my modern psychedelia playlist_
* BookRestaurant _book a diner for 1 on feb 28th 2037_
* GetWeather _will it be warm in michigantown at 07:00:00 am_
* PlayMusic i want to listen to the last track by michael hayyoronsky_
* RateBook i give the current essay a four_
* SearchCreativeWork _find a painting called the night ow_
* SearchScreeningEvent _show me the movie schedule for national iv_

Conversely, using only the category as input to the generation model, i.e., the OC baseline, is not able to produce high quality examples. For instance, with \(q\)=10, the OC model generates sentences like:

* AddToPlaylist _Add please rewind now my playlist_
* BookRestaurant _Book the menu for the night in the hotel room_
* GetWeather _Add a weather forecast from my backyard_
* PlayMusic _Play tunes and tracks to the chino sound bar_
* RateBook _Add some jazz music_
* SearchCreativeWork _Add my novel_
* SearchScreeningEvent _Add movies this week_

Again, EDA is only able to generate some minor variations of the training examples, and sometimes the swapping/substitution strategies are introducing issues. For example:

* AddToPlaylist _I want this record album on my indie alternative playlist_
* BookRestaurant _I need a table for during midday in Montana_
* GetWeather _Is it going tea be freezing at to time in Michigantown KS_
* PlayMusic _Play any song from rebekah hewitt_
* RateBook _Rate this series one stars_
* SearchCreativeWork _The me show song spiderman of the rings_
* SearchScreeningEvent _I want the neediness movie schedule for animated movies in the area_

**Natural Language Inference.** This task is the most challenging as the generated text is not only expected to be internally consistent, but also pair-wise consistent. In general, the syntactic complexity of the sentences is preserved, with an average length of 10 words per sentence. Similarly to the original training set, in many entailment examples generated by DATS the premise and the hypothesis share the same verb (e.g., _watching_ in the first example). On the contrary, the subject of the action is often changed during the contradictions (e.g., _boy_ vs _girl_) in the first contradictory pair. Below we report some example pairs:

Text Pairs in Entailment

* "_Two men watching a sports event in the background._" entails "_Two men are watching a game._"
* "_A young woman, wearing sunglassess, is raising her hand._" entails "_A woman is raising an arm_"
* "_A boy dressed to play soccer is playing" soccer._ entails "_The boy is playing outside._" 

Text Pairs in Contradiction
* "_Young Indian boy playing cricket and soccer._ contradicts "_Young Indian girl playing cricket._"
* "_A boy in a yellow shirt, orange and white, playing guitar._" contradicts "_A boy is sitting quietly._"
* "_A man is attempting a jump on his skateboard._" contradicts "_A man rides his bike over an obstacle course._"

Instead, the examples generated when only prompting the model with the category name are not as good. For example, the OC model with \(q\)=10 produces the following examples:

 Text Pairs in Entailment

* "_A man being photographed in a store in Seattle_" entails "_A man is sitting in a car._"
* "_A woman leaves a beer in the air._" entails "_A woman is leaving a beer._"
* "_A man walks along a street in St. Pete and walks past a city bus terminal._ entails "_Police are standing near the ground holding a man_"

Text Pairs in Contradiction

* "_A white van is driving by and a white man is sitting next to it_ contradicts "_A man in a blue polo shirt is sitting on the back of a white car._"
* "_from the middle of a street at a supermarket street, two girls wearing black and a yellow shirt._" contradicts "_A woman wearing a pink shirt looks down at a pole, waving a dollar bill._"
* "_s a little boy playing a ball._" contradicts "_A dog is jumping out of a field._"

Notice that the OC model is, in general, not able to produce label consistent pairs. For example, the first and third pairs reported above for the entailment category are not correct examples of this class.

In the following, we report also some examples generated with EDA. Again, EDA strategies introduces some issues in the sentences. For example,

 Text Pairs in Entailment

* "_An old man with gray hair wearing a scarf and black jacket_" entails "_An old woman is bundled up_'
* "_Two ladies are maam laughing on the street_" entails "_There are women_"
* "_Lady into the sky on city_ entails "_Lady looks into the sky_"

Text Pairs in Contradiction

* "_Young man doing a trick jump on a skateboard_" "_A man is writing record_"
* "_A by holds hands while walking couple buildings_" contradicts "_A couple dance in the street_"
* "_A a man in a black shirt and shorts sitting at blond table wine with a glass of eating_" contradicts "_a woman is drinking a beer_"

Besides introducing grammatical errors, EDA is also introducing label consistency errors (for example, the first entailment pair).

### Are generated examples really different from the original ones?

To better understand the advantages provided by DATS in the consistency-diversity trade-off, we here report an exhaustive qualitative evaluation.

Table 3 reports all the SA examples generated by different models on classes 5 and 1 when using \(q=10\) average training examples per class. Exception for a single case "(_It is a masterpiece, brilliantly directed, and incredibly well done._)" that is not a negative sentiment example as it was supposed to be), the examples generated by DATS (with \(n=1\) and \(l=1\)) are all label-consistent. Conversely, many examples generated with the OC baseline have a wrong sentiment. On the other side, the diversity introduced by DATS is impressive: all the examples are very different from the input example used to condition the NLG model while generating them. At the same time, they are also very novel with respect to the full



training set (used to train the NLG model) and therefore can be very useful when augmenting the training data of NLU models. On the opposite, BT and EDA introduce very minor modifications to the input text, resulting in a significant lower diversity.

Table 4 provides examples of generated questions for the QC task. This task is particularly different from SA as it involves 50 categories and the size of the training dataset causes many of them to be underrepresented. We generated a set of questions with DATS (\(q=100\), \(n=1\) and \(l=1\)) by selecting one instance for each category as the input to the model. Each row of the table thus reports the class and example used in input to the generator and the last column reports the generated example.

As it can be noticed, DATS generates examples different from the input ones: only 4 out of 50 (8 \(\%\)) examples are exactly copied and 3 out of 50 (6\(\%\)) are different with respect to only one word. Notice that the QC task is particularly sensitive to the syntactic structure of the sentence and this is generally preserved in the generation process. DATS very rarely generates a paraphrase of the original sentence. The target of the generated question almost always changes (e.g., when using "_Who is Nicolo Paganini?_" as input, DATS generates "_Who is Michael Jackson?_"). Finally, notice that in all cases DATS is able to generate a question which is coherent with the target category of the input question.

 



### Effects of Hyper-parameters

This section reports an analysis of the role of DATS hyper-parameters. In particular, we show the results by varying (_i_) the number of most similar elements \(n\) used to generate the pairs for training \({\cal M}\) and (_ii_) the number of elements \(l\) generated with nucleous sampling. Specifically, we tried \(n\in[1,2,5]\) and \(l\in[1,2,3,5,7,10]\). In figure 1 we report the difference in Accuracy between our approach and the NoDA baseline for each specific configuration. Each figure refers to a specific \(q\) (i.e., the average number of examples per class) value, and reports the average delta accuracy computed on the QC, SA and IC tasks.

In general, when using small datasets it is beneficial to use higher \(n\) values, i.e., to create more pairs for training the NLG model. In fact, when \(q\)=10 we can observe that the best improvement is obtained with \(n\)=2 or \(n\)=5.

Regarding the number \(l\) of examples generated through \({\cal M}\), we can observe that generating a higher number of examples seems to be beneficial in almost every case, especially when dealing with a small dataset. For example, when \(q\)=10 or \(q\)=50, our DA approach provides the best performance with \(l=7\) or \(l\)=10. More generally, we can observe a positive trend in generating at least \(l\)=5 examples with the generation model.

Figure 1: Average Accuracy difference between DATS and the NoDA baseline w.r.t. different dataset sizes. We vary the number \(n\) of pairs created with the learned similarity function and the number \(l\) of generated examples with generation model \({\cal M}\). The average is computed w.r.t. the QC, SA and IC tasks.



* [14] M. C. R. Jones, J. A. Laird, and D. S. Johnston, _The New Journal of Physics: Series B_**206**(3), 579-684
* [15] G. P. Kolesch, E. H. Simmons, and T. W. Taylor, _Journal of Statistical Mechanics: Theory and Experiment_**20**(1), 1-12
* [16] F. N. Smith, _Statistical mechanics of the statistical theory of a single particle_, Oxford University Press, Oxford, 1994.
* [17] I. V. Shalaev, _Theory of statistical physics_**19**(1), 1-17

[MISSING_PAGE_POST]