SlowGuess/NAACL_2022_OCR huggingface dataset:
Simone Filice
filicesf@amazon.com
Dept. of Enterprise Engineering
University of Rome, Tor Vergata
Even if recent Transformer-based architectures, such as BERT, achieved impressive results in semantic processing tasks, their fine-tuning stage still requires large scale training resources. Usually, Data Augmentation (DA) techniques can help to deal with low resource settings. In Text Classification tasks, the objective of DA is the generation of well-formed sentences that  $(i)$  represent the desired task category and  $(ii)$  are novel with respect to existing sentences. In this paper, we propose a neural approach to automatically learn to generate new examples using a pre-trained sequence-to-sequence model. We first learn a task-oriented similarity function that we use to pair similar examples. Then, we use these example pairs to train a model to generate examples. Experiments in low resource settings show that augmenting the training material with the proposed strategy systematically improves the results on text classification and natural language inference tasks by up to  $10\%$  accuracy, outperforming existing DA approaches.
Deep Learning models achieve state-of-the-art results in many domains, including Computer Vision and Natural Language Processing (NLP). Training these large models typically requires many examples, whose collection and annotation can be costly and time-consuming. Data augmentation (DA) has proven an efficient way to acquire more training samples without incurring in the prohibitive annotation cost in a variety of fields, including computer vision (Perez and Wang, 2017) and speech recognition (Rebai et al., 2017). In NLP, some DA techniques have been proposed too, as surveyed in (Feng et al., 2021): common approaches cre
ate synthetic data by manipulating real examples, using Text Editing (Wei and Zou, 2019) or Back-Translation (Sennrich et al., 2015); the resulting examples are automatically labeled by inheriting the class of the original example they were generated from.
Unfortunately, when using recent pre-trained language models, such as BERT (Devlin et al., 2018) or RoBERTa (Liu et al., 2019), the effectiveness of DA methods is extremely limited, and sometimes they can even hurt the results (Longpre et al., 2020). A possible explanation for this inefficacy is that these DA techniques introduce lexical and structural variability that accurate language models directly induce during pre-training. The usefulness of synthetic examples is strictly related to their diversity from the original training data. At the same time, diverging too much from the initial data might increase the risk of introducing noisy annotations, i.e., synthetic data not reflecting the class of the original examples they were generated from.
To directly tackle the trade-off between diversity and label consistency, in this paper we propose DATS<sup>1</sup> (Data Augmentation based on Task-specific Similarity), a novel data-augmentation technique for text classification based on Natural Language Generation (NLG) models. Starting from a reduced set of annotated examples, we first learn a task-oriented similarity function that we use to automatically create pairs of similar examples. Then, we use these pairs to train a generative model to generate an example similar to the input one. Finally, we employ this model to generate new synthetic examples and augment the training data. We show that pairing examples with respect to their task
https://github.com/cruz82/dats.
Human Language Technologies, pages 4587 - 4601
oriented similarity is striking in order to allow the generative model to automatically understand the lexical and structural variations that can be applied to an instance without changing its label. Experimental results on four different text classification datasets demonstrate that DATS achieves better results than several existing DA solutions and that it systematically improves NLU models based on state-of-the-art pre-trained language models. In the remaining, Section 2 summarizes related works, Section 3 describes our method and Section 4 provides the experimental evaluation.
Most of the existing approaches for DA perform some token-level manipulations on individual sentences (Kolomiyets et al., 2011; Kobayashi, 2018). For instance, Easy Data Augmentation (EDA) (Wei and Zou, 2019) applies simple operations including synonym replacement and random swap. These operations are also performed in (Ren et al., 2021) where the framework named Text AutoAugment (TAA) uses Bayesian Optimization algorithm to automatically search for the best manipulation policy. On the contrary, Wu et al. (2018) uses mask random tokens and use BERT to generate substituting words. Similarly, Kumar et al. (2020) use transformer-based models to apply token-level or span-level text content manipulation.
Other attempts operate at the entire sentence-level by paraphrasing the original text using backtranslation (BT) (Edunov et al., 2018; Shleifer, 2019); however, BT tends to skew towards high-frequency words which not only causes redundancy but also leads to lexical shrinkage in the augmented data (Liu et al., 2020). More recent approaches for data augmentation use generative models to create more diverse synthetic data. Anaby-Tavor et al. (2019) fine-tuned a GPT-2 model to generate text given a target class, and use a text classifier to filter out those synthetic examples whose predicted class does not match the target class. In our work, we show that by conditioning a target model not only on the class labels, but also on representative examples, it is possible to better control the diversity-consistency trade-off.
3 Learning to generate examples
Recent advances in NLG (Vaswani et al., 2017) demonstrated that sequence-to-sequence (seq2seq) models can generate natural sounding and meaning-
ful text given a prompt. As shown in (Keskar et al., 2019), the prompt can include style or content-related information that can help control the generation process.
In our work, we capitalize these techniques to augment a dataset  $\mathcal{D}$  for the training of a text classifier  $\mathcal{C}$ . Specifically, our goal is to fine-tune a pre-trained seq2seq model  $\mathcal{M}(*) = e$  that, given a prompt  $*$ , generates novel examples not in  $\mathcal{D}$ . The first application of this idea was investigated in Anaby-Tavor et al. (2019), where the authors trained a seq2seq model  $\mathcal{M}(c) = s_{o}$  that generates examples  $s_{o}$  of a given class  $c$ . While this approach provides interesting results, it must be said that the variability of the generated examples can be quite low: by conditioning  $\mathcal{M}$  only on  $c$ , it tends to produce the most frequent syntactic and lexical patterns associated with  $c$ , neglecting other modes this can exhibit. The key challenge here is to ensure diversity while preserving consistency, i.e., generating diverse examples that are valid representatives of the desired class.
We propose to train a seq2seq model  $\mathcal{M}$  that is conditioned not only on the class  $c$ , but also on an example  $s_i$  of that class, i.e., such that  $\mathcal{M}(c, s_i) = s_0$ . The model is thus expected to synthesize a new example  $s_0$ , which is consistent with the input class  $c$  and is also "inspired" by an existing example  $s_i$ . The problem now is how to build the dataset to train  $\mathcal{M}$ . Pairing two random examples  $s_i$  and  $s_o$  belonging to the same class might be a viable solution. However, in classification tasks, examples are not necessarily similar, and coupling together radically different examples risks destabilizing the training of  $\mathcal{M}$ . Ideally, we would like to identify different modes in the same class and pair only examples belonging to the same mode. An alternative way to reach the same outcome is the adoption of a semantic similarity function that can be used to select linguistically related example pairs. Several unsupervised metrics exist (Croce et al., 2011; Cer et al., 2018; Poerner et al., 2020), however, they are inherently task-independent and not adequate in capturing task-specific similarities: two sentences such as "The movie is good" and "The movie is not good" (or even "The movie is awful") should not be paired together, when dealing with sentiment classification, while they are good candidate pairs in the training of a topic classifier.
To define a task-oriented similarity measure, we leverage the text classifier  $\mathcal{C}$  we want to improve
via data augmentation. Independently on the underlying neural architecture, a neural classifier has an encoder  $\mathcal{E}$  that projects an example  $s\in \mathcal{D}$  in a  $d$  -dimensional space, i.e.,  $\mathcal{E}(s) = \vec{s}\in \mathbb{R}^d$  . In these spaces, simple linear classifiers (i.e., the output layers) identify the sub-spaces reflecting the target classes (Goodfellow et al., 2016; Goldberg, 2016). As a consequence,  $\mathcal{E}$  is expected to project examples in sub-spaces representative for individual classes. Our task-aware similarity measure is the cosine similarity operating on these representations. For instance, after training a BERT model on the question classification task (Li and Roth, 2006), the encoding [CLS] of the entire question "Who developed the vaccination against polio?" allows retrieving other questions corresponding to the most similar [CLS] embeddings, such as "Who invented the Moog Synthesizer?": they are clearly not paraphrases, and their purely lexical similarity is quite low, but they show a similar pattern characterizing the question class [HUM] (i.e., human).
Algorithm 1: Pseudo-code of DATS.
To generate an augmented dataset for a generic text classification task, we propose DATS (Data Augmentation based on Task-specific Similarity) that is described in Algorithm 1. First, we train a classifier  $\mathcal{C}$  on  $\mathcal{D}$ . We use the resulting encoder  $\mathcal{E}$  to project each training example  $e$  into the task-specific vector space, obtaining the embedding  $\vec{e}$ . Then, we split the training data  $\mathcal{D}$  into  $\mathcal{D}_{tr}$  and  $\mathcal{D}_{te}$ . Each example  $e_i\in \mathcal{D}_{tr}$  is paired with the  $n$  exam
plies  $e_j \in \mathcal{D}_{tr}$  of the same class having the highest cosine similarity computed on their corresponding embeddings vectors  $\mathcal{E}(e_i)$  and  $\mathcal{E}(e_j)$ . The resulting pairs are expected to lie in the same subspace and share some task-oriented linguistic relatedness. We use these example pairs to fine-tune a seq2seq model to solve the task  $\mathcal{M}(c_{e_i}, e_i) = e_j$ , where  $c_{e_i}$  is the category of  $e_i$ . Finally, examples in  $\mathcal{D}_{te}$  are provided in input to  $\mathcal{M}$  to generate the new synthetic dataset  $\mathcal{D}_{syn t}$ . By applying multiple splits of  $\mathcal{D}$  in a cross-fold scheme, we can use each training example to condition the model and generate new synthetic instances. As in almost all existing formulations, a generator can be used to generate a set of  $l$  variants for each input  $(c, e)$ , for simplicity referred as  $\mathcal{M}^l(c, e)$ . In particular, techniques such as nucleus-sampling (Holtzman et al., 2020) enable the generation of large sets of variants, generally characterized by a good diversity.
It is worth noting that no assumption is applied when selecting  $\mathcal{C}$  or  $\mathcal{M}$ . Even though in the experimental evaluation we consider only specific architectures (namely BERT and BART), this methodology can be applied to a wider plethora of models. Moreover, there is no restriction on the classification task type: the experimental section shows that DATS is applicable to classification tasks operating on both individual sentences and text pairs, e.g., in natural language inference (Bowman et al., 2015): given text pairs  $s_{i,1}$  and  $s_{i,2}$ , it is sufficient to extend the above process defining  $\mathcal{M}$  differently, i.e.,  $\mathcal{M}(c_{(s_{i,1}, s_{i,2})}, s_{i,1}, s_{i,2}) = (s_{j,1}, s_{j,2})$ .
We test our approach on four tasks: 50-class Question Classification (QC) over the TREC dataset (Li and Roth, 2006); 5-class Sentiment Classification (SA) over the SST dataset (Socher et al., 2013); 7-class Intent Classification (IC) over the SNIPS dataset (Coucke et al., 2018); sentence-pair classification for Natural Language Inference (NLI) over the 3-category SNLI dataset (Bowman et al., 2015). Details about the datasets are in Appendix A.1.
Baselines. We compare our approach with multiple baselines and simplified versions of DATS for ablation study: (i) Easy Data Augmentation (EDA) (Wei and Zou, 2019); (ii) Back-Translation (BT) (Sennrich et al., 2015) in an English-German-English setting $^{2}$ ; (iii) Random Pairing (RP) - DATS
2For EDA and BT we adopted the code in https://github.com/varunkumar-dev/
without task-oriented similarity functions where we created the training input-output pairs for  $\mathcal{M}$  by selecting two random examples of the same class from the training set;  $(iv)$  Only Class (OC) - DATS where the prompt is only the category name (i.e., no representative example), similar to Anaby-Tavor et al. (2019).
Experimental Setting. For QC, SA and IC, we report the Accuracy when using  $q = 10$ ,  $q = 50$  or  $q = 100$  average examples per class<sup>3</sup>. For the NLI task, since it is more challenging, we report the Accuracy also for  $q = 500$  and  $q = 1000$  average examples per category. We also report the performance of each model when using the entire (F) training set<sup>4</sup>. We use the bert-base-uncased model from the Huggingface library (Wolf et al., 2019) as the classifier  $\mathcal{C}$ , and bart-base (Lewis et al., 2020) as the NLG model  $\mathcal{M}$ . Both are trained for 10 epochs with early stopping (patience=3) and learning rate  $5e^{-5}$ . We adopt nucleous sampling<sup>5</sup> (Holtzman et al., 2020) with  $p = 0.90$ . We repeat the experiments 5 times with different seeds and we report the average classification accuracy.
Results. Table 1 shows the results of a BERT-based model without DA (NoDA), the DA baselines and our approach (DATS). We perform hyperparameters tuning on the development set of each task w.r.t. the number of similar examples  $n$  and the number of generated examples  $l$ . Task-agnostic DA approaches, like BT or EDA, seem slightly beneficial when using a transformer based classifier  $\mathcal{C}$ , as also reported in (Longpre et al., 2020). In some cases, they significantly hurt accuracy.
For instance, in QC or IC when  $q = 10$ , EDA Accuracy is lower than NoDA by about 1 and 6 points, respectively. The same is for BT, where the drop is even higher, i.e., 8 and 18 points. Using the category information in the generation process, i.e., RP and OC, provides variable results, with minor improvements only in few specific settings (e.g., RP on SA). Instead, DATS improves accuracy for almost all  $q$  and tasks and such improvements are
q	NoDA	EDA	BT	RP	OC	DATS
Question Classification (QC)
10	66.39	65.32	58.76*	66.72	62.64	69.04
50	90.64	90.48	89.84	90.28	90.48	90.68
100	91.60	92.04	91.48	91.08	91.44	92.28
F	91.60	91.32	91.48	90.72	91.36	92.28
Sentiment Analysis (SA)
10	26.64	27.95	27.50	29.82*	27.68	28.14
50	40.06	40.89	39.90	40.18	40.20	43.48*
100	42.58	44.38	43.79	43.16	44.26	46.30*
F	53.10	53.29	53.40	52.56	52.76	54.26
Intent Classification (IC)
10	81.92	75.94*	63.71*	81.92	70.06*	91.74*
50	95.18	95.14	94.69	95.42	95.28	95.62
100	95.38	95.54	95.57	95.54	95.78	95.62
F	97.32	97.80	97.42	97.96	98.00*	98.12*
Natural Language Inference (NLI)
10	40.67	39.42	36.07*	35.78*	36.30*	38.74
50	49.32	47.26	42.37*	43.58*	46.08*	48.40
100	57.78	58.92	56.23	50.64*	49.60*	58.36
500	71.48	72.23	73.23*	71.04	63.78*	73.66*
1k	75.10	76.26	76.60*	75.86	75.48	76.86*
Table 1: Accuracy of DA approaches: the best results are in bold while the results higher than NoDA are underlined. The  $\star$  symbol indicates statistically significant differences  $(\mathrm{p} < 0.05)$  with respect to NoDA according to an unpaired T-test.
often statistically significant. When  $q = 10$ , the improvement ranges between 1 and 10 points. In the few cases where a minor drop is observed, this is never statistically significant. By comparing DATS with RP it is clear that the task-specific similarity measure is striking to learn a good data generator<sup>7</sup>. DATS works well also on sentence-pair tasks, like NLI. However, given the complexity of this type of tasks, it requires more training examples (e.g.,  $q > 100$ ) to show a positive impact. Furthermore, DATS outperforms other competitive methods<sup>8</sup>. With  $q = 10$  Kumar et al. (2020) report  $67.30\%$  accuracy in QC and  $87.24\%$  in IC, which is  $1.74\%$  and  $4.5\%$  lower than DATS. In the same setting, the gap with Wu et al. (2018) is even larger, i.e.,  $4.71\%$  and  $5.95\%$ . Moreover, Ren et al. (2021) report  $52.55\%$  accuracy in SA using the entire training set, and DATS outperforms it by  $1.71\%$ .
Qualitative Analysis. To better understand the advantages provided by DATS in the consistency-diversity trade-off, we report a qualitative evalua
TransformersDataAugmentation
3To maintain the original class distribution we randomly sample  $10c$ ,  $50c$  or  $100c$  from the original training set, with  $c$  being the number of classes. We ensure that at least one example per class is sampled.
4We omit results on the full dataset for NLI (made of more than 500,000 examples) for which DA is not needed.
In preliminary evaluations, we tested alternative decoding strategies, e.g., standard beam search and k-sampling. Overall, nucleous sampling was superior in terms of diversity and consistency, in line with the literature (Holtzman et al., 2020).
An analysis of the hyper-parameters is in Appendix A.4
<sup>7</sup>We tried to substitute the task-oriented similarity with a cosine similarity on USE embeddings (Cer et al., 2018) - a general-purpose sentence encoder. Preliminary results on QC were in between DATS and RP, demonstrating that a task-oriented similarity is beneficial.
Due to lack of budget, we did not run experiments on these models. The reported comparisons are based on the results reported by the authors.
cat.	Original Data	DATS	BT
Positive	A colorful , vibrant introduction to a universal human impulse , lushly photographed and beautifully recorded .	This is a real film.	A colorful, lively introduction to a uni- versal human impulse, lavishly pho- tographed and beautifully captured .
A generous , inspiring film that un- folds with grace and humor and grad- ually becomes a testament to faith .	Is in the midst of a monumental ad- venture, and I can't wait to see what the next one looks like.	A generous, inspiring film that unfolds with grace and humor and gradually becomes a witness of faith.
A solid and refined piece of moviemak- ing imbued with passion and attitude .	It 's a film that develops with convic- tion and a deep, passionate heart.	A solid and refined piece of film steeped in passion and attitude.
Enormously enjoyable , high- adrenaline documentary .	It is a remarkable film...	Enormously entertaining, adrenaline- pumping documentary.
Negative	- spy action flick with Antonio Ban- deras and Lucy Liu never comes to-gether .	"It just felt... too heavy... for me to have it in my trunk."	- Espionage action with Antonio Ban- deras and Lucy Liu never comes to-gether.
If you value your time and money , find an escape clause and avoid seeing this trite , predictable rehash .	Isn't the best spy film ever made.	If you value your time and money, find an exit clause and avoid this mundane, predictable repetition.
No amount of burning , blasting , stab- bing , and shooting can hide a weak script .	It is a masterpiece, brilliantly di- rected, and incredibly well done.	No amount of firing, blasting, stabbing and shooting can hide a weak script.
This Tuxedo ... should have been sent back to the tailor for some major al- terations .	This is a film that I was unable to stomach... and very much enjoyed... until the last 30 minutes... when I finally got close to the film... and it turned into a garbage dump.	This tuxedo... should have been sent back to the tailor for some major changes.
Table 2: A qualitative analysis of the diversity in the Sentiment Analysis task over SST-5.
tion of the generated sentences for the SA task.
In particular, Table 2 reports some of the SA examples generated by different models on classes 5 and 1 when using  $q = 10$  average training examples per class. Exception for a single case "It is a masterpiece, brilliantly directed, and incredibly well done.") that is not a negative sentiment example as it was supposed to be), the examples generated by DATS (with  $n = 1$  and  $l = 1$ ) are all label-consistent. However, the diversity introduced by DATS is impressive: all the examples are very different from the input example used to condition the NLG model while generating them, and therefore can be very useful when augmenting the training data of NLU models. On the opposite, BT introduces very minor modifications to the input text, resulting in a significant lower diversity A larger and systematic qualitative analysis on the quality and diversity of the generated material is in Appendix A.2 and A.3.
This paper proposes DATS, a data augmentation method based on Natural Language Generation (NLG) models. A generative model is fine-tuned to produce examples similar to the input ones. The training input-output pairs are selected according to a task-oriented similarity function. This pairing allows the NLG model to learn the lexical and
structural variations that can be applied to an instance without changing its label. The experimental results suggest that the generated sentences are diverse and label consistent, and can improve state-of-the-art text classifiers, outperforming existing DA methods. In the future, we plan to apply DATS to further tasks (e.g., Question Answering) and neural architectures.
We would like to thank the "Istituto di Analisi dei Sistemi ed Informatica - Antonio Ruberti" (IASI) for supporting the experimentations through access to dedicated computing resources.
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2019. Not enough data? deep learning to the rescue! CoRR, abs/1911.03118.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua,