SlowGuess/NAACL_2022_OCR huggingface dataset:
Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.
Danilo Croce, Alessandro Moschitti, and Roberto Basili. 2011. Structured lexical similarity via convolution kernels on dependency trees. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1034-1046, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Edward H. Hovy. 2021. A survey of data augmentation approaches for NLP. CoRR, abs/2105.03075.
Yoav Goldberg. 2016. A primer on neural network models for natural language processing. J. Artif. Intell. Res., 57:345-420.
Ian J. Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press, Cambridge, MA, USA. http://www.deeplearningbook.org.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. CoRR, abs/1909.05858.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. CoRR, abs/1805.06201.
Oleksandr Kolomiyets, Steven Bethard, and Marie-Francine Moens. 2011. Model-portability experiments for textual temporal analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 271-276, Portland, Oregon, USA. Association for Computational Linguistics.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained transformer models. In Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems, pages 18-26, Suzhou, China. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Xin Li and Dan Roth. 2006. Learning question classifiers: the role of semantic information. Natural Language Engineering, 12(3):229-249.
Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. 2020. Data boost: Text data augmentation through reinforcement learning guided conditional generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9031-9041, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Shayne Longpre, Yu Wang, and Christopher DuBois. 2020. How effective is task-agnostic data augmentation for pretrained transformers? CoRR, abs/2010.01764.
Luis Perez and Jason Wang. 2017. The effectiveness of data augmentation in image classification using deep learning. CoRR, abs/1712.04621.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. Sentence meta-embeddings for unsupervised semantic textual similarity. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7027-7034, Online. Association for Computational Linguistics.
Ilyes Rebai, Yessine BenAyed, Walid Mahdi, and Jean-Pierre Lorre. 2017. Improving speech recognition using data augmentation and acoustic model fusion. Procedia Computer Science, 112:316-322. Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 21st International Conference, KES-20176-8 September 2017, Marseille, France.
Shuhuai Ren, Jinchao Zhang, Lei Li, Xu Sun, and Jie Zhou. 2021. Text AutoAugment: Learning compositional augmentation policy for text classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages
9029-9043, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. CoRR, abs/1511.06709.
Sam Shleifer. 2019. Low resource text classification with ulmfit and backtranslation. CoRR, abs/1903.09244.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. CoRR, abs/1901.11196.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2018. Conditional BERT contextual augmentation. CoRR, abs/1812.06705.
A.1 Task Description
In this section we report details and statistics of the dataset we adopted in the experimental section.
TREC. The TREC (Li and Roth, 2006) dataset contains 4,907, 545 and 500 examples for training, development and test, respectively. We adopted the fine-grained version of the dataset that contains 50 categories.
SST5. The Sentiment Analysis Treebank dataset (Socher et al., 2013) consists of 8,544, 1,101 and 2,210 examples for training, development and test, respectively. The dataset is characterized by 5 categories for sentiment, i.e., 1 (Very Negative) 2 (Negative), 3 (Neutral), 4 (Positive) and 5 (Very Positive) and it contains movie review sentences.
SNIPS. The SNIPS dataset (Coucke et al., 2018) consists of natural language commands for a voice assistant. The commands are classified into 7 categories, i.e., RateBook, BookRestaurant, AddTo Playlist, PlayMusic, GetWeather, SearchScreeningEvent, SearchCreativeWork. The dataset consists of 13,084, 700, 700 examples for training, development and test, respectively.
SNLI. The SNLI dataset (Bowman et al., 2015) consists of examples of pairs for the Natural Language Inference task. There are 3 categories: entailment, neutral and contradiction. The dataset consists of 550,152, 10,000 and 10,000 for training, development and test, respectively.
This section reports additional examples generated by DATS. In particular, we report the examples generated with  $n = 2$  and  $l = 10$ . Notice that, the generated examples are not in the original datasets. It is worth noting that some examples may result odd, such as a question like "Name the American president born in 2005?" but we only expected that they are linguistically sound and consistent with the corresponding class. In general, the number  $n$  strongly affects the example novelty: high values of  $n$  make the generator observe multiple times the same text as the target sentence, making the generation process more conservative. As a result, when using  $n > 20$  the generator tends to produce the same examples from the original dataset.
Question classification. Here is a list of 20 questions generated by DATS for different classes. Please refer to the original dataset description for the details about the specific classes.
- ABBR: exp What does D-DAY stand for?
- DESC: def What does the term "Italian Renaissance" mean?
- DESC: desc What process takes place after a hydrogen release?
- DESC:reason Why do people go to the bathroom at night?
- ENTY: animal What animal was the first domesticated creature in the world?
- ENTY: color What color is the cross on the French flag?
- ENTY: event What happened on February 27, 1991?
- ENTY:food What cereal is "sweet, soft and orange"?
- ENTY: plant What tree has the longest trunk?
- ENTY:subst. What are diamond rings made of?
- ENTY: veh What was the name of the U.S. Navy gunboat used by Dwight D. Eisenhower?
HUM:gr Name the two major companies in the energy industry.
HUM:ind What's the name of the author of "Harry Potter"?
- LOC: city What's the name of the largest city in Germany?
- LOC: country What country in 1991 recorded the largest number of cocaine seizures?
- NUM: count How many hundred ships sank in the Norwegian Sea in 1923?
- NUM: date What is the date of the first inauguration of President Nixon?
- NUM:money What is the cost of university admission to Stanford?
- NUM:period How long does it take to clean up a cache?
- NUM:weight What is the maximum weight for a healthy adult?
Although the above examples seem coherent, we can argue if they can be really new, i.e., useful for the task according to a Data Augmentation process. We thus focused on the specific class ENTY: lang that is underrepresented in the original dataset, with only the six original examples reported below:
What is the name of a language spoken by the Sioux?
- What is the only modern language that capitalizes its singular first-person pronoun?
What is one of the languages spoken by the Sioux called?
What's the official language of Algeria?
What are the two languages of Malta?
What is the main language of Sao Paulo, Brazil?
Below some of the synthetic examples generated by DATS with  $q = 10$ . It is worth noting that language model introduces some expressions like sub-dialect that are novel with respect to the original training material.
What is the language of Switzerland?
What is the oldest language in the Americas?
- What's the language spoken by the Kootenai people?
What is the sub-dialect of English?
Conversely, if we only use the category name as a prompt (i.e., the OC model), the generated questions will have a lower quality. For instance, with  $q = 10$  and  $l = 10$ , the OC model produces questions like:
1. What is the name of the island of Cote D 'Azubis?
2. What English language does the language speak?
3. What languages of English and French are spoken as well as Arabic?
4. What what languages would French be spoken?
They have several issues, including label-inconsistency (question 1), malformed syntax (question 4) and unclear semantics (question 2-3).
Similarly, the questions generated by EDA seems of lower quality. For example, with  $q = 10$ , the EDA model produces the questions:
1. What is the name of a language spoken by?
2. What is the name of a language spoken by the Sioux?
3. What is language mostly spoken in Brazil?
4. What language is mostly spoken in Brazil?
5. Name a gaelic language?
6. What is one of the spoken communication spoken by the Sioux called?
7. What is one of the languages spoken by the Sioux called?
8. What's the most commonly verbalise language in Belgium?
9. What's the most commonly spoken language in Belgium?
Similarly, to the OC case, these questions have several problems, including malformed syntax (question 1, 3, 8), incompleteness (question 1).
Sentiment Analysis. Below is a list of statements generated by DATS in the movie review domain. To simplify the qualitative analysis, we report only Negative (class 1 in the dataset), Neutral (class 3) and Positive (class 5) examples. In general, the method generates sound judgments, with a high syntactic variability even though a quite limited lexical variability, i.e., a manual inspection suggested
that most of the adjectives used to express judgments are reused in the generated examples. Sentences in general are less complex than the original ones. This is reflected by generally shorter generated sentences, i.e., with an average length of 16 words per review against the original 20 words per review.
Negative reviews:
- It's almost too derivative to stand on its own as a stark portrait of desperation and violence.
- A mediocre work of storytelling, lacking the slightest bit of wit or charm.
- A self-conscious, incoherent, self-interested fable.
Neutral reviews:
- It could be a lot worse if it were, well, more adventurous.
- If Tuck had ever made a movie about a vampire, it probably would look a lot like this one.
- It's very much like George Romero's final work, where he had a hand in making huge cuts to his movie – only it takes a whole lot more to feel good about.
Positive reviews:
- A gripping and wildly unpredictable comedy.
- It's lovely, funny, different, odd.
- A compelling example of why animation is a part of the human spirit.
The sentences generated using only the category as a prompt (i.e., OC model) have a lower quality. For instance, with the same  $q = 10$  the OC model sentences look like:
- Why, a good movie, an interesting and absorbing film
- The, says a filmmaker and the film's creator.
- The its full potential.
- No at its most extreme.
- This and an epic comedy of a documentary.
So a little razzle dazzle, but not at all riveting.
- No the greatest of the time
- This the first movie to touch a dark heart
- No in the form of a poem and a movie itself
Similarly, the sentences generated by EDA contain different issues, which can be explained by the substitution and word swapping strategies adopted by this method. For example,
- This tuxedo should have been sent back to the tailor for john major some major alterations
- If you value your time and money find an escape clause rehash avoid seeing this trite predictable and
- Spy litigate flick with Antonio Banderas and Lucy Liu never comes together
- It would work a better as much one hour tv documentary
- The fast runner transports the viewer into an unusual space
- Often overwrought and at times positively irritating the film turns into an engrossing thriller almost in spite of
- Documentary enjoyable high adrenaline enormously
- Quite simply a joy to watch follow and especially to listen to
- A solid and refined piece of moviemaking imbued with passion and attitude
Intent Classification. Below is a list of examples generated by DATS of each intent in the SNIPS dataset. Please refer to the original dataset description for the details about the specific classes. In general, the examples are different from the original ones introducing variability mostly on the involved named entities, e.g., proper nouns of the music authors or places. In general, the syntactic complexity is the same as the original material. Sometimes, odd dates are introduced (here 2037 as a date for a reservation).
- AddTo Playlist add this track to my modern psychedelia playlist
- BookRestaurant book a diner for 1 on feb 28th 2037
- GetWeather will it be warm in michigantown at 07:00:00 am
- PlayMusic i want to listen to the last track by michael hayvoronsky
- RateBook i give the current essay a four
- SearchCreativeWork find a painting called the night owl
SearchScreeningEvent show me the movie schedule for national tv
Conversely, using only the category as input to the generation model, i.e., the OC baseline, is not able to produce high quality examples. For instance, with  $q = 10$ , the OC model generates sentences like:
- AddTo Playlist Add please rewind now my playlist
- BookRestaurant Book the menu for the night in the hotel room
- GetWeather Add a weather forecast from my backyard
- PlayMusic Play tunes and tracks to the chino sound bar
- RateBook Add some jazz music
- SearchCreativeWork Add my novel
SearchScreeningEvent Add movies this week
Again, EDA is only able to generate some minor variations of the training examples, and sometimes the swapping/substitution strategies are introducing issues. For example:
- AddTo Playlist I want this record album on my indie alternative playlist
- BookRestaurant I need a table for during midday in Montana
- GetWeather Is it going tea be freezing at to time in Michigantown KS
- PlayMusic Play any song from rebekah hewitt
- RateBook Rate this series one stars
SearchCreativeWork The me show song spiderman of the rings
SearchScreeningEvent I want the neediness movie schedule for animated movies in the area
Natural Language Inference. This task is the most challenging as the generated text is not only expected to be internally consistent, but also pair-wise consistent. In general, the syntactic complexity of the sentences is preserved, with an average length of 10 words per sentence. Similarly to the original training set, in many entailment examples generated by DATS the premise and the hypothesis share the same verb (e.g., watching in the first example). On the contrary, the subject of the action is often changed during the contradictions (e.g., boy vs girl) in the first contradictory pair. Below we report some example pairs:
Text Pairs in Entailment
- "Two men watching a sports event in the background." entails "Two men are watching a game."
"A young woman, wearing sunglasses, is raising her hand." entails "A woman is raising an arm"
"A boy dressed to play soccer is playing" soccer. entails "The boy is playing outside."
Text Pairs in Contradiction
- "Young Indian boy playing cricket and soccer. contradicts "Young Indian girl playing cricket."
"A boy in a yellow shirt, orange and white, playing guitar." contradicts "A boy is sitting quietly."
- "A man is attempting a jump on his skateboard." contradicts "A man rides his bike over an obstacle course."
Instead, the examples generated when only prompting the model with the category name are not as good. For example, the OC model with  $q = 10$  produces the following examples:
"A man being photographed in a store in Seattle" entails "A man is sitting in a car."
- "A woman leaves a beer in the air." entails "A woman is leaving a beer."
- "A man walks along a street in St. Pete and walks past a city bus terminal. entails "Police are standing near the ground holding a man"
- "A white van is driving by and a white man is sitting next to it contradicts "A man in a blue polo shirt is sitting on the back of a white car."
- "from the middle of a street at a supermarket street, two girls wearing black and a yellow shirt." contradicts "A woman wearing a pink shirt looks down at a pole, waving a dollar bill."
- "s a little boy playing a ball." contradicts "A dog is jumping out of a field."
Notice that the OC model is, in general, not able to produce label consistent pairs. For example, the first and third pairs reported above for the entailment category are not correct examples of this class.
In the following, we report also some examples generated with EDA. Again, EDA strategies introduce some issues in the sentences. For example,
An old man with gray hair wearing a scarf and black jacket entails "An old woman is bundled up"
- "Two ladies are maam laughing on the street" entails "There are women"
"Lady into the sky on city entails "Lady looks into the sky"
- "Young man doing a trick jump on a skateboard" "A man is writing record"
- "A by holds hands while walking couple buildings" contradicts "A couple dance in the street"
- “A man in a black shirt and shorts sitting at blond table wine with a glass of eating” contradicts “a woman is drinking a beer”
Besides introducing grammatical errors, EDA is also introducing label consistency errors (for example, the first entailment pair).
A.3 Are generated examples really different from the original ones?
To better understand the advantages provided by DATS in the consistency-diversity trade-off, we here report an exhaustive qualitative evaluation.
Table 3 reports all the SA examples generated by different models on classes 5 and 1 when using  $q = 10$  average training examples per class. Exception for a single case " (It is a masterpiece, brilliantly directed, and incredibly well done.)" that is not a negative sentiment example as it was supposed to be), the examples generated by DATS (with  $n = 1$  and  $l = 1$ ) are all label-consistent. Conversely, many examples generated with the OC baseline have a wrong sentiment. On the other side, the diversity introduced by DATS is impressive: all the examples are very different from the input example used to condition the NLG model while generating them. At the same time, they are also very novel with respect to the full
training set (used to train the NLG model) and therefore can be very useful when augmenting the training data of NLU models. On the opposite, BT and EDA introduce very minor modifications to the input text, resulting in a significant lower diversity.
Table 4 provides examples of generated questions for the QC task. This task is particularly different from SA as it involves 50 categories and the size of the training dataset causes many of them to be underrepresented. We generated a set of questions with DATS ( $q = 100$ ,  $n = 1$  and  $l = 1$ ) by selecting one instance for each category as the input to the model. Each row of the table thus reports the class and example used in input to the generator and the last column reports the generated example.
As it can be noticed, DATS generates examples different from the input ones: only 4 out of  $50(8\%)$  examples are exactly copied and 3 out of  $50(6\%)$  are different with respect to only one word. Notice that the QC task is particularly sensitive to the syntactic structure of the sentence and this is generally preserved in the generation process. DATS very rarely generates a paraphrase of the original sentence. The target of the generated question almost always changes (e.g., when using "Who is Nicolo Paganini?" as input, DATS generates "Who is Michael Jackson?"). Finally, notice that in all cases DATS is able to generate a question which is coherent with the target category of the input question.
Table 4: Examples of DATS outputs generated sentence on the QC task for each category.
A.4 Effects of Hyper-parameters
This section reports an analysis of the role of DATS hyper-parameters. In particular, we show the results by varying  $(i)$  the number of most similar elements  $n$  used to generate the pairs for training  $\mathcal{M}$  and  $(ii)$  the number of elements  $l$  generated with nucleous sampling. Specifically, we tried  $n\in [1,2,5]$  and  $l\in [1,2,3,5,7,10]$ . In figure 1 we report the difference in Accuracy between our approach and the NoDA baseline for each specific configuration. Each figure refers to a specific  $q$  (i.e., the average number of examples per class) value, and reports the average delta accuracy computed on the QC, SA and IC tasks.
Figure 1: Average Accuracy difference between DATS and the NoDA baseline w.r.t. different dataset sizes. We vary the number  $n$  of pairs created with the learned similarity function and the number  $l$  of generated examples with generation model  $\mathcal{M}$ . The average is computed w.r.t. the QC, SA and IC tasks.
In general, when using small datasets it is beneficial to use higher  $n$  values, i.e., to create more pairs for training the NLG model. In fact, when  $q = 10$  we can observe that the best improvement is obtained with  $n = 2$  or  $n = 5$ .