SlowGuess/NAACL_2022_OCR huggingface dataset:
We found that, both RandMix and TreeMix are quite effective, but TreeMix outperforms RandMix on most datasets. For instance, TreeMix exceeds RandMix by  $0.8\%$  on SST2,  $0.6\%$  on TREC-f, and  $0.5\%$  on TREC-c. One exception is on IMDb, where the average sentence length is much longer. The reason for the poorer performance of TreeMix is due to the sparse parsing results on long sentences; since there are many subtrees, substituting any single part might bring very minimal change to the entire sentence.
5.3 Influence of Training Set Size
To examine the influence of TreeMix with different training set sizes, we uniformly sample  $1\%$ ,  $2\%$ ,  $5\%$ ,  $10\%$ , and  $20\%$  of the data from the training set to investigate TreeMix in low-resource situations. The entire test set is used to evaluate the model's generalization ability. Since TreeMix generates more examples for training, we use RandMix to generate the same number of extra samples as a comparison to ensure the data size is fair. The results are summarized in Figure 3.
We found that, (1) TreeMix outperforms RandMix in all settings, further demonstrating the advantage of the compositional substructure with the constituency information over the randomly selected spans. (2) Both mixup methods can significantly improve the model's performance in the case of extreme data scarcity (e.g.,  $1\%$  and  $2\%$ ). (3) When the amount of data is sufficient (e.g., more than
4We observed  $\lambda_{rand}\sim \mathcal{U}(0,0.3)$  is optimal and we use this settings for the experiment
Model	Single Sentence Classification	Pair Sentence Classification
SST2	TREC-f	TREC-c	IMDb	AG NEWS	MRPC	RTE	QNLI	QQP	MNLI
BERT	92.96†	92.36	97.08†	93.63	94.67	84.90	68.15	90.54	90.67	84.27†
BERT+EDA	92.20†	91.95	96.79†	93.62	94.67	-	-	-	-	-
BERT+AEDA	92.57†	92.15	97.20†	93.59	94.22	-	-	-	-	-
BERT+BT	92.48	92.15	96.68	-	-	82.13	67.40	-	-	-
BERT+GPT3Mix	93.25†	-	-	-	-	-	-	-	-	-
BERT+SSMix	93.14†	92.80	97.60†	93.74	94.64	84.31	68.40	90.60	90.75	84.54†
BERT+EmbedMix	93.03†	92.32	97.44†	93.72	94.72	85.34	68.37	90.44	90.58	84.35†
BERT+TMix	93.03†	92.68	97.52†	93.69	94.69	85.69	68.45	90.48	90.66	84.30†
BERT+TreeMix	93.92	93.20	97.95	94.34	94.72	85.34	70.62	91.36	90.88	84.45
$\dagger$  denotes the result is extracted from the original paper
Table 4: Results of comparison with baseline on full datasets, TREC-f and TREC-c indicates TREC-fine and TREC-coarse respectively. Scores are averaged over 5 random seeds. For GLUE tasks, we report accuracy of validation sets, and for other datasets we report test accuracy. EDA and AEDA will seriously damage the sentence relationship and harm the accuracy; GPT3Mix only reports full data experiments results on SST2 in original paper. We only report the results of back translation on small dataset due to the heavy computational cost.
Figure 3: Results on SST2 varying data size. Scores are averaged over 5 random seeds.
$5\%$ ), TreeMix outperforms RandMix by a significant margin. However, TreeMix only slightly outperforms RandMix when there is a severe lack of data (e.g.,  $1\%$  and  $2\%$ ). This is due to that the too small datasets often contain very limited structures, thus constraining TreeMix's ability to increase text patterns and compositional diversity. (4) The relative improvement of TreeMix over conventional training without augmentation diminishes as the amount of data increases, largely due to that additional augmented text patterns might overlap with those already existing in the dataset, resulting in limited improvement.
5.4 Influence of Cross-Category Mixing
Different from prior work Shi et al. (2021), TreeMix allows the composition of sentences from different label categories. To test whether this cross-label category mixup is more effective than
Datasets	BERT	TM(same)	TM(cross)
SST2	92.96	93.78	93.92
TREC-fine	92.36	92.60	93.20
TREC-coarse	97.08	97.74	97.95
IMDb	93.63	94.22	94.34
AGNEWS	94.67	94.47	94.71
MRPC	84.90	85.34	85.34
RTE	68.15	70.25	70.62
QNLI	90.54	90.87	91.36
QQP	90.67	90.85	90.88
MNLI	84.27	84.33	84.45
Table 5: Performance with TreeMix performed (1) within same classes TM(same) and (2) cross different classes TM(cross), averaged over 5 runs.
a within-label category mixup, we conducted ablation studies with TreeMix on samples in the same class. Table 5 shows the results. Across all datasets, we found that TreeMix that combines data from different classes is more effective than combining data from the same class, consistent with findings in Zhang et al. (2018). When given only labels from one category, current models have a tendency to make simple or spurious judgments based on the most frequently occurring words. However the semantics of the sentence are complicated beyond simple words. For example, the model is likely to classify a sentence like "I like this good movie" as positive because of the words "like" and "good", but if "good movie" is replaced with "bad
$^{5}$ Section D.2 in Appendix discuss the effect of constraining the length and label of the swapped subtree on the result
film", the model must perceive the different constituent parts within the sentence. This ability can only be obtained when the model is trained on the cross-category generated samples.
5.5 Influence of Length Ratio
Dataset	BERT	λ=[0.1,0.3]	λ=[0.3,0.5]
SST2	92.96	93.92	93.05
TREC-fine	92.36	93.2	92.25
TREC-coarse	97.08	97.95	96.94
IMDb	93.63	94.34	93.29
AG NEWS	94.67	94.72	94.53
MRPC	84.90	85.34	84.93
RTE	68.15	70.62	70.35
QNLI	90.54	91.36	90.78
QQP	90.67	90.88	90.54
MNLI	84.27	84.45	83.78
The only constraint we impose on TreeMix is the length ratio of the subtree controlled by  $\lambda$ . We select subtrees that are between  $10\%$  and  $30\%$  and between  $30\%$  and  $50\%$  of the length of the sentence, respectively. Table 6 shows the results.
On all datasets,  $\lambda = [0.1, 0.3]$  outperforms  $\lambda = [0.3, 0.5]$ , which is in line with Zhang et al. (2018)'s observation that giving too high mixup ration values can lead to underfitting. Another linguistic explanation for the scenario follows: When  $\lambda = [0.3, 0.5]$ , TreeMix may select longer text spans, which usually contain unique constituency components like  $SBAR$ ; The exchange of these spans will severely damage the sentence's semantic and grammatical structure, causing the model to become confused. As a result, TreeMix with larger switching spans performs poorly, and even worse than baseline on some datasets.
5.6 Compositional Generalization
To quantify TreeMix's overall ability of compositional generalization beyond classification tasks, we conducted experiments on SCAN (Lake and Baroni, 2018) dataset, which is a command execution dataset widely used to test for systematic compositionality. It contains simple source commands and target action sequences. We test on commonly used challenging splits: addprim-jump, addprimiturn-left, around-right, where primitive commands (e.g "jump") only appear alone during training but will be combined with other modifiers (e.g "jump
Table 6: Performance with different length ratio intervals  $\lambda$
Method	JUMP	TURN-L	AROUND-R
Baseline	0†	49%†	0†
WordDrop	0†	51%†	0†
SwitchOut	0†	16%†	0†
SeqMix	49%†	99%†	0†
TreeMix	72%	99%	0%
GECA	87%†	-	82†
GECA+WordDrop	51%†	-	61†
GECA+SwitchOut	77%†	-	73†
GECA+SeqMix	98%†	-	89†
GECA+TreeMix	99%	-	91%
$\dagger$  denotes the result is extracted from the original paper
Table 7: Experimental results (accuracy) on SCAN.
twice") during testing. A model that works well for this task should learn to compose the primitive commands with the modifiers and generates corresponding execution. With TreeMix, we can generate the compositional commands that are not seen in the training set.
The new command generation process is the same as in single sentence classification, except that we increase the length constraint  $\lambda_U$  to 1 to allow the exchange of the commands with only one word. After we synthesize new commands, we follow the rules in Lake and Baroni (2018) to translate valid commands into actions and filter out ungrammatical commands. We follow the settings in Andreas (2020) and use the following data augmentation methods as baselines: (1) WordDrop that drops words randomly; (2) SwitchOut (Wang et al., 2018) that randomly replaces words with other random words from the same vocabulary; (3) SeqMix (Guo et al., 2020) which creates new synthetic examples by softly combining in-put/output sequences from the training set, and (4) GECA (Andreas, 2020) that performs enumerated valid swaps.
As shown in Table 7, TreeMix outperforms SwitchOut and WordDrop for all splits. TreeMix by itself does not perform as well as GECA, but when being combined with GECA, it demonstrates very strong results. TreeMix outperforms SeqMix in all splits, due to the fact that TreeMix can more precisely find the linguistically rich compositional segments of a sentence, as evidenced by the results of the comparisons of TreeMix and SSMix in Section 5.1 and TreeMix and RandMix in Section 5.3. A closer look at these augmented samples show that TreeMix can generate all possible combinations of "jump" and other modifiers like "left"
and "around"; these previously unseen command combinations further validates TreeMix's ability to improve the dataset's compositional diversity. TreeMix demonstrates weak performances on the around-right split, where the model observes commands "around" and "right" in isolation at the training stage, and it has to derive the meaning of "around right" at the test time. Because the word "around" cannot be parsed as a single subtree for swap. Instead, it always appears in a subtree with the word "left", preventing TreeMix from generating the phrase "turn right". Despite its limitations on around-left, TreeMix performs well on all other splits and can be easily combined with other data augmentation methods, demonstrating the compositional generalization ability of TreeMix beyond classification tasks.
This work introduced TreeMix, a compositional data augmentation approach for natural language understanding. TreeMix leverages constituency parsing tree to decompose sentences into substructures and further use the mixup data augmentation technique to recombine them to generate new augmented sentences. Experiments on text classification and semantic parsing benchmarks demonstrate that TreeMix outperforms prior strong baselines, especially in low-resource settings and compositional generalization.
The authors would like to thank reviewers for their helpful insights and feedback. This work is funded in part by a grant from Salesforce.
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue! In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7383-7390. AAAI Press.
Jacob Andreas. 2020. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7556-7566, Online. Association for Computational Linguistics.
Toms Bergmanis, Katharina Kann, Hinrich Schütze, and Sharon Goldwater. 2017. Training data augmentation for low-resource morphological inflection. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflation, pages 31-39, Vancouver. Association for Computational Linguistics.
Hengyi Cai, Hongshen Chen, Yonghao Song, Cheng Zhang, Xiaofang Zhao, and Dawei Yin. 2020. Data manipulation: Towards effective instance learning for neural dialogue generation via learning to augment and reweight. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6334-6343, Online. Association for Computational Linguistics.
Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. 2021. An empirical survey of data augmentation for limited data learning in nlp.
Jiaao Chen, Zhenghui Wang, Ran Tian, Zichao Yang, and Diyi Yang. 2020a. Local additivity based data augmentation for semi-supervised NER. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1241-1251, Online. Association for Computational Linguistics.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020b. Mix-Text: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147-2157, Online. Association for Computational Linguistics.
Luoxin Chen, Weitong Ruan, Xinyue Liu, and Jianhua Lu. 2020c. SeqVAT: Virtual adversarial training for semi-supervised sequence labeling. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8801-8811, Online. Association for Computational Linguistics.
Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. 2020d. Compositional generalization via neural-symbolic stack machines. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Yong Cheng, Lu Jiang, Wolfgang Macherey, and Jacob Eisenstein. 2020. AdvAug: Robust adversarial augmentation for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5961-5970, Online. Association for Computational Linguistics.
Xiang Dai and Heike Adel. 2020. An analysis of simple data augmentation for named entity recognition. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3861-3867, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Mathieu Dehouck and Carlos Gomez-Rodriguez. 2020. Data augmentation via subtree swapping for dependency parsing of low-resource languages. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3818-3830, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kru-engkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si, and Chunyan Miao. 2020. DAGA: Data augmentation with a generation approach for low-resource tagging tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6045-6057, Online. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 567-573, Vancouver, Canada. Association for Computational Linguistics.
Steven Y. Feng, Varun Gangal, Dongyeop Kang, Teruko Mitamura, and Eduard Hovy. 2020. GenAug: Data augmentation for finetuning text generators. In Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 29-42, Online. Association for Computational Linguistics.
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 968-988, Online. Association for Computational Linguistics.
Demi Guo, Yoon Kim, and Alexander Rush. 2020. Sequence-level mixed sample data augmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5547-5552, Online. Association for Computational Linguistics.
Hongyu Guo, Yongyi Mao, and Richong Zhang. 2019. Augmenting data with mixup for sentence classification: An empirical study.
Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. 2020. Revisiting self-training for neural sequence generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Jonathan Herzig and Jonathan Berant. 2021. Span-based semantic parsing for compositional generalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 908-921, Online. Association for Computational Linguistics.
Wei-Ning Hsu, Hao Tang, and James R. Glass. 2018. Unsupervised adaptation with interpretable disentangled representations for distant conversational speech recognition. ArXiv, abs/1806.04872.
Wei-Ning Hsu, Yu Zhang, and James Glass. 2017. Unsupervised domain adaptation for robust speech recognition via variational autoencoder-based data augmentation. 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).
Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12-22, Berlin, Germany. Association for Computational Linguistics.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and efficient fine-tuning for pretrained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2177-2190, Online. Association for Computational Linguistics.
Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021. Aeda: An easier data augmentation technique for text classification.
Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. 2021. Co-mixup: Saliency guided joint mixup with supermodular diversity.
Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song, 2020. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5275-5285. PMLR.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 452-457, New Orleans, Louisiana. Association for Computational Linguistics.
Oleksandr Kolomiyets, Steven Bethard, and Marie-Francine Moens. 2011. Model-portability experiments for textual temporal analysis. In Proceedings of the 49th Annual Meeting of the Association for
Computational Linguistics: Human Language Technologies, pages 271-276, Portland, Oregon, USA. Association for Computational Linguistics.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained transformer models. In Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems, pages 18-26, Suzhou, China. Association for Computational Linguistics.
Brenden M. Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2879-2888. PMLR.
Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Jiancheng Lv, Nan Duan, and Ming Zhou. 2020a. Tell me how to ask again: Question data augmentation with controllable rewriting in continuous space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5798-5810, Online. Association for Computational Linguistics.
Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. 2020b. Data boost: Text data augmentation through reinforcement learning guided conditional generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9031-9041, Online. Association for Computational Linguistics.
Nikolaos Malandrakis, Minmin Shen, Anuj Goyal, Shuyang Gao, Abhishek Sethi, and Angeliki Metallinou. 2019. Controlled text generation for data augmentation in intelligent artificial agents. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 90-98, Hong Kong. Association for Computational Linguistics.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
Zhengjie Miao, Yuliang Li, Xiaolan Wang, and Wang-Chiew Tan. 2020. Snippext: Semi-supervised opinion mining with augmented data. In WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 617-628. ACM / IW3C2.
Takeru Miyato, Shin-Ichi Maeda, Masanori Koyama, and Shin Ishii. 2019. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 41(8):1979-1993.
Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. 2020. SSMBA: Self-supervised manifold based data augmentation for improving out-of-domain robustness. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1268-1283, Online. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Dinghan Shen, Ming Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. 2020. A simple but tough-to-beat data augmentation approach for natural language understanding and generation. ArXiv, abs/2009.13818.
Haoyue Shi, Karen Livescu, and Kevin Gimpel. 2020. On the role of supervision in unsupervised constituency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7611-7621, Online. Association for Computational Linguistics.
Haoyue Shi, Karen Livescu, and Kevin Gimpel. 2021. Substructure substitution: Structured data augmentation for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3494-3508, Online. Association for Computational Linguistics.
Haoyue Shi, Hao Zhou, Jiaze Chen, and Lei Li. 2018. On tree-based neural sentence modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4631-4641, Brussels, Belgium. Association for Computational Linguistics.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps.
A. F. M. Shahab Uddin, Mst. Sirazam Monira, Wheemyung Shin, TaeChoong Chung, and Sung-Ho Bae. 2021. Saliencymix: A saliency guided data augmentation strategy for better regularization.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
William Yang Wang and Diyi Yang. 2015. That's so annoying!!!: A lexical and frame-semantic embedding