SlowGuess/NAACL_2022_OCR huggingface dataset:
based data augmentation approach to automatic categorization of annoying behaviors using #petpeeve tweets. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2557-2563, Lisbon, Portugal. Association for Computational Linguistics.
Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. SwitchOut: an efficient data augmentation algorithm for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 856-861, Brussels, Belgium. Association for Computational Linguistics.
Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6382-6388, Hong Kong, China. Association for Computational Linguistics.
Zhanghao Wu, Shuai Wang, Yanmin Qian, and Kai Yu. 2019. Data augmentation using variational autoencoder for embedding based speaker verification. In INTERSPEECH.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. 2020. Unsupervised data augmentation for consistency training.
Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y. Ng. 2017. Data noising as smoothing in neural network language models. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyeong Park. 2021. Gpt3mix: Leveraging large-scale language models for text augmentation.
Soyoung Yoon, Gyuwan Kim, and Kyumin Park. 2021. SSMix: Saliency-based span mixup for text classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3225-3234, Online. Association for Computational Linguistics.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. *Qanet: Combining local convolution with global self-attention* for reading comprehension. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. 2019. Cutmix: Regularization strategy to train
strong classifiers with localizable features. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 6022-6031. IEEE.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for natural language understanding. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
A Augmentation examples
Original Sentence1	Original Sentence2	New Sentence
a love story and a murder mystery that expands into a meditation on the deep deceptions of innocence [1]	really an advantage to invest such subtlety and warmth in an animatronic bear when the humans are acting like puppets [0]	a love story and are acting like puppets that expands into a meditation on the deep deceptions of innocence [0.21 0.79]
the attempt to build up a pressure cooker of horrified awe [0]	had the ability to mesmerize , astonish and entertain [1]	the attempt to build up the ability of horrified awe [0.8 0.2]
rest contentedly with the knowledge that he ’s made at least one damn fine horror movie [1]	minor film [0]	rest contentedly with the knowledge that he ’s made minor film damn fine horror movie [0.13 0.87]
might just be better suited to a night in the living room than a night at the movies [0]	are made for each other . [1]	might just be better suited to each other room than a night at the movies [0.86 0.14]
is a touching reflection on aging , suffering and the prospect of death [1]	keep upping the ante on each other [1]	is each other on aging , suffering and the prospect of death [0 1]
is dark , brooding and slow , and takes its central idea waytoo seriously [0]	merely pretentious [0]	is dark , brooding and slow , and takes merely pretentious too seriously [1. 0.]
Table A.1: Examples of TreeMix on SST2 datasets, the number following sentence is label, bold tokens are selected phrase for substitution
B The necessity of merged loss techniques
We provide a detailed discussion of the techniques proposed in 3.2. We first investigate the noise contained in the augmentation dataset, then we figure out how the unbalance dataset will affect the performance. In the second part, we vary the weight parameter  $\gamma$  to see how it affects the model's learning process.
B.1 Noise and Unbalance
All mixup methods, as previously stated, introduce noise into the dataset. This noise in the text includes grammatical structure confusion and multiple semantic meanings in the sentences. The model will be overwhelmed by the noise if trained solely on the generated augmentation dataset, and will even perform worse than the baseline. In terms of the unbalance problem, we find that training the model without replicating the original dataset to the same size as the augmentation dataset hurts the model's performance. The results are shown in the table B.1.
SST2	TREC-f	TREC-c	IMDb	AG NEWS	MRPC	RTE	QNLI	QQP	MNLI
BERT	92.96	92.36	97.08	93.63	94.67	84.9	68.15	90.54	90.67	84.27
Merged Loss	93.92	93.2	97.95	94.34	94.72	85.34	70.63	91.36	90.88	84.45
Augmentation only	92.57	90.44	96.42	92.37	93.98	83.93	65.45	88.72	89.24	83.78
No Replicate	93.05	92.42	97.21	93.7	94.65	85.02	69.56	91.04	90.72	84.35
Table B.1: Merged Loss indicates results following techniques in 3.2, Augmentation indicates the model is trained on the generated dataset alone. No Replicate indicates Merged Loss without replication of the original training set.
B.2 Weight parameter
We vary weight parameter  $\gamma$  to find optimal balance point between diversity and linguistic grammar, the results are shown in figure 4. Performance on the two classification tasks follows a similar pattern. Both increase with increasing weight and then rapidly decrease with increasing weight after reaching the highest point. Performance is weaker than the baseline when the weight value exceeds 0.7. We find the model achieves the best performance with  $\gamma \in \{0.2, 0.5\}$ . For single sentence classification tasks, when  $\gamma = 0.5$  the model always gets higher accuracy, and  $\gamma = 0.2$  is better for these sentence relation classification datasets.
Figure 4: The performance when varying the value of the weight parameter on single sentence classification (left) and sentence relation classification (right)
Datasets	epoch	batch size	aug batch size	val steps	sequence length	aug weight
SST2	5	96	96	100	128	0.5
TREC-f	20	96	96	100	128	0.5
TREC-c	20	96	96	100	128	0.5
IMDb	5	8	8	500	512	0.5
AGNEWS	5	96	96	500	128	0.5
MRPC	10	32	32	100	128	0.2
RTE	5	32	32	50	128	-0.2
QNLI	5	96	96	100	128	0.2
QQP	5	96	96	300	128	0.2
MNLI	5	96	96	100	128	0.2
Table C.1: Best settings for different datasets
C Hyper-parameters for each datasets
We explore different parameter combinations and find the best ones for each task, as in Tab C.1. There are some exceptions, such as TREC datasets, where the model cannot converge even with 10 epochs, so we increase the training epochs to 20 for this dataset. IMDb's examples are extremely long, with an average length of more than 200 words. Along with this change, we increased the truncation length to 512 and the batch size to 8 to fully capture the semantic meaning. RTE is the most unusual. First, when we train using original RTE datasets, the accuracy deviation is really substantial, reaching up to  $4\%$ . Second, we find that  $\gamma = -0.2$  is optimum for this set, which contradicts previous findings.
Shi et al. (2021) has proposed a similar study that uses constituency information for mixup. There are a few significant differences between our approaches. To begin with, their method is too restricted; they only perform mixup between examples from the same category, and they require the substituted subtree's label to be the same. Second, because they are limited to the same class examples, they are unable to devise a method for adding a soft label to the example. Instead, we only use TreeMix in the previous settings with the length constraint. Several other constraints in the subtree selection process are investigated in this section, and we achieve better performance than Shi et al. (2021) by giving the subtree selection process more freedom, and we validate that their work is a special case of our method by examining how other constraints affect the performance. This section's values are the averages of five runs with seeds ranging from 0 to 4
D.1 What is the difference between different amounts of data?
TreeMix has the potential to generate an infinite amount of augmented data in theory. However, due to TreeMix's principle, it can only improve performance to a point when the size of the augmentation data
set reaches a certain limit. We investigated how many augmentation datasets the model needs. Table D.1 shows the results of producing twice and five times the augmentation data for experiments.
Dataset	Size	BERT	TM(x2)	TM(x5)
RTE	3.5k	68.15	70.57	70.62
MRPC	3.7k	84.90	85.22	85.37
TREC-f	5.5k	92.36	93.2	92.85
TREC-c	5.5k	97.08	97.71	97.95
IMDb	12.5k	93.63	94.34	94.24
SST2	67k	92.96	93.92	93.92
QNLI	105k	90.54	91.36	91.34
AGNEWS	120k	94.67	94.71	94.69
QQP	364k	90.67	90.88	90.83
MNLI	393k	84.27	84.45	84.41
Table D.1: Improvement of performance on all datasets with different amount of augmentation datasets,  $TM(x2)$  indicates generating twice as much augmentation data than the original data,  $TM(x5)$  indicates five times than original data, datasets in the table are listed in order of size
The key to getting the best results is to strike a balance between the original datasets and the augmentation datasets in terms of diversity and linguistic confusion. With more augmentation datasets, the model will learn more patterns while also observing more grammatically poor samples, which could negatively impact performance. We discovered that augmentation datasets twice the size of the original dataset produce the best results for larger datasets. This is in line with our previous theoretical analysis: large datasets inherently include more patterns and diversity, which helps the model generalize better. Maintaining the original linguistic grammar while increasing diversity in these datasets is, therefore, more important. When working with smaller datasets, it's better to train with more augmentation data. For models to train on these datasets, we believe diversity is more important than linguistic grammar.
TREC-fine is an exception. We attribute it to the datasets' excessive classes (up to 47 classes within only  $5.5\mathrm{k}$  training samples): each class has a very limited number of samples, and if we create overly augmented dataset samples, the limited samples of each category are insufficient to resist injected linguistic noise. As a result, for TREC-fine, x2 is preferable to x5. For a smaller dataset, we recommend generating five times as much augmentation data as possible, and for a larger dataset, we recommend generating twice as much augmentation data.
D.2 Is it beneficial to keep the swapped subtree's label or length the same?
Figure 5: Performance on SST2 with different subtree selection constraints, green part is bert performance, orange part is improvement of TreeMix when applying different constraints, TreeMix.label) indicates only select subtrees with same phrase label, TreeMix(length) indicates only select subtrees with same length. TreeMix indicates without any constraints
Each subtree has its own label (e.g., VP and NP) and corresponds to a specific text span. When selecting subtrees, we can use these characteristics as additional constraints. Figure 5 shows the results.
When we impose restrictions on the subtree selection process, the experimental results clearly show that performance suffers.
We hypothesize that this is because in datasets with similar sentence lengths, subtrees of the same phrase label or phrase length tend to have similar structures (e.g., tree height, relative position in the sentence). Although the exchange of such subtrees can retain the original linguistic grammar of the text to some extent (e.g., replacing a noun phrase with another noun phrase will not significantly disrupt the sentence) and maintain similar sentence length, it cannot exploit the potential compositional diversity in the datasets as efficiently as TreeMix without any constraints, resulting in lower diversity augmentation datasets and limited improvement compared to the baseline. In terms of the comparison of TreeMix(length) and TreeMix(length), we find that TreeMix(length) prefers simple phrases such as NP and VP because these are the most common phrases occurring in sentences, and this exchange will not improve the diversity of the datasets. For example, in "I like this apple," replacing "apple" with "orange" will not provide innovative text patterns.