Venue: NAACL 2022

# TreeMix: Compositional Constituency-based Data Augmentation for

Natural Language Understanding

Le Zhang

Fudan University

zhangle18@fudan.edu.cn &Zichao Yang

CMU

yangtze2301@gmail.com &Diyi Yang

Georgia Tech

dyang888@gatech.edu

###### Abstract

Data augmentation is an effective approach to tackle over-fitting. Many previous works have proposed different data augmentations strategies for NLP, such as noise injection, word replacement, back-translation etc. Though effective, they missed one important characteristic of language-compositionality, meaning of a complex expression is built from its sub-parts. Motivated by this, we propose a compositional data augmentation approach for natural language understanding called TreeMix. Specifically, TreeMix leverages constituency parsing tree to decompose sentences into constituent sub-structures and the Mixup data augmentation technique to recombine them to generate new sentences. Compared with previous approaches, TreeMix introduces greater diversity to the samples generated and encourages models to learn compositionality of NLP data. Extensive experiments on text classification and SCAN demonstrate that TreeMix outperforms current state-of-the-art data augmentation methods.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5243 - 5258

## 1 Introduction

Data augmentation (DA) has won great popularity in natural language processing (NLP) Chen et al. (2021); Feng et al. (2021) due to the increasing demand for data and the expensive cost for annotation. DA aims at increasing the quantity and diversity of the datasets by generating more samples based on existing ones, which helps make the training process more consistent and improves the model's capacity for generalization Xie et al. (2020). For instance, existing DA methods often leverage word-level manipulation Wei and Zou (2019); Kobayashi (2018); Karimi et al. (2021) and model-based sentence generation Edunov et al. (2018); Ng et al. (2020). As mixup-based Zhang et al. (2018) augmentation achieving huge success in computer vision Yun et al. (2019); Uddin et al. (2021); Kim et al. (2021), some recent works start to adapt mixup to NLP, such as at the hidden level Guo et al. (2019); Chen et al. (2020) and at the input level Yoon et al. (2021); Shi et al. (2021).

Despite these empirical success, DA methods still suffer from key limitations. Simple rules based augmentation methods Wei and Zou (2019); Kobayashi (2018); Karimi et al. (2021) show little to none effect over large pretrained language models. While mixup-based augmentation methods demonstrate huge potential, such interpolation at the hidden or input level has limited capability to capture explicit linguistic properties in text Guo et al. (2019); Chen et al. (2020); Yoon et al. (2021). Moreover, current DA methods exhibit limited ability in compositional generalization. Take a look at the following example from a BERT-based model that is fine-tuned using the SST2 dataset from the

\begin{table}
\begin{tabular}{|c|c|} \hline S1:They will find little interest in this poor film. \\ \hline \multicolumn{2}{|c|}{S2:It comes as a touching love story.} \\ \hline
**Method** & **Example** \\ \hline EDA & They will **this** find little \\ Wei and Zou (2019) & interest in **bad movie.** \\ \hline AEDA & They will find ?! little in \\ Karimi et al. (2021) & **!** this poor movie¿, \\ \hline Noise & **Thes will \#** little \\ Xie et al. (2017) & **intres** this poor film . \\ \hline SSMIX & They will find little interest \\ Yoon et al. (2021) & in **love** poor film \\ \hline Replacement & They will find **limited** interest \\ Kolomiyets et al. (2011) & in this **odd** film. \\ \hline Back Translation & They will **show** little interest \\ Eduunov et al. (2018) & in this **strange** film. \\ \hline TreeMix & They will find little interest \\  & in this **touching love story**. \\ \hline \end{tabular}
\end{table}
Table 1: Input-level DAs for Text-Classification. EDA includes random deletion, swapping, and insertion. AEDA randomly inserts punctuation. SSMix swaps tokens based on their saliency. The replacement method randomly substitutes words with synonyms. In Back-translation, the source sentences are first translated into another language, and then back again.



GLUE Benchmark:

\begin{tabular}{l c} \hline This film is good and everyone loves it. & 99\% \\ \hline This film is poor and I do not like it. & 99\% \\ \hline This film is good and I do not like it. & 99\% \\ \hline \end{tabular} The first two examples are correctly classified. Despite that the last one is composed of fragments from the first two, the model fails to produce a correct or plausible label (in terms of characterizing a sentence's sentiment), demonstrating poor performance in compositional generalization.

However, compositionality is one key aspect of language that the meaning of a complex sentence is built from its subparts. Prior work also shows that syntax trees (e.g., tree-based LSTMs) are helpful to model sentence structures for better text classification Shi et al. (2018). However, leveraging compositional structures for data augmentation has not received much attention in the language technologies communities, with a few exceptions in semantic parsing Andreas (2020); Herzig and Berant (2021).

To this end, we propose a compositional data augmentation method for natural language understanding, i.e., TreeMix (Figure 1). TreeMix is an input-level mixup method that utilizes constituency parsing information, where different fragments (phrase of a subtree) from different sentences are recombined to create new examples that were never seen in the training set; new soft labels will also be strategically created based on these fragments at the same time. In this way, TreeMix not only exploits compositional linguistic features to increase the diversity of the augmentation, but also provides reasonable soft labels for these mixed examples.

Empirically, we find that TreeMix outperforms existing data augmentation methods significantly on a set of widely used text classification benchmarks. To validate the compositional effectiveness of TreeMix, we experiment with SCAN Lake and Baroni (2018)--a task requires strong compositional generalization, and find that TreeMix exhibits reasonable ability to generalize to new structures built of components observed during training.

## 2 Related Work

### Generic Data Augmentation

Most prior work operates data augmentation at different levels Chen et al. (2021). **Token-level** DA methods manipulate tokens or phrases while preserving syntax and semantic meaning as well as labels of the original text, such as synonym words substitutions Wang and Yang (2015); Zhang et al. (2015); Fadaee et al. (2017); Kobayashi (2018); Miao et al. (2020) where synonyms are detected following pre-defined rules or by word embedding similarities. These methods has limited improvement Chen et al. (2021) over large pretrained language models (PLMs). Besides, introducing noise by random insertion, replacement, deletion, and swapping Wang et al. (2018); Wei and Zou (2019); Karimi et al. (2021); Xie et al. (2020) is expected to improve the robustness of the model. **Sentence-Level** DA methods increase the diversity by generating distinct examples, such as via paraphrasing Yu et al. (2018); He et al. (2020); Xie et al. (2020); Kumar et al. (2020); Chen et al. (2020b); Cai et al. (2020) or back translation Sennrich et al. (2016); Edunov et al. (2018). Other line of work used label-conditioned generation methods that train a conditional generation model such as GPT-2 or VAE to create new examples given labels as conditions Bergmanis et al. (2017); Liu et al. (2020); Ma et al. (2020); Ding et al. (2020); Anaby-Tavor et al. (2020). Although these methods can produce novel and diverse text patterns that do not exist in the original datasets, they require extensive training. **Hidden-Level** DA methods mainly manipulate hidden representations by perturbation Miyato et al. (2019); Zhu et al. (2020); Jiang et al. (2020); Chen et al. (2020c); Shen et al. (2020); Hsu et al. (2017, 2018); Wu et al. (2019); Malandrakis et al. (2019) and interpolation like mixup Zhang et al. (2018) to generates new examples Miao et al. (2020); Cheng et al. (2020); Chen et al. (2020b); Guo et al. (2019, 2020); Chen et al. (2020a).

### Compositional Data Augmentation

Compositional augmentation aims at increasing the diversity of the datasets and improving the compositional generalization capability of the resulting models Jia and Liang (2016); Andreas (2020). These methods often recombine different components from different sentences to create new examples following a set of pre-designed linguistic rules such as lexical overlaps Andreas (2020), neural-symbolic stack machines Chen et al. (2020d), and substructure substitution Shi et al. (2021). Compositional methods have been applied in a set of NLP tasks, such as sequence labeling Guo et al. (2020), semantic parsing Andreas (2020), constituency 

parsing (Shi et al., 2020, 2021), dependency parsing Dehouck and Gomez-Rodriguez (2020); Shi et al. (2021), named entity recognition Dai and Adel (2020), text generation Feng et al. (2020), and text classification Yoon et al. (2021); Shi et al. (2021). Our work also falls into this category.

The most relevant are Shi et al. (2021) and Yoon et al. (2021). However, Shi et al. (2021) only performs constituent substructure combinations with examples from the same category, thus inadequate in creating diverse enough augmentation with newly created labels.

Besides, Yoon et al. (2021) simply swaps the most and least salient spans, heavily relying on the model's performances in estimating salient spans, and failing to consider these sentences' linguistic structures. Our proposed TreeMix fills these gaps by allowing the composition of sentences from different label categories, by utilizing rich consistency based structures in text, and by strategically generating soft labels for these augmented instances.

## 3 Method

Our work is motivated by Mixup Zhang et al. (2018), which creates virtual samples by mixing inputs. Given two random drawn examples \(({\bf x}_{i},{\bf y}_{i})\) and \(({\bf x}_{j},{\bf y}_{j})\), where \({\bf x}\) denotes the input sample and \({\bf y}\) is the corresponding one-hot label, Mixup creates a new sample by:

\[\overline{{\bf x}}=\lambda{\bf x}_{i}+(1-\lambda){\bf x}_{j},\] \[\overline{{\bf y}}=\lambda{\bf y}_{i}+(1-\lambda){\bf y}_{j},\]

where \(\lambda\in[0,1]\). Mixup can be easily implemented in continuous space, hence some prior works Chen et al. (2020b) have extended it to NLP by performing interpolation in hidden space.

We improve upon Mixup by incorporating compositionality of language, a key characteristic that is essential to generalization but neural models often fall short in capturing Lake and Baroni (2018). Instead of interpolating with the whole sample, TreeMix, our newly proposed method, creates new sentences by removing phrases of sentences and reinserting subparts from other sentences. TreeMix makes use of constituency trees to decompose a sentence into meaningful constituent parts, which can then be removed and recombined to generate new augmentation samples. We aim to improve models' compositionality generalization ability by training on large amount of samples produced by TreeMix. An example of using TreeMix for single sentence classification is shown in Figure 1.

### TreeMix

Let \({\bf x}_{i}=\{x^{1}_{i},x^{2}_{i},...,x^{l}_{i}\}\) denotes a sequence with length \(l\) and its corresponding label in one-hot encoding as \({\bf y}_{i}\). We run a constituency parser on \({\bf x}_{i}\) to get its parsing tree as \(T({\bf x}_{i})\). In order to get meaningful subparts of a sequence, we traverse the parsing tree recursively and get all the subtrees with more than one child. Denote the collection of subtrees as \(S({\bf x}_{i})=\{t^{k}_{i}\}\), where \(t^{k}_{i}\) denotes the \(k\)-th subtree of sample \({\bf x}_{i}\). For a subtree \(t^{k}_{i}\), it covers a continuous span \(t^{k}_{i}\triangleq[x^{r_{k}}_{i},...,x^{s_{k}}_{i}]\) of \({\bf x}_{i}\) that starts with index \(r_{k}\) and ends with index \(s_{k}\). For example, as shown in the left part of Figure 1, the subtrees of the example sentence can cover spans such as this poor film, in this poor film, no interest etc.

Figure 1: Illustration of TreeMix for single sentence classification



For a given sample \(({\bf x}_{i},{\bf y}_{i})\), we randomly sample another data point \(({\bf x}_{j},{\bf y}_{j})\) from the training set. We run the constituency parser on both sentences and get their subtree sets \(S({\bf x}_{i})\) and \(S({\bf x}_{j})\), based on which we can sample subtrees to exchange. We introduce two additional hyper-parameters \(\lambda_{L}\) and \(\lambda_{U}\) to constraint the length of subtrees to sample. \(\lambda_{L}\) and \(\lambda_{U}\), measured in terms of length ratio of the subtree to the original sentences, sets the lower and upper limits of the subtrees to sample. Intuitively, \(\lambda\) controls the granularity of the phrases that we aim to exchange. We would like that the length of phrase to exchange to be reasonable. If it is too short, then the exchange cannot introduce enough diversity to the augmented sample; otherwise if it is too long, the process might inject too much noise to the original sentence. We set \(\lambda\) to be the ratio in order to be invariant to the length of original sentences. Table 2 shows some subtree examples with different length constraints. We define the length constrained subtree set as:

\[S_{\lambda}({\bf x})\triangleq\{t|t\in S({\bf x}),s.t..\frac{|t|}{|{\bf x}|} \in[\lambda_{L},\lambda_{U}]\}.\]

Here \(|.|\) denotes the length of a sequence or a subtree. For two sentences \({\bf x}_{i}\) and \({\bf x}_{j}\), we randomly sample two subtrees \(t^{k}_{i}\in S_{\lambda}({\bf x}_{i})\) and \(t^{l}_{j}\in S_{\lambda}({\bf x}_{j})\) and construct a new sample by replacing \(t^{k}_{i}\) with \(t^{l}_{j}\), i.e.

\[\bar{{\bf x}}\triangleq[x^{1}_{i},...,x^{r_{k}-1}_{i},\underbrace{x^{r_{l}}_{ j},...,x^{s_{l}}_{j}}_{t^{l}_{j}},x^{s_{k}+1}_{i},...x^{l}_{i}] \tag{1}\]

where \(t^{l}_{j}=[x^{r_{l}}_{j},...,x^{s_{l}}_{j}]\) replaces \(t^{k}_{i}=[x^{r_{k}}_{i},...,x^{s_{k}}_{i}]\). Figure. 1 shows an example of TreeMix, where the subtree a touching transcend love story replaces the subtree this poor film.

Label Creation for TreeMixCreating a valid label for the augmented sample \(\bar{{\bf x}}\) is a challenging problem. Similar to that of Mixup (Zhang et al., 2018), we use a convex combination of original labels of two sentences as the new label for the augmented sample:

\[\bar{{\bf y}}=\frac{l_{i}-|t^{k}_{i}|}{l_{i}-|t^{k}_{i}|+|t^{l}_{j}|}{\bf y}_{ i}+\frac{|t^{l}_{j}|}{l_{i}-|t^{k}_{i}|+|t^{l}_{j}|}{\bf y}_{j}, \tag{2}\]

where \(l_{i}\) is the length of \({\bf x}_{i}\) and \(|t^{k}_{i}|,|t^{k}_{j}|\) are the length of the subtrees. In the new sentence, \(l_{i}-|t^{k}_{i}|\) words from \({\bf x}_{i}\) are kept and \(|t^{l}_{j}|\) words from sentence \({\bf x}_{j}\) are inserted.

In Equation 2, \(\frac{l_{i}-|t^{k}_{i}|}{l_{i}-|t^{k}_{i}|+|t^{l}_{j}|}\) is the fraction of words that come from \({\bf x}_{i}\), which determines the weight of \({\bf y}_{i}\). The label is then created based on the conjecture that the change in labels is proportional to the length changes in the original sentences. We provided a set of augmentation examples from TreeMix in Table A.1 in Appendix.

Pairwise Sentence Classification TaskThe above mainly used single sentence classification as the running example for TreeMix. Here we argue that TreeMix can easily be extended to pairwise sentence classification problem, where the relationship between the sentences is the label.

Formally, for a given sample \(({\bf x}_{i},{\bf x}^{\prime}_{i},{\bf y}_{i})\), we randomly sample another sample \(({\bf x}_{j},{\bf x}^{\prime}_{j},{\bf y}_{j})\) and run the parser and get the subtree sets of each sentence \(S({\bf x}_{i}),S({\bf x}^{\prime}_{i})\) and \(S({\bf x}_{j}),S({\bf x}^{\prime}_{j})\). Then we randomly sample subtrees \(t^{k}_{i}\in S_{\lambda}({\bf x}_{i}),t^{k^{\prime}}_{i^{\prime}}\in S_{ \lambda}({\bf x}^{\prime}_{i})\) and \(t^{l}_{j}\in S_{\lambda}({\bf x}_{j}),t^{l^{\prime}}_{j^{\prime}}\in S_{ \lambda}({\bf x}^{\prime}_{j})\). We construct \(\bar{{\bf x}}\) by replacing \(t^{k}_{i}\) with \(t^{l}_{j}\) and \(\bar{{\bf x}}^{\prime}\) by replacing \(t^{k^{\prime}}_{i^{\prime}}\) with \(t^{l^{\prime}}_{j^{\prime}}\). The new label is created as:

\[\bar{{\bf y}}=\frac{l_{i}+l_{i^{\prime}}-|t^{k}_{i}|-|t^{k^{\prime}}_{i^{ \prime}}|}{l_{i}+l_{i^{\prime}}-|t^{k}_{i}|-|t^{k^{\prime}}_{i^{\prime}}|+|t^{ l}_{j}|+|t^{l^{\prime}}_{j^{\prime}}|}{\bf y}_{i}+\\ \frac{|t^{l}_{j}|+|t^{l^{\prime}}_{j^{\prime}}|}{l_{i}+l_{i^{ \prime}}-|t^{k}_{i}|-|t^{k^{\prime}}_{i^{\prime}}|+|t^{l}_{j}|+|t^{l^{\prime}}_{ j^{\prime}}|}{\bf y}_{j}.\]

The meanings of the notations are the same as in Equation 2.

\begin{table}
\begin{tabular}{|c|p{346.9pt}|} \hline They will find little interest in this poor film. \\ \hline \([\lambda_{L},\lambda_{U}]\) & **possible selected sub-trees** \\ \hline \([\)0.1,0.3\(]\) & (itttle interest),(this poor film) \\ \hline \([\)0.3,0.5\(]\) & (in this poor film) \\ \hline \([\)0.5,0.7\(]\) & (ittle interest in this poor film) \\ \hline \end{tabular}
\end{table}
Table 2: Examples of possible candidate subtrees with different \(\lambda\) intervals 

Our main algorithm is shown in Algorithm 1. Although not all sentences created by TreeMix are fluent or even valid new sentences, they contains subparts with different meanings that encourage the models to build rich representation of sentences in a compositional manner. Note that the augmented labels are convex combination of original labels, only when the model learns the representations of two parts together can they predict both labels with different weights.

### Training Objective

Our model is trained on a combination of the original samples and augmentation samples to obtain a trade-off between regularization and noise injection. The final training objective is:

\[{\cal L}= \mathop{\mathbb{E}}_{({\bf x},{\bf y})\sim D}\left[-{\bf y}^{ \intercal}\log P_{\theta}({\bf y}|{\bf x})\right]\] \[+\gamma\mathop{\mathbb{E}}_{(\bar{\mathbf{x}},\bar{\mathbf{y}}) \sim D^{\prime}}\left[-\bar{\mathbf{y}}^{\intercal}\log P_{\theta}(\bar{ \mathbf{y}}|\bar{\mathbf{x}})\right], \tag{4}\]

\(\gamma\) is the weight1 on the augmentation samples.

Footnote 1: Section B in Appendix presents discussions on how the objective and different weight parameter affects the result.

## 4 Experiment

### Datasets

To test TreeMix's effectiveness, we experiment with a variety of text classification benchmarks, as shown in Table 3. We use accuracy as a metric, and exclude datasets from GLUE Wang et al. (2019) that are not suitable for mixup, including CoLA that measures linguistic acceptability and will be ruined by mixup operations, and WNLI that is too small to show a method's validity.

### Experiment Setup

The proposed TreeMix method creates new samples by combining text spans based on the constituency tree's information, thus we use the Stanford CoreNLP toolkit2 to obtain parsing related information Manning et al. (2014). We use the pretrained language model _bert-base-uncased_ for sequence classification task from HuggingFace. With seeds ranging from 0 to 4 and \(\lambda_{L}=0.1,\lambda_{U}=0.3\), we use TreeMix to generate twice and five times more samples than the original training set3. We replicate the original dataset to the same size as the augmentation datasets in the training stage to ensure that the model receives the same amount of data from the original dataset and the augmentation dataset for each training batch.

Footnote 2: The specific version is 3.9.2

Footnote 3: Section D.1 in Appendix presents robustness check on how different amount of augmented data affects the result.

If not specified, we train the model for 5 epochs, with a maximum sequence length of 128 and batch size of 96. The model is optimized using the AdamW optimizer with an eps of 1e-8 and a learning rate of 2e-5. Table C.1 in Appendix contains detailed hyper-parameter settings for each dataset.

### Baseline

We compare TreeMix with the following benchmarks: (1) No augmentation (BERT): standard training without any augmentation, (2) EDA that randomly performs insertion, replacement, swap and deletion to the text. (3) AEDA that randomly inserts punctuation to the text. (4) Back translation(BT) Edunov et al. (2018): texts are translated between English and German using Transformer architectures trained on WMT16 English-German. (5) GPT3MixYoo et al. (2021) designs prompts and utilizes GPT3 to generate new examples to train the model. (6) SSMix Yoon et al. (2021) applies mixup based on the saliency Simonyan et al. (2014) of tokens, similar to PuzzleMix Kim et al. (2020) and SaliencyMix Uddin et al. (2021). (7) EmbedMix is the pretrained-language-model version of WordMixup in Guo et al. (2019), which performs mixup on the embedding level. (8) TMix Chen et al. (2020) first encodes two inputs separately, then performs the linear interpolation of two embeddings at a certain encoder layer, and fi

\begin{table}
\begin{tabular}{l c r r} \hline \hline Dataset & Task & Class & Size \\ \hline \multicolumn{4}{c}{**Single Sentence Classification**} \\ \hline SST2 & Sentiment & 2 & 67k/1.8k \\ TREC-fine & Question & 47 & 5.5k/500 \\ TREC-coarse & Question & 6 & 5.5k/500 \\ AG\_NEWS & News & 4 & 12k/4k \\ IMDb & Sentiment & 2 & 12.5k/12.5k \\ \hline \multicolumn{4}{c}{**Pair Sentence Classification**} \\ \hline RTE & NLI & 2 & 3.5k/300 \\ MRPC & Paraphrase & 2 & 3.7k/400 \\ QNLI & QA & 2 & 105k/5.5k \\ QQP & Paraphrase & 2 & 364k/40.4k \\ MNLI & NLI & 3 & 393k/9.8k \\ \hline \hline \end{tabular}
\end{table}
Table 3: Dataset name or split name, task category and number of label class, Size used for training and testing. For tasks from GLUE, Size indicates (#train:#validation); for TREC, AG_NEWS and IMDb, Size indicates (#train:#test).

 

nally forward-passes the combined embedding in the remaining layers.

## 5 Results and Analysis

### Performance On Full Dataset

The results of TreeMix on the entire datasets are shown in Table 4. TreeMix outperforms all baselines significantly on single sentence classification tasks, demonstrating the superiority of using compositional substructure for substitution and augmentation. For instance, On SST2, it improves by 0.98%. Compared to other methods, the improvement was more than doubled.

This is because that, unlike SSMix which substitutes the text spans based on the saliency, our TreeMix makes use of the constituency information to help identify linguistically informed sentence substructures, and by recombining these components, the compositional diversity of the datasets can be maximized. With our TreeMix generated samples, the model can see more combinations of the substructures in the training stage that aren't available in the original corpus, leading to better generalization ability.

When it comes to sentence relationship classification tasks, TreeMix is also very effective. For example, It improves by 2.47% on the RTE data set, whereas the best improvement of other methods is only 0.3%, and it improves by 0.82% on QNLI, where other data augmentation methods have little effect. We hypothesized that, when two constituent parts from one sentence pair are embedded into another sentence pair, the inherent relationship is also embedded. This better helps the models on how to to identify two pairs of relationships in a single sample, which further increases its capacity to categorize these challenging adversarial sentences. Since TreeMix works by increasing dataset diversity and providing models with more text patterns to learn, it has very significant improvements for these relatively small datasets such as RTE and TREC, compared to these large datasets such as AG NEWS,QQP and MNLI that already have a lot of diversity and text patterns.

### Influence of Constituency Information

To determine the importance of constituency information, we designed a Random Mixup (RandMix) that randomly selects text spans as long as the ratio of span length to sentence length is less than a particular threshold \(\lambda_{rand}\)4. The rest setting of RandMix is the same as TreeMix. We compare TreeMix and RandMix on single sentence classification datasets in Figure 2.

Footnote 4: We observed \(\lambda_{rand}\sim\mathcal{U}(0,0.3)\) is optimal and we use this settings for the experiment

We found that, both RandMix and TreeMix are quite effective, but TreeMix outperforms RandMix on most datasets. For instance, TreeMix exceeds RandMix by 0.8% on SST2, 0.6% on TREC-f, and 0.5% on TREC-c. One exception is on IMDb, where the average sentence length is much longer. The reason for the poorer performance of TreeMix is due to the sparse parsing results on long sentences; since there are many subtrees, substituting any single part might bring very minimal change to the entire sentence.

### Influence of Training Set Size

To examine the influence of TreeMix with different training set sizes, we uniformly sample 1%, 2%, 5%, 10%, and 20% of the data from the training set to investigate TreeMix in low-resource situations. The entire test set is used to evaluate the model's generalization ability. Since TreeMix generates more examples for training, we use RandMix to generate the same number of extra samples as a comparison to ensure the data size is fair. The results are summarized in Figure 3.

We found that, (1) TreeMix outperforms RandMix in all settings, further demonstrating the advantage of the compositional substructure with the constituency information over the randomly selected spans. (2) Both mixup methods can significantly improve the model's performance in the case of extreme data scarcity (e.g, 1% and 2%). (3) When the amount of data is sufficient (e.g, more than

Figure 2: Performance of RandMix and TreeMix on single sentence classification datasets, scores are averaged over 5 random seeds.

 

5%), TreeMix outperforms RandMix by a significant margin. However, TreeMix only slightly outperforms RandMix when there is a severe lack of data (e.g, 1% and 2%). This is due to that the too small datasets often contain very limited structures, thus constraining TreeMix's ability to increase text patterns and compositional diversity. (4) The relative improvement of TreeMix over conventional training without augmentation diminishes as the amount of data increases, largely due to that additional augmented text patterns might overlap with those already existing in the dataset, resulting in limited improvement.

### Influence of Cross-Category Mixing

Different from prior work Shi et al. (2021), TreeMix allows the composition of sentences from different label categories. To test whether this cross-label category mixup is more effective than a within-label category mixup, we conducted ablation studies with TreeMix on samples in the same class6. Table 5 shows the results. Across all datasets, we found that TreeMix that combines data from different classes is more effective than combining data from the same class, consistent with findings in Zhang et al. (2018). When given only labels from one category, current models have a tendency to make simple or spurious judgments based on the most frequently occurring words. However the semantics of the sentence are complicated beyond simple words. For example, the model is likely to classify a sentence like _"I like this good movie"_ as positive because of the words _"like"_ and _"good"_, but if _"good movie"_ is replaced with _"bad

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{3}{*}{Model} & \multicolumn{7}{c}{Single Sentence Classification} & \multicolumn{7}{c}{Pair Sentence Classification} \\ \cline{2-9}  & SST2 & TREC-f & TREC-c & IMDb & AG NEWS & MRPC & RTE & QNLI & QQP & MNLI \\ \hline BERT & 92.96\({}^{\dagger}\) & 92.36 & 97.08\({}^{\dagger}\) & 93.63 & 94.67 & 84.90 & 68.15 & 90.54 & 90.67 & 84.27\({}^{\dagger}\) \\ BERT+EDA & 92.20\({}^{\dagger}\) & 91.95 & 96.79\({}^{\dagger}\) & 93.62 & 94.67 & - & - & - & - & - \\ BERT+AEDA & 92.57\({}^{\dagger}\) & 92.15 & 97.20\({}^{\dagger}\) & 93.59 & 94.22 & - & - & - & - & - \\ BERT+PI & 92.48 & 92.15 & 96.68 & - & - & 82.13 & 67.40 & - & - & - \\ BERT+GPT3Mix & 93.25\({}^{\dagger}\) & - & - & - & - & - & - & - & - & - \\ BERT+SSMix & 93.14\({}^{\dagger}\) & 92.80 & 97.60\({}^{\dagger}\) & 93.74 & 94.64 & 84.31 & 68.40 & 90.60 & 90.75 & **84.54\({}^{\dagger}\)** \\ BERT+EmbedMix & 93.03\({}^{\dagger}\) & 92.32 & 97.44\({}^{\dagger}\) & 93.72 & **94.72** & 85.34 & 68.37 & 90.44 & 90.58 & 84.35\({}^{\dagger}\) \\ BERT+TMix & 93.03\({}^{\dagger}\) & 92.68 & 97.52\({}^{\dagger}\) & 93.69 & 94.69 & **85.69** & 68.45 & 90.48 & 90.66 & 84.30\({}^{\dagger}\) \\ \hline BERT+TreeMix & **93.92** & **93.20** & **97.95** & **94.34** & **94.72** & 85.34 & **70.62** & **91.36** & **90.88** & 84.45 \\ \hline \hline \end{tabular} \({}^{\dagger}\) denotes the result is extracted from the original paper

\end{table}
Table 4: Results of comparison with baseline on full datasets, TREC-f and TREC-c indicates TREC-fine and TREC-coarse respectively. Scores are averaged over 5 random seeds. For GLUE tasks, we report accuracy of validation sets, and for other datasets we report test accuracy. EDA and AEDA will seriously damage the sentence relationship and harm the accuracy; GPT3Mix only reports full data experiments results on SST2 in original paper. We only report the results of back translation on small dataset due to the heavy computational cost.

Figure 3: Results on SST2 varying data size. Scores are averaged over 5 random seeds.

 

film"_, the model must perceive the different constituent parts within the sentence. This ability can only be obtained when the model is trained on the cross-category generated samples.

### Influence of Length Ratio

The only constraint we impose on TreeMix is the length ratio of the subtree controlled by \(\lambda\). We select subtrees that are between 10% and %30 and between 30% and 50% of the length of the sentence, respectively. Table 6 shows the results.

On all datasets, \(\lambda=[0.1,0.3]\) outperforms \(\lambda=[0.3,0.5]\), which is in line with Zhang et al. (2018)'s observation that giving too high mixup ration values can lead to underfitting. Another linguistic explanation for the scenario follows: When \(\lambda=[0.3,0.5]\), TreeMix may select longer text spans, which usually contain unique constituency components like _SBAR_; The exchange of these spans will severely damage the sentence's semantic and grammatical structure, causing the model to become confused. As a result, TreeMix with larger switching spans performs poorly, and even worse than baseline on some datasets.

### Compositional Generalization

To quantify TreeMix's overall ability of compositional generalization beyond classification tasks, we conducted experiments on SCAN Lake and Baroni (2018) dataset, which is a command execution dataset widely used to test for systematic compositionality. It contains simple source commands and target action sequences. We test on commonly used challenging splits: _addprim-jump, addprim-turn-left, around-right_, where primitive commands (e.g "_jump_") only appear alone during training but will be combined with other modifiers (e.g "_jump twice_") during testing. A model that works well for this task should learn to compose the primitive commands with the modifiers and generates corresponding execution. With TreeMix, we can generate the compositional commands that are not seen in the training set.

The new command generation process is the same as in single sentence classification, except that we increase the length constraint \(\lambda_{U}\) to 1 to allow the exchange of the commands with only one word. After we synthesize new commands, we follow the rules in Lake and Baroni (2018) to translate valid commands into actions and filter out ungrammatical commands. We follow the settings in Andreas (2020) and use the following data augmentation methods as baselines: (1) WordDrop that drops words randomly; (2) SwitchOut Wang et al. (2018) that randomly replaces words with other random words from the same vocabulary; (3) SeqMix Guo et al. (2020) which creates new synthetic examples by softly combining in-put/output sequences from the training set, and (4) GECA Andreas (2020) that performs enumerated valid swaps.

As shown in Table 7, TreeMix outperforms SwitchOut and WordDrop for all splits. TreeMix by itself does not perform as well as GECA, but when being combined with GECA, it demonstrates very strong results. TreeMix outperforms SeqMix in all splits, due to the fact that TreeMix can more precisely find the linguistically rich compositional segments of a sentence, as evidenced by the results of the comparisons of TreeMix and SSMix in Section 5.1 and TreeMix and RandMix in Section 5.3. A closer look at these augmented samples show that TreeMix can generate all possible combinations of "_jump_" and other modifiers like "_lift_"

\begin{table}
\begin{tabular}{l c c c} \hline \hline Dataset & BERT & \(\lambda=\)[0.1,0.3] & \(\lambda=\)[0.3,0.5] \\ \hline SST2 & 92.96 & **93.92** & 93.05 \\ TREC-fine & 92.36 & **93.2** & 92.25 \\ TREC-coarse & 97.08 & **97.95** & 96.94 \\ IMDb & 93.63 & **94.34** & 93.29 \\ AG NEWS & 94.67 & **94.72** & 94.53 \\ MRPC & 84.90 & **85.34** & 84.93 \\ RTE & 68.15 & **70.62** & 70.35 \\ QNLI & 90.54 & **91.36** & 90.78 \\ QQP & 90.67 & **90.88** & 90.54 \\ MNLI & 84.27 & **84.45** & 83.78 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Performance with different length ratio intervals \(\lambda\)

\begin{table}
\begin{tabular}{l r l c} \hline \hline Method & JUMP & TURN-L & ABOUND-R \\ \hline Baseline & 0\({}^{\dagger}\) & 49\% \({}^{\dagger}\) & 0\({}^{\dagger}\) \\ WordDrop & 0\({}^{\dagger}\) & 51\% \({}^{\dagger}\) & 0\({}^{\dagger}\) \\ SwitchOut & 0\({}^{\dagger}\) & 16\% \({}^{\dagger}\) & 0\({}^{\dagger}\) \\ SeqMix & 49\%\({}^{\dagger}\) & 99\% \({}^{\dagger}\) & 0\({}^{\dagger}\) \\ TreeMix & **72\%** & **99\%** & **0\%** \\ \hline GECA & 87\%\({}^{\dagger}\) & - & 82\({}^{\dagger}\) \\ GECs+WordDrop & 51\%\({}^{\dagger}\) & - & 61\({}^{\dagger}\) \\ GECs+SwitchOut & 77\%\({}^{\dagger}\) & - & 73\({}^{\dagger}\) \\ GECs+SeqMix & 98\%\({}^{\dagger}\) & - & 89 \({}^{\dagger}\) \\ GECs+TreeMix & **99\%** & - & **91\%** \\ \hline \hline \end{tabular} \({}^{\dagger}\) denotes the result is extracted from the original paper

\end{table}
Table 7: Experimental results (accuracy) on SCAN.

 

and "_around_"; these previously unseen command combinations further validates TreeMix's ability to improve the dataset's compositional diversity. TreeMix demonstrates weak performances on the _around-right_ split, where the model observes commands "_around_" and "_right_" in isolation at the training stage, and it has to derive the meaning of "_around right_" at the test time. Because the word "_around_" cannot be parsed as a single subtree for swap. Instead, it always appears in a subtree with the word "_left_", preventing TreeMix from generating the phrase "_turn right_". Despite its limitations on _around-left_, TreeMix performs well on all other splits and can be easily combined with other data augmentation methods, demonstrating the compositional generalization ability of TreeMix beyond classification tasks.

## 6 Conclusion

This work introduced TreeMix, a compositional data augmentation approach for natural language understanding. TreeMix leverages constituency parsing tree to decompose sentences into substructures and further use the mixup data augmentation technique to recombine them to generate new augmented sentences. Experiments on text classification and semantic parsing benchmarks demonstrate that TreeMix outperforms prior strong baselines, especially in low-resource settings and compositional generalization.

## Acknowledgements

The authors would like to thank reviewers for their helpful insights and feedback. This work is funded in part by a grant from Salesforce.

## References

* A. Anaby-Tavor, B. Carmeli, E. Goldbraich, A. Kantor, G. Kour, S. Shlomov, N. Tepper, and N. Zwerdling (2020)Do not have enough data? deep learning to the rescue!. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Truth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7383-7390. Cited by: SS4.
* J. Andreas (2020)Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7556-7566. External Links: Link Cited by: SS1, SS2.
* T. Bergmanis, K. Kann, H. Schutze, and S. Goldwater (2017)Training data augmentation for low-resource morphological inflection.. In Proceedings of the CNL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, Vancouver, V. Association for Computational Linguistics, Cited by: SS1, SS2.
* H. Cai, H. Chen, Y. Song, C. Zhang, X. Zhao, and D. Yin (2020)Data manipulation: towards effective instance learning for neural dialogue generation via learning to augment and reweight. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 6334-6343. External Links: Link, Document Cited by: SS1, SS2.
* J. Chen, D. Tam, C. Raffel, M. Bansal, and D. Yang (2021)An empirical survey of data augmentation for limited data learning in nlp. Cited by: SS1, SS2.
* J. Chen, Z. Wang, R. Tian, Z. Yang, and D. Yang (2020)Local additivity based data augmentation for semi-supervised NER. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 1241-1251. External Links: Link, Document Cited by: SS1, SS2.
* J. Chen, Z. Yang, and D. Yang (2020)MixText: linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 2147-2157. External Links: Link, Document Cited by: SS1, SS2.
* L. Chen, W. Ruan, X. Liu, and J. Lu (2020)SeqVAT: virtual adversarial training for semi-supervised sequence labeling. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 8801-8811. External Links: Link, Document Cited by: SS1, SS2.
* X. Chen, C. Liang, A. Wei Yu, D. Song, and D. Zhou (2020)Compositional generalization via neural-symbolic stack machines. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual, Y. Cheng, L. Jiang, W. Macherey, and J. Eisenstein (Eds.), Vol. , AdvAug, pp. 5961-5970. External Links: Link, Document Cited by: SS1, SS2.
* X. Dai and Heike Adel (2020)An analysis of simple data augmentation for named entity recognition. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, pp. 3861-3867. External Links: Link, Document Cited by: SS1, SS2.

 

Mathieu Dehouck and Carlos Gomez-Rodriguez. 2020. Data augmentation via subtree swapping for dependency parsing of low-resource languages. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 3818-3830, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Ding et al. (2020) Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kruengkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si, and Chunyan Miao. 2020. DAGA: Data augmentation with a generation approach for low-resource tagging tasks. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6045-6057, Online. Association for Computational Linguistics.
* Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
* Fadaee et al. (2017) Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 567-573, Vancouver, Canada. Association for Computational Linguistics.
* Feng et al. (2020) Steven Y. Feng, Varun Gangal, Dongyeop Kang, Teruko Mitamura, and Eduard Hovy. 2020. GenAug: Data augmentation for finetuning text generators. In _Proceedings of Deep Learning Inside Out (DeeLO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_, pages 29-42, Online. Association for Computational Linguistics.
* Feng et al. (2021) Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for NLP. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 968-988, Online. Association for Computational Linguistics.
* Guo et al. (2020) Demi Guo, Yoon Kim, and Alexander Rush. 2020. Sequence-level mixed sample data augmentation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5547-5552, Online. Association for Computational Linguistics.
* Guo et al. (2019) Hongyu Guo, Yongyi Mao, and Richong Zhang. 2019. Augmenting data with mixup for sentence classification: An empirical study.
* He et al. (2020) Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. 2020. Revisiting self-training for neural sequence generation. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Herzig and Berant (2021) Jonathan Herzig and Jonathan Berant. 2021. Span-based semantic parsing for compositional generalization. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 908-921, Online. Association for Computational Linguistics.
* Hsu et al. (2018) Wei-Ning Hsu, Hao Tang, and James R. Glass. 2018. Unsupervised adaptation with interpretable disentangled representations for distant conversational speech recognition. _ArXiv_, abs/1806.04872.
* Hsu et al. (2017) Wei-Ning Hsu, Yu Zhang, and James Glass. 2017. Unsupervised domain adaptation for robust speech recognition via variational autoencoder-based data augmentation. _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_.
* Jia and Liang (2016) Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12-22, Berlin, Germany. Association for Computational Linguistics.
* Jiang et al. (2020) Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and efficient fine-tuning for pretrained natural language models through principled regularized optimization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 2177-2190, Online. Association for Computational Linguistics.
* Karimi et al. (2021) Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021. Aeda: An easier data augmentation technique for text classification.
* Kim et al. (2021) Jang-Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. 2021. Co-mixup: Saliency guided joint mixup with supermodular diversity.
* Kim et al. (2020) Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song. 2020. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_, volume 119 of _Proceedings of Machine Learning Research_, pages 5275-5285. PMLR.
* Kobayashi (2018) Sousuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 452-457, New Orleans, Louisiana. Association for Computational Linguistics.
* Kolomiyets et al. (2011) Oleksandr Kolomiyets, Steven Bethard, and Marie-Francine Moens. 2011. Model-portability experiments for textual temporal analysis. In _Proceedings of the 49th Annual Meeting of the Association for 

Computational Linguistics: Human Language Technologies_, pages 271-276, Portland, Oregon, USA. Association for Computational Linguistics.
* Kumar et al. (2020) Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained transformer models. In _Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems_, pages 18-26, Suzhou, China. Association for Computational Linguistics.
* Lake and Baroni (2018) Brenden M. Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018_, volume 80 of _Proceedings of Machine Learning Research_, pages 2879-2888. PMLR.
* Liu et al. (2020a) Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Jiancheng Lv, Nan Duan, and Ming Zhou. 2020a. Tell me how to ask again: Question data augmentation with controllable rewriting in continuous space. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5798-5810, Online. Association for Computational Linguistics.
* Liu et al. (2020b) Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. 2020b. Data boost: Text data augmentation through reinforcement learning guided conditional generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9031-9041, Online. Association for Computational Linguistics.
* Malandrakis et al. (2019) Nikolaos Malandrakis, Minmin Shen, Anuj Goyal, Shuyang Gao, Abhishek Sethi, and Angeliki Metallinou. 2019. Controlled text generation for data augmentation in intelligent artificial agents. In _Proceedings of the 3rd Workshop on Neural Generation and Translation_, pages 90-98, Hong Kong. Association for Computational Linguistics.
* Manning et al. (2014) Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In _Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
* Miao et al. (2020) Zhengjie Miao, Yuliang Li, Xiaolan Wang, and Wang-Chiew Tan. 2020. Snippext: Semi-supervised opinion mining with augmented data. In _WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020_, pages 617-628. ACM / IW3C2.
* Miyato et al. (2019) Takeru Miyato, Shin-Ichi Maeda, Masanori Koyama, and Shin Ishii. 2019. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 41(8):1979-1993.
* Ng et al. (2020) Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. 2020. SSMBA: Self-supervised manifold based data augmentation for improving out-of-domain robustness. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1268-1283, Online. Association for Computational Linguistics.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 86-96, Berlin, Germany. Association for Computational Linguistics.
* Shen et al. (2020) Dinghan Shen, Ming Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. 2020. A simple but tough-to-beat data augmentation approach for natural language understanding and generation. _ArXiv_, abs/2009.13818.
* Shi et al. (2020) Haoyue Shi, Karen Livescu, and Kevin Gimpel. 2020. On the role of supervision in unsupervised constituency parsing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7611-7621, Online. Association for Computational Linguistics.
* Shi et al. (2021) Haoyue Shi, Karen Livescu, and Kevin Gimpel. 2021. Substructure substitution: Structured data augmentation for NLP. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 3494-3508, Online. Association for Computational Linguistics.
* Shi et al. (2018) Haoyue Shi, Hao Zhou, Jiaze Chen, and Lei Li. 2018. On tree-based neural sentence modeling. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4631-4641, Brussels, Belgium. Association for Computational Linguistics.
* Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps.
* Shahab Uddin et al. (2021) A. F. M. Shahab Uddin, Mst. Sirazam Monira, Wheemyung Shin, TaeChoong Chung, and Sung-Ho Bae. 2021. Saliencymix: A saliency guided data augmentation strategy for better regularization.
* Wang et al. (2019) Alex Wang, Amampreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Wang and Yang (2015) William Yang Wang and Diyi Yang. 2015. That's so annoying!!!: A lexical and frame-semantic embedding 

based data augmentation approach to automatic categorization of annoying behaviors using #petpeeve tweets. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, pp. 2557-2563. External Links: Link Cited by: SS4.
* Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig (2018)SwitchOut: an efficient data augmentation algorithm for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 856-861. External Links: Link Cited by: SS4.
* J. Wei and K. Zou (2019)EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 6382-6388. External Links: Link Cited by: SS4.
* Z. Wu, S. Wang, Y. Qian, and Kai Yu (2019)Data augmentation using variational autoencoder for embedding based speaker verification. In INTERSPEECH, Cited by: SS4.
* Q. Xie, Z. Dai, E. Hovy, M. Luong, and Quoc V. Le (2020)Unsupervised data augmentation for consistency training. External Links: Link Cited by: SS4.
* Z. Xie, S. I. Wang, J. Li, D. Levy, A. Nie, D. Jurafsky, and A. Y. Ng (2017)Data noising as smoothing in neural network language models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, External Links: Link Cited by: SS4.
* K. Min Yoo, D. Park, J. Kang, S. Lee, and W. Park (2021)Gpt3mix: leveraging large-scale language models for text augmentation. External Links: Link Cited by: SS4.
* S. Yoon, G. Kim, and Kyumin Park (2021)SSMix: saliency-based span mixup for text classification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Online, pp. 3225-3234. External Links: Link Cited by: SS4.
* May 3, 2018, Conference Track Proceedings, External Links: Link Cited by: SS4.
* November 2, 2019, pp. 6022-6031. External Links: Link Cited by: SS4.
* May 3, 2018, Conference Track Proceedings, OpenReview.net, Cited by: SS4.
* X. Zhang, J. J. Zhao, and Y. LeCun (2015)Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 649-657. External Links: Link Cited by: SS4.
* C. Zhu, Y. Cheng, Z. Gan, S. Sun, T. Goldstein, and J. Liu (2020)Freelb: enhanced adversarial training for natural language understanding. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, External Links: Link Cited by: SS4.

 



## Appendix A Augmentation examples

## Appendix B The necessity of merged loss techniques

We provide a detailed discussion of the techniques proposed in 3.2. We first investigate the noise contained in the augmentation dataset, then we figure out how the unbalance dataset will affect the performance. In the second part, we vary the weight parameter \(\gamma\) to see how it affects the model's learning process.

### Noise and Unbalance

All mixup methods, as previously stated, introduce noise into the dataset. This noise in the text includes grammatical structure confusion and multiple semantic meanings in the sentences. The model will be overwhelmed by the noise if trained solely on the generated augmentation dataset, and will even perform worse than the baseline. In terms of the unbalance problem, we find that training the model without replicating the original dataset to the same size as the augmentation dataset hurts the model's performance. The results are shown in the table B.1.

### Weight parameter

We vary weight parameter \(\gamma\) to find optimal balance point between diversity and linguistic grammar, the results are shown in figure 4. Performance on the two classification tasks follows a similar pattern. Both increase with increasing weight and then rapidly decrease with increasing weight after reaching the highest point. Performance is weaker than the baseline when the weight value exceeds 0.7. We find the model achieves the best performance with \(\gamma\in\{0.2,0.5\}\). For single sentence classification tasks, when \(\gamma=0.5\) the model always gets higher accuracy, and \(\gamma=0.2\) is better for these sentence relation classification datasets.





## Appendix C Hyper-parameters for each datasets

We explore different parameter combinations and find the best ones for each task, as in Tab 1. There are some exceptions, such as TREC datasets, where the model cannot converge even with 10 epochs, so we increase the training epochs to 20 for this dataset. IMDb's examples are extremely long, with an average length of more than 200 words. Along with this change, we increased the truncation length to 512 and the batch size to 8 to fully capture the semantic meaning. RTE is the most unusual. First, when we train using original RTE datasets, the accuracy deviation is really substantial, reaching up to 4%. Second, we find that \(\gamma=-0.2\) is optimum for this set, which contradicts previous findings.

## Appendix D Ablation Study

Shi et al. (2021) has proposed a similar study that uses constituency information for mixup. There are a few significant differences between our approaches. To begin with, their method is too restricted; they only perform mixup between examples from the same category, and they require the substituted subtree's label to be the same. Second, because they are limited to the same class examples, they are unable to devise a method for adding a soft label to the example. Instead, we only use TreeMix in the previous settings with the length constraint. Several other constraints in the subtree selection process are investigated in this section, and we achieve better performance than Shi et al. (2021) by giving the subtree selection process more freedom, and we validate that their work is a special case of our method by examining how other constraints affect the performance. This section's values are the averages of five runs with seeds ranging from 0 to 4

### What is the difference between different amounts of data?

TreeMix has the potential to generate an infinite amount of augmented data in theory. However, due to TreeMix's principle, it can only improve performance to a point when the size of the augmentation data

Figure 4: The performance when varying the value of the weight parameter on single sentence classification (left) and sentence relation classification (right)

set reaches a certain limit. We investigated how many augmentation datasets the model needs. Table D.1 shows the results of producing twice and five times the augmentation data for experiments.

The key to getting the best results is to strike a balance between the original datasets and the augmentation datasets in terms of diversity and linguistic confusion. With more augmentation datasets, the model will learn more patterns while also observing more grammatically poor samples, which could negatively impact performance. We discovered that augmentation datasets twice the size of the original dataset produce the best results for larger datasets. This is in line with our previous theoretical analysis: large datasets inherently include more patterns and diversity, which helps the model generalize better. Maintaining the original linguistic grammar while increasing diversity in these datasets is, therefore, more important. When working with smaller datasets, it's better to train with more augmentation data. For models to train on these datasets, we believe diversity is more important than linguistic grammar.

TREC-fine is an exception. We attribute it to the datasets' excessive classes (up to 47 classes within only 5.5k training samples): each class has a very limited number of samples, and if we create overly augmented dataset samples, the limited samples of each category are insufficient to resist injected linguistic noise. As a result, for TREC-fine, x2 is preferable to x5. For a smaller dataset, we recommend generating five times as much augmentation data as possible, and for a larger dataset, we recommend generating twice as much augmentation data.

### Is it beneficial to keep the swapped subtree's label or length the same?

Each subtree has its own label (e.g., VP and NP) and corresponds to a specific text span. When selecting subtrees, we can use these characteristics as additional constraints. Figure 5 shows the results.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Dataset & Size & BERT & TM(x2) & TM(x5) \\ \hline RTE & 3.5k & 68.15 & 70.57 & **70.62** \\ MRPC & 3.7k & 84.90 & 85.22 & **85.37** \\ TREC-f & 5.5k & 92.36 & **93.2** & 92.85 \\ TREC-c & 5.5k & 97.08 & 97.71 & **97.95** \\ IMDb & 12.5k & 93.63 & **94.34** & 94.24 \\ SST2 & 67k & 92.96 & **93.92** & 93.92 \\ QNLI & 105k & 90.54 & **91.36** & 91.34 \\ AGNEWS & 120k & 94.67 & **94.71** & 94.69 \\ QQP & 364k & 90.67 & **90.88** & 90.83 \\ MNLI & 393k & 84.27 & **84.45** & 84.41 \\ \hline \hline \end{tabular}
\end{table}
Table D.1: Improvement of performance on all datasets with different amount of augmentation datasets, _TM(x2)_ indicates generating twice as much augmentation data than the original data, _TM(x5)_ indicates five times than original data, datasets in the table are listed in order of size

Figure 5: Performance on SST2 with different subtree selection constraints, green part is bert performance, orange part is improvement of TreeMix when applying different constraints, TreeMix(label) indicates only select subtrees with same phrase label, TreeMix(length) indicates only select subtrees with same length. TreeMix indicates without any constraints



When we impose restrictions on the subtree selection process, the experimental results clearly show that performance suffers.

We hypothesize that this is because in datasets with similar sentence lengths, subtrees of the same phrase label or phrase length tend to have similar structures (e.g., tree height, relative position in the sentence). Although the exchange of such subtrees can retain the original linguistic grammar of the text to some extent (e.g., replacing a noun phrase with another noun phrase will not significantly disrupt the sentence) and maintain similar sentence length, it cannot exploit the potential compositional diversity in the datasets as efficiently as TreeMix without any constraints, resulting in lower diversity augmentation datasets and limited improvement compared to the baseline. In terms of the comparison of _TreeMix[label]_ and _TreeMix[length]_, we find that _TreeMix[label]_ prefers simple phrases such as NP and VP because these are the most common phrases occurring in sentences, and this exchange will not improve the diversity of the datasets. For example, in "I like this apple," replacing "apple" with "orange" will not provide innovative text patterns.