SlowGuess/NAACL_2022_OCR huggingface dataset:
Le Zhang
zhangle18@fudan.edu.cn
Zichao Yang
yangtze2301@gmail.com
Data augmentation is an effective approach to tackle over-fitting. Many previous works have proposed different data augmentations strategies for NLP, such as noise injection, word replacement, back-translation etc. Though effective, they missed one important characteristic of language-compositionality, meaning of a complex expression is built from its sub-parts. Motivated by this, we propose a compositional data augmentation approach for natural language understanding called TreeMix. Specifically, TreeMix leverages constituency parsing tree to decompose sentences into constituent sub-structures and the Mixup data augmentation technique to recombine them to generate new sentences. Compared with previous approaches, TreeMix introduces greater diversity to the samples generated and encourages models to learn compositionality of NLP data. Extensive experiments on text classification and SCAN demonstrate that TreeMix outperforms current state-of-the-art data augmentation methods.
Data augmentation (DA) has won great popularity in natural language processing (NLP) (Chen et al., 2021; Feng et al., 2021) due to the increasing demand for data and the expensive cost for annotation. DA aims at increasing the quantity and diversity of the datasets by generating more samples based on existing ones, which helps make the training process more consistent and improves the model's capacity for generalization (Xie et al., 2020). For instance, existing DA methods often leverage word-level manipulation (Wei and Zou, 2019; Kobayashi, 2018; Karimi et al., 2021) and model-based sentence generation (Edunov et al., 2018; Ng et al., 2020). As mixup-based (Zhang et al., 2018) augmentation achieving huge success in computer vision (Yun et al., 2019; Uddin et al., 2021; Kim et al., 2021), some recent works start
S1:They will find little interest in this poor film.
S2:It comes as a touching love story.
Method	Example
EDA
(Wei and Zou, 2019)	They will this find little
interest in bad movie.
AEDA
(Karimi et al., 2021)	They will find ? little in
! this poor movie;.
Noise
(Xie et al., 2017)	Thes will fi little
intres _ this poor film .
SSMix
(Yoon et al., 2021)	They will find little interest
in love poor film
Replacement
(Kolomiyets et al., 2011)	They will find limited interest
in this odd film.
Back Translation
(Edunov et al., 2018)	They will show little interest
in this strange film.
TreeMix	They will find little interest
in this touching love story.
Table 1: Input-level DAs for Text-Classification. EDA includes random deletion, swapping, and insertion. AEDA randomly inserts punctuation. SSMix swaps tokens based on their saliency. The replacement method randomly substitutes words with synonyms. In Back-translation, the source sentences are first translated into another language, and then back again.
to adapt mixup to NLP, such as at the hidden level (Guo et al., 2019; Chen et al., 2020b) and at the input level (Yoon et al., 2021; Shi et al., 2021).
Despite these empirical success, DA methods still suffer from key limitations. Simple rules based augmentation methods (Wei and Zou, 2019; Kobayashi, 2018; Karimi et al., 2021) show little to none effect over large pretrained language models. While mixup-based augmentation methods demonstrate huge potential, such interpolation at the hidden or input level has limited capability to capture explicit linguistic properties in text (Guo et al., 2019; Chen et al., 2020b; Yoon et al., 2021). Moreover, current DA methods exhibit limited ability in compositional generalization. Take a look at the following example from a BERT-based model that is fine-tuned using the SST2 dataset from the
Human Language Technologies, pages 5243 - 5258
GLUE Benchmark:
This film is good and everyone loves it.	99%
This film is poor and I do not like it.	99%
This film is good and I do not like it.	99%
The first two examples are correctly classified. Despite that the last one is composed of fragments from the first two, the model fails to produce a correct or plausible label (in terms of characterizing a sentence's sentiment), demonstrating poor performance in compositional generalization.
However, compositionality is one key aspect of language that the meaning of a complex sentence is built from its subparts. Prior work also shows that syntax trees (e.g., tree-based LSTMs) are helpful to model sentence structures for better text classification (Shi et al., 2018). However, leveraging compositional structures for data augmentation has not received much attention in the language technologies communities, with a few exceptions in semantic parsing (Andreas, 2020; Herzig and Berant, 2021).
To this end, we propose a compositional data augmentation method for natural language understanding, i.e., TreeMix (Figure 1). TreeMix is an input-level mixup method that utilizes constituency parsing information, where different fragments (phrase of a subtree) from different sentences are recombined to create new examples that were never seen in the training set; new soft labels will also be strategically created based on these fragments at the same time. In this way, TreeMix not only exploits compositional linguistic features to increase the diversity of the augmentation, but also provides reasonable soft labels for these mixed examples.
Empirically, we find that TreeMix outperforms existing data augmentation methods significantly on a set of widely used text classification benchmarks. To validate the compositional effectiveness of TreeMix, we experiment with SCAN (Lake and Baroni, 2018)—a task requires strong compositional generalization, and find that TreeMix exhibits reasonable ability to generalize to new structures built of components observed during training.
2.1 Generic Data Augmentation
Most prior work operates data augmentation at different levels (Chen et al., 2021). Token-level DA
methods manipulate tokens or phrases while preserving syntax and semantic meaning as well as labels of the original text, such as synonymy words substitutions (Wang and Yang, 2015; Zhang et al., 2015; Fadaee et al., 2017; Kobayashi, 2018; Miao et al., 2020) where synonyms are detected following pre-defined rules or by word embedding similarities. These methods have limited improvement (Chen et al., 2021) over large pretrained language models (PLMs). Besides, introducing noise by random insertion, replacement, deletion, and swapping (Wang et al., 2018; Wei and Zou, 2019; Karimi et al., 2021; Xie et al., 2020) is expected to improve the robustness of the model. Sentence-Level DA methods increase the diversity by generating distinct examples, such as via paraphrasing (Yu et al., 2018; He et al., 2020; Xie et al., 2020; Kumar et al., 2020; Chen et al., 2020b; Cai et al., 2020) or back translation (Sennrich et al., 2016; Edunov et al., 2018). Other line of work used label-conditioned generation methods that train a conditional generation model such as GPT-2 or VAE to create new examples given labels as conditions (Bergmanis et al., 2017; Liu et al., 2020b,a; Ding et al., 2020; Anaby-Tavor et al., 2020). Although these methods can produce novel and diverse text patterns that do not exist in the original datasets, they require extensive training. Hidden-Level DA methods mainly manipulate hidden representations by perturbation (Miyato et al., 2019; Zhu et al., 2020; Jiang et al., 2020; Chen et al., 2020c; Shen et al., 2020; Hsu et al., 2017, 2018; Wu et al., 2019; Malandrakis et al., 2019) and interpolation like mixup (Zhang et al., 2018) to generate new examples (Miao et al., 2020; Cheng et al., 2020; Chen et al., 2020b; Guo et al., 2019, 2020; Chen et al., 2020a).
2.2 Compositional Data Augmentation
Compositional augmentation aims at increasing the diversity of the datasets and improving the compositional generalization capability of the resulting models (Jia and Liang, 2016; Andreas, 2020). These methods often recombine different components from different sentences to create new examples following a set of pre-designed linguistic rules such as lexical overlaps (Andreas, 2020), neural-symbolic stack machines (Chen et al., 2020d), and substructure substitution (Shi et al., 2021). Compositional methods have been applied in a set of NLP tasks, such as sequence labeling (Guo et al., 2020), semantic parsing (Andreas, 2020), constituency
Figure 1: Illustration of TreeMix for single sentence classification
Parsing (Shi et al., 2020, 2021), dependency parsing (Dehouck and Gomez-Rodriguez, 2020; Shi et al., 2021), named entity recognition (Dai and Adel, 2020), text generation (Feng et al., 2020), and text classification (Yoon et al., 2021; Shi et al., 2021). Our work also falls into this category.
The most relevant are Shi et al. (2021) and Yoon et al. (2021). However, Shi et al. (2021) only performs constituent substructure combinations with examples from the same category, thus inadequate in creating diverse enough augmentation with newly created labels.
Besides, Yoon et al. (2021) simply swaps the most and least salient spans, heavily relying on the model's performances in estimating salient spans, and failing to consider these sentences' linguistic structures. Our proposed TreeMix fills these gaps by allowing the composition of sentences from different label categories, by utilizing rich consistency based structures in text, and by strategically generating soft labels for these augmented instances.
Our work is motivated by Mixup (Zhang et al., 2018), which creates virtual samples by mixing inputs. Given two random drawn examples  $(\mathbf{x}_i, \mathbf{y}_i)$  and  $(\mathbf{x}_j, \mathbf{y}_j)$ , where  $\mathbf{x}$  denotes the input sample and  $\mathbf{y}$  is the corresponding one-hot label, Mixup creates a new sample by:
$$
\overline {{\mathbf {x}}} = \lambda \mathbf {x} _ {i} + (1 - \lambda) \mathbf {x} _ {j},
$$
$$
\overline {{\mathbf {y}}} = \lambda \mathbf {y} _ {i} + (1 - \lambda) \mathbf {y} _ {j},
$$
where  $\lambda \in [0,1]$ . Mixup can be easily implemented in continuous space, hence some prior works (Chen
et al., 2020b) have extended it to NLP by performing interpolation in hidden space.
We improve upon Mixup by incorporating compositionality of language, a key characteristic that is essential to generalization but neural models often fall short in capturing (Lake and Baroni, 2018). Instead of interpolating with the whole sample, TreeMix, our newly proposed method, creates new sentences by removing phrases of sentences and reinserting subparts from other sentences. TreeMix makes use of constituency trees to decompose a sentence into meaningful constituent parts, which can then be removed and recombined to generate new augmentation samples. We aim to improve models' compositionality generalization ability by training on large amount of samples produced by TreeMix. An example of using TreeMix for single sentence classification is shown in Figure 1.
3.1 TreeMix
Let  $\mathbf{x}_i = \{x_i^1, x_i^2, \ldots, x_i^l\}$  denotes a sequence with length  $l$  and its corresponding label in one-hot encoding as  $\mathbf{y}_i$ . We run a constituency parser on  $\mathbf{x}_i$  to get its parsing tree as  $T(\mathbf{x}_i)$ . In order to get meaningful subparts of a sequence, we traverse the parsing tree recursively and get all the subtrees with more than one child. Denote the collection of subtrees as  $S(\mathbf{x}_i) = \{t_i^k\}$ , where  $t_i^k$  denotes the  $k$ -th subtree of sample  $\mathbf{x}_i$ . For a subtree  $t_i^k$ , it covers a continuous span  $t_i^k \triangleq [x_i^{r_k}, \ldots, x_i^{s_k}]$  of  $\mathbf{x}_i$  that starts with index  $r_k$  and ends with index  $s_k$ . For example, as shown in the left part of Figure 1, the subtrees of the example sentence can cover spans such as this poor film, in this poor film, no interest etc.
They will find little interest in this poor film.
[λL, λU]	possible selected sub-trees
[0.1,0.3]	(little interest),(this poor film)
[0.3,0.5]	(in this poor film)
[0.5,0.7]	(little interest in this poor film)
Table 2: Examples of possible candidate subtrees with different  $\lambda$  intervals
For a given sample  $(\mathbf{x}_i,\mathbf{y}_i)$ , we randomly sample another data point  $(\mathbf{x}_j,\mathbf{y}_j)$  from the training set. We run the constituency parser on both sentences and get their subtree sets  $S(\mathbf{x}_i)$  and  $S(\mathbf{x}_j)$ , based on which we can sample subtrees to exchange. We introduce two additional hyper-parameters  $\lambda_L$  and  $\lambda_U$  to constraint the length of subtrees to sample.  $\lambda_L$  and  $\lambda_U$ , measured in terms of length ratio of the subtree to the original sentences, sets the lower and upper limits of the subtrees to sample. Intuitively,  $\lambda$  controls the granularity of the phrases that we aim to exchange. We would like that the length of phrase to exchange to be reasonable. If it is too short, then the exchange cannot introduce enough diversity to the augmented sample; otherwise if it is too long, the process might inject too much noise to the original sentence. We set  $\lambda$  to be the ratio in order to be invariant to the length of original sentences. Table 2 shows some subtree examples with different length constraints. We define the length constrained subtree set as:
$$
S _ {\lambda} (\mathbf {x}) \triangleq \{t | t \in S (\mathbf {x}), s. t. \frac {| t |}{| \mathbf {x} |} \in [ \lambda_ {L}, \lambda_ {U} ] \}.
$$
Here  $|.|$  denotes the length of a sequence or a subtree. For two sentences  $\mathbf{x}_i$  and  $\mathbf{x}_j$ , we randomly sample two subtrees  $t_i^k \in S_\lambda(\mathbf{x}_i)$  and  $t_j^l \in S_\lambda(\mathbf{x}_j)$  and construct a new sample by replacing  $t_i^k$  with  $t_j^l$ , i.e.
$$
\bar {\mathbf {x}} \triangleq [ x _ {i} ^ {1}, \dots , x _ {i} ^ {r _ {k} - 1}, \underbrace {x _ {j} ^ {r _ {l}} , \dots , x _ {j} ^ {s _ {l}}} _ {t _ {j} ^ {l}}, x _ {i} ^ {s _ {k} + 1}, \dots x _ {i} ^ {l} ] \tag {1}
$$
where  $t_j^l = [x_j^{r_l},\dots,x_j^{s_l}]$  replaces  $t_i^k = [x_i^{r_k},\dots,x_i^{s_k}]$ . Figure. 1 shows an example of TreeMix, where the subtree a touching transcend love story replaces the subtree this poor film.
Label Creation for TreeMix Creating a valid label for the augmented sample  $\bar{\mathbf{x}}$  is a challenging problem. Similar to that of Mixup (Zhang et al., 2018), we use a convex combination of original
Algorithm 1: Dataset construction
labels of two sentences as the new label for the augmented sample:
$$
\bar {\mathbf {y}} = \frac {l _ {i} - \left| t _ {i} ^ {k} \right|}{l _ {i} - \left| t _ {i} ^ {k} \right| + \left| t _ {j} ^ {l} \right|} \mathbf {y} _ {i} + \frac {\left| t _ {j} ^ {l} \right|}{l _ {i} - \left| t _ {i} ^ {k} \right| + \left| t _ {j} ^ {l} \right|} \mathbf {y} _ {j}, \tag {2}
$$
where  $l_{i}$  is the length of  $\mathbf{x}_i$  and  $|t_i^k |,|t_j^k |$  are the length of the subtrees. In the new sentence,  $l_{i} - |t_{i}^{k}|$  words from  $\mathbf{x}_i$  are kept and  $|t_j^l |$  words from sentence  $\mathbf{x}_j$  are inserted.
In Equation 2,  $\frac{l_i - |t_i^k|}{l_i - |t_i^k| + |t_j^l|}$  is the fraction of words that come from  $\mathbf{x}_i$ , which determines the weight of  $\mathbf{y}_i$ . The label is then created based on the conjecture that the change in labels is proportional to the length changes in the original sentences. We provided a set of augmentation examples from TreeMix in Table A.1 in Appendix.
Pairwise Sentence Classification Task The above mainly used single sentence classification as the running example for TreeMix. Here we argue that TreeMix can easily be extended to pairwise sentence classification problem, where the relationship between the sentences is the label.
Formally, for a given sample  $(\mathbf{x}_i,\mathbf{x}_i',\mathbf{y}_i)$ , we randomly sample another sample  $(\mathbf{x}_j,\mathbf{x}_j',\mathbf{y}_j)$  and run the parser and get the subtree sets of each sentence  $S(\mathbf{x}_i),S(\mathbf{x}_i')$  and  $S(\mathbf{x}_j),S(\mathbf{x}_j')$ . Then we randomly sample subtrees  $t_i^k\in S_\lambda (\mathbf{x}_i),t_{i'}^{k'}\in S_\lambda (\mathbf{x}_i')$  and  $t_j^l\in S_\lambda (\mathbf{x}_j),t_{j'}^{l'}\in S_\lambda (\mathbf{x}_j')$ . We construct  $\bar{\mathbf{x}}$  by replacing  $t_i^k$  with  $t_j^l$  and  $\bar{\mathbf{x}}'$  by replacing  $t_{i'}^{k'}$  with  $t_{j'}^{l'}$ . The new label is created as:
$$
\begin{array}{l} \bar {\mathbf {y}} = \frac {l _ {i} + l _ {i ^ {\prime}} - \left| t _ {i} ^ {k} \right| - \left| t _ {i ^ {\prime}} ^ {k ^ {\prime}} \right|}{l _ {i} + l _ {i ^ {\prime}} - \left| t _ {i} ^ {k} \right| - \left| t _ {i ^ {\prime}} ^ {k ^ {\prime}} \right| + \left| t _ {j} ^ {l} \right| + \left| t _ {j ^ {\prime}} ^ {l ^ {\prime}} \right|} \mathbf {y} _ {i} + \tag {3} \\ \frac {\left| t _ {j} ^ {l} \right| + \left| t _ {j ^ {\prime}} ^ {l ^ {\prime}} \right|}{l _ {i} + l _ {i ^ {\prime}} - \left| t _ {i} ^ {k} \right| - \left| t _ {i ^ {\prime}} ^ {k ^ {\prime}} \right| + \left| t _ {j} ^ {l} \right| + \left| t _ {j ^ {\prime}} ^ {l ^ {\prime}} \right|} \mathbf {y} _ {j}. \\ \end{array}
$$
The meanings of the notations are the same as in Equation 2.
Our main algorithm is shown in Algorithm 1. Although not all sentences created by TreeMix are fluent or even valid new sentences, they contain subparts with different meanings that encourage the models to build rich representation of sentences in a compositional manner. Note that the augmented labels are convex combination of original labels, only when the model learns the representations of two parts together can they predict both labels with different weights.
3.2 Training Objective
Our model is trained on a combination of the original samples and augmentation samples to obtain a trade-off between regularization and noise injection. The final training objective is:
$$
\begin{array}{l} \mathcal {L} = \underset {(\mathbf {x}, \mathbf {y}) \sim D} {\mathbb {E}} [ - \mathbf {y} ^ {\top} \log P _ {\theta} (\mathbf {y} | \mathbf {x}) ] \\ + \gamma \underset {(\bar {\mathbf {x}}, \bar {\mathbf {y}}) \sim D ^ {\prime}} {\mathbb {E}} \left[ - \bar {\mathbf {y}} ^ {\top} \log P _ {\theta} (\bar {\mathbf {y}} | \bar {\mathbf {x}}) \right], \tag {4} \\ \end{array}
$$
$\gamma$  is the weight on the augmentation samples.
To test TreeMix's effectiveness, we experiment with a variety of text classification benchmarks, as shown in Table 3. We use accuracy as a metric, and exclude datasets from GLUE (Wang et al., 2019) that are not suitable for mixup, including CoLA that measures linguistic acceptability and will be ruined by mixup operations, and WNLI that is too small to show a method's validity.
The proposed TreeMix method creates new samples by combining text spans based on the constituency tree's information, thus we use the Stanford CoreNLP toolkit² to obtain parsing related information (Manning et al., 2014). We use the pretrained language model bert-base-uncased for sequence classification task from HuggingFace. With seeds ranging from 0 to 4 and  $\lambda_{L} = 0.1$ ,  $\lambda_{U} = 0.3$ , we use TreeMix to generate twice and five times more samples than the original training set³. We replicate the original dataset to the same size as the augmentation datasets in the training stage to
Dataset	Task	Class	Size
Single Sentence Classification
SST2	Sentiment	2	67k/1.8k
TREC-fine	Question	47	5.5k/500
TREC-coarse	Question	6	5.5k/500
AG_NEWS	News	4	12k/4k
IMDb	Sentiment	2	12.5k/12.5k
Pair Sentence Classification
RTE	NLI	2	3.5k/300
MRPC	Paraphrase	2	3.7k/400
QNLI	QA	2	105k/5.5k
QQP	Paraphrase	2	364k/40.4k
MNLI	NLI	3	393k/9.8k
Table 3: Dataset name or split name, task category and number of label class, Size used for training and testing. For tasks from GLUE, Size indicates (#train:#validation); for TREC, AG_NEWS and IMDb, Size indicates (#train:#test).
ensure that the model receives the same amount of data from the original dataset and the augmentation dataset for each training batch.
If not specified, we train the model for 5 epochs, with a maximum sequence length of 128 and batch size of 96. The model is optimized using the AdamW optimizer with an eps of 1e-8 and a learning rate of 2e-5. Table C.1 in Appendix contains detailed hyper-parameter settings for each dataset.
4.3 Baseline
We compare TreeMix with the following benchmarks: (1) No augmentation (BERT): standard training without any augmentation, (2) EDA that randomly performs insertion, replacement, swap and deletion to the text. (3) AEDA that randomly inserts punctuation to the text. (4) Back translation(BT) (Edunov et al., 2018): texts are translated between English and German using Transformer architectures trained on WMT16 English-German. (5) GPT3Mix(Yoo et al., 2021) designs prompts and utilizes GPT3 to generate new examples to train the model. (6) SSMix (Yoon et al., 2021) applies mixup based on the saliency (Simonyan et al., 2014) of tokens, similar to PuzzleMix (Kim et al., 2020) and SaliencyMix (Uddin et al., 2021). (7) EmbedMix is the pretrained-language-model version of WordMixup in Guo et al. (2019), which performs mixup on the embedding level. (8) TMix (Chen et al., 2020b) first encodes two inputs separately, then performs the linear interpolation of two embeddings at a certain encoder layer, and fi
<sup>1</sup>Section B in Appendix presents discussions on how the objective and different weight parameter affects the result.
2The specific version is 3.9.2
<sup>3</sup>Section D.1 in Appendix presents robustness check on how different amount of augmented data affects the result.
nally forward-passes the combined embedding in the remaining layers.
5.1 Performance On Full Dataset
The results of TreeMix on the entire datasets are shown in Table 4. TreeMix outperforms all baselines significantly on single sentence classification tasks, demonstrating the superiority of using compositional substructure for substitution and augmentation. For instance, On SST2, it improves by  $0.98\%$ . Compared to other methods, the improvement was more than doubled.
This is because that, unlike SSMix which substitutes the text spans based on the saliency, our TreeMix makes use of the constituency information to help identify linguistically informed sentence substructures, and by recombining these components, the compositional diversity of the datasets can be maximized. With our TreeMix generated samples, the model can see more combinations of the substructures in the training stage that aren't available in the original corpus, leading to better generalization ability.
When it comes to sentence relationship classification tasks, TreeMix is also very effective. For example, It improves by  $2.47\%$  on the RTE data set, whereas the best improvement of other methods is only  $0.3\%$ , and it improves by  $0.82\%$  on QNLI, where other data augmentation methods have little effect. We hypothesized that, when two constituent parts from one sentence pair are embedded into another sentence pair, the inherent relationship is also embedded. This better helps the models on how to identify two pairs of relationships in a single sample, which further increases its capacity to categorize these challenging adversarial sentences. Since TreeMix works by increasing dataset diversity and providing models with more text patterns to learn, it has very significant improvements for these relatively small datasets such as RTE and TREC, compared to these large datasets such as AG NEWS,QQP and MNLI that already have a lot of diversity and text patterns.
5.2 Influence of Constituency Information
To determine the importance of constituency information, we designed a Random Mixup (RandMix) that randomly selects text spans as long as the ratio of span length to sentence length is less than
Figure 2: Performance of RandMix and TreeMix on single sentence classification datasets, scores are averaged over 5 random seeds.
a particular threshold  $\lambda_{rand}^4$ . The rest setting of RandMix is the same as TreeMix. We compare TreeMix and RandMix on single sentence classification datasets in Figure 2.