Boosting Text Augmentation via Hybrid Instance Filtering Framework
Year: 2023

Abstract:
Text augmentation is an effective technique for addressing the problem of insufficient data in natural language processing. However, existing text augmentation methods tend to focus on few-shot scenarios and usually perform poorly on large public datasets. Our research indicates that existing augmentation methods often generate instances with shifted feature spaces, which leads to a drop in performance on the augmented data (for example, EDA generally loses ≈ 2% in aspect-based sentiment classification). To address this problem, we propose a hybrid instance-filtering framework (BOOSTAUG) based on pre-trained language models that can maintain a similar feature space with natural datasets. BOOSTAUG is transferable to existing text augmentation methods (such as synonym substitution and back translation) and significantly improves the augmentation performance by ≈ 2− 3% in classification accuracy. Our experimental results on three classification tasks and nine public datasets show that BOOSTAUG addresses the performance drop problem and outperforms state-ofthe-art text augmentation methods. Additionally, we release the code to help improve existing augmentation methods on large datasets.

heading: 1 Introduction
text: Recent pre-trained language models (PLMs) (Devlin et al., 2019; Brown et al., 2020; He et al., 2021; Yoo et al., 2021) have been able to learn from large amounts of text data. However, this also leads to a critical problem of data insufficiency in many lowresource fine-tuning scenarios (Chen et al., 2020; Zhou et al., 2022a; Miao et al., 2021; Kim et al., 2022; Wang et al., 2022b; Yang et al., 2022). Despite this, existing augmentation studies still encounter failures on large public datasets. While some studies(Ng et al., 2020; Body et al., 2021; Chang et al., 2021; Luo et al., 2021) have attempted
∗Corresponding author
to leverage the language modeling capabilities of PLMs in text augmentation, these methods still suffer from performance drops on large datasets.
To explore the root cause of this failure mode, we conducted experiments to explain the difference between “good” and “bad” augmentation instances. Our study found that existing augmentation methods (Wei and Zou, 2019; Coulombe, 2018; Li et al., 2019; Kumar et al., 2019; Ng et al., 2020) usually fail to maintain the feature space in augmentation instances, which leads to bad instances. This shift in feature space occurs in both edit-based and PLM-based augmentation methods. For example, edit-based methods can introduce breaking changes that corrupt the meaning of the text, while PLM-based methods can introduce outof-vocabulary words. In particular, for the edit-
1652
based methods, the shifted feature space mainly comes from breaking text transformations, such as changing important words (e.g., ‘but ’ ) in sentiment analysis. As for PLM-based methods, they usually introduce out-of-vocabulary words due to word substitution and insertion, which leads to an adverse meaning in sentiment analysis tasks.
To address the performance drop in existing augmentation methods caused by shifted feature space, we propose a hybrid instance-filtering framework (BOOSTAUG) based on PLMs to guide augmentation instance generation. Unlike other existing methods (Kumar et al., 2020), we use PLMs as a powerful instance filter to maintain the feature space, rather than as an augmentor. This is based on our finding that PLMs fine-tuned on natural datasets are familiar with the identical feature space distribution. The proposed framework consists of four instance filtering strategies: perplexity filtering, confidence ranking, predicted label constraint, and a cross-boosting strategy. These strategies are discussed in more detail in section Section 2.3. Compared to prominent studies, BOOSTAUG is a pure instance-filtering framework that can improve the performance of existing text augmentation methods by maintaining the feature space.
With the mitigation of feature space shift, BOOSTAUG can generate more valid augmentation instances and improve existing augmentation methods’ performance, which more augmentation instances generally trigger performance sacrifice in other studies (Coulombe, 2018; Wei and Zou, 2019; Li et al., 2019; Kumar et al., 2020)). According to our experimental results on three finegrained and coarse-grained text classification tasks, BOOSTAUG1 significantly alleviates feature space shifts for existing augmentation methods.
Our main contributions are: • We propose the feature space shift to explain the
performance drop in existing text augmentation methods, which is ubiquitous in full dataset augmentation scenarios. • We propose a universal augmentation instance filter framework to mitigate feature space shift and significantly improve the performance on the ABSC and TC tasks. • Our experiments show that the existing text augmentation methods can be easily improved by employing BOOSTAUG.
1We release the source code and experiment scripts of BOOSTAUG at: https://github.com/yangheng95/ BoostTextAugmentation.
Algorithm 1: The pseudo code of BOOSTAUG
1 Split D into k folds, D := {F i}ki=1; 2 Daug := ∅; 3 for i← 1 to k do 4 Diaug := ∅, Diboost := F i; 5 Randomly pick up k − 2 folds except F i to constitute Ditrain; 6 Divalid := F \ (F i
⋃Ditrain); 7 Use the DeBERTa on Ditrain and Divalid to build the surrogate language model; 8 forall dorg ∈ Diboost do 9 Diaug := F (diorg, Ñ ,Θ);
10 forall daug ∈ Diaug do 11 Use the surrogate language model to
predict P(daug), C(daug), and the ℓ̃aug of daug;
12 if P(daug) ≥ α ∥ C(daug) ≤ β ∥ ℓ̃daug ̸= ℓ̃dorg then 13 Diaug := Diaug \ {daug}; 14 Daug := Daug ⋃Diaug; 15 Daug := Daug ⋃Diboost; 16 return Daug

heading: 2 Proposed Method
text: The workflow of BOOSTAUG is shown in Figure 2 and the pseudo code is given in Algorithm 1. Different from most existing studies, which focus on unsupervised instance generation, BOOSTAUG serves as an instance filter to improve existing augmentation methods. The framework consists of two main phases: 1) Phase #1: the training of surrogate language models; 2) Phase #2: surrogate language models guided augmentation instance filtering. The following paragraphs will provide a detailed explanation of each step of the implementation.

heading: 2.1 Surrogate Language Model Training
text: At the beginning of Phase #1, the original training dataset is divided into k ≥ 3 folds where the k−2 ones are used for training (denoted as the training fold) while the other two are used for the validation and augmentation purposes, denoted as the validation and boosting fold, respectively2 (lines 4- 6). Note that the generated augmentation instances, which will be introduced in Section 2.2, can be
2We iteratively select the i-th fold, i ∈ 1, · · · , k, as the boosting fold (line 3 in Algorithm 1). The validation fold is used to select the best checkpoint of the surrogate language model to filter the augmented instances. This process is repeated k times to ensure that all the folds have been used for validation and boosting at least once, thus avoiding data overlapping between the training and validation folds.
identical to the instances in training folds the surrogate language model. This data overlapping problem will lead to a shifted feature space. We argue that the proposed k-fold augmentation approach, a.k.a. “cross-boosting”, can alleviate the feature space shift of the augmentation instances, which will be validated and discussed in detail in Section 4.3. The main crux of Phase #1 is to build a surrogate language model as a filter to guide the elimination of harmful and poor augmentation instances.
We construct a temporary classification model using the DeBERTa (He et al., 2021) architecture. This model is then fine-tuned using the data in the k − 2 training folds and the validation fold to capture the semantic features present in the data (line 7). It is important to note that we do not use the original training dataset for this fine-tuning process. Once the fine-tuning is complete, the language model constructed from the DeBERTa classification model is then utilized as the surrogate language model in the instance filtering step in Phase #2 of BOOSTAUG.
This is different from existing works that use a pre-trained language model to directly generate augmentation instances. We clarify our motivation for this from the following two aspects. • In addition to modeling the semantic feature, the
surrogate language model can provide more information that can be useful for the quality control of the augmentation instances, such as text per-
plexity, classification confidence, and predicted label. • Compared to the instance generation, we argue that the instance filtering approach can be readily integrated with any existing text augmentation approach.

heading: 2.2 Augmentation Instance Generation
text: As a building block of Phase #2, we apply some prevalent data augmentation approaches as the back end to generate the augmentation instances in BOOSTAUG (line 9). More specifically, let Dorg := {diorg}Ni=1 be the original training dataset. diorg := ⟨si, ℓi⟩ is a data instance where si indicates a sentence and ℓi is the corresponding label, i ∈ 1, · · · , N . By applying the transformation function F (·, ·, ·) upon diorg as follows, we expect to obtain a set of augmentation instances Diaug for diorg:
Diaug := F (diorg, Ñ ,Θ), (1)
where Ñ ≥ 1 is used to control the maximum number of generated augmentation instances. In the end, the final augmentation set is constituted as Daug := ⋃N i=1Diaug (line 14). Note that depending on the specific augmentation back end, there can be more than one strategy to constitute the transformation function. For example, EDA (Wei and Zou, 2019) has four transformation strategies, including synonym replacement, random insertion, random swap, and random deletion. Θ consists
of the parameters associated with the transformation strategies of the augmentation back end, e.g., the percentage of words to be modified and the mutation probability of a word.

heading: 2.3 Instance Filtering
text: Our preliminary experiments have shown that merely using data augmentation can be detrimental to the modeling performance, no matter how many augmentation instances are applied in the training process. In addition, our experiments in Section 4.3 have shown a surprising feature space shift between the original data and the augmented instances in the feature space. To mitigate this issue, BOOSTAUG proposes an instance filtering approach to control the quality of the augmentation instances. It consists of three filtering strategies, including perplexity filtering, confidence ranking, and predicted label constraint, which will be delineated in the following paragraphs, respectively. Note that all these filtering strategies are built on the surrogate language model developed in Phase #1 of BOOSTAUG (lines 12 and 13).

heading: 2.3.1 Perplexity Filtering
text: Text perplexity is a widely used metric to evaluate the modeling capability of a language model (Chen and Goodman, 1999; Sennrich, 2012). Our preliminary experiments have shown that low-quality instances have a relatively high perplexity. This indicates that perplexity information can be used to evaluate the quality of an augmentation instance. Since the surrogate language model built in Phase #1 is bidirectional, the text perplexity of an augmentation instance daug is calculated as:
P(daug) = s∏
i=1
p (wi | w1, · · · , wi−1, wi+1, · · · , ws) ,
(2) where wi represents the token in the context. s is the number of tokens in daug and p (wi | w1, · · · , wi−1, wi+1, · · · , ws) is the probability of wi conditioned on the preceding tokens, according to the surrogate language model, i ∈ 1, · · · , s. Note that daug is treated as a lowquality instance and is discarded if P(daug) ≥ α while α ≥ 0 is a predefined threshold.

heading: 2.3.2 Confidence Ranking
text: We observe a significant feature space shift in the augmentation instances. These instances will be allocated with low confidences by the surrogate
language model. In this case, we can leverage the classification confidence as a driver to control the quality of the augmentation instances. However, it is natural that long texts can have way more augmentation instances than short texts, thus leading to the so-called unbalanced distribution. Besides, the confidence of most augmentation instances is ≥ 95%, which is not selective as the criterion for instance filtering. To mitigate the unbalanced distribution in augmentation instances and make use of confidence, we develop a confidence ranking strategy to eliminate the redundant augmentation instances generated from long texts while retaining the rare instances having a relatively low confidence. More specifically, we apply a softmax operation on the output hidden state learned by the surrogate language model, denoted as H(daug), to evaluate the confidence of daug as:
C(daug) = argmax (
exp(Hdaug)∑c 1 exp(Hdaug)
) , (3)
where c is the number of classes in the original training dataset. To conduct the confidence ranking, 2 × Ñ instances are generated at first, while only the top Ñ instances are selected to carry out the confidence ranking. By doing so, we expect to obtain a balanced augmentation dataset even when there is a large variance in the confidence predicted by the surrogate language model. After the confidence ranking, the augmentation instances with Cdaug ≤ β are discarded while β ≥ 0 is a fixed threshold.

heading: 2.3.3 Predicted Label Constraint
text: Due to some breaking text transformation, text augmentation can lead to noisy data, e.g., changing a word "greatest" to "worst" in a sentence leads to an adverse label in a sentiment analysis task. Since the surrogate language model can predict the label of an augmentation instance based on its confidence distribution, we develop another filtering strategy that eliminates the augmentation instances whose predicted label ℓ̃daug is different from the ground truth. By doing so, we expect to mitigate the feature space bias.

heading: 2.4 Feature Space Shift Metric
text: To quantify the shift of the feature space, we propose an ensemble metric based on the overlapping ratio and distribution skewness of the t-SNE-based augmented instances’ feature space.
The feature space overlapping ratio measures the diversity of the augmented instances. A larger overlapping ratio indicates that more natural instances have corresponding augmented instances. On the other hand, the distribution skewness measure describes the uniformity of the distribution of the augmented instances. A smaller distribution skewness indicates that the natural instances have approximately equal numbers of corresponding augmented instances. To calculate the feature space shift, we first calculate the overlapping ratio and distribution skewness of the natural instances and their corresponding augmented instances. The feature space shift is calculated as follows:
S = 1−O + sk, (4)
where O and sk are the feature space convex hull overlapping ratio and feature space distribution skewness, which will be introduced in the following subsections.

heading: 2.4.1 Convex hull overlapping calculation
text: To calculate the convex hull overlapping rate, we use the Graham Scan algorithm3 (Graham, 1972) to find the convex hulls for the test set and target dataset in the t-SNE visualization, respectively.
Let P1 and P2 represent the convex hulls of two datasets in the t-SNE visualization; we calculate the overlapping rate as follows:
O = P1 ∩ P2P1 ∪ P2 , (5)
where ∩ and ∪ denote convex hull intersection and union operations, respectively. O is the overlapping rate between P1 and P2.

heading: 2.4.2 Distribution skewness calculation
text: The skewness of an example distribution is computed as follows:
sk = m3
m 3/2 2
, (6)
mi = 1
N
N∑
n=1
(xn − x̄)i, (7)
where N is the number of instances in the distribution; sk is the skewness of an example distribution. mi and x̄ are the i-th central moment and mean of the example distribution, respectively. Because the t-SNE has two dimensions (namely x and y
3https://github.com/shapely/shapely.
axes), we measure the global skewness of the target dataset (e.g., training set, augmentation set) by summarizing the absolute value of skewness on the x and y axes in t-SNE:
skg = |skx|+ |sky|, (8)
where skg is the global skewness of the target dataset; skx and sky are the skewness on the x and y axes, respectively.
By combining the convex hull overlapping ratio and distribution skewness, the proposed feature space shift metric offers a comprehensive view of how well the augmented instances align with the original data distribution. This metric can be used to evaluate the effectiveness of different data augmentation approaches, as well as to inform the fine-tuning process for better model performance.

heading: 3 Experimental Setup

heading: 3.1 Datasets
text: Our experiments are conducted on three classification tasks: the sentence-level text classification (TC), the aspect-based sentiment classification (ABSC), and natural language inference (NLI). The datasets used for the TC task include SST2, SST5 (Socher et al., 2013) from the Stanford Sentiment Treebank, and AGNews10K4(Zhang et al., 2015). Meanwhile, the datasets used for the ABSC task are Laptop14, Restaurant14(Pontiki et al., 2014), Restaurant15 (Pontiki et al., 2015), Restaurant16 (Pontiki et al., 2016), and MAMS (Jiang et al., 2019). The datasets5 used for the NLI task are the SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) datasets, respectively. The split of these datasets is summarized in Table 1. The commonly used Accuracy (i.e., Acc) and macro F1 are used as the metrics for evaluating the performance of different algorithms following existing research (Wang et al., 2016; Zhou et al., 2022a). Additionally, all experiments are repeated five times with different random seeds. Detailed information on the hyper-parameter settings and sensitivity tests of α and β can be found in Appendix A.
4We use the first 10, 000 examples to build the AGNews10K dataset (7, 000 for training, 1, 000 for validation, and 2, 000 for testing), which is large enough compared to other datasets.
5We select the first 1000 training examples as the training set and keep the original validation/testing sets for experimental efficiency.

heading: 3.2 Augment Backends
text: We use BOOSTAUG to improve five state-of-theart baseline text augmentation methods, all of which are used as the text augmentation backend of BOOSTAUG. Please find the introductions of these baselines in Appendix B and refer to Table 6 for detailed performance of BOOSTAUG based on different backends.
We also compare BOOSTAUG enhanced EDA with the following text augmentation methods: • EDA (TextAttack6) (Wei and Zou, 2019) per-
forms text augmentation via random word insertions, substitutions, and deletions. • SynonymAug (NLPAug7) (Niu and Bansal, 2018) replaces words in the original text with their synonyms. This method has been shown to be effective in improving the robustness of models on certain tasks. • TAA (Ren et al., 2021) is a Bayesian optimization-based text augmentation method. It searches augmentation policies and automatically finds the best augmentation instances. • AEDA (Karimi et al., 2021) is based on the EDA, which attempts to maintain the order of the words while changing their positions in the context. Besides, it alleviates breaking changes such as critical deletions and improves the robustness. • AMDA (Si et al., 2021) linearly interpolates the representations of pairs of training instances, which has a diversified augmentation set compared to discrete text adversarial augmentation. In our experiments, LSTM, BERT-BASE(Devlin et al., 2019), and DeBERTa-BASE(He et al., 2021) are used as the objective models for the TC task. FastLCF is an objective model available for the
6https://github.com/QData/TextAttack 7https://github.com/makcedward/nlpaug
ABSC task.

heading: 4 Experimental Results

heading: 4.1 Main Results
text: From the results shown in Table 2, it is clear that BOOSTAUG consistently improves the performance of the text augmentation method EDA across all datasets and models. It is also worth noting that some traditional text augmentation methods can actually harm the performance of the classification models. Additionally, the performance improvement is relatively small for larger datasets like SST-2, SST-5, and MAMS. Furthermore, the performance of LSTM is more affected by text augmentation, as it lacks the knowledge gained from the large-scale corpus that is available in PLMs.
When comparing the different text augmentation methods, it is apparent that EDA performs the best, despite being the simplest method. On the other hand, SplitAug performs the worst for LSTM because its augmentation instances are heavily biased in the feature space due to the word splitting transformation. The performance of SpellingAug is similar to EDA. This can be attributed to the fact that PLMs have already captured some common misspellings during pretraining. Additionally, PLM-based augmentation methods like WordsEmbsAug tend to generate instances with unknown words, further exacerbating the feature space shift of the augmented texts.
We also compare the performance of BOOSTAUG with several state-of-the-art text augmentation methods. The results of these comparisons can be found in Table 3. From the results, it can be seen that even when using EDA (Wei and Zou, 2019) as the backend, BOOSTAUG outperforms other state-of-the-art methods such as AEDA (Karimi et al., 2021), AMDA (Si et al., 2021), and Bayesian optimization-based TAA (Ren et al., 2021) on the full SST2 dataset.

heading: 4.2 Ablation Study
text: To gain a deeper understanding of the working mechanism of BOOSTAUG, we conduct experiments to evaluate the effectiveness of crossboosting, predicted label constraint, confidence ranking, and perplexity filtering. The results, which can be found in Table 4, show that the performance of the variant MonoAug is significantly lower than that of BOOSTAUG. This is because MonoAug trains the surrogate language model using the entire
training set, leading to a high degree of similarity between the original and augmentation instances. This data overlapping problem, as discussed in Section 2.1, results in biased instance filtering and overfitting of the instances to the training fold data distribution. Additionally, the variant without the perplexity filtering strategy performs the worst, indicating that the perplexity filtering strategy is crucial in removing instances with syntactical and grammatical errors. The performance of the variants without the predicted label constraint and confidence ranking is similar, with the label constraint helping to prevent the mutation of features into an adverse meaning and the confidence ranking helping to eliminate out-of-domain words and reduce feature space shift.

heading: 4.3 Feature Space Shift Investigation
text: In this subsection, we explore the feature space shift problem in more detail by using visualizations and the feature space shift metric. We use t-SNE to visualize the distribution of the features of the testing set and compare it to different augmented variants. The full results of feature space shift metrics are available in Figure 6. The results of feature space shift metrics in our experiment show that the augmentation instances generated by BOOSTAUG have the least shift of feature space. Specifically, the overlapping ratio and skewness in relation to the testing set are consistently better than those of the training set. This explains the performance improvement seen when using BOOSTAUG in previous experiments. In contrast, the augmentation instances generated by EDA, which was the best peer text augmentation method, have a worse overlapping rate compared to even the training set. This explains the performance degradation when using EDA on the baseline classification models. It is also noteworthy that the quality of the augmentation instances generated by MonoAug is better than EDA.

heading: 4.4 Effect of Augmentation Instances Number
text: To further understand the effectiveness of BOOSTAUG, we conduct an experiment to analyze the relationship between the number of augmentation instances generated and the performance of
the classification models. We use Acc and F1 as the evaluation metrics and plot the trajectories of these metrics with error bars against the number of augmentation instances generated for an example by using BOOSTAUG. The results are shown in Figure 3. For comparison, the trajectory visualization plots of MonoAug and EDA can also be found in Figure 7. From the results, it is clear to see that the performance of the classification models improves as the number of augmentation instances increases, but eventually reaches a saturation point. Furthermore, it is observed that the performance improvement achieved by BOOSTAUG is consistently better than that of MonoAug and EDA. This further confirms the effectiveness of BOOSTAUG in mitigating the feature space shift problem and improving the performance of the classification models.
However, it is also important to consider the computational budgets required to generate a large
number of augmentation instances, as this can impact the overall efficiency of the text augmentation method being used.

heading: 4.5 Hyper-parameter Sensitivity Analysis
text: We find that there is no single best setting for the two hyper-parameters, α and β, in different situations such as different datasets and backend augmentation methods. To explore the sensitivity of these hyper-parameters, we conducted experiments on the Laptop14 and Restaurant14 datasets and show the Scott-Knott rank test (Mittas and Angelis, 2013) plots and performance box plots in Figure 4 and Figure 5, respectively. We found that the best value of α highly depends on the dataset. For the Laptop14 and Restaurant14 datasets, a value of α = 0.5 was found to be the best choice according to Figure 4. However, it’s worth noting that the smaller the value of α, the higher the computation complexity due to the need for more
augmentation instances. To balance efficiency and performance, we recommend a value of α = 0.99 (α = 1 means no augmentation instances survive) in BOOSTAUG, which reduces computation complexity. Additionally, we found that β is relatively easy to determine, with a value of β = 4 being commonly used.

heading: 5 Related Works
text: As pretraining has advanced, text augmentation techniques have become an increasingly popular area of research (Sennrich et al., 2016; Coulombe, 2018; Li et al., 2019; Wei and Zou, 2019; Kumar et al., 2020; Lewis et al., 2020; Xie et al., 2020; Bi et al., 2021; Ren et al., 2021; Haralabopoulos et al., 2021; Wang et al., 2022c; Yue et al., 2022; Zhou et al., 2022a; Kamalloo et al., 2022; Wang et al., 2022a). Many of these techniques focus on lowresource scenarios (Chen et al., 2020; Zhou et al., 2022a; Kim et al., 2022; Zhou et al., 2022b; Wu et al., 2022; Yang et al., 2022; Wang et al., 2022b; Yang et al., 2022). However, they tend to fail when applied to large public datasets (Zhou et al., 2022a). Recent prominent works (Sennrich et al., 2016; Kumar et al., 2020; Lewis et al., 2020; Ng et al., 2020; Body et al., 2021; Chang et al., 2021; Luo et al., 2021; Wang et al., 2022b) recognize the significance of pre-trained language models (PLMs) for text augmentation and propose PLM-based methods to improve text augmentation. However, the quality of augmentation instances generated by unsupervised PLMs cannot be guaranteed. Some re-
search (Dong et al., 2021) has attempted to use adversarial training in text augmentation, which can improve robustness, but these methods are more suitable for low-sample augmentation scenarios and cause shifted feature spaces in large datasets.
While recent studies have emphasized the importance of quality control for augmentation instances (Lewis et al., 2021; Kamalloo et al., 2022; Wang et al., 2022b), there remains a need for a transferable augmentation instance-filtering framework that can serve as an external quality controller to improve existing text augmentation methods.
Our work aims to address the failure mode of large dataset augmentation and improve existing augmentation methods more widely. Specifically, BOOSTAUG is a simple but effective framework that can work with a variety of existing augmentation backends, including EDA (Wei and Zou, 2019) and PLM-based augmentation (Kumar et al., 2020).

heading: 6 Conclusion
text: Existing text augmentation methods usually lead to performance degeneration in large datasets due to numerous low-quality augmentation instances, while the reason for performance degeneration has not been well explained. We find low-quality augmentation instances usually have shifted feature space compare to natural instances. Therefore, we propose a universal augmentation instance filter framework BOOSTAUG to widely enhance existing text augmentation methods. BOOSTAUG is an external and flexible framework, all the existing text augmentation methods can be seamless improved. Experimental results on three TC datasets and five ABSC datasets show that BOOSTAUG is able to alleviate feature space shift in augmentation instances and significantly improve existing augmentation methods.

heading: Acknowledgements
text: This work was supported by UKRI Future Leaders Fellowship (MR/X011135/1, MR/S017062/1), NSFC (62076056), Alan Turing Fellowship, EPSRC (2404317), Royal Society (IES/R2/212077) and Amazon Research Award.

heading: 7 Limitations
text: We propose and solve the feature space shift problem in text augmentation. However, there is a limitation that remains. BOOSTAUG cannot preserve
the grammar and syntax to a certain extent. We apply the perplexity filtering strategy, but it is an implicit constraint and cannot ensure the syntax quality of the augmentation instances due to some breaking transformations, such as keyword deletions and modifications. However, we do not need precise grammar and syntax information in most classification tasks, especially in PLM-based classification. For some syntax-sensitive tasks, e.g., syntax parsing and the syntax-based ABSC (Zhang et al., 2019; Phan and Ogunbona, 2020; Dai et al., 2021), ensuring the syntax quality of the augmented instances is an urgent problem. Therefore, BOOSTAUG may not be an best choice for some tasks or models requiring syntax as an essential modeling objective (Zhang et al., 2019). In other words, the syntax quality of BOOSTAUG depends on the backend.

heading: A Hyperparameter Settings
text: A.1 Hyperparameter Settings for BOOSTAUG
Some important parameters are set as follows. • k is set to 5 for the k-fold cross-boosting on all
datasets. • The number of augmentation instances per exam-
ple Ñ is 8. • The transformation probability of each token in a
sentence is set to 0.1 for all augmentation methods. • The fixed confidence and perplexity thresholds are set as α = 0.99 and β = 5 based on grid search. We provide sensitivity test of α and β in Appendix C.2. • The learning rates of base models LSTM and DeBERTa-BASE are set as 10−3 and 10−5, respectively. • The batch size and maximum sequence modeling length are 16 and 80, respectively. • The L2 regularization parameter λ is 10−8; we use Adam as the optimizer for all models during the training process.

heading: B Baseline Backends
text: We use BOOSTAUG to improve five state-of-the-art baseline text augmentation methods, all of which are used as the text augmentation back end of BOOSTAUG. Please refer to Table 6 for detailed experimental results. • EDA(TextAttack8) (Wei and Zou, 2019) performs
text augmentation via random word insertions, substitutions and deletions. • SynonymAug(NLPAug9) (Niu and Bansal, 2018) replaces words in the original text with their synonyms. This method has been shown to be effective in improving the robustness of models on certain tasks. • SpellingAug (Coulombe, 2018): it substitutes words according to spelling mistake dictionary. • SplitAug (Li et al., 2019) (NLPAug): it splits some words in the sentence into two words randomly. • BackTranslationAug (Sennrich et al., 2016) (NLPAug): it is a sentence level augmentation method based on sequence translation. • ContextualWordEmbsAug (Kumar et al., 2020) (NLPAug): it substitutes similar words ac-
8https://github.com/QData/TextAttack 9https://github.com/makcedward/nlpaug
cording to the PLM (i.e., Roberta-base (Liu et al., 2019)) given the context.

heading: C Additional Experiments
text: C.1 Natural Language Inference Experiments
The experimental results in Table 5 show that the performance of both BERT and DeBERTa models can be improved by applying BOOSTAUG. With BOOSTAUG, the accuracy of the BERT model on SNLI improves from 70.72% to 73.08%, and on MNLI from 51.11% to 52.49%. The DeBERTa model also shows significant improvement with EDA, achieving 86.39% accuracy on SNLI and 78.04% on MNLI. These results demonstrate the effectiveness of BOOSTAUG in improving the generalizability of natural language inference models, and its compatibility with different state-of-the-art pre-trained models such as BERT and DeBERTa.
C.2 Hyper-parameter Sensitivity Experiment
We provide the experimental results of BOOSTAUG on the Laptop14 and Restaurant14 datasets in Figure 5.
C.3 Performance of BOOSTAUG on Different Backends
To investigate the generalization ability of BOOSTAUG, we evaluate its performance based on the existing augmentation backends. From the results shown in Table 6, we find that the performance of these text augmentation back ends can be improved by using our proposed BOOSTAUG. Especially by cross-referencing the results shown in Table 2, we find that the conventional text augmentation methods can be enhanced if appropriate instance filtering strategies are applied.
Another interesting observation is that PLMs are not effective for text augmentation, e.g., WordEmdsAug is outperformed by EDA in most comparisons10. Moreover, PLMs are resourceintense and usually cause a biased feature space. This is because PLMs can generate some unknown words, which are outside the testing set, during the pre-training stage. Our experiments indicate that using PLM as an augmentation instance filter, instead of a text augmentation tool directly, can help alleviate the feature space shift.
C.4 Visualization of feature space Figure 6 shows the feature space shift of the ABSC datasets, where the augmentation back end of BOOSTAUG is EDA.
C.5 Trajectory Visualization of RQ4 Figure 7 shows the performance trajectory visualization of MonoAug and EDA. Compared to BOOSTAUG, MonoAug and existing augmentation methods usually trigger performance sacrifice while augmentation instances for each example are more than 3.
10In fact, we also tried some other PLM-based augmentation back ends, e.g., BackTranslationAug, and we come up with same observation.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3
3 B1. Did you cite the creators of artifacts you used? 3

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: The licenses of all artifacts are well announced on their websites

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. The documentation of all artifacts are well organized on their websites
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 3
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix B.1

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: This work was supported by UKRI Future Leaders Fellowship (MR/X011135/1, MR/S017062/1), NSFC (62076056), Alan Turing Fellowship, EPSRC (2404317), Royal Society (IES/R2/212077) and Amazon Research Award