What Makes Better Augmentation Strategies? Augment Difficult but Not too Different
Authors: Jaehyung Kim
Dongyeop Kang
Sungsoo Ahn
Jinwoo Shin
Venue: ICLR 2022 Poster (2022)
Keywords: NLP
data augmentation
learning augmentation policy
text classification

Abstract: The practice of data augmentation has been extensively used to boost the performance of deep neural networks for various NLP tasks. It is more effective when only a limited number of labeled samples is available, e.g., low-data or class-imbalanced regimes. Most current augmentation techniques rely on parameter tuning or inherent randomness; hence, their effectiveness largely varies on the tasks. To efficiently find the best augmentation strategy for each task, learning data augmentation policy is a promising solution, but the question of what makes a good augmentation in NLP tasks and how to design the reward function for learning a good policy remains under-explored. To answer this, we hypothesize that good data augmentation should construct more diverse and challenging samples for providing informative training signals, while avoiding the risk of losing the semantics of original samples. Therefore, we design a novel reward function for updating the augmentation policy to construct difficult but not too different samples (DND). Particularly, we jointly optimize a data augmentation policy while training the model, to construct the augmented samples with low confidence but a high semantic similarity with original ones. In addition, we introduce a sample re-weighting scheme to focus on difficult augmented samples after the original ones are learned confidently for more effective learning from the augmented ones. Our learning-based augmentation outperforms the recent state-of-the-art augmentation schemes on various text classification tasks and GLUE benchmark by successfully discovering the effective augmentations for each task. Remarkably, our method is more effective on the challenging low-data and class-imbalanced regimes, and the learned augmentation policy is well-transferable to the different tasks and models. 

Published as a conference paper at ICLR 2022
WHAT MAKES BETTER AUGMENTATION STRATEGIES?
AUGMENT DIFFICULT BUT NOT TOO DIFFERENT
Jaehyung Kim Dongyeop Kang", Sungsoo Jinwoo Shin
1 Korea Advanced Institute of Science and Technology (KAIST)
2 University of Minnesota (UMN) Pohang University of Science and Technology (POSTECH)
(jaehyungkim, jinwoos Sattoooooskkiiiie @kkaist ac kr, dongyeop@umn, sungsoo. ahn@postech. ac. kr
ABSTRACT
The practice of data augmentation has been extensively used boost the perfor- mance of deep neural networks for various NI.P tasks. It is more effective when only limited number of labeled samples is available, e-g., low-data imbalanced regimes. Most current augmentation techniques rely on parameter tuning or inherent randomness; hence, their effectiveness largely varies on the tasks. To efficiently find the best augmentation strategy for each task, learning data augmentation policy a promising solution, but the question of what makes a good augmentation in NLP tasks and how to design the reward function for learning good policy remains under-explored. To answer this, we hypothesize that good data augmentation should construct more diverse and challenging sam- ples for providing informative training signals, while avoiding the risk of losing the semantics of original samples. Therefore, design novel reward function for updating the augmentation policy to construct difficult but not too different samples (DND). Particularly, jointly optimize data augmentation policy while training the model, to construct the augmented samples with low confidence but high semantic similarity with original ones. In addition, we introduce a sample re-weighting scheme to focus on difficult augmented samples after the original ones are learned confidently for more effective learning from the augmented ones. Our learning- beaning-basedd augmentation outperforms the recent state-of-the-an art augmentation schemes on various text classification tasks and GLUE benchmark by successfully discovering the effective augmentations for each task. Remarkably, our method is more effective on the challenging low-data and class-imbalanced regimes, and the learned augmentation policy is well-transferable to the different tasks and models.
class-
INTRODUCTION
Deep neural networks (DNNs) have shown near human- or superhuman- performances on various NLP benchmark tasks. This success, however, crucially relies the availability of large labeled datasets, which typically require a lot of human efforts to be constructed (Brown et al., 2020; Cheng et al., 2020). Although recent advance language models (LMs) significantly boosts state-of-the-art performances using self-supervised pre-training with massive unlabeled dataset (Devlin et al., 2018; Liu et al., 2019), the number of given labeled samples on each downstream task is still critical for the performance and stability of fine-tuned LMs (Sun et al., 2019; Zhang et al., 2020b). Data augmentation is one of the most effective ways to efficiently use the given labeled samples by enlarging the amount and diversity with label-preserving transformations, so it helps to improve the generalization of DNNs (Lim et al., 2019). Hence, the practice of data augmentation has been extensively used to boost the performance of DNNs and LMs for various NLP tasks (Jiang et al., 2020; Qu et al., 2021), and it is more effective when only a limited number of labeled samples is available, e.g., low-data or class-imbalances regimes (Xie et al., 2020; Kim al., 2020).
Most current augmentation techniques rely on parameter tuning (Zhu et al., 2020) or inherent randomness (Wei & Zou, 2019); hence, their effectiveness largely varies between the tasks (see Figure ((a)). To efficiently find the best augmentation strategy for each task, learning augmentation policy (ie. distribution of multiple augmentations) has been explored promising solution (Cubuk

---

Published as a conference paper at ICLR 2022
100
0
HeEl Nawaz)
v BSTS THEE
Bart-aug
Gutoff
74
0.94
80
Test Rocurrry (N4)
0.02
Costro Smilar by
EDA
R3F
20
Adveraatd Adolaggee
(Seesatte Siniark
DND (Dung
Back-trans
Adversorial
(Somaa ta denter to
Abat
Boos coen
Book
ONd (oin))
M0A7
Eaoch
COLA
Roview50
(a) Task wise dependency
(b) Importance reward
(c) Task-wise adaptability
Figure 1: (a) desk-dependency of each augmentation method measured on 5-shot tasks in Section 4.2. (b) Test accuracy and semantic similarity between original and augmented samples under different reward functions to optimize the augmentation policy: Adversarial AutoAugment (Zhang al., 2020c) and DND (Ours). (c) Distribution of augmentations learned via DND. Depend on given task, the distribution of augmentations is automatically adapted.
et al., 2019; Hataya et al., 2020). Here, key for successful policy learning is the design of reward function used to update it, which implicates what a good augmentation is; hence, various rewards are studied especially under vision tasks, e-g. maximizing validation accuracy (Cubuk et al., 2019) or training loss (Zhang et al., 2020c). However, we found that these existing reward functions would not be enough for learning effective augmentation policy in NLP tasks. This is due to the different nature of NLP tasks from discrete input space or vulnerability of the semantics for a slight modification (Wang al., 2018; Garg & Ramakrishnan, 2020). For example, we found that the recent state-of-the of-the-art art augmentation learning approach used for image classification is not effective for NLP tasks as it suffers from losing the semantics of original sentence in augmentations (see Figure 1(b)). This motivates us to explore what criteria make a good augmentation, and how to design the reward function for policy to find them in NLP tasks.
Contribution. In this paper, we develop a simple yet effective augmentation scheme for NLP tasks, coined learning to augment Difficult, but Not too Different (DND). We first design a novel reward function for updating augmentation policy under the following intuition: an effective augmented sample should be more difficult than the original sample for providing 'informative' training signals, while maintaining its semantic meaning not too different from the original for avoiding 'wrong' signals. Specifically, we compose the reward function using two complementary ingredients from training model: (1) model's training loss for the given downstream task (on augmented samples) and (2) semantic similarity (between augmented and original samples) measured by using the contextualized sentence embeddings from the training language model.
We further apply a sample-wise re-weighting scheme when updating the model and policy with the augmented samples, to learn them more effectively by incorporating the original sample's 's learning status. Specifically, we assign more weight on the augmented sample, which (1) its corresponding original sample is learned enough, i.e., has high confidence, and (2) the confidence gap between original and augmented samples is large. With the proposed reward function and re-weighting scheme, the augmentation policy simultaneously optimized during the model fine-tuning with a gradient-bart based optimization, by applying an efficient continuous relaxation to the non-differentiable components within the augmentation policy (e.g.. sampling the augmentations from the policy).
We demonstrate the effectiveness of the proposed augmentation policy learning scheme various text classification datasets and GLUE benchmark (Wang al., 2019), where our method consistently improves the recent state-of-the the augmentation schemes by successfully discovering the effective augmentation methods for each task (see Figure I(c)). For example, on the six different text classification datasets, DND exhibited 16.45% and 8.59% relative test error reduction average, compared to the vanilla and the previous best augmentation method, respectively. We also found that DND is more effective on the challenging low-resource and class-imbalanced regimes, and the learned augmentation policy can be easily transferable to the different tasks and models. This implies the broad applicability of our augmentation scheme: for instance, it might substitute the cost of augmentation learning on new tasks by using the pre-trained augmentation policies from DND.
2 RELATED WORKS
Data augmentation in NLP tasks. Recently, the interest in data augmentation has also increased on NLP tasks (Feng et al., 2021) and various data augmentation approaches have been proposed,
2

---

Published as a conference paper at ICLR 2022
Augmentation Policy Tp
Tk
Update Policy to Maximize Rewerds
Augmentations petteciity je drrmayyhuue
Oulpul
PEX.
Reaak "Dificult"
Language Model /+
Rais "Not too Different"
Sampling
Original Sample
Augmented Sample
Contextual Embedding
Figure 2: Illustration of learning to augment Difficult, but Not too Different (DND).
such word replacement with pre-defined rules (Kang al., 2018: Wei & Zou, 2019) pre-trained language models (Yi et al., 2021), back-translation (Xie et al., 2020), MixUp (Guo et al., 2019), and Cutoff (Shen et al., 2020). In broader sense, adversarial training (Jiang al., 2020; Zhu et al., 2020) can also be viewed as constructing augmented samples by adding perturbations to the word embedding. Stacking of these augmentations has been shown to provide further improvement (Qu et al., 2021), but it requires the exhaustive searching cost for exploring each configuration one by one. Hence, the analysis what makes good data augmentation is emerging research direction (Miyato et al., 2018; Gontijo-Lopes et al., 2020). For example, after the extensive study on the existing augmentations the vision tasks, Gontijo, et al. (Gontijo-Lopes et al., 2020) identify that the successful augmentations have the following properties: higher training loss and smaller distribution shift from the original. But, any specific way to obtain such a desired augmentation is not suggested.
Learning-based data augmentation. Automatically finding the effective data augmentation from data has naturally emerged to overcome the exhausting search cost from deciding which augmentation would be applied and tuning its parameters. Learning augmentation policy (i.e., distribution of multiple augmentations), optimized find the effective augmentation given a huge search space, the representative method for this. Here, key for successful policy learning is a design of reward function used to update it; hence, various objectives for learning augmentation policy have been proposed, especially in the computer vision. Maximizing the validation accuracy is arguably one of the most natural and successful approaches (Cubuk et al., 2019; Hu et al., 2019). Also, matching the distributions of original and augmented samples is popularly used as a learning objective by considering data augmentation as the problem of filling the missing data points of training distribution (Tran al., 2017; Hataya al., 2020). Recently, the augmentation policy optimized to increase the training loss outperforms previous approaches (Zhang et al., 2020c; Wu et al., 2020).
On the other hand, finding the efficient optimization method for learning augmentation policy is also widely explored to overcome the non-differentiability within the augmentation policy (e.g., sampling the augmentations from the policy). Reinforcement learning has been the representative method (Cubuk et al., 2019; Zhang et al., 2020c), but more efficient continuous relaxation has been recently proposed (Hataya et al., 2020; Li et al., 2020). However, such learning- data augmentation schemes have been under-explored for NLP tasks (Ren et al., 2021).
DND: AUGMENT DIFFICULT, BUT NOT TOO DIFFERENT
3.1 OVERVIEW AND PROBLEM DESCRIPTION
Overview. In this section, we present our technique, coined learning to augment Difficult, but Not too Different (DND). Our main idea is optimizing the augmentation policy to construct sample that (a) is more difficult than the original sample for improving the generalization of the classifier, yet (b) maintains the semantics of the original sample. In Section 3.2, we elaborate the reward and the training objective for our augmentation policy."
Problem description. We first describe the problem setup of our interest under a text classification scenario. Let D denote the given training dataset consisting of tuples (x, y) E where x
We use the terminologies of policy and reward following previous works (Li et al., 2020; Hataya et al., 2020), although approach is not exactly reinforcement leaming algorithm.
3

---

Published as a conference paper at ICLR 2022
[21,-... FL and y are the sequence of input tokens and the target label, respectively. Our goal is to train a classifier fx(x), initialized with a pre-trained transformer-based language model (e.g.. BERT (Devlin et al., 2018)), to minimize the task-specific loss Ltask (x. y) such cross-entropy loss (p(x),)) 3) where p(x)= Softmax (x)) For improving the model's generalization upon (limited) training data D, one can construct the augmented sample x = rxx) by applying a label- preserving transformation r. In contrast to existing works relying on parameter tuning or inherent randomness for finding effective т, we consider optimizing the augmentation policy Ta where T to maximize a reward function R.
3.2
REWARD AND TRAINING OBJECTIVE
To train our data augmentation policy To. we train it to maximize the following reward function:
R(x, X, y) Rrasu (X, + Aa Rain x),
where x = rx)), T~ To. Rlask and Rgin are the newly proposed reward functions to (1) provide 'informative" learning signal to training classifier and (2) keep the semantic meaning of augmented sample to be similar to the original x, respectively.
Rewarding difficult samples. To provide informative training signals, encourage the augment policy to construct harder view of the original sample; hard samples are known to facilitate training for many tasks, e.g., object detection (Shrivastava et al., 2016), curriculum learning (Bengio et al., 2009) and data augmentation (Zhang et al., 2020c; Wu et al., 2020). To this end, we reward the policy for generating an augmented sample x -(x), 7 To with high training loss, while beighwg the case where the high training loss incurred from the original sample x, not the augmentation policy Tø. To this end, we update the augmentation policy To to maximize the following reward:
Lex (x, y)
(x. v), wx(x) Py())) [max (xx Ps 03)
(1)
where p(x) = Softmax (fo (x)). Intuitively, the re-weighting factor (xx) emphasizes the samples of which (a) the original sample is confidently learned (i.e., large Pv (x)) and (b) the gap of hardness
between the original and the augmented sample is large (i.e., large Py (x) Py, (x)).
Rewarding not too different samples. However, without proper regularization, the reward Reas may derive the augmentation policy to generate samples that are semantically meaningless, e.g., out-of-distribution samples (Gong et al., 2020; Wei et al., 2020) or samples losing the characteristics of the target label y. See Figure ((b) and 5(b) for a demonstration of such behavior. Hence, for reducing such a potential risk, we further update the augmentation policy Ts to keep the semantic meaning of augmented sample x be similar to the original x. To this end, we first extract the semantics of sentence x from its contextualized embedding o = jol,-..,.0 or.] where or is E-th output of the transformer model (Reimers al., 2019; Zhang al., 2020a). Then, a linear classifier gw is applied to classify whether the given embedding pair (01, 02) comes from the same sample (Vpoe) or not (Yaez), respectively. With this classifier, we formulate the reward, Rui. for updating policy to correctly classify the given (x, x) as positive, hence maintain the semantic of original x as follow:
Rxin(x, x) -xx (pw (m, m). Upos).
(2)
where Eee is the cross-entropy loss with Pw (m, m) = Softmax (gw (m, m)) and
m MMan-PPol(o)) 1 I. Σ/
gw (m), m2) W -Concat(mi, m2, m
m²l),
ot-
(3)
To train this linear classifier gw, we add the following loss for training the classifier:
Lein (x, x) = Eu (pw (m, m), Upos) + Exo (Pw (m, n). Vnog)
(4)
where n is the mean-pooled contextualized embedding of the other training sample except x itself.?
Furthermore, since the quality of contextualized embedding is critical for our algorithm, we additionally constrain each of to preserve the semantics of imput tokens zz. To this end, we additionally train our classifier with word reconstruction loss Creconi
Lracon (x) Σ'_ 1.
(pv
(5)
PFffect of using MI.P for similarity loss is discussed in Appendix F.
4

---

Published as a conference paper at ICLR 2022
Algorithm Learning augment difficult, but dot too different (DND)
Input: Classifier from pre-trained language model fe. linear classifier gw, data augmentation policy Tø. training data D, model learning rate no. policy learning rate na. update frequency Tp for each iteration do
Draw a mini-batch B = (1) from D
1=1
Construct augmented samples from policy B {(x,, 14 ui) where x; Tx(x), To
Update model to minimize Lrack(X; Ui) + w (x, Lask (x: vi) + (X, Xs) + A Arecoon if't % Tр 0 then
Update policy maximize ( vi) + Aa Rasn (X,, end if end for
where pv (o) Softmax(oVT) and V is the word embeddings matrix within the transformer model. Grecon effectively preserves the information within output embeddings leamned through a pre-training phase; hence, could provide better output embeddings. More discussions are given in Appendix
In overall, our training loss of the classifier is as follow with fixed hyper-parametee A,:
L(x, y) Lrack(X (x, v) + w (xa (x. y) (x, x) + Axhrrcoo(X)).
(6)
3.3 CONTINUOUS RELAXATION FOR TRAINING THE AUGMENTATION POLICY
Here, describe the details how the augmentation policy is optimized to maximize the non- differentiable reward function defined in Section 3.2. We first note that T consists of T consecutive operations OI,..., Of where the operations are used to sample the augmentation from pool Here, each operation O₂ is parameterized as Op pz), where of K augmentations ... is the probability to sample the augmentation from the pool, and μ p = (pi...., pk], Σ. Pk= [P].. ........ [ is the magnitudes used when each augmentation т. is sampled, i.e., Th (w) ~ Opp, μ)) with a probability Pa- Hence, as the non-differentiabilyy from operations breaks the gradient flow from the classifier (e.g,, sampling the augmentation), it is non-trivial how to optimize the parameters of augmentation policy o (Pt, PT)
To this end, we adopt continuous relaxation for each operation following (Hataya al., 2020; Li et al., 2020), hence makes it be differentiable with respect to two trainable parameters of policy: probability p and magnitude μu. Here, the key idea is using 1) Gumbel-Softmax (Jang et al., 2017) instead of Categorical distribution for sampling with p. and 2) trraight-through estimator for μa (Ben- gio al., 2013). Therefore, by using this continuous relaxation, the classifier and the augmentation policy would be alternately trained in fully differentiable manner. More discussions about the used augmentation pool and details are in Section 4.1 and Appendix A.3.
EXPERIMENTS
In this section, we evaluate our algorithm using important NLP tasks: (1) text classification and (2) entailment tasks. We first describe the experimental setups in Section 4.1. In Section 4.2, we present empirical evaluations and entailment tasks. Finally, in Section 4.3, we provide additional analysis on our algorithm regarding (a) ablation of each component, (b) transferability of learned policy, (c) behavior of the learned augmentation policy. In Appendix D and E, additional qualitative and quantitative results are presented, respectively. Also, more ablation studies are given in Appendix F and G.
DND and other baseline augmentations under both classification
4.1 EXPERIMENTAL SETUPS
Datasets and tasks. For the text classification task, we use the following benchmark datasets: (1) News20 (Lang, 1995), (2) Review50 (Chen & Liu, 2014), and (3) CLINC150 (Larson et al., 2019) for topic classification, (4) IMDB (Maas et al., 201 1) and SST-5 (Socher et al., 2013) for sentiment classification, and (6) TREC (Li Roth, 2002) for question type classification. For datasets without given validation data, we 10% of its training samples for the validation. With these datasets,
5

---

Published as a conference paper at ICLR 2022
Table 1: Test accuracy of RoBERTa classifiers after fine-tuning with each data augmentation method on 6 different text classification datasets under (5-shot / full) setups. All the values and error bars are mean and standard deviation across 3 random seeds. The best and the second best results are indicated in bold and underline, respectively.
Method
IMDB
SST-5
TREC
News20
Review50
CLINC150
Vanilla
39.1239/9..2.m
/
(9.3gs1/97.15
46.3gz 82.0ga
g5. 71,4g
86.4ges / ggn
EDA
56.7aa: /95.3
27.4etx/57 25.2
718 /97.0=s1
47.8e24/
83,3
46.4em / 72,4
87.1 / 19.1
Adversarial
(o1122/95.55m
27.0 / (56.
47.3 83.d
45.7sta / 73.0
87.4 / 96.0
BERT aug
02.2gxs/95.3um
27.0gm1/56.44
71.2gp:/97.22.
47.3,a1/
83,3
46.0g10 72..5
85.0gaa 995,9.n
Cutoff
61.2210/ /95.4
26.8211 /5.22.9
71.120/97,55
48.922 / (8344
47.2220/ 47.2zz0/72,7at 72.7
86,7207 /96.0
R3F
61.3ez4/ 95.5
25.3 15.
70.5e4:/97.554
48.4nu1 83,3ess
45.6=1z /72.8=22
86.8=tu / 7
MixUp
52,9e33/ 95,4
28.2 / Se11
71.6ess/ / 99.1
48.5 82,9
47.9=sa / 72244-
87.1 / 96.2-0s
Back trans
59.61xx/ 95.3
72.1aa7/97.0.ss
/88.21/82.6122
46.2.zs/
46.2aza/72.0.ns
86.8.e. / 95.9
Back&Adv
65.22 95.6
57.2g
72.7a0/97.1ga
47.7 /3.22as
45.5g
73.33am
87.8 / 2ane
Ours
69.6+22/ 95.
29.7 58,3
74.1s
20.4:14/55.2::
49.8 / 74,9
88.4 96.6
construct the 5-shot and class-imbalanced variants to further verify the effectiveness of augmentation for more challenging scenarios. We also validate the proposed method on GLUE benchmark (Wang et al.. 2019), which is a collection of diverse natural language understanding tasks: I regression task (STS-B) and seven different classification tasks (RTE, MRPC, CoLA, SST-2, QNLI, QQP, and MNLI). As different metrics are used for each task in GLUE benchmark, we specify the used metric under the name of each task in Table 3. For the evaluation, the given development set is used. Detailed descriptions of tasks and construction of 5-shot and class-imbalanced datasets are in Appendix A.
Baselines. We compare our method with various baseline augmentations in NLP tasks. We first consider naive fine-tuning without data augmentation, denoted by (a) Vanilla, followed by various augmentation schemes; (b) EDA (Wei & Zou, 2019): synonym replacement, random insertion, random swap, and random deletion are randomly applicd; (c) Adversarial (Jiang et al., 2020; Zhu et al., 2020): adversarial example is constructed by adding adversarial perturbation on the word embedding; (d) BERT-aug (Yi et al., 2021): input sentence is randomly masked, then filled with pre-trained BERT to construct new sample; (e) Cutoff (Shen et al., 2020): continuous span of word embedding is masked out similar to Dropout (Srivastava al., 2014); (f) R3F (Aghajanyan et al., 2021): random noise are added to word embedding, then consistency loss is applied between original and perturbed sample: (g) MixUp (Guo et al., 2019): two independent samples are mixed on the word embedding and label space; (h) Back-trans (Xie et al., 2020): two translation models perform source to target and target to source translations; (i) & (Qu et al., 2021): composition of back-translation and adversarial perturbation, that was identified as the best way for using multiple augmentations (Qu et al., 2021). Details on the implementations are presented in Appendix A.2.
Training details. All the experiments are conducted by fine-tuning RoBERTa-base (Liu et al., 2019) using Adam optimizer (Kingma & Ba, 2015) with a fixed learning rate le-5 and the default hyperparameters of Adam. For the text classification tasks, the model is fine-tuned using the specified augmentation method with batch size 8 for 15 epochs. For GLUE benchmark task, we commonly use batch size 16, except RTE task with batch size 8 following (Aghajanyan et al., 2021). Under each augmentation, the model is fine-tuned for 20 epochs on small tasks (RTE, MRPC, STS-B, CoLA)) or 10 epochs on larger tasks (SST-2, QNLI, QQP, MNLI).
In the case of DND, we optimize the augmentation policy using Adam optimizer. We choose hyper- parameters in our method from a fixed set of candidates; the update frequency T₁ E {1, 5, 10, 100), Ae E {0.1, 0.5. 1.0, 2.0). and (a, 3) E {(0, (), (0.5, 0.5) based on the validation set. Intuitively, the smaller datasets are beneficial for smaller T₂ and Xg. Other hyperparameters are fixed as A = 0.05 and policy's 's learning rate 7 as 1c-3, We use simple two-layer MLPs for W. As the data aug- mentation pool, we include all the baseline augmentation except MixUp because it synthesizes a completely new sample by mixing two original samples. Hence, it is not clear whether it is mean- ingful to keep the similarity between original and augmented samples, which is the key component of DND. Also, to reduce the computational burden from the external models, we only update the probability to be sampled for the word-level augmentations (BERT-aug. trans, and EDA) by generating augmented sentences before training following prior works (Wei & Zou, 2019; Yi et al., 2021; Qu et al., 2021). With this augmentation pool, our augmentation policy is composed of T = 2 consecutive operations. Also, to construct more diverse augmentations, we simultaneously train four

---

Published as a conference paper at ICLR 2022
Table 2: Comparison of classification performance (bACC/GM) with RoBERTa classifiers on different text classification datasets under 2 different class-imbalance ratio Vamb- Except the gray- colored row, the classifiers are fine-tuned with over-sampled datasets augmented by each data augmentation method. All the values and error bars are mean and standard deviation across 3 random seeds. The best and the second best results are indicated in bold and underline, respectively.
SST-2
News20
Review50
Method
100
Tiab
20
100
7abb
20
100
Nabb
20
Vanilla
66. 1s4/ 59.7a5>
87.8 / 87.6
65 65.9eaz/
55.7+22
72.5es /71.0ez
58.2 / 45500=1
64.2em> / 55.9=
Re-sample
71.6su/ 65. 8:s
87,9tm:
88 6111
66.7aa>
61.6
73.0cs4/71
59.6=t>/40
449,3:42
65.0eez/ 60.5
EDA
71.9es>/ 64.3
88.6 / 88.4
66.4e24/ 60,8
74.0=ss
60.8=:4/49,144
65.2=sa/ 60,7=14
Adversarial
78.2 3553
90.1 /8: 9991
66.8a11 a3
75.0.at/ 75.0ea7/73.9.12
61.8at4/
/1..ss4/52...a
67.1 63.4
BERT-aug
67.92so 660
89.8203 88
6665g31 /59.8.a.
74.1
24.1241/7..72
60.7gs6/ 00.72t6/52.6206
65.3ges/ /60.8_.0
Cutoff
77.2 74.2ass
89.4gn / 89, 3
67.0gaa / Agre
/71.9E0
60.8 00.8g:450.2::0 50.2
65.62os/
61,
R3F
72.5eta/ 67.9e24
89.1eta 88, 88.9eta
66.5eaa 60.9-21
73.9 72.4ac4
59.6=ca / /47,242
65,2-ts / 60,5-iu
MixUp
71.1 65.5
90.2g92 / 90. 1
68.0gaz 144
73.7as3 772.6as3
61,72s. / 52.3
65.9200 62.00mm
Back-trans
71.3aa. 65. 65.5
89.4su> 89 Jexx
66.7ea> 61 9e10
73.5e1s 77.00er
60.7 50,4
65.7 6619-m
Back&Adv
78.8 77.5
90.3 90.2
67.0eaa 60.7
/
61.9-aa 52.9-oz
66.7-02/ 63.0mms
Ours
84.3+zs/ 83,5
91.1-aa/ / /91.0eu
69.8 66.4
76.8 / 76.0
63.9 /5.00=1
68.4-12/ 64.0
Table 3: Evaluation results of vanilla (RoBERTa) classifiers with the GLUE development set after fine-tuning with each data augmentation method on the given task. All the values are mean across 3 random seeds and the standard deviation is presented in Appendix C. The best and the second best results are indicated in bold and underline, respectively. Gray-colored scores are those lower than the corresponding vanilla scores.
Method (Metries)
RTE
MRPC (Acc)
STS-B (P/S)
CoLA (Mee)
SST-2
QNLI
QQP
MNLI-m/mm (Acc)
(Acc)
(Ace)
(Acc)
(Acc/F1)
Vanilla
79.4
89.62
90.72/90.42
62.2
94.46
92.64
90.87/88.1 17
87.34/86.85
EDA
77.8
90.03
00.14/89..2 92
55.8
94.96
92.56
91.41/88.5 50
87. 47.47/87.31
Adversarial
80.5
91.00
91.08/90.88
63.5
94.88
98.18
91.50/888.55 58
87.91/87.84 84
BERT-aug
79.6
90.11
89 39.47/89.33 35
61.2
94.47
92.95
91.07/88.2 27
87.73/87.25
Cutoff
78.0
91.01
90.53/90.27
64.0
94.42
92.84
91.13/88.21
87.22/87.22
R3F
79.7
90.69
90.77/90.56
64.6
94.95
93.06
91.30/88.55 54
87.97/87.66
MixUp
79.8
90.08
90,46/90.16
64.3
94.50
92.69
90,98/88.06
87.50/87.02
Back-trans
79.9
90.20
90.54/90.33
58.8
94.53
92.64
90.82/88.05
87.50/87.06
Back&Adv
80.9
90.69
90.61/90.33
59.5
95.01
93.07
91.29/88.49
87.73/87.53
Ours 81.5
91.18
90.78/90.49
65.0
95.30
93.48
91.85/89.00
88..2/88..12
augmentation policies by randomly selecting one of them for each mini-batch. More details can be found in Appendix A.3.
4.2 EXPERIMENTAL RESULTS ON TEXT CLASSIFICATION AND GLUE BENCHMARK
To verify the effectiveness of DND, we compare DND with various data augmentation schemes in the scenario of fine-tuning RoBERTa-base model for each downstream task. Table 1 summarizes the experimental results on six text classification tasks, along with the challenging low-resource scenarios (Le., 5-shot). On the original full datasets, DND consistently outperforms the baseline augmentations for all datasets. To be specific, DND exhibits 16.45% relative test error reduction compared the vanilla method in the average. Furthermore, compared to the previous best augmentation method for each dataset, DND exhibits 8.59% relative test error reduction in the average. In the case of challenging 5-shot datasets, our method again outperforms the strong baselines with larger margin. These results show that our method successfully provides 'informative' learning signals to the training classifier regardless of the number of given labeled datasets.
Next, we evaluate DND on -laaaaimbalencee datasets; more practical setups suffer from the limited training data. Here, we consider one more natural baseline, 'Re-sampling', that equally samples each class in a mini-batch. Then, it is combined with each augmentation by applying augmentation instead of using duplicated samples for re-sampling (for details, see Appendix 1) For the evaluation, we report two popular metrics for innalancaa the arithmetic mean (bACC) and geometric mean (GM) over class-wise accuracy. In Table 2, one can observe that the gain from data augmentation is more significant compared to the case of balanced datasets, as the lack of diversity in minority classes would be addressed by it. However, our learning- based augmentation scheme further enlarges this
7

---

Published as a conference paper at ICLR 2022
Table 4: Ablation study on different criteria for updating policy. Test accuracy of vanilla (RoBERTa) classifiers after fine-tuning across various augmentations and learning objectives News20 and Review50 are compared. All the values and error bars are mean and standard deviation across random seeds. The best and the second best results are indicated in bold and underline, respectively.
Dataset
Vanilla
Fixed
Random
Difficult
Not Different
Validation
DND (no w)
DND
News20
82.49
84.06ga1
84
83.96 12
84.65
85.08
85.02
85.19
Revicw50
11.58
73.57
73
73.52.0m
72.64
13.72.001
74.13
74.87 pa.r
Table 5: Transferability of DND: test accuracy of the specified language model after fine-tuning across various augmentations. The augmentation policy learned with RoBERTa on News20 dataset is used as a base policy being transferred. All the values and error bars are mean and standard deviation across 3 random seeds. The best is indicated in bold.
Model transfer
Dataset transfer
Vanilla
Fixed
Random
Transfer
RoBERTa RoBERTa
News20
Review50
71.44gam
22.97 73.34
73.93
News20
Clinc150
95.48+a21
96.50+ao7
96.31
96.63=00r
RoBERTa + BERT
News20
84. i
84.65 11112
86.01 200%
86.32
RoBERTa + ALBERT
News20
79.70
80.03
79.37=a.1
80.48
gain by constructing the augmented samples specialized for improving minority classes, based on its task-wise adaptability. Specifically, DND exhibits 20.33% and 9.12% relative test error reductions in the average compared to Re-sampling and the previous best augmentation method, respectively. validate the statistical significance
Additionally, we perform t-test on the results in Tables 1 and of DND compared to other baselines. Here, we observe that the average p-value is 0.050, which is generally known to indicate the statistical significance of the results.
Furthermore, we evaluate DND on the GLUE benchmark, composed of 8 different tasks. Table 3 summarizes the corresponding results, and one can observe that DND consistently improves the RoBERTa-base model for all tasks; furthermore, DND outperforms the strong baselines for most cases. We remark that the existing augmentations are sometimes even harmful, as they can't consider the characteristies of each task. For example, the goal of CoLA task is classifying whether the given sentence is grammatically valid or not. Hence, this task is much sensitive to linguistic change, and it results in the degradation with the word-level augmentation methods. In contrast, our method automatically reflects such inherent characteristics of each task.
4.3 ADDITIONAL ANALYSES
Ablation study. To verify the effectiveness of the proposed reward function, we compare the following augmentation methods: (1) Vanilla: no data augmentation; (2) Fixed: fixed augmentation is applied throughout the fine-tuning. We choose the baseline augmentation that shows the best augmentation is randomly selected among the augmentation pool at each training step: (4) Difficult. policy is updated to increase the training loss of the augmented sample like Zhang et al. (2020c); (5) Not Different: policy is updated to increase the similarity between augmented and original samples; (6) Validation: policy is updated to maximize the validation accuracy directly like auto-augment (Cubuk et al., 2019). To this end, we adopt LDM (Hu et al., 2019), which adapts an off-the-shelf reward learning algorithm from RL for joint data manipulation learning and model training. As LDM originally uses different augmentation pool, we adapt using the same augmentation pool with DND. (7) DND (no w): policy is updated to increase both training loss and semantic similarity without ce-weighting scheme in Eq. Here, we commonly use the extra losses (Eq. 4 and Eq. 5) at training for the fair comparison. Effect of these losses is shown in Appendix F.
performance in Table 1; (3) Random:
From Table 4, one can verify that the effectiveness of Difficult is quite limited due to it easily diverges to construct too different examples as shown in Figure ((b) and 5(b). Also, the augmented samples from Not Different do not much improve the performance as the learning signals from these samples would be duplicated with the original samples. However, when we properly combine these rewards, it successfully provides 'informative' learning signals to the classifier (DND (no w)), and it can be further improved by the proposed re-weighting component (DND). Remarkably, it is observable that DND clearly outperforms Validation. These results demonstrate that our reward design is more effective than maximizing the validation accuracy for learning the augmentations. Furthermore,
8

---

Published as a conference paper at ICLR 2022
1/5
Advamadt
RDF
0.
Adkmmmm
R3F
Cutoff (0-0.27,--0..
Bacolrans
BERT-a.g
ED4
& Adversarial p=0.300
0.
Probobiitty p
04
Back-trans (p=0.29)
Cutoff (pm0.36,
02
Back-trans (pm0.22)
Adversarial (p=0 p=0.28)
0
20000
43030
60000
23000
40003
65000
Newly discovered
Heeratonss
herations
Re-discovered
(a) Dynamics of probability p
(b) Dynamies of magnitude μ
(c) Discovered augmentations
Figure 3: Analysis about the learned augmentation policy on Review50. (a. b) Dynamics of probability and magnitude during fine-tuning, respectively. (c) Top-3 representative augmentations, discovered
by our augmentation policy. Best viewed in color.
emphasize that such methods maximizing validation accuracy usually require more computations than training loss-based methods due to an inherent bi-level optimization (Zhang et al., 2020c; Hataya et al., 2020); for example, LDM is 3x slower than DND under the same update frequency.
Transferability across datasets and architectures. Here, we provide additional experiments to verify the transferability of the learned policy with our method. To this end, we first save the status of the augmentation policy for each update during the training on the source data and then directly apply it to other tasks or models as dynamic augmentations. From Table 5, one can verify that the transferred augmentation outperforms the baseline augmentations and is even competitive with the performance of the jointly updated policy. Also, the learned policy could be transferred to the different models, such BERT (Devlin et al., 2018) and ALBERT( imply that the proposed augmentation scheme would be greatly beneficial for successfully applying the data augmentation to the new task or model without the exhaustive search and update.
al., 2019). These results
More analysis on learned policy. In Figure 3, we further present more analysis on the learned policy, Specifically, we focus on the parameters the first operation O1. More detailed results can be found in Appendix B. We first note that the probability is dynamically changed at an early stage, which implies the exploration step naturally occurred. After enough iterations, most of the augmentations are converged, and few candidates are only actively updated. In contrast, each augmentation's ' magnitude ja progressively updated and almost converged at the last training iterations. This stability is actually the result of our reward function for similarity, the policy quickly diverges without this (see Appendix B). Interestingly, from successful guidance of the proposed reward, we observe that our augmentation policy could re-discover the previously known best composition of augmentations (which was found by exhaustive grid search). Also, it can newly discover the task-specific augmentations as shown in Figure 3(c). It reveals the reason for success from DND, along with the efficiency of this method.
5 CONCLUSION
We propose DND, a simple and effective augmentation learning scheme for NLP tasks. We design of novel reward function for optimizing augmentation policy to (a) construct the difficult augmented samples for providing 'informative' signals, while (b) preserving their semantic meaning to be not too different from the original for avoiding 'wrong' signals. From the experiments, we find our augmentation method to be much effective in challenging low-data and class-imbalanced regimes. Also, the learned augmentation policy transfers well across various tasks and models. As the augmentation policy learning for NLP tasks is under-explored in the literature, we expect our work to contribute exploring this direction of research.
Furthermore, since the proposed concept of difficult but not too different augmentation is task- and domain- aggossicc, we do believe that DND would benefit in other tasks (e-g., self-super learning) and domains (e.g., vision and graph) as well. On the other hand, incorporating the pre-trained models for augmentation learning in different domains (e.g.. SimCLR (Chen et al., 2020)) would be an interesting future direction, as the fine-tuning of such pre-trained models have shown remarkable successes recently.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
Data augmentation is generally effective for improving the performance on the target task; hence it now becomes a de-facto standard for training DNNs, in various input domains (Cubuk al., 2019; Park et al., 2019; You et al., 2020). However, it also has potential risk amplify the undesirable property of the model at the same time, e.g., gender bias (Bordia & Bowman, 2019), since the enlarged diversity by data augmentation is proportional to the size of used samples for augmentation again. Since the practice of data augmentation is usually based on pre-defined fixed augmentations, hard to remove such inherency of undesirable properties explicitly.
In this respect, the learning- augmentation scheme would be greatly beneficial, as it can easily alleviate this risk by setting the proper objective functions with respect the desired property (Ziegler et al., 2019). For example, one could construct the 'fair' data augmentation scheme by jointly adding the reward for fairness and task accuracy, as we have explored in this work. Hence, we believe our proposed learning- based augmentation scheme would be a step towards improving a such fairness aspect data augmentation, especially for NL.P task.
REPRODUCIBILITY STATEMENT
We describe the implementation details of the method in Section 4.1 and Appendix A.3. Also, provide the details of the datasets and baselines in Appendix A. and A.2, respectively. We also provide our code the supplementary material. All the used packages are along with the code. In our experiments, we use a single GPU (NVIDIA TITAN Xp) and CPU cores (Intel Xeon E5-2630 v4).
ACKNOWLEDGMENTS
This work was mainly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 1002201--002066 Artificial Intelligence Innovation Hub; 10-2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)).
REFERENCES
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. In International Conference on Learning Representations (ICLR), 2021.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Currieulum learning. In Proceedings of the International Conference on Machine Learning (ICML), pp. 41-48, 2009.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308. 3432, 2013.
Shikha Bordia and Samuel Bowman. Identifying and reducing gender bias in word-level language models. arXiv preprint trXiv:194.03003. 2019.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning (ICML), pp- 1597-1607. PMLR, 2020.
Zhiyuan Chen and Bing Liu. Mining topies in documents: standing the shoulders of big data. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1116-1125, 2014.
10

---

Published as a conference paper at ICLR 2022
Yong Cheng, Lu Jiang, Wolfgang Macherey, and Jacob Eisenstein. Advaug: Robust adversarial aug- mentation for neural machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation strategies from data. In Conference Computer Vision and Pullern Recognition (CVPR), pp. 113-123, 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. RandAugment: Practical automated data augmentation with a reduced search space. In Conference on Computer Vision and Puttern Recognition (CVPR), pp. 702-703, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. arXiv preprint ar 0.0000, 2018.
Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. A survey of data augmentation approaches for nlp. In Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
Siddhant Garg and Goutham Ramakrishnan. Bae: Bert-based adversarial examples for text clas- sification. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2020.
Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Keepaugment: A simple information- preeeerrng data augmentation approach. arXiv preprint vrXiv:2011.11770 2020.
Raphael Gontijo-Lppes, Sylvia Smullin, Ekin D Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprint arXi::2002.08973 2020.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Augmenting data with mixup for sentence classifica- tion: An empirical study. arXiv preprint arXiv: arXiv:/905.000 190.9999549911 2019.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoaugment: Learning augmentation strategies using backpropagation. International Conference on Computer Vision (ICCV), pp. 1-16. Springer, 2020.
Zhiting Hu, Bowen Tan, Ruslan Salakhutdinov, Tom Mitchell, and Eric Xing. Learning data manipulation for augmentation and weighting. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations (ICLR), 2017.
Haoming Jiang. Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regu- larized optimization. In Annual Meeting of the Association for Computational Linguistics (ACL., 2020.
Dongyeop Kang, Tushar Khot, Ashish Sabharwal, and Eduard Hovy. AdvEntuRe: Adversarial training for textual entailment with knowledge-guided examples. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18- 1225. URL https: www. ac web. org/
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to- minor translation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.

---

Published as a conference paper at ICLR 2022
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11949 2019.
Ken Lang. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings 1995, pp. 331-339. Elsevier, 1995.
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, et al. An evaluation dataset for intent classification and out-of-scops prediction. arXiv preprint arXiv:1909.0202, 2019.
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguisties, 2002.
Yonggang Li, Guosheng Hu, Yongtao Wang. Timothy Hospedales, Neil M Robertson, and Yongxin Yang. Differentiable automatic data augmentation. In International Conference Computer Vision (ICCV), pp. 580-595. Springer, 2020.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv: 1692, 2019.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 1188):197--1999 2018
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint 1904v:/904.0077 08779, 2019.
Yanru Qu, Dinghan Shen, Yelong Shen, Sandra Sajeev, Jiawei Han, and Weizhu Chen. Coda: Contrast-enhanced and diversity-promoting data augmentation for natural language understanding. In International Conference on Learning Representations (ICLR), 2021.
Nils Reimers, Iryna Gurevych, Nils Reimers, Iryna Gurevych, Nandan Thakur, Nils Reimers, Jo- hannes Daxenberger, and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert-networks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
Shuhuai Ren, Jinchao Zhang, Lei Li, Xu Sun, and Jie Zhou. Text autoaugment: Learning composi- tional augmentation policy for text classification. Conference on Empirical Methods Natural Language Processing (EMNLP), 2021.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. A simple but tough- to-beat data augmentation approach for natural language understanding and generation. arXiv preprint arXhv:2009.13818, 2020.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 761-769, 2016.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1631-1642, 2013.
12

---

Published as a conference paper at ICLR 2022
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955 2014.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for classification? In China National Conference on Chinese Computational Linguistics, pp. 194-206. Springer, 2019.
Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian Reid. A bayesian data augmentation approach for learning deep models. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations (ICLR), 2019.
Xinyi Wang, Hieu Pham, Zihang Dai. and Graham Neubig. Switchout: an efficient data augmentation algorithm for neural machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Jason Wei and Kai Zou. Fda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv: 2019.
Longhui Wei, An Xiao, Lingxi Xie, Xin Chen, Xiaopeng Zhang, and Qi Tian. Circumventing outliers of autoaugment with knowledge distillation. arXiv preprint arXiv: /342, 2(8), 2020.
Sen Wu, Hongyang Zhang, Gregory Valiant, and Christopher Ré. On the generalization effects of linear transformations data augmentation. In Proceedings of the International Conference on Machine Learning (ICML), pp. 10410-10420. PMLR, 2020.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Mingyang Yi, Lu Hou, Lifeng Shang. Xin Jiang, Qun Liu, and Zhi-Ming Ma. Reweighting augmented samples by minimizing the maximal expected loss. In International Conference on Learning Representations (ICLR), 2021.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations (ICLR), 2020a.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Weinberger, and Yoav Artzi. Revisiting few-sample bert fine-tuning. arXiv preprint arXiv:2006.05987 2020b.
Xinyu Zhang, Qiang Wang, Jian Zhang, and Zhao Zhong. Adversarial autoaugment. In International Conference on Learning Representations (ICLR), 2020c.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In International Conference Learning Representations (ICLR), 2020.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:190.00859; 2019.
13

---

Published as a conference paper at ICLR 2022
Supplementary Material
What Makes Better Augmentation Strategies? Augment Difficult but Not too Different
A DETAILS ON EXPERIMENTAL SETUPS
A.1 DATASET
Text classification task. We have used the following 6 text classification datasets in Section 4. Here, we set the maximum length the input sentence as 128 except IMDB dataset with 512. In case of 5-shot setup, it is further limited to 256 following (Hu et al., 2019).
(1) News20 (Lang, 1995) is a collection of 18,846 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. Originally, it is given with 11.314 training samples and 7,532 test samples. News20 dataset is officially available at http: / qwone. com/ cote 2 20Newsgroups/.
(2) Review50 (Chen & Liu, 2014) consists of Amazon reviews of 50 classes of electrical prod- Each class has 1,000 reviews and it is divided into 700 and 300 reviews for training and test samples, respectively. Review50 dataset is officially available at https: /tle:://gtttttte /github..../.. com/ zhouyonglong/ /iiggiiiiinniiiiiieiiieiiinnnnnm Topics- innss----DDocutttt
(3) CLINC150 (Larson et al., 2019) is a intent classification (text classification) dataset with 150 in- domain intent classes. In our experiments, we use the small version of CLINCI50 which is officially provided at https: / tuub. com/ cl saliin//ae ooss- - eval. Here, each of the 150 in-domain intent classes given with 50 training, 20 validation, and 30 test samples.
(4) SST-5 (Socher et al., 2013) is dataset for sentiment classification on movie reviews, which are annotated with five labels; very positive, positive, neutral, negative, or very negative. In the given SST-5 dataset, there are 8,544 training, 1,101 validation, and 2,210 test samples. We have used SST-5 dataset at https: http/yytrcccccco wttp//eet/tt.o/ooooooo
/datasets httmmmmm #sst.
(5) IMDB (Maas et al., 2011) is a dataset for binary sentiment classification on movie reviews. It composed of 25,000 labeled training samples and 25,000 test samples. We use IMDB dataset provided at https: /pytorch..hccco wwtgtoot//////o//o/////ttttt / 8 ////dattsssss htmlf # #imdb.
(6) TREC (Li & Roth, 2002) is a dataset for classifying the six question types, whether the question is about person, location, numeric information, and etc. There are 5,452 training samples and 500 test samples in TREC dataset at https: http:::c///t..cocoo/ot.o www.acchhhrrttteoo wwwp/ottt////////o///////tetttt/ /O. 8 1/..1//aaae daaaasets s htmll # trec.
GLUE benchmark. GLUE (Wang et al., 2019) is composed of 8 different tasks for natural language understanding: MNLI, MRPC, STS-B, CoLA,, SST-2, QNLI, QQP, and RTE. Here, we also set the maximum length of the input sentence as 128 for each task, except Col.A with 64. Here, we offer a brief summary for each task. More detailed description for each task can be checked at https: / gluebenchmark. com/tasks.
(1) MNLI is a binary entailment task which is composed of 393k training and 20k development samples. Given pair of sentences, the goal is predict whether the sentence is entailment, contradiction, or neutral with respect to another one.
(2) MRPC is binary classification task for deciding whether the given sentence pair is semantically cquivalent or not. It is composed of 3.7k training and 408 development samples.
(3) STS-B is regression task that scores (1 to 5) the semantic similarity between sentences given a pair. is composed of 7k training and 1.5k development samples.
(4) CoLA is binary single sentence classification task, where the goal is predict whether the given English sentence is linguistically valid or not. It is composed of 8.5k training samples and lk development samples.
14

---

Published as a conference paper at ICLR 2022
(5) SST-2 is binary single sentence classification task about movie reviews with human labels of their sentiment (positive or negative). is composed of 67k training and 872 development samples.
(6) QNLI is binary classification task that decides whether the given (question, sentence) pair contains the correct answer or not. It is composed of 108k training and 5.7k development samples.
(7) QQP is binary classification task where the goal is to determine if two questions in given pair are semantically equivalent. It is composed of 364k training and 40k development samples.
(8) RTE is a binary entailment task similar to MNLI, composed of 2.5k training and 276 development samples.
Construction of 5-shot and class-imbalanced datasets. In Section 4, we evaluate method on 5-shot (in Table 1) and iasalaemblldceed regimes (in Table 2) which are artificially constructed from the existing benchmark datasets for text classification task. In case of 5-shot datasets, randomly select five samples per each class to construct the datasets.
In case of class-imbalanced datasets, we consider "synthetically long-tailed" variants of SST-2, News20, and Review50 datasets in order to evaluate our algorithm under various levels of imbalance. For constructing the dlas-imbalanced training dataset, we assume the ordered numbers of data in each class as N1 2 ... 2 Nx. We use a single parameter Vimb 2 1, called the imbalance ratio, to control K-1 the class imbalan of the dataset: once Yinb and N1 are given, set N₂ Nı Vimb so that N1 = Yimb NK as done by Cui et al. (2019). Namely, larger Yiab indicates more imbalanced class distribution. We use Ni- 1000 for SST-10 and N1 400 for News20 and Review50, respectively. For 7/mb, we use the specified numbers in Table 2.
Re-sampling with data augmentation: Following (Kim et al., 2020), we combine each data aug- mentation with batch-wise re-sampling, to handle the class-imbalance problem. Namely, in order to simulate the generation of N1 NN samples for any = 1. K,, we perform the data augmentation with probability N,
No for all i in a given class-balanced mini-batch B = ((12,
A.2 BASELINES
(1) EDA (Wei & Zou, 2019): Our implementation of EDA is based on the official implementation by the author at https: / //ttts:: /tt////ttt////// com/ ja sonwei winwwiii2 20. 4 operations in EDA (synonym replacement, random insertion, random swap, random deletion) is randomly applied for each word with a probability Peda. which is selected among {0.15,0.. 0.45).
(2) Adversarial (Zhu al., 2020): We choose hyper-parameter 5 for the norm constraint of adver- sarial perturbation among {0.1, 0.2, 0.3} based on its performance on the validation dataset. We construct the adversarial example with a single step update under Eg norm.
(3) BERT-aug (Yi et al., 2021): Each token is randomly masked with a probability Prask and the masked token is replaced by the most likely token based on the output of pre-trained BERT model (https:/ hithub. com/ makcedward. nipaug), Paaek is tuned among (0.15, 0.3, 0.45).
(4) Cutoff (Shen et al., 2020): We choose hyper-parameter a for cutoff ratio among (0.1, 0.3) based on its performance on the validation dataset. It is observable that the dataset with shorter length usually benefit from the smaller value, and larger better for dataset with longer length. We adopt the official code at https: //gi// /ggthub..... com/ dwwggaansh..../////e/a/a Cutoff.
(5) R3F (Aghajanyan et al., 2021): We sample the noise from the uniform distribution 1 = [O. le-5), then it is added to the word embedding to construct the augmented samples. Unlike other augmentation, the consistency loss is used for training. The official code is available at https: //github. com/ httmm/pytocc///ce fairseq..
(6) MixUp (Guo et al., 2019): We adopt WordMixUp that mix two different sentences on their word embeddings. We use the haper-parameter Oaiz of mixing policy among {0.5, 1.0, 2.0). which decides the mixing ratio Anix where Amix ~ Beta(Oziz, Oniz).
(7) Back-trans (Xie al., 2020): We use the en-de and de-en single models trained on WMT19 and they are available at https: / ///////tttttt /github. com/ honn/pttorch fairseq. We use the default setups: beam search with beam size and keeping only top-1 hypothesis.
15

---

Published as a conference paper at ICLR 2022
(8) &ack&Adv (Qu et al., 2021): After bocc-translation is applied to the original sample, adversarial perturbation further added to its word embedding. Here, we choose rrper-paramete 6 for the norm constraint of adversarial perturbation among {0.1, 0.2, 0.3} based its performance on the validation dataset.
A.3 DND
For learning data augmentation via proposed DND, we use the following data augmentation pool: Cutoff (Shen et al., 2020), Back-translation (Xie et al., 2020), BERT-aug (Yi al., 2021), Adversarial (Zhu et al., 2020), EDA (Wei & Zou, 2019) and random noise (used R3F (Aghajanyan al., 2021)). Here, exclude MixUp in the augmentation pool because it synthesizes completely new sample by mixing two original samples. Hence, is not clear whether it meaningful to keep the similarity between original and augmented samples, which is the key component of DND. Except for MixUp, we think any other augmentations can be in our augmentation pool. Also, although both BERT-aug and EDA have their own magnitude, do not update these magnitudes, as it incurs the computational burden from the additional inference of external model (e.g.. BERT) for each training iteration. Instead, we adopt the common practice in text augmentations (Wei & Zou, 2019; Yi et al., 2021; Qu et al., 2021), which generate the augmented sentences using the pre-determined magnitude before training the model, similar the case of Back-trans, trans. Hence, we only optimize the probability parameters for those word-level augmentations (BERT-aug. Back-trans, and EDA)
With this augmentation pool, our data augmentation composed with T consecutive augmenta- tions by finding their optimal probability and magnitude p and μ., respectively. Since the word-level augmentations (BERT-aug, Back-trans, and EDA) can't be applied after the embeddingglevel augmen- tations are applied, we limit the search space of O₂ to the embedding-level augmentations. Also, to construct more diverse augmentations, we simultaneously train 4 augmentation policies by randomly selecting the one of them for each mini-batch. Also, empirically observed that the training of augmentation policy can be improved by introducing the auxiliary operation (Hataya et al., 2020; Li et al., 2020) that decides whether to apply the selected augmentation not, after sampling the augmentation from p.º Hence, we adopt this operation for training augmentation policy, but not necessary for the augmentation learning. Finally, we remark that other training losses for augmented samples, such as consistency loss (Jiang et al., 2020; Xie et al., 2020) or contrastive loss (Qu al., 202 1), can be used for training the model (Cras)) and the augmentation policy (Reae), although we have used the most simplest cross-entrue cross-entropy loss, i.e., Lessk = Exe. to focus on the effect of augmentation itself.
"This operation is originally modeled by Bernoulli distribution, but can be approximated with Relaxed Berouuli distribution done in (Hataya al., 2020; Li al., 2020)
16

---

Published as a conference paper at ICLR 2022
B MORE ANALYSIS ON LEARNED AUGMENTATION POLICY
In this section, we further provide the dynamics of learned augmentation policies with respect to probability p and magnitude ls under reward functions: (1) Difficult which was defined in Section 4.3, and (2) Difficult, but not too Different (Ours). As we have partially shown in Section 4.3, our reward function successfully guides the policy to learn the effective augmentation with progressive and steady updates on the parameters (in Figure 4). In contrast, as one can see in Figure 5, rewarding the augmentation policy just to maximize the training loss would mislead policy to generate samples that are semantically meaningless, e-g., out-of-distributinn samples or samples losing the characteristics
the target label y. Specifically, in Figure 5(b), one can actually see evidence for this; the size of adversarial perturbation becomes too large. Hence, this empirical observation again shows the importance of proper reward function to update augmentation policy and the validity of the proposed components in DND (maximizing semantic similarity and re-weighting scheme).
Adversania (I=1)
Cutelf (I+1)
R3F (-1)
Asversaria [I+1)
Anversaiial (I-2)
Advarsadia (t=2)
(I=2)
R3F (t-2)
Cusoft (t=1)
(1=2)
Back trans
But
EDA
R3F (E-1)
R3F (1:2)
Probability
Magnitude
0.4
0.2
20000
40000
60000
20000
40000
60000
Iterations
Iterations
(a) Dynamics of probability p
(b) Dynamics of magnitude μ
Figure 4: Dynamics of learned augmentation policy via DND.
Adversaria (t-1)
Cutoff (t+1)
RDF [[-]
Adversonal [I=1]
Adversarial (+=)
(I-))
(w22)/
(+2)
(=1)
(=2)
Back-trans
BERT-aug
EDA
R3i ([1))
R3i (t-2)
Probability
Magnitude
1.0
0.4
%
20000
40000
Iterations
Iterations
(a) Dynamics of probability p
(b) Dynamics of magnitude μ
Figure 5: Dynamics of learned augmentation policy via Difficult.
17

---

Published as a conference paper at ICLR 2022
C COMPLEMENTARY RESULTS ON GLUE BENCHMARK
In this section, we provide the complementary results GLUE benchmark (Wang et al., 2019), First, Table 6 shows the results on the development set including the standard deviation across 3 random seeds, have omitted for better presentation of the experimental results. Next, we also validate the effectiveness of ours method by comparing it with vanilla on the test set, which is available at https: htt///ggueeeeaaaaaa
com/ submit. As shown in Table 7, ours improves performance of vanilla classifier in most cases.
Table Evaluation results of vanilla (RoBERTa) classifiers with the GLUE development set after fine-tuning with each data augmentation method on the given task. All the values and error bars are mean and standard deviation across 3 random seeds. The best and the second best results are indicated in bold and underline, respectively.
Method (Metrics)
RTE
MRPC
STS-B (P/S)
CoLA (Mcc)
SST-2 (Acc)
QNLI
QQP (Acc/FI)
MNLI-m/mm (Acc)
(Acc)
(Acc)
(Ace)
Vanilla
79.4
89.62
90.72/90.42
62.2
94.46
92.64
90.87/88.1 17
87.34/86.88 85
+1.03
+0.42
+0. 10.12/-0.15
1.27
+0.14
+0.07
+0.03/++.007
+0.08/+0.066
EDA
77.8
90.03
90.14/89.92
55.8
94.96
92.56
91.41/88.50
87.46/87.31
-0.74
10.12
10. 10.13/20.12
11.29
10.07
10.12
10.10/10.20
10.01/11.11
Adversarial
80.5
91.00
91.08/90.68
63.5
94.88
93.18
91.50/88.55
87.91/87.80
+0.74
+0.31
+0.08/=0.04
+0062
+0.21
+0.11
10.01/+0.15
+0.02/+0.14
BERT-aug
79.6
90.11
89.47/89.35
61.2
94.27
92.95
91.07/88.21
87.73/87.22 25
+1.28
+0.70
#0.25/=0.33
1.25
+0.28
+0.13
10.03/10.12
0..00
Cutoff
78.0
91.01
90.53/90.27
64.0
94.42
92.84
91.13/88.27
87.22/87. .2
10.76
10.51
13.29/-0.23
+1.27
10.22
10.03
10.02/10.08
10.05/+0.02
R3F
79.7
90.69
90.77/90.56
64.6
94.95
93.06
91.30/88.54 54
$7.97/87.66
#1.18
10.35
10.08/10.06
10.45
10.01
00.01/1007
10.28/11.27
MixUp
79.8
90.08
90.46/90.16
64.3
94.50
92.69
90.98/88 06
87,50/87.0 .22
10.72
10.62
10.16/10.16
10.83
10.28
10.14
10.05/10.33
10.07/10.05
Back-trans
79,9
90.20
90.54/90.33
58.8
94.53
92.64
90.82/88 05
87.50/87, .66
10.90
10.20
10.12/20.17
10.20
10.21
10.02/10001
10.10/11.12
Adc
80.9
90.69
90.61/90.33
59.5
95.01
93.07
91.29/88.49
87.73/87.53
+1.03
+0.53
=0.07/-0.05
+1.01
+0006
#0.14
+0.01/+0.03
10.13/40.12
Ours
81.5
91.18
90.78/90.49
65.0
95.30
93.48
91.85/89.00
88.21/88.12
=0.87
+0.12
+0.04/=0.02
+0.25
+0.12
+0.12
+0.05/+0.07
+0.12/40.08
Table Evaluation results of vanilla (RoBERTa) classifiers with the GLUE set after fine-tuning with each data augmentation method on the given task. For evaluation, we submit the prediction averaged across random seeds. The better results are indicated in bold.
Method (Metries)
RTE
MRPC (Acc)
STS-B (P/S)
CoLA (Mcc)
SST-2 (Acc)
QNLI (Acc)
QQP (Acc/FI)
MNLI-m/mm (Acc)
(Acc)
Vanilla Ours
72.4
87.1
88.9/88.2
58.8
94.8
92.3
89.6/71.8
86.9/86.4
74.0
87.7
88.6/87.8
60.1
95.3
93.3
89.7/72.8
87.9/87.2
18

---

Published as a conference paper at ICLR 2022
D MORE QUALITATIVE RESULTS
Difficult, but not different samples. In this section, to further verify whether the augmented samples with the desired property are constructed from the learned policy, construct the augmented samples using each augmentation method and compare the average difficulty and similarity in Figure 6(a). We observe that the augmented samples from our augmentation policy actually have desired property: they show significantly low confidence, i.e, more difficult, while it moderately preserves the semantic of original samples. Furthermore, we present the augmented samples from BERT-aug by varying the difficulty and similarity in Figure 6(b). Here, one can verify that the difficulty is solely not enough as too many changes can alter its semantic. Also, the similarity solely not enough as tends to construct too easy samples which can't 't provide 'informative' signals to the training model. However, by considering two aspects jointly, reasonable and effective augmented samples can be constructed. More qualitative results are presented in Figure 8. Also, the plot of augmented samples by each augmentation scheme is presented in Figure 7.
Ours Berf-auug
Outefl Adversarial
Backtrans
Original
off, wouldn't turn without shaking it. Cheesy plastic. Actually really bright when
about
1.00
(p: 0.98, 1.00)
work.
0.35
about minutes. Turned wouldn't back Differen without shaking it. Cheesy plastic. Actually really cold after
Difficult
(p: 0.01, 0.73) did
juet minutes. Tumedd off, wouldn't & not Differant without shaking it. Old plastic. Actually really bright when did 0.98)
Easy
0.85
work.
0.80
Difficult Worked about months. Tumeed off, wouldn't back & not Different on without shaking it. Cheesy plastic. But looking bright when 0.53, 0.97)
Siilerity
did work.
(a) Difficulty and Similarity
(b) Example of augmented samples (Label: 'Lamp")
Figure Qualitative results difficult, but not too different samples; (a) Average difficulty (difficulty confidence and similarity of the augmented samples under each augmentation method on News20. (b) Real examples on Review50. Different criteria are used for the selection, and two most salient words of original sentence are remarked in bold. Critically altered words are colored in red. Here, p and s represent the confidence and similarity, respectively. Best viewed in color.
1.0
Adversarial
Back&Adv
Back-trans
EDA
Cutoff
Ours
0.8
BERT-aug
Confidence
0.6
0.4
0.2
0.0
0.85
0.9
0.95
1.0
Similarity
Figure 7: Difficulty (difficulty 1 confidence !) and similarity of the augmented samples under each augmentation method on Review50. For the better visualization, 250 training examples are randomly selected and then augmented.
19

---

Published as a conference paper at ICLR 2022
Original (p: 0.99, s: 1.00)
Everything seems to work fine, the only SDHC device have is my N82 and they work great logether. N95 8GB eat your heart out!
Difficult
Everything seems to work fine, the closest SDHC chips have is my N82 but they work great
& Different together. N95 can eat your brain out!
(o: 0.01. s: 0.20)
Easy
Everything seems work fine, the only SDHC card have an NB2 where they work great together. You 8GB eat your heart allve!
& nor Different (p: 0.99, s: 0.91)
Difficult & not Different
Everything continues to be fine, the best SDHC device have is my NB2 and you work great together. N95 8GB eat your brain out!
(o: 0.54, s: 0.76)
Label: "Memory Card"
Original am using this card in my new build and it works flawlessly. I love ASUS products anyway. (o: 0.97, s: 1.00) this not disappiinte.. hnghlyy recommend card you are looking for wireless
adapter.
Difficul & Different also this has not disappointed. I highly recommend this card when you are looking for a play
using card in my build but it works flawlessly. love ASUS products anyway.
(p: 0.00, s: 0.82) phone.
Easy am using this chip in a new build so works flawlessly. love ASUS products anyway, and
& not Different this child not disappointed. I highly recommend board if you are looking for wireless
(o: 0.98, s: 0.96) adapter.
Difficutt am using this card in my new build and it operates flawlessly. I love ASUS as well, and this not Different has not disappointed. Should highly recommend this card you are pick up a wireless
(p: 0.66, s: 0.96) adapter.
Label: "Memory Card"
Original When bought this product didn't read -LRB- and I'm pretty sure it wasn't clarfied anywhere (o: 0.99, s: 1.00) RRB- that this remote wouldn't work in my Nikon 23100. So, haven't been able to test it.
When when bought my product I read -LRB- and lb pretty sure wouldn't looking & Different anywhere -RRB- that this product wouldn't work in my Nikon d3100. So, shouldn't been
Difficult
(o: 0.01, s: 0.79) able
myself.
Easy
When sold product I didn't read -but and pretty sure script wasn't clarified anywhere RRB- that this remote wouldn't work against my Nikon d3100. So, haven's got allowed to
& nor Different
(p: 0.99, s: 0.95)
test
Difficult
Until I bought this thing I tead -LRB- and I'm pretty sure it waan't clarified anywhere-
not Different (o: 0.59. s: 0.96)
RRB- that phone remote wouldn't appear in my new d3100. So, I haven't been able test it.
Label: "Remote Control"
have two of these systems, bought one for work and one for home. They work great. Yes, (o: 0.99, s: 1.00) one of them had the same trouble as you may read some comments. found If you eject
Original
the disks before you shut off, they never get stuck
the machine where they did get stuck, unplugged it for two days and plugged it back in and It spit out cd.
Difficult
have two of these discs, bought one for work and other for rent. Things were great. Yes, one of them had the biggest trouble with you may read in some records. But found if you eject get stuck in it. lost, unplugged it for five days and plugged back in and it spit out the tape.
& Differett
(o: 0.03, s: 0.90) the disks before you shut it off, they
machine where they did get
Easy & not Different
He released two of these systems, bought one for businesses and one for home. They work great. Yes, one of them had those same trouble as you may add in some comments. I found disks before you got on, they never get stuck in them. the one machine where they did were caught, I unplugged it for two days only plugged it back and blew out the cd.
(o: 0.99. s: 0.96) you
Difficult not Different (p: 0.69, s: 0.94) tried three disks before you shut it off, they all get stuck in it. one machine where it did became stuck, unplugged it for two days to plugged it back in and it spit tue cd.
edited two of his systems, bought one for work and one from home. They work great. Yes, one of them had the same trouble like you may read some comments. We found you
Label: "Home Theater System"
Figure 8: Examples of augmented samples on Review50 (Chen & Liu, 2014). Different criteria are used for the selection, and three most salient words of original sentence remarked in bold. Critically altered words are colored in red. Here, p and s represent the confidence and similarity, respectively.
20

---

Published as a conference paper at ICLR 2022
E MORE QUANTITATIVE RESULTS
In this section, we present more quantitative results about the proposed DND. Here, all experiments conducted News20 (Lang, 1995) dataset. Also, all the values and error bars are mean and standard deviation across random seeds. The best result is indicated in bold.
Table 8: Experimental results to verify the compatibility among different language models. Test accuracy of specified classifiers after fine-tuning across various data augmentations are compared.
Method
RoBERTa-large
BERT-base
ALBERT-base
Vanilla
83.53
83.4440.30
78.79
Adversarial
84. 14
84.55
79.73
Cutoff
25.09
85.49
79.66=0094
R3F
84.58
84.53
78.99
Ours
85.79
85.93
80.422t0:6
Compatibility of DND about model choice. In Section 4, we mainly conduct experiments with RoBERTa-baaseLLu al., 2019) model. Hence, to verify the compatibility of DND about model choice, we further provide the experimental results under the different choice of pre-trained language models: RoBERTa-large (Liu et al., 2019), BERT-base (Devlin et al., 2018), and ALBERT-base (Lan et al., 2019). RoBERTa-large has 24 Transformer blocks with 1024 hidden size and 16 self-attention heads (total parameters=340MM BERT-base has the same architecture with RoBERTa- (12 blocks with 768 hidden size and 12 self-attention heads, hence 1 10M total parameters), but it's pre-trained different way. ALBERT-base had an identical architecture with BERT-base except 128 hidden size (total parameters= 1 2M). In overall, the model size has following order: RoBERTa-large > RoBERTa-base = BERT-base > ALBERT-basee augmentations that top-3 baselines in Table 1: Adversarial (Zhu et al., 2020), Cutoff (Shen et al., 2020), and R3F (Aghajanyan et al., 2021). Here, DND outperforms all baselines and it shows that DND successfully learns the effective data augmentation regardless of the model choice.
Table 8, we compare DND with the following
21

---

Published as a conference paper at ICLR 2022
Word Reconstrurtion antionn Accuracy (%)
Sho
20
Reveext Newe20
With Crow
(Es (90
Vert
0.25
0.5
0.75
1,3
0.25
0.5
0.75
1.0
Epochs
Epochs
Figure 9: Accuracy of gw (Eq. 3) across train- Figure 10: Accuracy of word reconstruction (Eq. ing iterations on News50 and Review50 datasets.
5) across training iterations on Review 50 dataset. Here, we only report the results during one epoch for better visualization.
Here, we only report the results during one epoch for better visualization.
F DETAILED ANALYSIS ON EXTRA LOSSES
In this section, we provide the detailed analysis on two extra losses Cain (Eq. 4) and Lrecon (Eq. 5) used when training the model, especially to investigate the following:
For Liin (and Rein), using gw with MLP is better than simpler choices such as cosine- similarity &2 distance?
What is the contribution Crecon in DND?
We remark that Lein is introduced to train the linear classifier gw, which is used for rewarding not too different samples with Rein (Eq. 2). Lrecon is introduced to preserve the original semantics of input token, as the quality of contextualized output embedding is critical for our algorithm. Here, all experiments are conducted on News20 (Lang, 1995) and Review50 (Chen & Liu, 2014) datasets. Also, all the values and error bars are mean and standard deviation across three random seeds. The best is indicated in bold.
Discussions with simpler choices for similarity loss. Although other simpler choices, such as cosine-similarity or & distance on the output embeddings, considerable as the similarity loss, using MLP allows a better trade-off between the original task performance and learning the similarity between the embeddings. Namely, the simpler choices could regulaauua the output embedding and result in an underfitting for the target task. However, by leveraging MI.P, more information for the target task be formed and maintained in the output embeddings. Hence, Iw with MIPP can get properly trained during the fine-tuning for the target task, even with small number of training iterations. For example, Review50 dataset, we empirically observed that gw achieves 98.1 % accuracy only after one epoch of training, as shown in Figure 9, To empirically validate this intuition, we conduct the following experiments with DND using 1) cosine similarity and & distance between m, and my, instead of using MLP. As shown in the Table 9, the performance with the simpler choices actually decreases compared to using MLP. We also remark that the effectiveness of MIP for measuring the similarity on the embeddings has been demonstrated in the other context as well, c.g., SimCLR (Chen et al., 2020).
Table 9: Ablation study on different choices for similarity loss.
Dataset Cosine similarity
l distance
MLP
News20
84.42.a.10 74.04
83,991029 73,81
85.19
Review50
74.87
More intuition and effect of reconstruction loss. First, we present a more detailed description how reconstruction loss works. Input token 13 V is V-dimensional one-hot vector with vocabulary size V. V E V is the word embeddings matrix of pre-trained transformer model that maps d-dimensional input embeddings Rdd This feed-forwarded the remaining
22

---

Published as a conference paper at ICLR 2022
parts of the transformer (e.g., self-attention blocks) and the corresponding output embeddings is o E Rd. Hence, the resulting word reconstruction loss can be formulated as Zze (Pv(())) (Eq. 5) where pv Softmax ). Namely, it is predicting one-hot label (given by input token 2:) from the dot product of output embeddings and word embedding matrix V.
Intuitively, reconstruction loss Lrecon effectively preserves the information within output embeddings learned through the pre-training phase; hence, could provide a better output embedding. We empirically observed that the word reconstruction (i.e., prediction) from output embeddings becomes impossible as the model is fine-tuned in the target task (See Figure 10). We remark that this word reconstruction was almost perfectly possible before the fine-tuning, the language model pre- trained using a similar reconstruction loss (c.g., masked token prediction in BERT). Thus, this observation implies that this semantic information lost. Our reconstruction loss effectively prevents such information loss and improves the quality of output embeddings. To empirically demonstrate the effectiveness of this term, we further provide the experimental results of DND without reconstruction loss (i.e., A = 0). As shown in Table 10, the reconstruction loss clearly improves the performance of DND by providing better output embeddings.
Table 10: Ablation study the effect of reconstruction loss
Dataset
A
A, 0.05
News20
85.1940.5 74.52 74.87
84.88+019
Review50
Effeet of extra losses. While the extra losses Lein (Eq. 4) and Crecon (Eq. 5) are introduced for updating policy with DND, these losses could contribute to the learning of better representations and lead to the improvement themselves. To verify this, we conduct the additional experiments about these auxiliary losses. Here, when each loss is applied, the fixed coefficients fein and 0.05 for Crecon- Table 11, is verified that each of the proposed losses could improve the performance contributing the learning of better representations, and two auxiliary losses complementary to each other.
used: 1 for
Table 11: Effect of extra losses Lain and Lracon
Dataset
Lein
Crocon
Cutoff
Back-translation
Adversurial
News20
83,362041
82.57-0.0
83.37
22
82.77
83.63
a315
82.86=047
83.44
V
84.06=0.1
83.01=025
83.79#ase
23

---

Published as a conference paper at ICLR 2022
G MORE ABLATION STUDY
In this section, we provide more ablation study on the design choices of DND. Here, all experiments conducted on News20 (Lang, 1995) and Review50 (Chen & Liu, 2014) datasets. Also, the same values are used for the others except for the specified hyper-parameters. All the values and error bars are mean and standard deviation across three random seeds. The results with originally used values are underlined.
Number of augmentation policies nt. As we have mentioned Appendix A.3, we simultaneously train multiple augmentation policies (n2 4). To see the effect of this, we compare the cases with less (ny 1) and more (nT 10) number of policies in Table 12. Overall, the improvement from DND is not sensitive to the number of augmentation policies, but using multiple policies shows slightly better results as would construct more diverse augmentations.
Table 12: Effect of different number of augmentation policies.
Dataset
ny 1
BT
WT 10
News20
84.9 9320s
85.19
85. 35.16_000
Review50
74.84
74.87
74.90=cu1
Number of operations T. described in Section 4.1, our augmentation policy is composed of two consecutive operations. Here, compare this choice (I' 2) with less (T' = 1) and more (T' 3, 4) number operations. As shown Table 13, one can verify that a single operation is not enough to construct complexly augmented samples; hence the empirical gain is relatively small compared to the original choice. With more operations, almost saturated improvement is observed. Remarkably, similar tendencies also observed in vision tasks (Hataya et al., 2020; Cubuk al., 2020).
Table 13: Effect of different number of operations T'.
Dataset
T =
T=
T= 3
T = 4
Ncws20
84.68
85. 19
85.19202 74.93
85.22:a10 75.01
Review50
74.24
74.87
Different values for and 3. We additionally provide the results with different values for and
3, which are introduced for weightnt scheme in Reaex (Eq. 1). In Table 14, it is observable that each component of the re- weighting scheme is solely effective, and the gain from re- weightin is not much sensitive to the choice of a and (3. We remark the original choice (a, B) = (0.5, 0.5) shows the best empirical gains.
Table 14: Effect of different values for hyper-paramedde and 3.
Dataset
(a, (8) (0.5, 0)
(a, () (0,0.5)
(a, 9) (0.5, 0.5) (a, B) (1,
News20
85.08
84.95
19.
85.09
Review50
74.34_0.10
74.76 74.76
74.87
74.77+015
24

---

Published as a conference paper at ICLR 2022
Different values for A, in the reconstruction loss. We conduct additional experiments with different values of Ar although we use a fixed value (Λ, = 0.05) for the reconstruction loss (Eq. 5) in all the experiments. As shown in Table 15, the reconstruction loss with moderate values of X, clearly improves the performance, but too large value decreases the performance as excessively updates the model for reconstruction loss. Hence, recommend using the originally fixed value (), 0.05).
Table 15: Effect of different values for hyper-parametee X,.
Dataset
X, = 0
A, = 0.05
A, = 0.1
X = 0.5
News20
84.88=019
19
85.10
84.522t025
Review50
74.52
74.87
74.89
Update frequency 7, and weight X₂ for policy Tø. As described in Section 4.1, we mainly tune Tp and A for each task. Table and Table 17 show the results of different values of them. Here, we observe the consistent improvement of DND with varied Tp, which indicates efficiency of DND. In the case of As the performance of DND still significantly outperforms the other baselines with all directly affects the trade-off
Ag. but the gain is varied with the chosen λę this between 'informativeness' and 'risk' of the augmented samples.
Table 16: Effect of different Tp-
Table 17: Effect of different X
Dataset
Tp = Tp
Tр 10
Dataset
A = 0.1
A = 0.5
A = 1.0
News20
84.9910m
85.19
85.1 Lois
News20
85,06 10.7
85.19 10.3
84.97 1015
Review50
74.72+om
74.87
74.89+0m0
Review50
74.17
74.87
74.35
Effeet of auxiliary operation. As we have mentioned in Appendix A.3, adopt the auxiliary operation for our implementation, which has been originally used by Hataya al. (2020). Here, we provide the results without this auxiliary operation. As one can see in Table 18, this operation is not significant to the gain from DND.
Table 18: Effect of auxiliary operation (Hataya et al., 2020).
Dataset
DND (no auxiliary)
DND
News20 Review50
85.14 14
$5.19
74.79
20.87
25


Reviews:
[{"review_id": "Wx9J3331ME", "reviewer": "Reviewer_XAnt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-\t__Clarity__. The writing is clear and easy to follow.\n-\t__Well-motivated approach with extensive experiments.__ Although the approaches to learn an augmentation policy using an adversarial objective, enforce the augmented samples to be similar to the original samples and use continuous relaxation to train the augmentation policy are studied separately in image domain, the proposed method is a viable way to apply the ideas to text data and shows good experimental results on various NLP tasks.\n\nWeaknesses/comments:\n-\t__Comparison with other learning objectives.__ The ultimate goal of data augmentation is to improve the generalization power of a model. How does the proposed Difficult and Not Different objective compare with the objective that improves the validation performance directly like AutoAugment or TAA [1].\n-\t__Sensitivity of DND hyperparameters.__ Despite being an Automated Data Augmentation method, there are several important hyperparameters, e.g. $\\lambda_r, \\lambda_{sim}, \\lambda_s, T_p$, to be tuned. How sensitive is the proposed method to these hyperparameters? Whether these hyperparameters needed to be tuned carefully?\n-\t__Ablation study for extra loss terms.__ In the ablation study section, are the ‘Vanilla’, ‘Random’ and ‘Fixed’ baselines also being trained with the extra loss terms $L_{sim}$ and $L_{recon}$? While it is a motivated decision to introduce these terms during the training of $f_\\theta$, can these losses also contribute to the learning of better representations and lead to the improvement? It would be useful if the effects of these extra losses are discussed in the ablation study. \n-\t__Exclusion of MixUp from the augmentation pool.__ Authors mention that MixUp is not included in the augmentation pool as it alters the semantics of the original sentences. However, MixUp can create difficult examples by sample and label mixing. MixUp also shows good results for some datasets under Table 1. As a learning-based data augmentation method, is it the responsibility for the search algorithm to learn the use of MixUp in a data-driven way? Can the inclusion of MixUp in the augmentation pool improves the performance? If the validity of a candidate augmentation method has to be evaluated before adding to the augmentation pool, does it contradict with the goal of fully learnable data augmentation?\n-\t__Hyperparameter tuning for other baselines.__ Some of the baseline methods, like EDA, BERT-Aug, MixUp, Back&Adv also involve hyperparameters. According to Appendix A2, it seems that these hyperparameters are not tuned. As the hyperparameters of DND (e.g. $\\lambda_s$ and $T_p$) are tuned for each dataset, it is fairer to tune the hyperparameters for the other baselines. \n-\t__Formulation of the probability and magnitude for EDA.__  Is there a specific reason that EDA is assigned with a single probability and magnitude parameter? Can the use of different $p$ and $\\mu$ values for synonym replacement, random insertion, random swap, and random deletion further improves the augmentation policy?\n-\t__Definition of the magnitude parameters for the augmentation candidates.__ It is unclear how the magnitude parameters are defined for the augmentation candidates. For example, is the mask probability of BERT-Aug taken as the magnitude? What are the ranges of the magnitudes? It would be useful to include these in the appendix.\n\nReference\n\n[1] Shuhuai Ren, Jinchao Zhang, Lei Li, Xu Sun, Jie Zhou. Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification. _arXiv preprint arXiv:2109.00523,_ 2021", "summary_of_the_paper": "The proposed method DND learns an augmentation policy to augment a target text dataset. The augmentation policy is optimized to create difficult yet not too different examples from the original data. To further improve the performance, a sample re-weighting scheme is introduced to focus on harder examples. Experiments show that learned augmentation policy can achieve good results on various NLP tasks and transfer well across datasets and architectures.", "summary_of_the_review": "Although the approach to learn an augmentation policy using the adversarial and similarity objectives is studied in the image domain, the proposed method contributes to the adaptation of learnable augmentation in the NLP domain and shows good experimental results. Overall, I tend to recommend for acceptance for the initial rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IiVMBbkz2A", "reviewer": "Reviewer_SErr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe proposed method is intuitive, simple to apply and effective over many baseline augmentation methods on various datasets, especially on low-resource, class-imbalanced regimes. Besides the good performance on benchmarks, the authors also conducted good ablation study and analysis on learned policies, showing that the selected policy coincides with previous work (via exhaustive grid search).\n\nWeakness and Concerns:\n1. The main issue lies in the choice of baseline. As a learning-based augmentation method, it will be more convincing to compare with other learning-based method (using same pool), e.g. auto-augment, adversarial auto-augment etc. If the method can still outperform them, it can better demonstrate the effectiveness of their reward design.\n2. Limited technical novelty: The whole idea is very straight-forward and most of the techniques are borrowed from previous works. ", "summary_of_the_paper": "The authors proposed a new reward function to learn the distribution of augmentation policies for NLP tasks, which can generate \"difficult\" and semantic similar samples to facilitate training. They further introduced a sample-wise re-weighting scheme to leverage the learning status of original sample. A continuous relaxation is applied to optimize the trainable parameters in policy. The proposed method outperformed SOTA on text classification and entailment tasks. They also conducted extensive studies and analysis, demonstrating the effectiveness of their method on low-resource and class-imbalanced regimes, as well as its transferability.", "summary_of_the_review": "This paper proposed a simple and effective reward function for learning-based augmentation selection in NLP. Extensive experiments and analysis demonstrated its effectiveness on text classification and entailment, especially on low-resource and class-imbalanced regimes.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ErFi8cI8UJH", "reviewer": "Reviewer_PytR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##########################################################################  \nStrengths: \n- The paper is very well written. The motivations and contributions of the paper are very clear and important. Data augmentation policy learning is a relatively under-explored area in the literature, and I think this paper makes a very useful contribution to exploring this research area. The proposed approach is simple enough to be replicated easily in follow-up research. The provided code in the supplementary material is well written and easy to understand too.\n- The paper provides comprehensive experiments covering a diverse set of tasks. It includes results using 8 baseline/state-of-the-art models on 6 datasets with different text classification tasks, and on 8 tasks present in the GLUE benchmark. It provides the results on multiple class-imbalance/low-data setups. Further, it provides an ablation study to understand the contribution of different components of the proposed model, to provide evidence of the importance of each of those components. Finally, the paper also provides some results to show the transferability of the approach, and some analysis of learning dynamics. In most of these results, the proposed approach outperforms the baseline/state-of-the-art models.\n\n##########################################################################  \nWeaknesses / Suggestions:\n- Minor correction in page 4: \"Rewarding not too different samples\" section. It's mentioned to see \"Figure 1(b) and 6(b)\". But the figure 6 is not present in the main paper. Further, both \"1(b)\" and \"6(b)\" are linked to 1(b) figure only.\n- `gW` from equation (3) is sent directly to the cross entropy loss function in equation (4). I think it would be more clear if a sigmoid function is added on `gW` either in equation (3) or (4). Further, in equation (3), did you consider simply calculating the dot product of `m1` and `m2`, instead of concatenating them and using a neural network?\n- I think the equation (5) is a bit unclear. Could you clarify the following:\n    - Is the idea here similar to minimizing the dot product between the output embedding and input embedding for a given word?\n    - Can you add more description of what exactly `V` and `x` are? `x` is referred to as the input token, but its used as a label in the cross entropy loss, implying that it’s a binary value. How is the input token converted to a binary label?\n    - Given that the output and input embeddings of a transformer are in different spaces, could you provide some justification of how calculating the dot product between those vectors would be appropriate?\n- A minor comment, but the usage of policy/reward made me think that this paper uses some reinforcement learning approach. Upon reading the paper, the usage of those terms do seem appropriate. I am not suggesting that this needs a change, but I just wanted to point out that some readers might get confused a bit by this terminology.\n- Regarding the GLUE benchmark results, it would be interesting to see how this model fares in the GLUE leaderboard. Could you add those results if available?\n- In the final conclusion section, could you add some directions in which this work can be extended?", "summary_of_the_paper": "The paper provides a simple yet effective approach towards data augmentation in NLP tasks. In particular, it proposes an augmentation strategy that encourages constructing augmented samples with low confidence (which makes them challenging, hence more informative) and high semantic similarity with the original samples (which ensures the semantics are not lost in augmentation). Through comprehensive experiments, the authors show that their approach outperforms recent state-of-the-art techniques, especially on low-data and class-imbalanced regimes.", "summary_of_the_review": "Overall, I vote for accepting. I think the paper proposes a very interesting approach to improve data augmentation in NLP tasks by learning a policy that decides how to combine different augmentation techniques in a task dependent manner to generate samples that are informative while retaining the semantic information from the original sample. Since this research area is relatively new, I think this paper provides a really good baseline, and a framework for other papers to improve upon, as its easy to implement and allows adding more augmentation techniques in the policy pool, improving the loss/regularizer functions, etc. I have a few minor issues regarding the paper, and I described them in detail below. Hopefully the authors can address my concerns in the rebuttal period.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "9cBk2yWyZI8", "reviewer": "Reviewer_A1ft", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well-written and easy to understand. Experiments are conducted on different datasets and training schemes. The proposed method improves upon existing data augmentation techniques.\n\nThe proposed algorithm in this paper is a combination of known techniques, i.e., all the data augmentation techniques are well-established, and the relaxation method used to train the policy is also not new. Because the technical novelty of this paper is limited (which is acceptable), empirical evaluation needs to be substantially strengthened.\n\nSome designs are not well-motivated. \n1. In the similarity loss (Eq. 4), a two layer MLP is used to serve as a linear classifier. Could the authors explain why this is better than simpler choices such as cosine-similarity or $\\ell_2$ distance?\n2. Continue on Eq. 4, what is the accuracy of $g_W$ after training? It is possible that $g_W$ cannot get properly trained because of the limited number of fine-tuning epochs.\n3. The reconstruction loss (Eq. 5) is confusing. Could the authors provide more intuition on this? Also, there should be experiments that demonstrate the necessity of this loss term.\n\nThe training looks ad-hoc. There are several designs that seem arbitrary. Some additional experiments are needed.\n1. Several existing methods are selected as the data augmentation pool (Cutoff, Back-trans, BERT-aug, Adv, EDA and R3F). There should be experiments that show the importance of these policies, i.e., are they all necessary? In Figure 3a, the probability of selecting R3F, Adv, EDA and Cutoff is low (Cutoff nearly drops to 0) towards the end of training. Is it possible to only include Back-trans and BERT-aug?\n2. The authors mention that they need to simultaneously train 4 augmentation policies. What will happen if we only train one (or more than 4) policy?\n3. The operation count is set to $T=2$ in the experiments. Analysis are needed on other values of $T$.\n4. The authors mention an auxiliary operation (Hataya et al., 2020) is added. An ablation study is needed regarding the significance of this operation.\n5. In the sample weight $w$ (Eq. 1), there are two hyper-parameters $\\alpha$ and $\\beta$. Experiments are needed to see how these two hyper-parameters change model performance.\n6. There should be experiments on the weight ($\\lambda_r$) of the reconstruction loss (Algorithm 1).\n\nAdditional comments:\n1. In Table 1, performance gain is not clear. Variance of the results is very large, and it is hard to tell the significance of the gain. The authors should conduct significance tests and report the p-values.\n2. In the current version, the training loss function is only shown in the Algorithm box. Include it at the beginning of Section 3.2 will make the presentation clearer.\n\nI will raise my score if the authors can conduct the ablation experiments and address my concerns.", "summary_of_the_paper": "This paper proposes a data augmentation technique --- difficult but not too different. The authors claim that an augmented sample should be difficult (in terms of loss) and semantically similar to the original sample. The authors propose two rewards functions to reward difficult samples and similar samples, respectively. A relaxation technique is further used such that the augmentation policy can be trained. Experiments are conducted on the GLUE development set and several other text classification tasks. The proposed method improves upon existing ones.", "summary_of_the_review": "The paper applies existing differentiable data augmentation methods (Hataya et al., 2020) to text classification. Overall, the technical novelty is low. There are several additional experiments (in particular ablation studies) needed to justify the design choices and the performance gain. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: We appreciate the authors for addressing the comments raised by the reviewers during the discussion period, which includes providing more experimental results to address the concerns. We believe the publication of this paper can contribute to the important topic of data augmentation.

The authors are highly recommended to consider all the comments and suggestions made by the reviewers when further revising their paper for publication.
Decision: Accept (Poster)