CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
Authors: Yanru Qu
Dinghan Shen
Yelong Shen
Sandra Sajeev
Weizhu Chen
Jiawei Han
Venue: ICLR 2021 Poster (2021)
Keywords: data augmentation
natural language understanding
consistency training
contrastive learning

Abstract: Data augmentation has been demonstrated as an effective strategy for improving model generalization and data efficiency.  However, due to the discrete nature of natural language, designing label-preserving transformations for text data tends to be more challenging. In this paper, we propose a novel data augmentation frame-work dubbed CoDA, which synthesizes diverse and informative augmented examples by integrating multiple transformations organically.  Moreover, a contrastive regularization is introduced to capture the global relationship among all the data samples.  A momentum encoder along with a memory bank is further leveraged to better estimate the contrastive loss. To verify the effectiveness of the proposed framework, we apply CoDA to Transformer-based models on a wide range of natural language understanding tasks. On the GLUE benchmark, CoDA gives rise to an average improvement of 2.2%while applied to the Roberta-large model. More importantly, it consistently exhibits stronger results relative to several competitive data augmentation and adversarial training baselines (including the low-resource settings). Extensive experiments show that the proposed contrastive objective can be flexibly combined with various data augmentation approaches to further boost their performance, highlighting the wide applicability of the CoDA framework.

Published as a conference paper at ICLR 2021
CoDA: CONTRAST-ENHANCED AND DIVERSITY- PROMOTING DATA AUGMENTATION FOR NATURAL LANGUAGE UNDERSTANDING
Yanru Dinghan Shen", Yelong Shen", Sandra , Jiawei Han', Weizhu Chen"
University of Illinois, Jabana-Champaign. Microsoft Dynamics 365 Al
([yanruqu2, falliiniis... idu.s 2 [(dishen, yeshe, ssajeev, wzchen] wtachhnnnmiccoooft..oom @microsoft.com
ABSTRACT
Data augmentation has been demonstrated effective strategy for improving model generalization and data efficiency. However, due to the discrete nature natural language, designing label-preserving transformations for text data tends to be more challenging. framework dubbed CoDA, which synthesizes diverse and informative augmented examples by integrating multiple transformations organically. Moreover, a con- trastive regularization objective introduced to capture the global relationship among all the data samples. A momentum encoder along with memory bank is further leveraged to better estimate the contrastive loss. To verify the effective- ness of the proposed framework, we apply CoDA to Transformer-based models on a wide range natural language understanding tasks. the GLUE bench- mark, CoDA gives rise to an average improvement of 2.2% while applied to the RoBERTa-large model. More importantly. it consistently exhibits stronger results relative to several competitive data augmentation and adversarial training base- lines (including the low-resource settings). Extensive experiments show that the proposed contrastive objective can be flexibly combined with various data aug- mentation approaches to further boost their performance, highlighting the wide applicability of the CoDA framework.
this paper, we propose a novel data augmentation
INTRODUCTION
Data augmentation approaches have successfully improved large-scale neural-network-bast mod- 2017; Xie et al. 2019: Berthelot et al. 2019 [Sohn et al.] 2020) [He et al.] 2020)
els, (Laine & Aila
Khosla al. 2020 Chen al. (020b)). however, the majority of existing research is geared towards
computer vision tasks. The discrete nature of natural language makes it challenging to design effec-
tive pebe--peeserging transformations for text sequences that can help improve model generalization (Hu et al. 2019 Xie et al., 2019). On the other hand, fine-tuning powerful, peer-paramtterized language models proves to be difficult, especially when there limited amount of task saec-specific data available. It may result in representation collapse (Aghajanyan et al. 2020) or require special finetuning techniques (Sun et al. [2019] Hao et al. 2019)). In this work, we aim to take a further step towards finding effective data augmentation strategies through systematic investigation.
In essence, data augmentation can be regarded as constructing neighborhoods around training instance that preserve the ground truth label. With such a characterization, adversarial training (Zhu ct al., 20200 Jiang et al. 2020 Liu et al., 2020; Cheng et al.. 2020) also performs label-preserving transformation in embedding space, and thus considered as an alternative to data augmentation methods in this work. From this perspective, the goal of developing effective data augmentation strategies be summarized answering three fundamental questions:
i) What are some label- preserving transformations, that can be applied to text, to compose useful augmented samples?
Work was done during an internship at Microsoft Dynamics 365 "To name few, BERT (Devlin et al. 2019)] 340M parameters, T5 (Raffel et al. 2019 11B parameters,
GPT-3 (Brown
2020 175B parameters.

---

Published as a conference paper at ICLR 2021
if) Are these transformations complementary in nature, and can find some strategies to consoli- date them for producing more diverse augmented examples?
fii) How can we incorporate the obtained augmented samples into the training process in an effective and principled manner?
Previous efforts in augmenting text data were mainly focused on answering the first question Yu et al. 2018; Xie et al.) 2019; Kumar et al.) 2019, Wei & Zoul 2019; Chen et al.) 2020a] Shen et al., 2020). Regarding the second question, different label-preserving transformations have been proposed, but it remains unclear how to integrate them organically. In addition, it has been shown that the diversity of augmented samples plays vital role in their effectiveness (Xie et al. 2019 Gontij-Lopes et al., 2020). In the case of image data, several strategies that combine different augmentation methods have been proposed, such as applying multiple transformations sequentially (Cubuk et al., 2018 2020 Hendrycks et al. 2020), learning data augmentation policies (Cubuk) (et al.]|| 2018), randomly sampling operations for each data point (Cubuk et al. (020). However, these methods cannot be naively applied to text data, since the semantic meanings of a sentence are much more sensitive to local perturbations (relative to an image).
As for the third question, consistency training is typically employed to utilize the augmented samples (Laine & Aila, 2017] (endryek et al. 2020; Xie et al. 2019 (Shnn et al. 2020, Miyato et al. 2018). This method encourages the model predictions to be invariant certain label- preservng transfor- mations. However, existing approaches only examine a pair of original and augmented samples in isolation, without considering other examples in the entire training set. As result, the representa- tion of an augmented sample may be closer to those of other training instances, rather than the one it is derived from. Based on this observation, we advocate that, in addition consistency training, training objective that can globally capture the intrinsic relationship within the entire set of original and augmented training instances can help leverage augmented examples more effectively.
In this paper, we introduce a novel Contrast-enhaneee and prommotiy Data Augmentation (CoDA) framework for natural language understanding. To improve the diversity of augmented samples, we extensively explore different combinations of isolated label-preserving transformations in an unified approzch. We find that stacking distinct label- preservng transformations produces par- ticularly informative samples. Specifically, the most diverse and high-quality augmented samples are obtained by stacking an adversarial training module over the back-translation transformation. Besides the monsistency-regularize
loss for repelling the model to behave consistently within lo- cal neighborhoods, we propose a contrastive learning objective to capture the global relationship among the data points in the representation space. We evaluate CoDA on the GLUE benchmark (with RoBERTa (Liu et al., 2019) as the testhed), and CoDA consistently improves the generaliza- tion ability of resulting models and gives rise significant gains relative to the standard fine-tuning procedure. Moreover, our method also outperforms various single data augmentation operations, combination schemes, and other strong baselines. Additional experiments in the low-resource set- tings and ablation studies further demonstrate the effectiveness of this framework.
2 METHOD
In this section, we focus our discussion on the natural language understanding (NLU) tasks, and par- ticularly, under a text classification scenario. However, the proposed data augmentation framework can be readily extended to other NLP tasks as well.
2.1 BACKGROUND: DATA AUGMENTATION AND ADVERSARIAL TRAINING
Data Augmentation Let D {æ;, Je denote the training dataset, where the input ex- ample x; is a sequence of tokens, and yi is the corresponding label. To improve model's robust- ness and generalization ability, several data augmentation techniques (e.g., fack-translation (Sen- nrich et al. 2016 Edunov et al.] 2018: Xie et al.) 2019). mixup (Guo et al.] 2019). c-BERT (Wu al.) 2019h) have been proposed. Concretely, label-pres transformations are performed (on the original training sequences) to synthesize a collection of augmented samples, denoted by
D' [x/ 1/.,E....N Thus, model can learn from both the training set D and the augmented set
D', with (-) the predicted output distribution of the model parameterized by 0:
8" arg min
Σ (EDD
C(po (ap)))))
Σ
((po v6)
(1)
0
( (EE)))
2

---

Published as a conference paper at ICLR 2021
(x(x)
consistancy losa
xx(x))
consistency
pv(8)
The
IadeeTtanaaate
that
Advemiaial
place until now.
was nover interested that
Garrann
now frog has never been
inter usted this place.
Input
oriodd
max Cif(+ +),
ombod opp
(a) Back- translation
(b) Adversarial training
PV(K)
condi sonniy loss
Taci-Traralate
Advensaraal
The welld thingg is that fro
strange thing ls unat
rever intereeted that place undil
German
been irtenestdd this place.
crrbod
Dax t).s))
emboo
(c) Stacking of back-translation and adversarial training
Figure I: Illustration of data augmentation combined with adversarial training.
Several recent research efforts focused encouraging model predictions to be invariant to stochastic or domain-specific data transformations (Xie et al. [2019] Laine & Aila 2017 Tarvainen 2020; Hendrycks et al. 2020). example: xÚ Back Trans( then x; is a paraphrase of z;. The model can be regularized to have consistent predictions for (a;, æ;)), , by minimizing the distribution dis-
& Valpola 2017, (Shnn et al., 2020 Miyato al., 2018 Jiang
Take transsttalattion
crepancy Res(po (p::), (orr))) which typically adopts KL divergence (see Fig. lap.
Adversarial Training In another line, adversarial training methods are applied to text data (Zhu al. 2020 Jiang al., 2020 Cheng et al., 2020; Aghajanyan et 2020) for improving model's robustness. Compared with data augmentation techniques, adversarial training requires no domain knowledge generate additional training examples. Instead, relies on the model itself to produce adversarial examples which the model are most likely to make incorrect predictions. Similar to data augmentation, adversarial training also typically utilizes the cross-entropy and consistency- algorithms, the adversarial loss [Goodfellow et al. 2015) (Eqn. 2] and virtual adversarial loss (Miyato et al. 2018
based objectives for training. As the two most popular
(Eqn. 3) can be expressed as follows (see Fig. lbb:
RaT Ci, v:) L (po (m:), vs). s.h., ||: x.,l S S R2vT(2;; k.) Res (pe (po( (.), po (z.)). s.t., || x,ll e.
(2)
(3)
Generally, there is no closed-form to obtain the exact adversarial example è; in either Eqn. 2 or However, it usually can be approximated by a low-order approximation of the objective function with respect to x,, For example, the adversarial example in Equ. 2 can be approximated by:
2 x.
where g Vz, (po((()()))
(4)
2.2 DIVERSITY-PROMOTING CONSISTENCY TRAINING
As discussed in the previous section, data augmentation and adversarial training share the same intuition of producing neighbors around the original training instances. Moreover, both approaches share very similar training objectives. Therefore, it is natural to ask the following question: are different data augmentation methods and adversarial training equal in nature? Otherwise, are they complementary to each other, and thus can be consolidated together to further improve the model's generalization ability? Notably, it has been shown, the CV domain, that combining different data augmentation operations could lead to more diverse augmented examples (Cubuk et al. 2018 2020 Hendrycks et al. 2020). However, this is especially challenging for natural language, given that the semantics of sentence can be entirely altered by slight perturbations.
To answer the above question, we propose several distinct strategies to combine different data trans- formations, with the hope to produce more diverse and informative augmented examples. Specifi- cally, we consider different types of preservig transformations: tacc-trranslttion (Sennrich et al. 2016; Edunov al. 2018 Xie et al. 2019). c-BERT word replacement (Wu et al. 2019), mixup (Guo et al.) 20192019] Chen
[Cen et al. 2020a). cutoff (Shen et al., 2020), and adversarial training The 3 combination strategies schematically illustrated
(7hu et al. 2020 Jiang et al. 2020
3

---

Published as a conference paper at ICLR 2021
01
X2
01
Q2
X
Random combination
Mixup interpolation
(c) Sequential stacking
Figure 2: Illustration of different strategies to combine various habel-preserving transformations.
in Figure 2
For random combination, a particular label-preserving transformation is randomly se- lected, among all the augmentation operations available, for each mini-batch. As the mixup interpolation, given two samples æ; and æ; drawn in mini-batch, linear interpolation is performed between their input embedding matrices e; and e; (Zhang a is the interpolation parameter, usually drawn from a Beta distribution.
2017): e - ae; + L-ajejj. where
Moreover, we consider stacking different label-preserving transformations in sequential manner (see Figure 2ch. It is worth noting that due to the discrete nature of text data, some stacking orders infeasible. For example, quence to the fack-translation module. Without loss of generality, we choose the combination where adversarial training is stacked over (see Fig. Ic]. Formally, given training example (æ,, Mk), the consistency training objective for such a stacking operation can be written as:
reasonable to provide an wwversarially-perurrbii
embedding se-
to demonstrate the sequential stacking operation
x BackTrans(aa)),
argmax Rat R1, Vi).
coonsistency (a,,, 3; y/i (poo(:;) vi) + ((po + (pos
(6)
where the first term corresponds to the cross-entropy loss, the second term is the adversarial loss, Res denotes the consistency loss term between (æ; @;). Note that è; is obtained through two different label-presenr babbl-pressrving transformations applied æ, and thus deviates farther from and should be more diverse than x! Inspired by (Bachman et al. 2014 Zheng et al., 2016 Kannan et al., 2018 Hendrycks al. 2020), employ the Jensen-Shannon divergence for Res- since is upper bounded and tends to be more stable and consistent relative to the KL divergence:
Res (ps(
(po( + KL(po(
where M (p)( + po (=:)))) //2 Later we simply use x/ to represent the transformed example.
2.3 CONTRASTIVE REGULARIZATION
contrastive loss
Consistency loss only provides local regularization, iee. æ; and of should have close predictions. How- the relative positions between x and other training instances Ij (j # i) have not been exam- ined. In this regard, we propose leverage trastive learning objective to better utilize the aug- mented examples. Specifically, assume that the model should encourage an augmented sample x; to be closer, in the representation space, its original sample relative to other data points æ, (j * i) in the training set. This reasonable assumption since intuitively, the model should be robust enough to successfully determine from which original data augmented sample is produced.
similarity
update
{q q
[ky 1..
memory bank
momentum
query
{x,, x(}
Figure 3: Illustration of the contrastive learn- ing module.
The contrastive learning module is illustrated in Fig. 3 As demonstrated by prior efforts on con- trastive learning, adopting a large batch size is especially vital for its effectiveness (Chen et al.) 2020b Khosla et al. 2020). Therefore, we introduce a memory bank that stores the history em- beddings, thus enabling much larger number of negative samples. Moreover, to avoid the encoder from changing too rapidly (which may result in inconsistency embeddings), a momentum encoder module is incorporated into algorithm. Concretely, let fo(.) and fa(.) denote the transformation

---

Published as a conference paper at ICLR 2021
parameterized by the query encoder and key encoder, respectively. Note that A and & represent their parameters. The momentum model parameters are not learned by gradients. Instead, they are updated through the momentum rule: 8 t 7 + (1 22)0 at each training step. We omit the details here and refer the interested readers to the work by (He et al. (020) for further explanation. Given sample a; and its augmented example zzy. the query and key can be obtained follows:
= [o(;),
q = -o(()))
k; = fo(x;)
(8)
Thus, the contrastive training objective can be written as:
coomras (I,, (2, z/. M) RcT(q, k, M) + (q: M,,,
(9)
exp(sim(q:: q: k:)/T)
Ret(q ky M) -log (()E) exp(sim(q:: k²)/T)²
(10)
where r is the temperature, and M is the memory bank which the history keys are stored. Cosine similarity is chosen for sim(-). Note that ReT(q k,, is similarly defined as Rer(q: k, M) (with q, replaced by q in Equ. I0). In Eqn. [ the first term corresponds to the contrastive loss calculated on the original examples (self-contrastive loss), while the second term is computed loss). Under such a framework, the pair of original
the augmented sample (augment-contrastee encouraged to stay closer in the learned embedding space, relative to all other training instances. As a result, the model is regularized globally through considering the embeddings of all the training examples available.
and augmented samples
By integrating both the consistency training objective and the contrastive regularization, the overall training objective for the CoDA framework can be expressed as:
0 argming
Σ
Cconsistaccy (2,, x',, yv) + ee (z,, x/,MM
(11)
(DD
where À is a hyperparameter to be chosen. It is worth noting that the final objective has taken both the local (consistency loss) and global (contrastive loss) information introduced by the augmented examples into consideration.
3 EXPERIMENTS
To verify the effectiveness of CoDA, We evaluate it on the widely-adopted GLUE benchmark (Wang
jet al. 2018), which consists of multiple natural language understanding (NLU) tasks. The details of these datasets can be found in Appendix B RoBERTa (Liu et al.] 2019) is employed as the
testbed for our experiments. However, the proposed approach can be flexibly integrated with other
models as well. We provide more implementation details in Appendix C Our code will be released to encourage future research.
In this section, we first present our exploration of several different strategies to consolidate vari- ous data transformations (.1c Next, we conduct extensive experiments to carefully select the contrastive objective for NLU problems in Sec 3.2 Based upon these settings, we further evalu- ate CoDA on the GLUE benchmark and compare it with a set of competitive baselines in Sec 33.3 Additional experiments in the low-resource settings and qualitative analysis (Sec (344 are further conducted to gain a deep understanding of the proposed framework.
3.1 COMBINING LABEL-PRESERVING TRANSFORMATIONS
We start by implementing and comparing several data augmentation baselines. As described in the previous section, we explore different approaches: tickstrrnnsltion, c-BERT word replacement, Mixup, Cutoff and adversarial training. More details can be found in Appendix A The standard cross-entropy loss, along with the consistency regularization term (Eq. 66 is utilized for all meth- ods to ensure fair comparison. We employ the MNLI dataset and RoBERTa- model for the comparison experiments with the results shown in Table
All these methods have achieved improvements over the RoBERTa- model, demonstrating the effectiveness of leveraging label- preservin transformations for NLU, Moreover, translrannlttion, cutoff and adversarial training exhibit stronger empirical results relative to mixup and c-BERT.
To improve the diversity of augmented examples, we explore several strategies to combine multi- ple transformations: /) random combination, ii) mixup interpolation, and ii) sequential stacking, as
5

---

Published as a conference paper at ICLR 2021
shown in Fig. 2 mations. This may be attributed to the fact that different label- aabe-ppree transformations regularize the model in distinet ways, and thus the model may not be able to leverage different regularization terms simultaneously.
In Table the score of naive random combination lies between single transfor-
Besides, among all the other combination strategies, we observe that gains can be ob- tained by integrating back-translation and ad- versarial training together. Concretely, mixing back trck-translation and adversarial training sam- ples (in the input embedding space) slightly im- prove the accuracy from 88.5 to 88.6 6. More importantly, the result is further improved to 88.8 with these two transformations stacking togethef (see See 2.2). With significance test, we find stack (back, adv) performs consistently better than other combinations (t-test of 10 runs, p-values < 0.02). This observation in- dicates that the stacking operation, especially in the case of back-translation and adversarial training. can produce more diverse augment ex- amples.
MNLI-m (Acc)
Method
MMD
RoBERTa-base
87.6
Single Transformation
back-translation
88.5
0.63
+ c-BERT
88.0
0.01
cutoff
88.4
0.02
mixup (ori, ori)
88.2
0.06
adversarial
88.5
0.65
Multiple Transformations
random (back, cut, adv)
88.4
mix (ori, back)
88.4
0.11
mix (back, adv)
88.6
0.81
stack (back, cut)
88.5
0.62
+ stack (back, adv)
88.8
1.14
stack (back, cut, adv)
88.5
1.14
stack (back, adv, cut)
88.4
1.14
Intuitively, the augmented sample, with two se- quential transformations, deviates more from the corresponding training data, and thus tends to be more effective at improving the model's generalization ability. To verify this hypothesis, we further calculate the MMD) (Gretton et al.) 2012) between augmented samples and the original training instances. It can be observed that stack (back, adv), stack (back, cut, adv) and stack (back, adv, cut) have all produced examples the farthest from the original training instances (see Table I.. However, we conjecture that the latter two may have altered the semantic meanings too much, thus leading to inferior results. In this regard, stack (back, adv) is employed as the data transformation module for all the experiments below.
Table I: Comparison of different transformations
on the MNLI-m development set. Abbr: original training instances (ori), back-translation (back), cutoff (cut), mixup (mix), adversarial (adv).
3.2 CONTRASTIVE REGULARIZATION DESIGN
In this section, we aim incorporate the global information among the entire set of original and augmented samples via a contrastive regularization. First, we explore a few hyperparameters for the proposed contrastive objective. Since both the memory bank and the momentum encoder are vital components, we study the impacts of different hyperparameter values on both the temperature and the momentum. As shown in Fig. 4aa temperature of 1.0 combined with the momentum of 0.99 can achieve the best empirical result. We then examine the size effect of the memory bank, and observe a larger memory bank size leads to a better capture of the global information and results in higher performance boos (see Fig. (b).
(see
After carefully choosing the best setting based on the above experiments, we apply the contrastive learning objective to several GLUE datasets. We also implement several prior works on contrastive learning to compare, including the MoCo loss (He et al. 2020) and the supervised contrastive (Sup- Con) loss (Khosla et al. 2020). all implemented with memory banks. Note that remove the consistency regularization for this experiment to better examine the effect of the contrastive regular- ization term (i.e., a exhibits the largest performance improvement. This observation demonstrates for NLJ,, our data transformation module can be effectively equipped with the contrastive regularization.
0, À A 0). As presented in Table our contrastive objective consistently
"In practice. we can use Machine Translation (MT) models trained on large parallel corpus (e.g., English- French, Anglish-German) to back fack-translate the input sentence. Since tacktrranltttinn requires decoding, it be performed offline. If the input contains multiple sentences, we split into sentences, perform back- translation, and ensemble those paraphrases back.
We set the default memory bank size as 65536. As smaller datasets, we choose the size no larger than the number of training data (e.g., MRPC has 3.7k examples, and the memory size as 2048).

---

Published as a conference paper at ICLR 2021
88.3
88.2
temp 0.01
MNLI-m
INLI-m dev Accuracy
88.2
temp 0.07
88.0
MNLI-mm
88.1
temp
Accuracy (%6)
87.8
88.0
87.6
0.9
0.99
0,999
0.9999
256 5121024
4096
16384
65536
Momentum
Memory Bank Size (log scale)
(a) Momentum encoder
(b) Memory bank
Figure 4: Hyperparameter exploration for the contrastive loss, evaluated on the MNLI-m develop-
ment set. Note: All models use the RoBERTa-bass model as the encoder.
MNLI-m (Ace)
QNLI
SST-2
RTE
MRPC
Method
(Acc)
(Acc)
(Acc)
(Acc)
KoBBRTTabaas
87.6
92.8
94.8
78.7
90.2
MoCo [ee
2020
88.2
93.3
95.1
80.8
90.9
SupCon (Khosla et al. 2020
88.
93.2
95.2
80.5
90.2
Contrastive (ours)
88.1
93.6
95.3
82.0
91.7
Table 2: Comparison among different contrastive objectives on the GLUE development set.
3.3 GLUE BENCHMARK EVALUATION
With both components within the CoDA algorithm being specifically tailored to the natural language understanding applications, we apply it the RoBERTa- model (Liu et al., (219). Compar- isons are made with several competitive approaches on the GLUE benchmark. Specifically, we consider vack-translation, cutoff (Shen al., 2020), FreeI.B (Zhu et al., 2020). SMART (Jiang et al. 2020), and R3F (Aghajanyan et al. 2020) as the baselines, where the last three belong to adversarial training. The results are presented in Table
tat----ttttttnbaaaeeeeeee and adversarial penssarral-tramingg -tta-trrnnnaaaae based
It is worth noting that trecssrannlttoon is based our implementation, where both the cross-entropy and consistency regularization terms are utilized.
MNLI-m/ mm (Acc)
QQP (Acc/F1)
QNLI
SST-2
MRPC (Acc/F1)
CoLA (Mec)
RTE (Acc)
STS-B (P/S)
Method
Avg
(Acc)
(Acc)
RoBERTa-large
90.2/-
92.2/-
94.7
96.4
-/90.9
68
86.6
92.4/-
88.9
Back-Trans
91.1/90.4
92/-
95.3
97.1
90.9/93.5
69.4
91.7
92,8/92.6
90.4
Cutoff
91.1/-
92.4/-
95.3
96.9
91.4/93.8
71.5
91.0
92.8/-
90.6
FreeL.B
90.6/-
92.6/-
95
96.7
91.4/-
71.1
88.1
92.7/-
SMART
91.1/91.3
92.4/89.8
95.6
96.9
89.2/92.1
70.6
92
92.8/92.6
90.4
R3F
91.1/91.3
92.4/89.9
95.3
97.0
91.6/-
71.2
88.5
CoDA
91.3/90.8
92.5/89.9
95.3
97.4
91.9/94
72.6
92.4
93/92.7
91.1
Table 3: Main results of single models on the GLUE development set. Note: The best result on each task is in bold and "_" denotes the missing results. The average score is calculated based on the same setting as RoBERTa.
We find that CoDA brings significant gains to the RoBERTa-large model, with the averaged score on the GLUE dev set improved from 88.9 to 91.1. More importantly. CoDA consistently outperforms these strong baselines (indicated by a higher averaged score), demonstrating that our algorithm can produce informative and high-quality augmented samples and leverage them effectively as well. Concretely, on datasets with relatively larger numbers of training instances (> 100K), i.e.. MNLI, QQP and QNLI, different approaches show similar gains over the RoBERTa-large model. However, on smaller tasks (SST-2, MRPC, CoLA, RTE, and STS-B), CoDA beats other data augmentation or adversarial-basee methods by a wide margin. We attribute this observation to the fact that the synthetically produced examples more helpful when the tasks-specific data is limited. Thus, when smaller datasets are employed for fine-tuning large-scale language models, the superiority of the proposed approach manifested larger extent.
3.4 ADDITIONAL EXPERIMENTS AND ANALYSIS
Low-resource Setting To verify the advantages of CoDA when a smaller number of task- specific data is available, further conduct low-resource experiment with the MNLI

---

Published as a conference paper at ICLR 2021
92.3
MNLI-m dev Accuracy 196
85
ReBERTa
90.0
Back
80
QNL dev Accuracy
87.5
CoDA
Adv
85.0
75-
RaBERTa
82.5
Back
80.0
Adv
70
CoDA
77.5
0.2
0.5
1.0
2.0
5.0
10.0
20.0
0.2
0.5
1.0
2.0
5.0 10.0 20.0
Train (%, log scale)
Train (%, scale)
Figure 5: Low-resource setting experiments on the MNLI (left) and QNLI (right) dev sets.
and QNLI datasets. Concretely, different proportions of training data are sampled and utilized for training. We apply CoDA RoBERTa-base and compare it with back- translation and adversarial training across various training set sizes. The corresponding re- sults are presented in Fig. 5 We observe that back-translation and adversarial training ex- hibit similar performance across different proportions. More importantly, CoDA demonstrates stronger results consistently, further highlighting its effectiveness with limited training data.
The Effectiveness of Contrastive Objective To investigate the general upplicability of the proposed contrastive regularization objective, we further ap-
94.0
weo contrastive contractive
ply it to different data augmentation methods. The
ONU Accuracy
RoBERTa-bee base model and QNLI dataset are lever-
93.6
aged for this set of experiments, and the results are
93.4
shown in Fig. 0 We observe that the contrastive learning objective boosts the empirical performance of the resulting algorithm regardless of the data aug- mentation approaches it is applied to. This further validates our assumption that considering the global information among the embeddings of all examples is beneficial for leveraging augmented samples more effectively.
93.2
93.0
Back
Cutoff DA Dattranstormtiins
Adv
Ours
Figure 6: Evaluation of the proposed con- trastive objective while applied to different data augmentation approaches.
4 RELATED WORK
Data Augmentation in NLP Different data augmentation approaches have been proposed for text data, such as fack-translation (Sennrich et al. 2016 Edunov et al., 2018 Xie et al. 2019), c-BERT word replacement (Wu al. 2019), mixup (Guo al. 2019 Chen al. 2020a), Cutoff (Shen al,, 2020). Broadly speaking, adversarial training Zhu et al. 2020 Jiang et al.) 2020) also synthesizes additional examples via perturbations at the word embedding layer. Although effective, how these data augmentation transformations may be combined together to obtain further improvement has been rarely explored. This could be attributed to the fact that a sentence's s semantic meanings are quite sensitive to small perturbations. Consistency-regulatte
loss (Bachman et al.) 2014: Rasmus et al. 2015] Laine & Aila 2017] (Tarvainen & Valpola| (017)) is typically employed as the training objective, which ignores the global information within the entire dataset.
Contrastive Learning Contrastive methods learn representations by contrasting positive and neg- ative examples, which has demonstrated impressive empirical success in computer vision tasks (Hénaff et al., 2019 He et al. 2020). Under an unsupervised setting, Contrastive learning ap- proaches learn representation by maximizing mutual information between local-global hidden rep- resentations (Hjelm et al. 2019] Oord et al. 2018 HHénaff al.] (019). It can be also leveraged to learn invariant representations by encouraging consensus between augmented samples from the same input (Bachman al.) 2019: Tian et al. 2019). He et al. (2020); Wu et al. (2018) proposes to utilize memory bank enable much larger number of negative samples, which is shown ben- efit the transferability learned representations as well (Khosla et al. (2020). Recently, contrastive
learning also employed improve language model pre-training (Iter al. 2020
5 CONCLUSION
In this paper, proposed CoDA, Contrast-enhanced and Diversity promoting data Augmenta- tion framework. Through extensive experiments, we found that stacking adversarial training over a back-translation module give rise more diverse and informative augmented samples. Besides,
8

---

Published as a conference paper at ICLR 2021
introduced precially-esiinnd contrastive loss incorporate these examples for training in principled manner. Experiments on the GLUE benchmark showed that CoDA consistently improves over several competitive data augmentation and adversarial training baselines. Moreover, it is ob- served that the proposed contrastive objectíve can be leveraged to improve other data augmentation approaches as well, highlighting the wide applicability of the CoDA framework.
REFERENCES
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. arXiv preprint 2rX0v:2008.03 2020.
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Advances in neural information processing systems, pp. 3365-3373, 2014.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp. 15535-15545, 2019.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. Advances in Neural Information Processing Systems, pp. 5049-505) 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Jiaao Chen, Zichao Yang. and Diyi Yang. MixText: Lingaistically-inoomeed interpolation of hid- den space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2147-2157, Online, July 2020a. Asso- ciation for Computational Linguistics. doi: 0.18653/v1/0020 mai- URL https:
7/ www. awwweb.wwb.... org/ http://tthhoohotgooee acl-ma 1 194
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 200% 2020b.
Yong Cheng, Lu Jiang, Wolfgang Macherey, and Jacob Eisenstein. Advaug: Robust adversarial augmentation for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, 2020.
E. Cubuk, Barret Zoph, Dandelion Mané, Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data. ArXiv,
2018.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, 2019.
Terrance Dries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04:: 04552, 2017.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- cessing, 2018.
Raphael Gontejo-Lopes, Sylvia Smullin, Ekin D Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprint arXiv:2002.08973. 2020.

---

Published as a conference paper at ICLR 2021
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. 2015.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Augmenting data with mixup for sentence classifi- cation: An empirical study. arXiv preprint arXiv: 1rXiv:1905.0894/, 2019.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of bert. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.0922 2019.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In 8th International Conference on Learning Representations, ICLR, 2020.
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR, 2019.
Karan Grewal, Phil Bachman, Adam
Vu Cong Duy Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. Iterative back- translation for neural machine translation. Proceedings of the 2nd Workshop on Neural Ma- chine Translation and Generation, pp. 18-24, 2018.
Zhiting Hu, Bowen Tan, Russ Salakhutdinov, Tom M Mitchell, and Eric P Xing. Learning data manipulation for augmentation and weighting. In Advances in Neural Information Processing Systems, pp. 15764-15775, 2019.
Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. Pretraining with contrastive sentence ob- jectives improves discourse performance of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI, 2020.
Haoming Jiang, Pengeheng He., Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regu- larized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, ACL, 2020.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv:/803.06377. 2018.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint orXiv:2004.11366 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Interna- tional Conference on Learning Representations, ICLR 2015, 2014.
Ashutosh Kumar, Satwik Bhattamishra, Manik Bhandari, and Partha Talukdar. Submodular optimization-basee based diverse paraphrasing and its effectiveness in data augmentation. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
10

---

Published as a conference paper at ICLR 2021
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. Adversarial training for large neural language models. arXiv preprinl arXiv:2004 a0Xiv:2004.08994 2020.
Yinhan Liu, Myle Out Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
2019.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and supervsse learning. IEEE transactions on pattern analysis and machine intelligence, 11(8):1979-1993 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 87700778,,7 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. In Advances in neural information processing systems, pp. 3546-3554, 2015.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL, 2016.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. A simple but tough-to- beat data augmentation approach for natural language understanding and generation, 2020.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the Thirty-Secone AAAI Conference on Artifi- cial Intelligence, (AAAI-18), 2018.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.0766 2020.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for text classification? In China National Conference Chinese Computational Linguistics, pp. 194-206. Springer, 2019.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averager consis- tency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pp. 1195-1204, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv: (ttp://906.084444 2019.
arXiv preprint
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018, pp. 353, 2018
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing, EMNIP-IJCNIPP.. 2019.
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. Conditional bert contextual augmentation. In International Conference on Computational Science, pp. 84-95. Springer, 2019.
11

---

Published as a conference paper at ICLR 2021
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742. 2018.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904 /2848, 2019.
Adams Wei Yu, David Dohan, Minh-
Minh-Thang Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. Qanet: Combining local convolution with global self-attention for reading compre- hension. In 6th International Conference on Learning Representations, ICLR 2018, 2018.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 171.09471 2017.
Stephan Zheng, Yang Song. Thomas Leung. and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 4480-4488, 2016.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Thomas Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for language understanding. 8th International Conference on Learning Representations, ICLR, 2020.
12

---

Published as a conference paper at ICLR 2021
A DATA AUGMENTATION DETAILS
We select the following representative data augmentation operations as basic building blocks of our data augmentation module. We denote a; = "5.1....,,;..l, as the input text sequence, and e, .e........ e.,]] as corresponding embedding vectors.
Back translation is widely applied in machine translation (MT) (Sennrich et al. 2016 Hoang et al. 2018; |duuno et al.) 2018), and introduced to text classification recently (Xie et al. 2019). Back-Trans uses 2 MTT models to translate the input example another pivot language, and then translate it back, 2; - Pivot Language -> z
C-BERT Word Replacement (Wu al. 2019) is representative of the word replacement aug- mentation family. C-BERT pretrains a conditional BERT model to learn contextualized represen- tation P(x ... P [MASK] ......... 2.,1,, y/) conditioning classes. This method then randomly substitutes words of obtain x' .FF.......... I..I
Cutoff VDees & Taylor 2017) randomly drops units in continuous span on the input, while Shen et al. (2020) adapts this method to text embeddings. For input embeddings this method randomly set continuous span of elements to Os, ey je,,1 ........e De.,I...eijj Cii-i, .....0.... Cij+o e,,.), where the window size x I, and the start position j E [1, - wi| is randomly selected. For trans- former encoders that involve position embeddings, we also set input mask as at corresponding positions.
Mixup Zhang et al. 2017) interpolates two image as well as their labels. Guo et al. (2019) borrows this method to text. For 2 input embeddings (e;, ej), mixup interpolates the embedding vectors e = ae; + (1 - a)e; where sampled from a Beta distribution. Also, the labels are interpolated for the augmented sample y/ ay; + (1 a)
Adversarial training generates adversarial examples for input embeddings. simply, ef arg -|: ((f(e)), yt). We mainly follow the implementation of Zhu et al. (220)). Be- sides, when computing the adversarial example e', the dropout variables recorded and reused later when encoding e'.
Maximum mean discrepancy (MMD) Gretton et al. 2012 is a widely used discrepancy measure for 2 distributions. We adopt the multi-kemat MMD implementation based on Shen al. 2018 to quantify the distance of data distributions before and after DA transformations.
B DATASET DETAILS
The datasets and statistics are summarized in Table
Sentence Pair
Corpus
Task
#Train
#Dev
#Test
#Class
Metrics
MNLI
NLI
393k
20k
20k
Accuracy
QQP
Paraphrase
364k
40k
391k
Accuracy/F1
QNLI
QA/NLI
V
108k
5.7k
5.7k
2
Accuracy
SST
Sentiment
X
67k
872
1.8k
Accuracy
MRPC
Paraphrase
7
3.7k
408
1.7k
2
Accuracy/Fi
CoLA
Acceptability
8.5k
1k
1k
2
Matthews corr
RTE
NLI
2.5k
276
3k
Accuracy
Pearson/Spe-
STS-B
Similarity
7k
1.5k
1.4k
arman corr
Table 4: GLUE benchmark summary.
*EDA Wei Zou 2019) uses synonym replacement, another word replacement technique. choose
C-BERT for this family to take the advantages of contextual representation.
13

---

Published as a conference paper at ICLR 2021
C IMPLEMENTATION DETAILS
Our implementation is based on RoBERTa (Luu et al. 2019). We use ADAM (Kingma & Ba 2014) our optimizer. We follow the hyper-parameter study of RoBERTa and default the following parameters: batch size (32), learning rate (Ie-5), epochs (5), warmup ratio (0.06), weight decay (0.1) and we keep other parameters unchanged with RoBERTa. For Back-Trans, we use the en-de single models trained on WMT19 and released in FairSeq. More specifically, we use beam search (beam size 5) and keep only the top-1 hypothesis. We slightly tune Adversarial parameters on MNLI based on FreeL.B and fix them on other datasets, since adversarial training is not our focus. For contrastive regularization, we implement based on MoCo. In GLUE evaluation, we mainly tune the weights of 3 regularization terms, [0, 1], E [0, 3|, À E [0, 0.03] (Eq. 6 [T.. Besides, for smaller tasks (MRPC, CoLA, RTE, STS-B), we use the best performed MNLI model to initialize their parameter
'RoBERTa: http://cwww..oooouugggggeettnnnn
FairSeq: https://gthhub.com/// http::/jiihul.commyytoobb//ggggg MoCo: http://ghhbbeemmicccceerrrrrrrn We will
FreeI.B:
release model and code for further study.
14


Reviews:
[{"review_id": "uslwszAU-EW", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe augmentation of NLP samples is an important task with no clear \"applicable to all\" mechanism. This is in sharp contrast to computer vision where techniques like rotation, modification of hue, saturation as well as umpteen other techniques exist. This work tries to address the issue by proposing a technique that carefully amalgamates multiple previously known approaches to generate diverse label preserving examples. The experimental results on RoBERTa highlight the applicability and importance of this data augmentation approach on the downstream task of text classification (GLUE).\n\nStrengths:\n\n1. Empirical results. Performance better than previous approaches (although minor).\n2. Paper Clarity\n3. Each formulation is backed by a strong intuitive understanding.\n4. Contrastive training (negative sampling) is one of the crucial contributions of this work. It seems to be making every previously known augmentation approach better. \nPlease feel free to highlight other major contributions.\n\nWeaknesses (Minor):\n\n1. Ad-hoc regularization parameter selection is necessary for getting performance gains. This makes it difficult to conclusively prove that this is an \"applicable to all\" data augmentation scheme.\n2. It would have been better to see the performance gains on more difficult text-classification tasks (non-GLUE), or underperforming models (non-BERT based). Since the gains are not much. It becomes difficult to fathom if the gains are actually due to good objective function or a case of chance for choosing better examples.\n\n\nComments/Questions:\n\n1. What is the augmentation size being used in the setup? I suspect the size plays an important role in such setups and this hasn't been discussed much in the paper. Also, please show the performance trends based on different augmentation sizes.\n\n2. How do you measure the diversity (as mentioned in the paper title) in the generated samples? \n\n3. Rather than using the ad-hoc approach for selecting which augmentation \"stacking\" scheme is helpful, it would have been better to compare/use an approach highlighted in \"Learning to Compose Domain-Specific Transformations for Data Augmentation\" [NeuRIPS 2017].\n\n\nCorrection:\n\n1. Related Work: Contrastive learning - Under an unsupervised setting, ontrastive -> contrastive\n\nOverall:\n\nThis work highlights the importance of incorporating contrastive training for data augmentation.\n\nPlease let me know if I have misunderstood something(s)", "extra_scores": {}, "extra_text": {}}, {"review_id": "scZsGOYdKGd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel data augmentation framework, which explores different combinations of isolated label-preserving transformations to improve the diversity of augmented samples.  The authors find that stacking distinct label-preserving transformations produces particularly informative samples.  The paper also introduces a contrastive learning objective to capture the global relationship among the data points in representation space.  \n\nIn my opinion, the exploration of different combinations of isolated label-preserving transformations is the major contribution of this paper, which may inspire future works for data augmentation. However, the contrastive regularization object is a bit incremental, and I cannot see a big difference compared with Moco or SupCon.   \n\nStrength:\n\n+ The idea of stacking distinct label-preserving transformations is intuitive.\n+ The integration of the consistency training objective and the contrastive regularization objective is interesting.\n\nWeakness:\n\n- Lack of novelty, the contrastive regularization object is a bit incremental, and this object is very similar to MoCo or SupCon.\n- The model has first to obtain the augmented samples, which is computation expensive for large-scale datasets and may hinder the practical application of the model. Moreover, the overall improvements are relatively small compared with R3F, and there is a lack of variance analysis.\n\nQuestions:\n\nWhat is the computational complexity of CoDA?\n\nWhy using MMD distance in section 3.1?\n\nIs stacking distinct label-preserving transformations the default setting for CoDA in your GLUE experiments? What if other strategies (mix, random) work better in datasets like QNLI, RTE. MRPC, and so on. Why not report results on those datasets?\n\nWhat is the major difference between your contrastive regularization and MoCo or SupCon?\n\nAs the improvements are relatively small, could you please provide the test of statistical significance？\n\nWhat if you stack cut first and then back? Does the order affect the performance?", "extra_scores": {}, "extra_text": {}}, {"review_id": "_7lW_Y77tZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper proposes a contrastive learning-based approach to combine different data augmentation techniques for NLP tasks. While the widely used consistency loss focuses on a single example, the proposed contrastive objective allows capturing the relationships among all data samples which helps in producing diverse and informative examples. For experiments, the paper explores 5 data augmentation approaches with Roberta-large as the classification model. Empirical results on the standard GLUE benchmark leads to an impressive 2.2% average improvement. Authors also found that back-translation and adversarial training combination leads to better performance than other DA combinations.  \n\n\nStrengths:\n1. The proposed framework can be applied with any text data augmentation methods. It's a solid work that will help the NLP community in developing better DA techniques. For example, [Kumar et al. 2020] shows that any pre-trained model can be used for data augmentation. I believe seq2seq model like T5, BART based augmentation combined with CoDA, will further push the state of the art for text DA. \n2. Paper provides clear motivations and describes their methods, experiments in detail. Authors study DA in both low-resource and rich-resource setting. Ablation studies have been conducted to investigate gains from different components.  \n3. Authors plan to release their code which is good for reproducibility.  \n\nWeakness: \nMy understanding is that all numbers reported in the paper are from a single experiment. As a reader, I would like to see some variance with the results. Apart from this, I don't see any major issues with the paper.   \n\nQuestions:\n1. Since one of your goals is to improve the diversity of the augmented data, have you tried replacing more words in the c-bert model? By nature, c-bert is bound to replace max 15% of the tokens while maintaining the sentence length. Methods such as back-translation or seq2seq models do not have such restrictions.  Also, have you considered using a pre-trained seq2seq model for DA as in [Kumar et al. 2020]\n2. Fig 5, back-translation, and adversarial training have similar performance. This result is intriguing.  Do you have some further insights into it? \n\nTypos:\n- Sec2.2. \"the first term correspond\" -> corresponds \n- Sec 4, Contrastive Learning para, \"ontrastive learning\" -> \"Contrastive learning\"\n\nReferences (additional DA citations):\n1. Kumar, V., Choudhary, A., & Cho, E. (2020). Data Augmentation using Pre-trained Transformer Models. ArXiv, abs/2003.02245.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper concerns data augmentation techniques for NLP. In particular, the authors introduce a general augmentation framework they call CoDA and demonstrate its utility on a few benchmark NLP tasks, reporting promising empirical results. The authors addressed some key concerns (e.g., regarding hyperparameters, reporting of variances) during the discussion period. The consensus, then, is that this work provides a useful and relatively general method for augmentation in NLP and the ICLR audience is likely to find this useful.
Decision: Accept (Poster)