CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding

1 INTRODUCTION . Data augmentation approaches have successfully improved large-scale neural-network-based models , ( Laine & Aila , 2017 ; Xie et al. , 2019 ; Berthelot et al. , 2019 ; Sohn et al. , 2020 ; He et al. , 2020 ; Khosla et al. , 2020 ; Chen et al. , 2020b ) , however , the majority of existing research is geared towards computer vision tasks . The discrete nature of natural language makes it challenging to design effective label-preserving transformations for text sequences that can help improve model generalization ( Hu et al. , 2019 ; Xie et al. , 2019 ) . On the other hand , fine-tuning powerful , over-parameterized language models1 proves to be difficult , especially when there is a limited amount of task-specific data available . It may result in representation collapse ( Aghajanyan et al. , 2020 ) or require special finetuning techniques ( Sun et al. , 2019 ; Hao et al. , 2019 ) . In this work , we aim to take a further step towards finding effective data augmentation strategies through systematic investigation . In essence , data augmentation can be regarded as constructing neighborhoods around a training instance that preserve the ground-truth label . With such a characterization , adversarial training ( Zhu et al. , 2020 ; Jiang et al. , 2020 ; Liu et al. , 2020 ; Cheng et al. , 2020 ) also performs label-preserving transformation in embedding space , and thus is considered as an alternative to data augmentation methods in this work . From this perspective , the goal of developing effective data augmentation strategies can be summarized as answering three fundamental questions : i ) What are some label-preserving transformations , that can be applied to text , to compose useful augmented samples ? ∗Work was done during an internship at Microsoft Dynamics 365 AI . 1To name a few , BERT ( Devlin et al. , 2019 ) : 340M parameters , T5 ( Raffel et al. , 2019 ) : 11B parameters , GPT-3 ( Brown et al. , 2020 ) : 175B parameters . ii ) Are these transformations complementary in nature , and can we find some strategies to consolidate them for producing more diverse augmented examples ? iii ) How can we incorporate the obtained augmented samples into the training process in an effective and principled manner ? Previous efforts in augmenting text data were mainly focused on answering the first question ( Yu et al. , 2018 ; Xie et al. , 2019 ; Kumar et al. , 2019 ; Wei & Zou , 2019 ; Chen et al. , 2020a ; Shen et al. , 2020 ) . Regarding the second question , different label-preserving transformations have been proposed , but it remains unclear how to integrate them organically . In addition , it has been shown that the diversity of augmented samples plays a vital role in their effectiveness ( Xie et al. , 2019 ; Gontijo-Lopes et al. , 2020 ) . In the case of image data , several strategies that combine different augmentation methods have been proposed , such as applying multiple transformations sequentially ( Cubuk et al. , 2018 ; 2020 ; Hendrycks et al. , 2020 ) , learning data augmentation policies ( Cubuk et al. , 2018 ) , randomly sampling operations for each data point ( Cubuk et al. , 2020 ) . However , these methods can not be naively applied to text data , since the semantic meanings of a sentence are much more sensitive to local perturbations ( relative to an image ) . As for the third question , consistency training is typically employed to utilize the augmented samples ( Laine & Aila , 2017 ; Hendrycks et al. , 2020 ; Xie et al. , 2019 ; Sohn et al. , 2020 ; Miyato et al. , 2018 ) . This method encourages the model predictions to be invariant to certain label-preserving transformations . However , existing approaches only examine a pair of original and augmented samples in isolation , without considering other examples in the entire training set . As a result , the representation of an augmented sample may be closer to those of other training instances , rather than the one it is derived from . Based on this observation , we advocate that , in addition to consistency training , a training objective that can globally capture the intrinsic relationship within the entire set of original and augmented training instances can help leverage augmented examples more effectively . In this paper , we introduce a novel Contrast-enhanced and Diversity-promoting Data Augmentation ( CoDA ) framework for natural language understanding . To improve the diversity of augmented samples , we extensively explore different combinations of isolated label-preserving transformations in an unified approach . We find that stacking distinct label-preserving transformations produces particularly informative samples . Specifically , the most diverse and high-quality augmented samples are obtained by stacking an adversarial training module over the back-translation transformation . Besides the consistency-regularized loss for repelling the model to behave consistently within local neighborhoods , we propose a contrastive learning objective to capture the global relationship among the data points in the representation space . We evaluate CoDA on the GLUE benchmark ( with RoBERTa ( Liu et al. , 2019 ) as the testbed ) , and CoDA consistently improves the generalization ability of resulting models and gives rise to significant gains relative to the standard fine-tuning procedure . Moreover , our method also outperforms various single data augmentation operations , combination schemes , and other strong baselines . Additional experiments in the low-resource settings and ablation studies further demonstrate the effectiveness of this framework . 2 METHOD . In this section , we focus our discussion on the natural language understanding ( NLU ) tasks , and particularly , under a text classification scenario . However , the proposed data augmentation framework can be readily extended to other NLP tasks as well . 2.1 BACKGROUND : DATA AUGMENTATION AND ADVERSARIAL TRAINING . Data Augmentation Let D = { xi , yi } i=1 ... N denote the training dataset , where the input example xi is a sequence of tokens , and yi is the corresponding label . To improve model ’ s robustness and generalization ability , several data augmentation techniques ( e.g. , back-translation ( Sennrich et al. , 2016 ; Edunov et al. , 2018 ; Xie et al. , 2019 ) , mixup ( Guo et al. , 2019 ) , c-BERT ( Wu et al. , 2019 ) ) have been proposed . Concretely , label-preserving transformations are performed ( on the original training sequences ) to synthesize a collection of augmented samples , denoted by D′ = { x′i , y′i } i=1 ... N . Thus , a model can learn from both the training set D and the augmented set D′ , with pθ ( · ) the predicted output distribution of the model parameterized by θ : θ∗ = arg min θ ∑ ( xi , yi ) ∈D L ( pθ ( xi ) , yi ) + ∑ ( x′i , y ′ i ) ∈D′ L ( pθ ( x ′ i ) , y ′ i ) ( 1 ) Several recent research efforts were focused on encouraging model predictions to be invariant to stochastic or domain-specific data transformations ( Xie et al. , 2019 ; Laine & Aila , 2017 ; Tarvainen & Valpola , 2017 ; Sohn et al. , 2020 ; Miyato et al. , 2018 ; Jiang et al. , 2020 ; Hendrycks et al. , 2020 ) . Take back-translation as example : x′i = BackTrans ( xi ) , then x ′ i is a paraphrase of xi . The model can be regularized to have consistent predictions for ( xi , x′i ) , by minimizing the distribution discrepancyRCS ( pθ ( xi ) , pθ ( x′i ) ) , which typically adopts KL divergence ( see Fig . 1a ) . Adversarial Training In another line , adversarial training methods are applied to text data ( Zhu et al. , 2020 ; Jiang et al. , 2020 ; Cheng et al. , 2020 ; Aghajanyan et al. , 2020 ) for improving model ’ s robustness . Compared with data augmentation techniques , adversarial training requires no domain knowledge to generate additional training examples . Instead , it relies on the model itself to produce adversarial examples which the model are most likely to make incorrect predictions . Similar to data augmentation , adversarial training also typically utilizes the cross-entropy and consistencybased objectives for training . As the two most popular adversarial-training-based algorithms , the adversarial loss ( Goodfellow et al. , 2015 ) ( Eqn . 2 ) and virtual adversarial loss ( Miyato et al. , 2018 ) ( Eqn . 3 ) can be expressed as follows ( see Fig . 1b ) : RAT ( xi , x̃i , yi ) = L ( pθ ( x̃i ) , yi ) , s.t. , ‖x̃i − xi‖ ≤ , ( 2 ) RVAT ( xi , x̃i ) = RCS ( pθ ( x̃i ) , pθ ( xi ) ) , s.t. , ‖x̃i − xi‖ ≤ . ( 3 ) Generally , there is no closed-form to obtain the exact adversarial example x̂i in either Eqn . 2 or 3 . However , it usually can be approximated by a low-order approximation of the objective function with respect to xi . For example , the adversarial example in Eqn . 2 can be approximated by : x̂i ≈ xi + g ‖g‖2 , where g = ∇xiL ( pθ ( xi ) , yi ) . ( 4 ) 2.2 DIVERSITY-PROMOTING CONSISTENCY TRAINING . As discussed in the previous section , data augmentation and adversarial training share the same intuition of producing neighbors around the original training instances . Moreover , both approaches share very similar training objectives . Therefore , it is natural to ask the following question : are different data augmentation methods and adversarial training equal in nature ? Otherwise , are they complementary to each other , and thus can be consolidated together to further improve the model ’ s generalization ability ? Notably , it has been shown , in the CV domain , that combining different data augmentation operations could lead to more diverse augmented examples ( Cubuk et al. , 2018 ; 2020 ; Hendrycks et al. , 2020 ) . However , this is especially challenging for natural language , given that the semantics of a sentence can be entirely altered by slight perturbations . To answer the above question , we propose several distinct strategies to combine different data transformations , with the hope to produce more diverse and informative augmented examples . Specifically , we consider 5 different types of label-preserving transformations : back-translation ( Sennrich et al. , 2016 ; Edunov et al. , 2018 ; Xie et al. , 2019 ) , c-BERT word replacement ( Wu et al. , 2019 ) , mixup ( Guo et al. , 2019 ; Chen et al. , 2020a ) , cutoff ( Shen et al. , 2020 ) , and adversarial training ( Zhu et al. , 2020 ; Jiang et al. , 2020 ) . The 3 combination strategies are schematically illustrated in Figure 2 . For random combination , a particular label-preserving transformation is randomly selected , among all the augmentation operations available , for each mini-batch . As to the mixup interpolation , given two samples xi and xj drawn in a mini-batch , linear interpolation is performed between their input embedding matrices ei and ej ( Zhang et al. , 2017 ) : e′i = aei+ ( 1−a ) ej , where a is the interpolation parameter , usually drawn from a Beta distribution . Moreover , we consider stacking different label-preserving transformations in a sequential manner ( see Figure 2c ) . It is worth noting that due to the discrete nature of text data , some stacking orders are infeasible . For example , it is not reasonable to provide an adversarially-perturbed embedding sequence to the back-translation module . Without loss of generality , we choose the combination where adversarial training is stacked over back-translation to demonstrate the sequential stacking operation ( see Fig . 1c ) . Formally , given a training example ( xi , yi ) , the consistency training objective for such a stacking operation can be written as : x′i = BackTrans ( xi ) , x̂i ≈ argmaxx̃iRAT ( x ′ i , x̃i , yi ) , ( 5 ) Lconsistency ( xi , x̂i , yi ) = L ( pθ ( xi ) , yi ) +αL ( pθ ( x̂i ) , yi ) + βRCS ( pθ ( xi ) , pθ ( x̂i ) ) , ( 6 ) where the first term corresponds to the cross-entropy loss , the second term is the adversarial loss , RCS denotes the consistency loss term between ( xi , x̂i ) . Note that x̂i is obtained through two different label-preserving transformations applied to x , and thus deviates farther from x and should be more diverse than x′i . Inspired by ( Bachman et al. , 2014 ; Zheng et al. , 2016 ; Kannan et al. , 2018 ; Hendrycks et al. , 2020 ) , we employ the Jensen-Shannon divergence forRCS , since it is upper bounded and tends to be more stable and consistent relative to the KL divergence : RCS ( pθ ( xi ) , pθ ( x̂i ) ) = 1 2 ( KL ( pθ ( xi ) ‖M ) + KL ( pθ ( x̂i ) ) ‖M ) ) , ( 7 ) where M = ( pθ ( xi ) + pθ ( x̂i ) ) /2 . Later we simply use x′i to represent the transformed example . 2.3 CONTRASTIVE REGULARIZATION Consistency loss only provides local regularization , i.e. , xi and x′i should have close predictions . However , the relative positions between x′i and other training instances xj ( j 6= i ) have not been examined . In this regard , we propose to leverage a contrastive learning objective to better utilize the augmented examples . Specifically , we assume that the model should encourage an augmented sample x′i to be closer , in the representation space , to its original sample xi , relative to other data points xj ( j 6= i ) in the training set . This is a reasonable assumption since intuitively , the model should be robust enough to successfully determine from which original data an augmented sample is produced . The contrastive learning module is illustrated in Fig . 3 . As demonstrated by prior efforts on contrastive learning , adopting a large batch size is especially vital for its effectiveness ( Chen et al. , 2020b ; Khosla et al. , 2020 ) . Therefore , we introduce a memory bank that stores the history embeddings , thus enabling much larger number of negative samples . Moreover , to avoid the encoder from changing too rapidly ( which may result in inconsistency embeddings ) , a momentum encoder module is incorporated into our algorithm . Concretely , let fθ ( . ) and fθ̄ ( . ) denote the transformation parameterized by the query encoder and key encoder , respectively . Note that θ and θ̄ represent their parameters . The momentum model parameters θ̄ are not learned by gradients . Instead , they are updated through the momentum rule : θ̄ ← γθ̄ + ( 1− γ ) θ at each training step . We omit the details here and refer the interested readers to the work by ( He et al. , 2020 ) for further explanation . Given a sample xi and its augmented example x′i , the query and key can be obtained as follows : qi = fθ ( xi ) , q ′ i = fθ ( x ′ i ) , ki = fθ̄ ( xi ) . ( 8 ) Thus , the contrastive training objective can be written as : Rcontrast ( xi , x′i , M ) = RCT ( qi , ki , M ) +RCT ( q′i , ki , M ) , ( 9 ) RCT ( qi , ki , M ) = −log exp ( sim ( qi , ki ) /τ ) ∑ kj∈M ⋃ { ki } exp ( sim ( qi , kj ) /τ ) , ( 10 ) where τ is the temperature , andM is the memory bank in which the history keys are stored . Cosine similarity is chosen for sim ( · ) . Note that RCT ( q′i , ki , M ) is similarly defined as RCT ( qi , ki , M ) ( with qi replaced by q′i in Eqn . 10 ) . In Eqn . 9 , the first term corresponds to the contrastive loss calculated on the original examples ( self-contrastive loss ) , while the second term is computed on the augmented sample ( augment-contrastive loss ) . Under such a framework , the pair of original and augmented samples are encouraged to stay closer in the learned embedding space , relative to all other training instances . As a result , the model is regularized globally through considering the embeddings of all the training examples available . By integrating both the consistency training objective and the contrastive regularization , the overall training objective for the CoDA framework can be expressed as : θ∗ = argminθ ∑ ( xi , yi ) ∈D Lconsistency ( xi , x′i , yi ) + λRcontrast ( xi , x ′ i , M ) . ( 11 ) where λ is a hyperparameter to be chosen . It is worth noting that the final objective has taken both the local ( consistency loss ) and global ( contrastive loss ) information introduced by the augmented examples into consideration .

Summary: Paper proposes a contrastive learning-based approach to combine different data augmentation techniques for NLP tasks. While the widely used consistency loss focuses on a single example, the proposed contrastive objective allows capturing the relationships among all data samples which helps in producing diverse and informative examples. For experiments, the paper explores 5 data augmentation approaches with Roberta-large as the classification model. Empirical results on the standard GLUE benchmark leads to an impressive 2.2% average improvement. Authors also found that back-translation and adversarial training combination leads to better performance than other DA combinations.