CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding

1 INTRODUCTION . Data augmentation approaches have successfully improved large-scale neural-network-based models , ( Laine & Aila , 2017 ; Xie et al. , 2019 ; Berthelot et al. , 2019 ; Sohn et al. , 2020 ; He et al. , 2020 ; Khosla et al. , 2020 ; Chen et al. , 2020b ) , however , the majority of existing research is geared towards computer vision tasks . The discrete nature of natural language makes it challenging to design effective label-preserving transformations for text sequences that can help improve model generalization ( Hu et al. , 2019 ; Xie et al. , 2019 ) . On the other hand , fine-tuning powerful , over-parameterized language models1 proves to be difficult , especially when there is a limited amount of task-specific data available . It may result in representation collapse ( Aghajanyan et al. , 2020 ) or require special finetuning techniques ( Sun et al. , 2019 ; Hao et al. , 2019 ) . In this work , we aim to take a further step towards finding effective data augmentation strategies through systematic investigation . In essence , data augmentation can be regarded as constructing neighborhoods around a training instance that preserve the ground-truth label . With such a characterization , adversarial training ( Zhu et al. , 2020 ; Jiang et al. , 2020 ; Liu et al. , 2020 ; Cheng et al. , 2020 ) also performs label-preserving transformation in embedding space , and thus is considered as an alternative to data augmentation methods in this work . From this perspective , the goal of developing effective data augmentation strategies can be summarized as answering three fundamental questions : i ) What are some label-preserving transformations , that can be applied to text , to compose useful augmented samples ? ∗Work was done during an internship at Microsoft Dynamics 365 AI . 1To name a few , BERT ( Devlin et al. , 2019 ) : 340M parameters , T5 ( Raffel et al. , 2019 ) : 11B parameters , GPT-3 ( Brown et al. , 2020 ) : 175B parameters . ii ) Are these transformations complementary in nature , and can we find some strategies to consolidate them for producing more diverse augmented examples ? iii ) How can we incorporate the obtained augmented samples into the training process in an effective and principled manner ? Previous efforts in augmenting text data were mainly focused on answering the first question ( Yu et al. , 2018 ; Xie et al. , 2019 ; Kumar et al. , 2019 ; Wei & Zou , 2019 ; Chen et al. , 2020a ; Shen et al. , 2020 ) . Regarding the second question , different label-preserving transformations have been proposed , but it remains unclear how to integrate them organically . In addition , it has been shown that the diversity of augmented samples plays a vital role in their effectiveness ( Xie et al. , 2019 ; Gontijo-Lopes et al. , 2020 ) . In the case of image data , several strategies that combine different augmentation methods have been proposed , such as applying multiple transformations sequentially ( Cubuk et al. , 2018 ; 2020 ; Hendrycks et al. , 2020 ) , learning data augmentation policies ( Cubuk et al. , 2018 ) , randomly sampling operations for each data point ( Cubuk et al. , 2020 ) . However , these methods can not be naively applied to text data , since the semantic meanings of a sentence are much more sensitive to local perturbations ( relative to an image ) . As for the third question , consistency training is typically employed to utilize the augmented samples ( Laine & Aila , 2017 ; Hendrycks et al. , 2020 ; Xie et al. , 2019 ; Sohn et al. , 2020 ; Miyato et al. , 2018 ) . This method encourages the model predictions to be invariant to certain label-preserving transformations . However , existing approaches only examine a pair of original and augmented samples in isolation , without considering other examples in the entire training set . As a result , the representation of an augmented sample may be closer to those of other training instances , rather than the one it is derived from . Based on this observation , we advocate that , in addition to consistency training , a training objective that can globally capture the intrinsic relationship within the entire set of original and augmented training instances can help leverage augmented examples more effectively . In this paper , we introduce a novel Contrast-enhanced and Diversity-promoting Data Augmentation ( CoDA ) framework for natural language understanding . To improve the diversity of augmented samples , we extensively explore different combinations of isolated label-preserving transformations in an unified approach . We find that stacking distinct label-preserving transformations produces particularly informative samples . Specifically , the most diverse and high-quality augmented samples are obtained by stacking an adversarial training module over the back-translation transformation . Besides the consistency-regularized loss for repelling the model to behave consistently within local neighborhoods , we propose a contrastive learning objective to capture the global relationship among the data points in the representation space . We evaluate CoDA on the GLUE benchmark ( with RoBERTa ( Liu et al. , 2019 ) as the testbed ) , and CoDA consistently improves the generalization ability of resulting models and gives rise to significant gains relative to the standard fine-tuning procedure . Moreover , our method also outperforms various single data augmentation operations , combination schemes , and other strong baselines . Additional experiments in the low-resource settings and ablation studies further demonstrate the effectiveness of this framework . 2 METHOD . In this section , we focus our discussion on the natural language understanding ( NLU ) tasks , and particularly , under a text classification scenario . However , the proposed data augmentation framework can be readily extended to other NLP tasks as well . 2.1 BACKGROUND : DATA AUGMENTATION AND ADVERSARIAL TRAINING . Data Augmentation Let D = { xi , yi } i=1 ... N denote the training dataset , where the input example xi is a sequence of tokens , and yi is the corresponding label . To improve model ’ s robustness and generalization ability , several data augmentation techniques ( e.g. , back-translation ( Sennrich et al. , 2016 ; Edunov et al. , 2018 ; Xie et al. , 2019 ) , mixup ( Guo et al. , 2019 ) , c-BERT ( Wu et al. , 2019 ) ) have been proposed . Concretely , label-preserving transformations are performed ( on the original training sequences ) to synthesize a collection of augmented samples , denoted by D′ = { x′i , y′i } i=1 ... N . Thus , a model can learn from both the training set D and the augmented set D′ , with pθ ( · ) the predicted output distribution of the model parameterized by θ : θ∗ = arg min θ ∑ ( xi , yi ) ∈D L ( pθ ( xi ) , yi ) + ∑ ( x′i , y ′ i ) ∈D′ L ( pθ ( x ′ i ) , y ′ i ) ( 1 ) Several recent research efforts were focused on encouraging model predictions to be invariant to stochastic or domain-specific data transformations ( Xie et al. , 2019 ; Laine & Aila , 2017 ; Tarvainen & Valpola , 2017 ; Sohn et al. , 2020 ; Miyato et al. , 2018 ; Jiang et al. , 2020 ; Hendrycks et al. , 2020 ) . Take back-translation as example : x′i = BackTrans ( xi ) , then x ′ i is a paraphrase of xi . The model can be regularized to have consistent predictions for ( xi , x′i ) , by minimizing the distribution discrepancyRCS ( pθ ( xi ) , pθ ( x′i ) ) , which typically adopts KL divergence ( see Fig . 1a ) . Adversarial Training In another line , adversarial training methods are applied to text data ( Zhu et al. , 2020 ; Jiang et al. , 2020 ; Cheng et al. , 2020 ; Aghajanyan et al. , 2020 ) for improving model ’ s robustness . Compared with data augmentation techniques , adversarial training requires no domain knowledge to generate additional training examples . Instead , it relies on the model itself to produce adversarial examples which the model are most likely to make incorrect predictions . Similar to data augmentation , adversarial training also typically utilizes the cross-entropy and consistencybased objectives for training . As the two most popular adversarial-training-based algorithms , the adversarial loss ( Goodfellow et al. , 2015 ) ( Eqn . 2 ) and virtual adversarial loss ( Miyato et al. , 2018 ) ( Eqn . 3 ) can be expressed as follows ( see Fig . 1b ) : RAT ( xi , x̃i , yi ) = L ( pθ ( x̃i ) , yi ) , s.t. , ‖x̃i − xi‖ ≤ , ( 2 ) RVAT ( xi , x̃i ) = RCS ( pθ ( x̃i ) , pθ ( xi ) ) , s.t. , ‖x̃i − xi‖ ≤ . ( 3 ) Generally , there is no closed-form to obtain the exact adversarial example x̂i in either Eqn . 2 or 3 . However , it usually can be approximated by a low-order approximation of the objective function with respect to xi . For example , the adversarial example in Eqn . 2 can be approximated by : x̂i ≈ xi + g ‖g‖2 , where g = ∇xiL ( pθ ( xi ) , yi ) . ( 4 ) 2.2 DIVERSITY-PROMOTING CONSISTENCY TRAINING . As discussed in the previous section , data augmentation and adversarial training share the same intuition of producing neighbors around the original training instances . Moreover , both approaches share very similar training objectives . Therefore , it is natural to ask the following question : are different data augmentation methods and adversarial training equal in nature ? Otherwise , are they complementary to each other , and thus can be consolidated together to further improve the model ’ s generalization ability ? Notably , it has been shown , in the CV domain , that combining different data augmentation operations could lead to more diverse augmented examples ( Cubuk et al. , 2018 ; 2020 ; Hendrycks et al. , 2020 ) . However , this is especially challenging for natural language , given that the semantics of a sentence can be entirely altered by slight perturbations . To answer the above question , we propose several distinct strategies to combine different data transformations , with the hope to produce more diverse and informative augmented examples . Specifically , we consider 5 different types of label-preserving transformations : back-translation ( Sennrich et al. , 2016 ; Edunov et al. , 2018 ; Xie et al. , 2019 ) , c-BERT word replacement ( Wu et al. , 2019 ) , mixup ( Guo et al. , 2019 ; Chen et al. , 2020a ) , cutoff ( Shen et al. , 2020 ) , and adversarial training ( Zhu et al. , 2020 ; Jiang et al. , 2020 ) . The 3 combination strategies are schematically illustrated in Figure 2 . For random combination , a particular label-preserving transformation is randomly selected , among all the augmentation operations available , for each mini-batch . As to the mixup interpolation , given two samples xi and xj drawn in a mini-batch , linear interpolation is performed between their input embedding matrices ei and ej ( Zhang et al. , 2017 ) : e′i = aei+ ( 1−a ) ej , where a is the interpolation parameter , usually drawn from a Beta distribution . Moreover , we consider stacking different label-preserving transformations in a sequential manner ( see Figure 2c ) . It is worth noting that due to the discrete nature of text data , some stacking orders are infeasible . For example , it is not reasonable to provide an adversarially-perturbed embedding sequence to the back-translation module . Without loss of generality , we choose the combination where adversarial training is stacked over back-translation to demonstrate the sequential stacking operation ( see Fig . 1c ) . Formally , given a training example ( xi , yi ) , the consistency training objective for such a stacking operation can be written as : x′i = BackTrans ( xi ) , x̂i ≈ argmaxx̃iRAT ( x ′ i , x̃i , yi ) , ( 5 ) Lconsistency ( xi , x̂i , yi ) = L ( pθ ( xi ) , yi ) +αL ( pθ ( x̂i ) , yi ) + βRCS ( pθ ( xi ) , pθ ( x̂i ) ) , ( 6 ) where the first term corresponds to the cross-entropy loss , the second term is the adversarial loss , RCS denotes the consistency loss term between ( xi , x̂i ) . Note that x̂i is obtained through two different label-preserving transformations applied to x , and thus deviates farther from x and should be more diverse than x′i . Inspired by ( Bachman et al. , 2014 ; Zheng et al. , 2016 ; Kannan et al. , 2018 ; Hendrycks et al. , 2020 ) , we employ the Jensen-Shannon divergence forRCS , since it is upper bounded and tends to be more stable and consistent relative to the KL divergence : RCS ( pθ ( xi ) , pθ ( x̂i ) ) = 1 2 ( KL ( pθ ( xi ) ‖M ) + KL ( pθ ( x̂i ) ) ‖M ) ) , ( 7 ) where M = ( pθ ( xi ) + pθ ( x̂i ) ) /2 . Later we simply use x′i to represent the transformed example . 2.3 CONTRASTIVE REGULARIZATION Consistency loss only provides local regularization , i.e. , xi and x′i should have close predictions . However , the relative positions between x′i and other training instances xj ( j 6= i ) have not been examined . In this regard , we propose to leverage a contrastive learning objective to better utilize the augmented examples . Specifically , we assume that the model should encourage an augmented sample x′i to be closer , in the representation space , to its original sample xi , relative to other data points xj ( j 6= i ) in the training set . This is a reasonable assumption since intuitively , the model should be robust enough to successfully determine from which original data an augmented sample is produced . The contrastive learning module is illustrated in Fig . 3 . As demonstrated by prior efforts on contrastive learning , adopting a large batch size is especially vital for its effectiveness ( Chen et al. , 2020b ; Khosla et al. , 2020 ) . Therefore , we introduce a memory bank that stores the history embeddings , thus enabling much larger number of negative samples . Moreover , to avoid the encoder from changing too rapidly ( which may result in inconsistency embeddings ) , a momentum encoder module is incorporated into our algorithm . Concretely , let fθ ( . ) and fθ̄ ( . ) denote the transformation parameterized by the query encoder and key encoder , respectively . Note that θ and θ̄ represent their parameters . The momentum model parameters θ̄ are not learned by gradients . Instead , they are updated through the momentum rule : θ̄ ← γθ̄ + ( 1− γ ) θ at each training step . We omit the details here and refer the interested readers to the work by ( He et al. , 2020 ) for further explanation . Given a sample xi and its augmented example x′i , the query and key can be obtained as follows : qi = fθ ( xi ) , q ′ i = fθ ( x ′ i ) , ki = fθ̄ ( xi ) . ( 8 ) Thus , the contrastive training objective can be written as : Rcontrast ( xi , x′i , M ) = RCT ( qi , ki , M ) +RCT ( q′i , ki , M ) , ( 9 ) RCT ( qi , ki , M ) = −log exp ( sim ( qi , ki ) /τ ) ∑ kj∈M ⋃ { ki } exp ( sim ( qi , kj ) /τ ) , ( 10 ) where τ is the temperature , andM is the memory bank in which the history keys are stored . Cosine similarity is chosen for sim ( · ) . Note that RCT ( q′i , ki , M ) is similarly defined as RCT ( qi , ki , M ) ( with qi replaced by q′i in Eqn . 10 ) . In Eqn . 9 , the first term corresponds to the contrastive loss calculated on the original examples ( self-contrastive loss ) , while the second term is computed on the augmented sample ( augment-contrastive loss ) . Under such a framework , the pair of original and augmented samples are encouraged to stay closer in the learned embedding space , relative to all other training instances . As a result , the model is regularized globally through considering the embeddings of all the training examples available . By integrating both the consistency training objective and the contrastive regularization , the overall training objective for the CoDA framework can be expressed as : θ∗ = argminθ ∑ ( xi , yi ) ∈D Lconsistency ( xi , x′i , yi ) + λRcontrast ( xi , x ′ i , M ) . ( 11 ) where λ is a hyperparameter to be chosen . It is worth noting that the final objective has taken both the local ( consistency loss ) and global ( contrastive loss ) information introduced by the augmented examples into consideration .

Summary: The augmentation of NLP samples is an important task with no clear "applicable to all" mechanism. This is in sharp contrast to computer vision where techniques like rotation, modification of hue, saturation as well as umpteen other techniques exist. This work tries to address the issue by proposing a technique that carefully amalgamates multiple previously known approaches to generate diverse label preserving examples. The experimental results on RoBERTa highlight the applicability and importance of this data augmentation approach on the downstream task of text classification (GLUE).