Contrastive  Learning  with Adversarial Perturbations for Conditional Text Generation

1 INTRODUCTION . The sequence-to-sequence ( seq2seq ) models ( Sutskever et al. , 2014 ) , which learn to map an arbitrary-length input sequence to another arbitrary-length output sequence , have successfully tackled a wide range of language generation tasks . Early seq2seq models have used recurrent neural networks to encode and decode sequences , leveraging attention mechanism ( Bahdanau et al. , 2015 ) that allows the decoder to attend to a specific token in the input sequence to capture long-term dependencies between the source and target sequences . Recently , the Transformer ( Vaswani et al. , 2017 ) , which is an all-attention model that effectively captures long-term relationships between tokens in the input sequence as well as across input and output sequences , has become the de facto standard for most of the text generation tasks due to its impressive performance . Moreover , Transformerbased language models trained on large text corpora ( Dong et al. , 2019 ; Raffel et al. , 2020 ; Lewis et al. , 2020 ) have shown to significantly improve the model performance on text generation tasks . However , a crucial limitation of seq2seq models is that they are mostly trained only with teacher forcing , where ground truth is provided at each time step and thus never exposed to incorrectly generated tokens during training ( Fig . 1- ( a ) ) , which hurts its generalization . This problem is known as the “ exposure bias ” problem ( Ranzato et al. , 2016 ) and often results in the generation of lowquality texts on unseen inputs . Several prior works tackle the problem , such as using reinforcement learning ( RL ) to maximize non-differentiable reward ( Bahdanau et al. , 2017 ; Paulus et al. , 2018 ) . ∗Equal contribution Another approach is to use RL or gumbel softmax ( Jang et al. , 2017 ) to match the distribution of generated sentences to that of the ground truth , in which case the reward is the discriminator output from a Generative Adversarial Network ( GAN ) ( Zhang et al. , 2018 ; 2017 ; Yu et al. , 2017 ) . Although the aforementioned approaches improve the performance of the seq2seq models on text generation tasks , they either require a vast amount of effort in tuning hyperparameters or stabilize training . In this work , we propose to mitigate the exposure bias problem with a simple yet effective approach , in which we contrast a positive pair of input and output sequence to negative pairs , to expose the model to various valid or incorrect sentences . Naı̈vely , we can construct negative pairs by simply using random nontarget sequences from the batch ( Chen et al. , 2020 ) . However , such a naı̈ve construction yields meaningless negative examples that are already well-discriminated in the embedding space ( Fig . 1- ( b ) ) , which we highlight as the reason why existing methods ( Chen et al. , 2020 ) require large batch size . This is clearly shown in Fig . 2 , where a large portion of positive-negative pairs can be easily discriminated without any training , which gets worse as the batch size decreases as it will reduce the chance to have meaningfully difficult examples in the batch . Moreover , discriminating positive and naı̈ve negative pairs becomes even more easier for models pretrained on large text corpora . To resolve this issue , we propose principled approaches to automatically generate negative and positive pairs for constrastive learning , which we refer to as Contrastive Learning with Adversarial Perturbation for Seq2seq learning ( CLAPS ) . Specifically , we generate a negative example by adding a small perturbation to the hidden representation of the target sequence , such that its conditional likelihood is minimized ( Denoted as the red circle in Fig . 1- ( c ) ) . Conversely , we construct an additional positive example ( Denoted as green circle in Fig . 1- ( c ) ) by adding a large amount of perturbation to the hidden representation of target sequence such that the perturbed sample is far away from the source sequence in the embedding space , while enforcing it to have high conditional likelihood by minimizing Kullback-Leibler ( KL ) divergence between the original conditional distribution and perturbed conditional distribution . This will yield a negative example that is very close to the original representation of target sequence in the embedding space but is largely dissimilar in the semantics , while the generated positive example is far away from the original input sequence but has the same semantic as the target sequence . This will generate difficult examples that the model fails to correctly discriminate ( Fig . 1- ( c ) , Fig.2 ) , helping it learn with more meaningful pairs . To verify the efficacy of our method , we empirically show that it significantly improves the performance of seq2seq model on three conditional text generation tasks , namely machine translation , text summarization and question generation . Our contribution in this work is threefold : • To mitigate the exposure bias problem , we propose a contrastive learning framework for conditional sequence generation , which contrasts a positive pair of source and target sentence to negative pairs in the latent embedding space , to expose the model to various valid or incorrect outputs . • To tackle the ineffectiveness of conventional approach for constructing negative and positive examples for contrastive learning , we propose a principled method to automatically generate negative and positive pairs , that are more difficult and allows to learn more meaningful representations . • We show that our proposed method , CLAPS , significantly improves the performance of seq2seq model on three different tasks : machine translation , text summarization , and question generation . 2 RELATED WORK . Exposure Bias There are several prior works to tackle the exposure bias ( Ranzato et al. , 2016 ) . Bengio et al . ( 2015 ) introduce scheduled sampling where the model is initially guided with the true previous tokens but uses the tokens generated by the seq2seq model as the conditional input for the next token , as training goes on . Paulus et al . ( 2018 ) ; Bahdanau et al . ( 2017 ) leverage RL to maximize non-differentiable rewards , so it enables to penalize the model for incorrectly generated sentences . Another works ( Zhang et al. , 2017 ; 2018 ; Yu et al. , 2017 ) train GANs to match the distribution of generated sequences to that of ground truth . Since sampling tokens from the generator is not differentiable , they resort RL or gumbel-softmax to train the networks in end-to-end fashion . However , they require either a large amount of effort to tune hyperparameters or stabilize training . However , Choshen et al . ( 2020 ) show that RL for machine translation does not optimize the expected reward and the performance gain is attributed to the unrelated effects such as increasing the peakiness of the output distribution . Moreover , ( Caccia et al. , 2019 ) show that by tuning the temperature parameter , the language models trained with MLE can be tuned to outperform GAN-based text generation models . Adversarial Perturbation Many existing works , such as ( Madry et al. , 2018 ) , address the robustness of neural networks to adversarial examples , which are generated by applying a small perturbations to the input samples . While adversarial robustness has been mostly explored in image domains , Miyato et al . ( 2017 ) adopted adversarial training to text domains . However instead of targeting robustness to perturbed samples , they utilize the adversarial examples as augmented data , and enforce consistency across the predictions across original unlabeled example and its perturbation , for semisupervised learning . Recently Zhu et al . ( 2019 ) and Jiang et al . ( 2020 ) leverage adversarial training to induce the smoothness of text classifiers , to prevent overfitting to training samples . While they are relevant to ours , these methods do not have the notion of positive and negative examples as they do not consider contrastive learning , and only target text classification . Moreover , they are computationally prohibitive since they use PGD for adversarial training , which requires iterative optimization for each individual sample . Recently , Aghajanyan et al . ( 2020 ) propose a simpler yet effective method based on Gaussian noise perturbation to regularize neural networks without expensive PGD steps , which is shown to outperform methods from Zhu et al . ( 2019 ) and Jiang et al . ( 2020 ) . Although our work is similar to these prior works in that we add perturbations to the text embeddings , note that we used the adversarially-generated samples as negative examples of our contrastive learning framework rather than trying to learn the model to be robust to them . Contrastive Learning Contrastive learning has been widely used . It is to learn a representation by contrasting positive pairs and negative pairs . Chopra et al . ( 2005 ) ; Weinberger & Saul ( 2009 ) ; Schroff et al . ( 2015 ) leverage a triplet loss to separate positive examples from negative examples in metric learning . Chen et al . ( 2020 ) show that contrastive learning can boost the performance of selfsupervised and semi-supervised learning in computer vison tasks . In natural language processing ( NLP ) , contrastive learning has been widely used . In Word2Vec ( Mikolov et al. , 2013 ) , neighbouring words are predicted from context with noise-contrastive estimation ( Gutmann & Hyvärinen , 2012 ) . Beyond word representation , Logeswaran & Lee ( 2018 ) sample two contiguous sentences for positive pairs and the sentences from other document as negative pairs . They constrast positive and negative pairs to learn sentence representation . Moreover , contrastive learning has been investigated in various NLP tasks — language modeling ( Huang et al. , 2018 ) , unsupervised word alignment ( Liu & Sun , 2015 ) , caption generation ( Mao et al. , 2016 ; Vedantam et al. , 2017 ) , and machine translation ( Yang et al. , 2019 ) . 3 METHOD . 3.1 BACKGROUND : CONDITIONAL TEXT GENERATION . The goal of conditional text generation with a seq2seq model is to generate an output text sequence y ( i ) = ( y ( i ) 1 , . . . , y ( i ) T ) with length T conditioned on the input text sequence x ( i ) = ( x ( i ) 1 , . . . , x ( i ) L ) with length L. A typical approach to the conditional text generation is to leverage the encoderdecoder architecture to parameterize the conditional distribution . We maximize the conditional log likelihood log pθ ( y|x ) for a given N observations { ( x ( i ) , y ( i ) ) } Ni=1 as follows : LMLE ( θ ) = N∑ i=1 log pθ ( y ( i ) |x ( i ) ) pθ ( y ( i ) 1 , . . . , y ( i ) T |x ( i ) ) = T∏ t=1 pθ ( y ( i ) t |y ( i ) < t , x ( i ) ) pθ ( y ( i ) t |y ( i ) < t , x ( i ) ) = softmax ( Wh ( i ) t + b ) h ( i ) t = g ( y ( i ) t−1 , M ( i ) ; θ ) , M ( i ) = f ( x ( i ) ; θ ) ( 1 ) where f , g denote the encoder and the decoder respectively and M ( i ) = [ m ( i ) 1 · · ·m ( i ) L ] ∈ Rd×L is the concatenation of the hidden representations of the source tokens x ( i ) .

Summary: This paper proposes to add contrastive learning to the sequence-to-sequence generation problem. More specifically, the authors apply a contrastive loss on the globally pooled hidden representation of the generated hidden states. The key novelty is to apply adversarial gradients to obtain both hard negative and hard positive examples. The proposed method can improve a state-of-art pretrained transformer model (T5) on 3 tasks: machine translation (WMT16 En-Ro), abstractive summarization (XSum), and question generation (SQuAD).