Encoder-Agnostic Adaptation for Conditional Language Generation

1 INTRODUCTION . Large-scale language models have been shown to dramatically improve the performance of natural language understanding ( NLU ) systems on a broad range of tasks ( Peters et al. , 2018 ; Devlin et al. , 2018 ; Radford & Salimans , 2018 ; McCann et al. , 2017 ) . The dominant paradigm is to pretrain a self attention-based language model on a large corpus of unlabeled text and then finetune the language model with an additional task-specific classification head on supervised data . Optimizing the effectiveness of this approach has been the focus of much study ( Houlsby et al. , 2019 ; Wang et al. , 2019 ; Chronopoulou et al. , 2019 ) . Given the success of pretraining for NLU tasks , how can large language models best be adapted for conditional language generation ? Ideally , one should only need to train a large language model once and then apply it as part of the decoder to a range of tasks with different source modalities ( e.g. , text , images , bits ) . In the encoder/decoder framework , a task-specific encoder can encode source information into a continuous vector . The central question is thus how to adapt a pretrained decoder to effectively utilize arbitrary source information from an encoder ( encoder-agnostic adaptation ) . Considering the high quality of samples from large languae models ( Radford et al. , 2019 ) , it is natural to expect encoder-agnostic adaptation to improve the coherence and grammaticality of conditional text generation over training a decoder from scratch , even when the source modality is not text ( e.g . image captioning or class-conditional generation ) . Unfortunately , past results indicate otherwise . Edunov et al . ( 2019 ) show , for example , that a straightforward extension of contextual representations ( Peters et al. , 2018 ) to the conditional generation setting actually hurts performance compared to a model without any pretraining . Other pretraining approaches for language generation ( Song et al. , 2019 ; Dong et al. , 2019 ; Lample & Conneau , 2019 ) have demonstrated strong performance on text-to-text tasks , but these methods are constrained to tasks where the source is natural language and do not address the encoder-agnostic setting . In this work , we consider several different approaches to the problem of encoder-agnostic adaptation . We first observe that standard adaptation approaches perform poorly on this task . We hypothesize that because these techniques require relearning significant parts of the network structure to inject contextual conditioning , they move the parameters too far from the pretrained values . In contrast , Radford et al . ( 2019 ) observe that even trivial conditioning with the original model produces reasonable zero-shot generations without finetuning . These results motivate an approach that learns the correct conditioning to control the model ’ s output , which we call pseudo self attention . The idea is to learn a task-specific encoder that injects pseudo history into a pretrained self attention model . Because self attention works with sets of any size , the model can immediately utilize or ignore this history . Finetuning adapts the model to this new input while training a task-specific encoder . Experiments utilize the GPT-2 ( Radford et al. , 2019 ) transformer as a pretrained model . We consider four diverse generation tasks spanning a range of source modalities : class-conditional generation , document summarization , story generation , and image paragraph captioning . Across all tasks , we find that pseudo self attention outperforms the other pretraining methods and is the most consistent . As a practical tool , pseudo self attention improves performance compared to a baseline without pretraining by large margins without sacrificing adherence to the source , even for tasks with large amounts of supervised data . We further demonstrate that the approach is data-efficient and produces qualitatively more coherent outputs . Code is available at https : //github.com/anon37234/ encoder-agnostic-adaptation . 2 RELATED WORK . Pretrained Decoder Transfer learning for NLG Natural language generation ( NLG ) tasks have a long history of incorporating unconditional language models with conditional input ( Bahl et al. , 1983 ; Koehn et al. , 2003 ) . These approaches traditionally use the noisy channel model ( i.e. , Bayes ’ rule ) , and n-gram models as the language model . Recent adaptations of these ideas include the Neural Noisy Channel ( Yu et al. , 2017 ) as well as “ fusion ” methods ( Koehn et al. , 2003 ; Gulcehre et al. , 2015 ; Sriram et al. , 2018 ; Stahlberg et al. , 2018 ) in which the output logits of a language model and a conditional model are combined to calculate the output probabilities . We consider this class of transfer learning as a baseline in a preliminary experiment ( see Section 4.1 ) , but focus on alternative “ deep ” approaches that incorporate the language model weights as an integral part of the model instead of an add-on at the end . Along these lines , Ramachandran et al . ( 2017 ) propose a finetuning-based method for machine translation with LSTMs , in which some of the layers of the LSTM are initialized with pretrained language model weights . As their method is specific to LSTMs , however , it is incompatible with modern transformer architectures . Pretraining-Based Transfer Learning for NLG Zhang et al . ( 2019 ) use BERT in the encoder and decoder of a summarization model via a unique cloze generative process . They demonstrate strong summarization performance , but the value of pretraining relative to other model components is not clear , and the cloze process significantly reduces the practicality of the model . More related , Edunov et al . ( 2019 ) experiment with a representation-based approach for applying ELMo ( Peters et al. , 2018 ) to the source and target sides of a standard seq2seq model separately . Their approach consistently improves performance when applied to the source , but hurts performance when applied to the decoder . We consider such a representation approach as a baseline in this work . Most recently , several studies experiment with BERT-like masking approaches that are compatible with natural language generation ( Song et al. , 2019 ; Dong et al. , 2019 ; Lample & Conneau , 2019 ) . While these works demonstrate impressive performance , they are constrained to text-to-text tasks because they do not have a way to handle arbitrary conditional information . Whereas these works study pretraining methods that optimize transfer for text-to-text tasks , our study considers the separate problem of adapting a fixed pretrained model to arbitrary source conditioning . Concurrent with this work , Golovanov et al . ( 2019 ) propose a similar approach to pseudo self attention and report initial experiments with dialogue generation . This study complements ours with positive results on dialogue generation , though we aim for experimental data over a wide range of language generation tasks and input modalities and comparison to strong encoder-agnostic baselines . 3 METHODS . We assume that we have a pretrained language model , p ( y ) = p ( y1 , . . . , yT ; θ ) , that the model is an autoregressive neural network , and that it is based on self attention to implement conditioning , i.e. , SA ( Y ) = softmax ( ( YWq ) ( YWk ) > ) ( YWv ) , where input Y ∈ T ×D for hidden dimensionD , Wk , Wv , Wq ∈ D×D′ are parameters , representing the key , value , and query projections respectively , and the output is T ×D′ . 1 We are interested in using this model to estimate the conditional probability p ( y | x ) for an arbitrary input x for which we have a small amount of supervised ( x , y ) pairs . The goal is to learn a model on this new data that best makes use of the pretrained model p ( y ) with a method that is agnostic to the form of x . All models are based on the encoder/decoder architecture , and for each we follow the same high-level procedure : First , some of the weights of the decoder are initialized with weight values from a pretrained language model . Next , a task-specific encoder and all non-pretrained decoder weights are randomly initialized . Finally , the entire model is trained/finetuned end-to-end using the supervised data for the given task . In all cases , the input and output embeddings are tied . Each approach uses all of the pretrained weights , differing only in where and how they use pretrained weights in the decoder . Further experimental details are included in appendix Section A. Baseline 1 : Repr-Transformer The first approach considered ( Fig 1a ) utilizes the pretrained LM to produce a general-purpose representation of the target text before introducing the source information . For this method , a standard transformer decoder is used with the target word embeddings replaced by the output representation of the pretrained language model . In preliminary experiments , we considered both fixing and updating these representations and found that a fixed weightedaveraging ( ” ELMo-Style ” ) method performed better , consistent with Edunov et al . ( 2019 ) . One possible downside to this approach is that the conditioning information from the encoder is injected after all of the pretrained weights . Baseline 2 : Context-Attn The second approach ( Fig 1b ) considers initializing a standard transformer decoder with the shared weights of a pretrained LM . The newly added context attention weights at each layer are randomly initialized . Compared to Repr-Transformer , the conditioning information is injected alongside the pretrained weights . However , the randomly initialized context attention block may interfere with the carefully co-tuned pretrained weights of the rest of the model . This interference may introduce optimization challenges and lead to reduced performance . 1In practice many of these units ( ” heads ” ) are stacked together via concatenation across dimension followed by a final linear projection Wf ∈ D ×D . Proposed Model : Pseudo-Self A more radical approach to incorporating conditional information is the “ zero-shot ” model proposed by Radford et al . ( 2019 ) . Instead of learning a representation for x and passing it into a context attention block they note that an auto-regressive model , p ( yt | y < t ) , is already a conditional model . If x is the same modality as y ( e.g. , both language ) , one can condition on x by prepending the source to target : p ( yt |x , y < t ) = p ( yt | x y < t ) .2 While this does not produce competitive models and is limited in its applicability , it is surprising that it works at all . Taking inspiration from this approach , we propose learning this contextualization in an encoderagnostic way . Our approach , pseudo self attention , simply injects learned encoder conditioning directly into the pretrained self attention of the model . Assume that we have a matrix X ∈ S ×D representing a size S encoding of x , define pseudo self attention as PSA ( X , Y ) = softmax ( ( YWq ) [ XUk YWk ] > ) [ XUv YWv ] , where Uk , Uv ∈ D × D′ are new parameters which project encoder outputs into the decoder self attention space . Because attention is inherently variable length , these additional inputs can be injected without changing the module and only act additively on the attention output . The full model is shown in Figure 1c . Compared to Context-Attn , the proposed approach only introduces new parameters in the self attention block , which we expect leads to only minimal interference . As the pretrained LM weights encode for generation capability , deviating less from this initialization may lead to better generation performance . We explore this quantitatively in Section 5 .

Summary: This paper proposes a new architecture to train decoder models on language generation using a pre-trained encoder (such as BERT or GPT-2). They introduce a novel block called `````"pseudo self-attention" that allow injecting the input for conditional generation in the self-attention layer (i.e. softmax of YW_q (XU_k | YW_k)^T (XU_v | YW_v) instead of softmax(YW_q(YW_k)^T)YW_v). They extensively evaluate their approach on a large set of tasks showing improvements across all of them (which includes class-conditional generation, summarization, story generation and paragraph generation). They also provide interesting ablation studies.