CoCon: A Self-Supervised Approach for Controlled Text Generation

1 INTRODUCTION . Transformer-based ( Vaswani et al. , 2017 ; Tay et al. , 2020 ) pretrained language models ( LMs ) have led a wave of new advances in natural language processing tasks as a means to extract contextualized word embeddings ( Devlin et al. , 2018 ; Dai et al. , 2019b ; Yang et al. , 2019 ) and as text generators ( Radford et al. , 2019 ; Brown et al. , 2020 ) . These LMs are trained on huge amounts of text corpora to predict next tokens through a log-likelihood objective . Given its remarkably fluent text generation , there is growing interest in controlling output texts of such LMs ( Keskar et al. , 2019 ; Dathathri et al. , 2019 ) . Approaches like training a modified LM from scratch to incorporate target text attributes ( Keskar et al. , 2019 ) can be expensive while finetuning pretrained LMs for specific attributes ( Ziegler et al. , 2019 ) limits the scope of text control . Without changing the architecture or weights of pretrained LMs , one promising approach ( PPLM ) ( Dathathri et al. , 2019 ) controls generated text through attribute models . Though effective in controlling high-level text attributes such as topic and sentiment , the same target attribute may generate text samples with vastly different content at the word- and phrase-levels , leaving a gap for more fine-grained control over the content of LM-generated texts . We conceptualize Content-Conditioner ( CoCon ) as an approach to narrow this gap by guiding pretrained LMs ’ text outputs through the incorporation of content input . This content input can take the form of a text sequence whose content we would like to condition on for text generation . Essentially , CoCon comprises two parts : 1 ) a pretrained LM and 2 ) a interleave CoCon layer . By employing a pretrained LM , CoCon incorporates the representations of a content input into the encoded text representations through the CoCon layer before passing the content-conditioned representations into LMβ for generation . To train the CoCon block , we propose a self-supervised learning approach where training data consist of text samples generated by the pretrained LM itself ( § 3.1 ) . By splitting each text sequence into two segments ( [ xa ; xb ] ) , CoCon learns through a self reconstruction objective to help the LM reconstruct missing latter segments ( xb ) by taking xb itself as the content input . We use content masking for CoCon and also propose other loss functions such as cycle reconstruction to condition content from divergent sources while producing high-quality texts . Since the CoCon block ’ s size is a small fraction of the LM and no finetuning is conducted on the LM ’ s weights , the training cost is significantly lower than training an LM from scratch . We show that CoCon ’ s fine-grained content control can be extended to also influence higher-level text attributes ∗Corresponding author : guoweial001 @ ntu.edu.sg 1Codes and models are available at : https : //github.com/alvinchangw/COCON ICLR2021 such as topic and sentiment in a zero-shot manner , and compare it with strong controlled generation baselines . Furthermore , CoCon is versatile in assimilating multiple content inputs , and its strength of content-conditioning can be flexibly adjusted through a content bias term during inference . In this paper , we demonstrate the CoCon approach with the GPT-2 345M model ( Radford et al. , 2019 ) as the pretrained LM . Given CoCon ’ s modular nature , it can be used with other Transformer-based LMs or even other controlled generation methods . All in all , the core contributions of this paper are : • We propose CoCon for content-conditioned language generation . • We introduce a self-supervised learning approach where CoCon learns to complete text sequences when given information about future tokens . • Through ablation studies and comparisons with strong baselines like PPLM and CTRL ( Keskar et al. , 2019 ) , we investigate how CoCon controls high-level attributes such as topic and sentiment while generating texts that have high content similarity to conditioning text . 2 RELATED WORK . There is a line of work that aims to generate output text of desired attributes with neural networks . Some of the earliest efforts involve conditional generative models ( Kikuchi et al. , 2016 ; Ficler & Goldberg , 2017 ) where the networks are trained on text data labeled with the target attributes . These models can be trained via reinforcement learning ( Ziegler et al. , 2019 ) or the generative adversarial network ( Yu et al. , 2017 ) framework . Unlike CoCon , the requirement of predetermined attributes in those methods limits the possible types of generated texts . CTRL ( Keskar et al. , 2019 ) is a recent approach that generated controlled fluent texts through the use of control codes which are meta-data prepended to the text during generation . Though it produces high-quality text with its GPT-2-like architecture , its control codes are also predetermined during the training . Closest to our work is Plug and Play Language Model ( PPLM ) ( Dathathri et al. , 2019 ) which seeks to control text on already pretrained LM without finetuning through relatively small ‘ pluggable ’ attribute models . While PPLM ’ s flexible design also enables controlled generation without retraining or finetuning the LM like in CoCon , our approach aims to control the generation at a content level , beyond high-level text attributes . Another core difference lies in the training where CoCon ’ s self-supervised learning absolves the need for labeled data , such as the ones employed to train PPLM ’ s attribute discriminator models . Weighted decoding ( Ghazvininejad et al. , 2017 ; Holtzman et al. , 2018 ) seeks to control the output text token by upweighting the probabilities of targeted words during the decoding step but has been shown to produce incoherent text ( See et al. , 2019 ) . Conditioning language generation has been used in question generation to enhance faithfulness by attending to textual context such as predicates , subject types or object types ( Elsahar et al. , 2018 ) rather than the content input used here in CoCon . Small adapter layers ( Bapna et al. , 2019 ) have been previously proposed for multilingual translation to also save on model size and training resources but differ from CoCon ’ s self-supervised training as they rely on annotated sentence pairs of different languages for training . Text style transfer is a related area that controls texts ’ attributes by translating text from one style to another ( Dai et al. , 2019a ) . A few of such studies employ auto-encoders to separate texts ’ style and non-style latent representation ( Shen et al. , 2017 ; Hu et al. , 2017 ; Yang et al. , 2018 ) . This disentanglement enables style changes to the text at the latent space while retaining most of its content . Another work identifies attribute markers ( Li et al. , 2018 ) which are n-grams correlated with a particular style in a text corpus and edit texts ’ style by substituting them . Essentially , style transfer alters existing texts rather than generating texts and requires predefined attributes . 3 CONTENT CONDITIONER ( COCON ) . In the following sections , we discuss the motivation for CoCon , its model architecture and how we train the CoCon block . Motivation In text generation with language models , given the prompt text x : t−1 = { x1 , . . . , xt−1 } , the following text { xt , . . . , xl } is generated in an auto-regressive manner ( Man- ning et al. , 1999 ; Bengio et al. , 2003 ) : p ( xt , . . . , xl|x1 , . . . , xt−1 ) = l∏ i=t p ( xi|x1 , . . . , xi−1 ) . ( 1 ) Previous studies on controlled text generation in LM showed that p ( x ) can be conditioned on target attributes ( Dathathri et al. , 2019 ) or control codes ( Keskar et al. , 2019 ) to control the text ’ s sentiment or topic , i.e. , p ( xt , . . . , xl|x1 , . . . , xt−1 ) = l∏ i=1 p ( xi|a , { x1 , . . . , xi−1 } ) , ( 2 ) where a is the target attribute . While these methods show that the generation is fluent and can be aligned with the target attribute well , the output texts { xt , . . . , xl } are controlled at a global attribute ( e.g. , sentiment/topic ) level rather than at a more local content ( e.g. , words/phrases ) level . Since there is a vast number of possible { xt , . . . , xl } candidates which would align well with both the prompt text and target attribute , this results in generated text samples that contain very different content during the stochastic token sampling process . This motivates an approach to condition on an content input c for more fine-grained control over text generation : p ( xt , . . . , xl|x1 , . . . , xt−1 ) = l∏ i=1 p ( xi|c , { x1 , . . . , xi−1 } ) , ( 3 ) where c can be a text sequence whose content we would like to condition on during text generation . Next , we propose the model architecture of Content-Conditioner ( CoCon ) as an approach for this control . Model Architecture Our proposed Content-Conditioner ( Figure 1 ) controls the content of the generated text while maintaining fluency by incorporating a pretrained Transformer-based language model ( LM ) , GPT-2 ( Radford et al. , 2019 ) in our experiments . Such LMs have shown remarkable natural text generation in the auto-regressive manner ( Eq . 1 ) where the next token xt is sampled based on the logits ot = LM ( x : t−1 ) . These LMs are essentially stacks of Transformer blocks , each consisting of layer normalization ( Ba et al. , 2016 ) , multi-head self-attention ( Vaswani et al. , 2017 ) and position-wise feed forward operations . An LM ’ s generation can be broken down into two separate parts : layers before the CoCon block ( LMα ) and layers after ( LMβ ) . The LMα acts as a feature extractor that takes in the input sequence ’ s embeddings and outputs its intermediate representation at a breakpoint , i.e. , h : t−1 = LMα ( x : t−1 ) . Subsequently , LMβ takes in this representation and outputs the logits for the next token , i.e. , ot = LMβ ( h : t−1 ) , yielding ot = LM ( x : t−1 ) = LMβ ( LMα ( x : t−1 ) ) = LMβ ( h : t−1 ) . ( 4 ) From Eq . 4 , we can see that the representation ( h ) is a medium to control next token logits ( o ) and hence the text generation process . Indeed , we transform h by conditioning it with the content input ( c ) through a CoCon block such that h′ : t−1 = CoCon ( h ( c ) : lc , h : t−1 ) , ( 5 ) where h ( c ) : lc = LMα ( c ) is the content representations and lc is the length of the content text sequence . We parameterize the CoCon block as a single Transformer block with an attention and position-wise feed-forward operation . Similar to a typical LM attention layer , the query ( Q ) , key ( K ) , value ( V ) matrices are computed through linear transformations on the representations h : t−1 , where Q , K , V ∈ R ( t−1 ) ×d and d is the representations ’ dimension . To attend to the content representations ( h ( c ) : lc ) , the content keys and values ( K ( c ) , V ( c ) ∈ Rlc×d ) are also computed , and concatenated to the original attention matrices before computing the CoCon attention output : K′ = [ K ( c ) ; K ] , V′ = [ V ( c ) ; V ] , A = Softmax ( QK′ > ) V′ = Softmax ( W ) V′ , ( 6 ) where A = { a1 , . . . , at−1 } and W ∈ R ( t−1 ) × ( lc+t−1 ) represents the attention weights . The final CoCon outputs are computed with a position-wise feed-forward layer . By concatenating to the representations prior to t − 1 and passing them to LMβ , the next logits , and consequently word token x̃t , is now conditioned on c : h′i = FF ( ai ) , õt = LMβ ( [ h : t−2 ; h ′ t−1 ] ) , pθ , ψ ( x̃t|c , x : t−1 ) = Softmax ( õt ) , ( 7 ) where θ and ψ are the paramterization of the CoCon block and LM respectively . Similar to a GPT-2 Transformer block , our CoCon block includes layer normalization before its multi-headed attention and feed-forward layers . Figure 1 summarizes the CoCon architecture which enables auto-regressive text generation by using x̃i as the token input ( xi ) to generate x̃i+1 where i ≥ t. Multiple Content Inputs CoCon ’ s flexible design enables multiple content inputs for a single generation . In the case where we have N content inputs ( c1 , . . . , cN ) , the output text can be conditioned by these contents through their attention keys and values , similar to Eq . 6 : K′ = [ K ( c 1 ) . . .K ( c N ) ; K ] , V′ = [ V ( c 1 ) . . .V ( c N ) ; V ] , A = Softmax ( QK′ > ) V′ . ( 8 ) Strength of Content Conditioning Within CoCon ’ s attention mechanism , we can vary the extent of content conditioning on the output text by biasing the attention weights in W ( Eq . 6 ) that correspond to the content input ( c ) . More specifically , the influence of c on the output text can be altered through the attention ’ s softmax weighting on the content values ( V ( c ) ) . During generation , a positive bias term ( τcontent ) can optionally be added to the content attention weights W : , : lc ∈ R ( t−1 ) ×lc to increase influence of V ( c ) , boosting content conditioning , while a negative term can conversely reduce the content-conditioning effect . We discuss examples of varying τcontent in § 4.4 .

Summary: This paper tackles the problem of controlled text generation by converting it into a conditional text generation similar to (Keskar et al.19).  It proposes an architectural modification to the transformer LM used in GPT2,  Specifically, a CoCon layer is added as a separate transformer block in the middle allowing self-attention to be performed between the textual context representations LM_α(c) and the generations LM_α(x_{:t-1}) this is performed through concatenating the key and value matrices with the keys and values of the encoded textual context. Authors provide 4 different losses to train this additional layer.