CoCon: A Self-Supervised Approach for Controlled Text Generation

1 INTRODUCTION . Transformer-based ( Vaswani et al. , 2017 ; Tay et al. , 2020 ) pretrained language models ( LMs ) have led a wave of new advances in natural language processing tasks as a means to extract contextualized word embeddings ( Devlin et al. , 2018 ; Dai et al. , 2019b ; Yang et al. , 2019 ) and as text generators ( Radford et al. , 2019 ; Brown et al. , 2020 ) . These LMs are trained on huge amounts of text corpora to predict next tokens through a log-likelihood objective . Given its remarkably fluent text generation , there is growing interest in controlling output texts of such LMs ( Keskar et al. , 2019 ; Dathathri et al. , 2019 ) . Approaches like training a modified LM from scratch to incorporate target text attributes ( Keskar et al. , 2019 ) can be expensive while finetuning pretrained LMs for specific attributes ( Ziegler et al. , 2019 ) limits the scope of text control . Without changing the architecture or weights of pretrained LMs , one promising approach ( PPLM ) ( Dathathri et al. , 2019 ) controls generated text through attribute models . Though effective in controlling high-level text attributes such as topic and sentiment , the same target attribute may generate text samples with vastly different content at the word- and phrase-levels , leaving a gap for more fine-grained control over the content of LM-generated texts . We conceptualize Content-Conditioner ( CoCon ) as an approach to narrow this gap by guiding pretrained LMs ’ text outputs through the incorporation of content input . This content input can take the form of a text sequence whose content we would like to condition on for text generation . Essentially , CoCon comprises two parts : 1 ) a pretrained LM and 2 ) a interleave CoCon layer . By employing a pretrained LM , CoCon incorporates the representations of a content input into the encoded text representations through the CoCon layer before passing the content-conditioned representations into LMβ for generation . To train the CoCon block , we propose a self-supervised learning approach where training data consist of text samples generated by the pretrained LM itself ( § 3.1 ) . By splitting each text sequence into two segments ( [ xa ; xb ] ) , CoCon learns through a self reconstruction objective to help the LM reconstruct missing latter segments ( xb ) by taking xb itself as the content input . We use content masking for CoCon and also propose other loss functions such as cycle reconstruction to condition content from divergent sources while producing high-quality texts . Since the CoCon block ’ s size is a small fraction of the LM and no finetuning is conducted on the LM ’ s weights , the training cost is significantly lower than training an LM from scratch . We show that CoCon ’ s fine-grained content control can be extended to also influence higher-level text attributes ∗Corresponding author : guoweial001 @ ntu.edu.sg 1Codes and models are available at : https : //github.com/alvinchangw/COCON ICLR2021 such as topic and sentiment in a zero-shot manner , and compare it with strong controlled generation baselines . Furthermore , CoCon is versatile in assimilating multiple content inputs , and its strength of content-conditioning can be flexibly adjusted through a content bias term during inference . In this paper , we demonstrate the CoCon approach with the GPT-2 345M model ( Radford et al. , 2019 ) as the pretrained LM . Given CoCon ’ s modular nature , it can be used with other Transformer-based LMs or even other controlled generation methods . All in all , the core contributions of this paper are : • We propose CoCon for content-conditioned language generation . • We introduce a self-supervised learning approach where CoCon learns to complete text sequences when given information about future tokens . • Through ablation studies and comparisons with strong baselines like PPLM and CTRL ( Keskar et al. , 2019 ) , we investigate how CoCon controls high-level attributes such as topic and sentiment while generating texts that have high content similarity to conditioning text . 2 RELATED WORK . There is a line of work that aims to generate output text of desired attributes with neural networks . Some of the earliest efforts involve conditional generative models ( Kikuchi et al. , 2016 ; Ficler & Goldberg , 2017 ) where the networks are trained on text data labeled with the target attributes . These models can be trained via reinforcement learning ( Ziegler et al. , 2019 ) or the generative adversarial network ( Yu et al. , 2017 ) framework . Unlike CoCon , the requirement of predetermined attributes in those methods limits the possible types of generated texts . CTRL ( Keskar et al. , 2019 ) is a recent approach that generated controlled fluent texts through the use of control codes which are meta-data prepended to the text during generation . Though it produces high-quality text with its GPT-2-like architecture , its control codes are also predetermined during the training . Closest to our work is Plug and Play Language Model ( PPLM ) ( Dathathri et al. , 2019 ) which seeks to control text on already pretrained LM without finetuning through relatively small ‘ pluggable ’ attribute models . While PPLM ’ s flexible design also enables controlled generation without retraining or finetuning the LM like in CoCon , our approach aims to control the generation at a content level , beyond high-level text attributes . Another core difference lies in the training where CoCon ’ s self-supervised learning absolves the need for labeled data , such as the ones employed to train PPLM ’ s attribute discriminator models . Weighted decoding ( Ghazvininejad et al. , 2017 ; Holtzman et al. , 2018 ) seeks to control the output text token by upweighting the probabilities of targeted words during the decoding step but has been shown to produce incoherent text ( See et al. , 2019 ) . Conditioning language generation has been used in question generation to enhance faithfulness by attending to textual context such as predicates , subject types or object types ( Elsahar et al. , 2018 ) rather than the content input used here in CoCon . Small adapter layers ( Bapna et al. , 2019 ) have been previously proposed for multilingual translation to also save on model size and training resources but differ from CoCon ’ s self-supervised training as they rely on annotated sentence pairs of different languages for training . Text style transfer is a related area that controls texts ’ attributes by translating text from one style to another ( Dai et al. , 2019a ) . A few of such studies employ auto-encoders to separate texts ’ style and non-style latent representation ( Shen et al. , 2017 ; Hu et al. , 2017 ; Yang et al. , 2018 ) . This disentanglement enables style changes to the text at the latent space while retaining most of its content . Another work identifies attribute markers ( Li et al. , 2018 ) which are n-grams correlated with a particular style in a text corpus and edit texts ’ style by substituting them . Essentially , style transfer alters existing texts rather than generating texts and requires predefined attributes . 3 CONTENT CONDITIONER ( COCON ) . In the following sections , we discuss the motivation for CoCon , its model architecture and how we train the CoCon block . Motivation In text generation with language models , given the prompt text x : t−1 = { x1 , . . . , xt−1 } , the following text { xt , . . . , xl } is generated in an auto-regressive manner ( Man- ning et al. , 1999 ; Bengio et al. , 2003 ) : p ( xt , . . . , xl|x1 , . . . , xt−1 ) = l∏ i=t p ( xi|x1 , . . . , xi−1 ) . ( 1 ) Previous studies on controlled text generation in LM showed that p ( x ) can be conditioned on target attributes ( Dathathri et al. , 2019 ) or control codes ( Keskar et al. , 2019 ) to control the text ’ s sentiment or topic , i.e. , p ( xt , . . . , xl|x1 , . . . , xt−1 ) = l∏ i=1 p ( xi|a , { x1 , . . . , xi−1 } ) , ( 2 ) where a is the target attribute . While these methods show that the generation is fluent and can be aligned with the target attribute well , the output texts { xt , . . . , xl } are controlled at a global attribute ( e.g. , sentiment/topic ) level rather than at a more local content ( e.g. , words/phrases ) level . Since there is a vast number of possible { xt , . . . , xl } candidates which would align well with both the prompt text and target attribute , this results in generated text samples that contain very different content during the stochastic token sampling process . This motivates an approach to condition on an content input c for more fine-grained control over text generation : p ( xt , . . . , xl|x1 , . . . , xt−1 ) = l∏ i=1 p ( xi|c , { x1 , . . . , xi−1 } ) , ( 3 ) where c can be a text sequence whose content we would like to condition on during text generation . Next , we propose the model architecture of Content-Conditioner ( CoCon ) as an approach for this control . Model Architecture Our proposed Content-Conditioner ( Figure 1 ) controls the content of the generated text while maintaining fluency by incorporating a pretrained Transformer-based language model ( LM ) , GPT-2 ( Radford et al. , 2019 ) in our experiments . Such LMs have shown remarkable natural text generation in the auto-regressive manner ( Eq . 1 ) where the next token xt is sampled based on the logits ot = LM ( x : t−1 ) . These LMs are essentially stacks of Transformer blocks , each consisting of layer normalization ( Ba et al. , 2016 ) , multi-head self-attention ( Vaswani et al. , 2017 ) and position-wise feed forward operations . An LM ’ s generation can be broken down into two separate parts : layers before the CoCon block ( LMα ) and layers after ( LMβ ) . The LMα acts as a feature extractor that takes in the input sequence ’ s embeddings and outputs its intermediate representation at a breakpoint , i.e. , h : t−1 = LMα ( x : t−1 ) . Subsequently , LMβ takes in this representation and outputs the logits for the next token , i.e. , ot = LMβ ( h : t−1 ) , yielding ot = LM ( x : t−1 ) = LMβ ( LMα ( x : t−1 ) ) = LMβ ( h : t−1 ) . ( 4 ) From Eq . 4 , we can see that the representation ( h ) is a medium to control next token logits ( o ) and hence the text generation process . Indeed , we transform h by conditioning it with the content input ( c ) through a CoCon block such that h′ : t−1 = CoCon ( h ( c ) : lc , h : t−1 ) , ( 5 ) where h ( c ) : lc = LMα ( c ) is the content representations and lc is the length of the content text sequence . We parameterize the CoCon block as a single Transformer block with an attention and position-wise feed-forward operation . Similar to a typical LM attention layer , the query ( Q ) , key ( K ) , value ( V ) matrices are computed through linear transformations on the representations h : t−1 , where Q , K , V ∈ R ( t−1 ) ×d and d is the representations ’ dimension . To attend to the content representations ( h ( c ) : lc ) , the content keys and values ( K ( c ) , V ( c ) ∈ Rlc×d ) are also computed , and concatenated to the original attention matrices before computing the CoCon attention output : K′ = [ K ( c ) ; K ] , V′ = [ V ( c ) ; V ] , A = Softmax ( QK′ > ) V′ = Softmax ( W ) V′ , ( 6 ) where A = { a1 , . . . , at−1 } and W ∈ R ( t−1 ) × ( lc+t−1 ) represents the attention weights . The final CoCon outputs are computed with a position-wise feed-forward layer . By concatenating to the representations prior to t − 1 and passing them to LMβ , the next logits , and consequently word token x̃t , is now conditioned on c : h′i = FF ( ai ) , õt = LMβ ( [ h : t−2 ; h ′ t−1 ] ) , pθ , ψ ( x̃t|c , x : t−1 ) = Softmax ( õt ) , ( 7 ) where θ and ψ are the paramterization of the CoCon block and LM respectively . Similar to a GPT-2 Transformer block , our CoCon block includes layer normalization before its multi-headed attention and feed-forward layers . Figure 1 summarizes the CoCon architecture which enables auto-regressive text generation by using x̃i as the token input ( xi ) to generate x̃i+1 where i ≥ t. Multiple Content Inputs CoCon ’ s flexible design enables multiple content inputs for a single generation . In the case where we have N content inputs ( c1 , . . . , cN ) , the output text can be conditioned by these contents through their attention keys and values , similar to Eq . 6 : K′ = [ K ( c 1 ) . . .K ( c N ) ; K ] , V′ = [ V ( c 1 ) . . .V ( c N ) ; V ] , A = Softmax ( QK′ > ) V′ . ( 8 ) Strength of Content Conditioning Within CoCon ’ s attention mechanism , we can vary the extent of content conditioning on the output text by biasing the attention weights in W ( Eq . 6 ) that correspond to the content input ( c ) . More specifically , the influence of c on the output text can be altered through the attention ’ s softmax weighting on the content values ( V ( c ) ) . During generation , a positive bias term ( τcontent ) can optionally be added to the content attention weights W : , : lc ∈ R ( t−1 ) ×lc to increase influence of V ( c ) , boosting content conditioning , while a negative term can conversely reduce the content-conditioning effect . We discuss examples of varying τcontent in § 4.4 .

Summary: The paper proposed a way to control the content output of a DNN-based language model (GPT-2 in the experiment, but not limited to it). It places an layer (CoCon) that can take an arbitrary phrase as the hint after generating the embedding but before generating the text. Experiments showed that the control is effective at directing the generated text. Examples confirmed that too.