GeDi: Generative Discriminator Guided Sequence Generation

1 INTRODUCTION . Natural language generation has seen great progress with the advent of Transformers ( Vaswani et al. , 2017 ) and large scale training ( Radford et al. , 2017 ; 2018 ; 2019 ; Brown et al. , 2020 ) . Large language models ( LMs ) like GPT-2 ( Radford et al. , 2019 ) and GPT-3 ( Brown et al. , 2020 ) are able to learn the distribution of their training set well enough to generate realistic text . However , simply imitating the distribution of the training data during generation has many drawbacks ; large-scale text training sets are crawled from the web which is imbued with toxicity , bias , hate , and misinformation . Methods for better controlling or filtering generation are valuable for making LMs trained on such data safer and more generally useful for downstream applications . Existing approaches to controlling LMs have limitations . Class-conditional LMs ( CC-LMs ) such as CTRL ( Keskar et al. , 2019 ) attempt to control text generation by conditioning on a control code , which is an attribute variable representing a data source . However , CTRL is not as useful for controlling what not to generate ( i.e . toxicity ) . Furthermore , using a specific control code can reduce sample diversity across prompts , as samples will generally resemble the data source of the control code . Another approach is to use discriminators to steer generation , but existing methods to do this are very computationally intensive . Weighted decoding ( Holtzman et al. , 2018 ) requires feeding candidate next tokens into a discriminator , and thus scales linearly in computation with the number of tokens to be re-weighted . Plug and Play LM ( Dathathri et al. , 2020 , PPLM ) applies up to 10 updates to the generating LM ’ s latent states per time step using gradients from a discriminator , also making it many times slower than generating from the LM directly . We present GeDi1 as an algorithm for efficiently guiding generation from large LMs to make them safer and more controllable . Our proposed method uses CC-LMs as generative discriminators ( GeDis ) to guide language generation towards desired attributes . We use GeDis to compute classification likelihoods for all candidate next tokens during generation using Bayes rule , saving many 1pronounced “ Jedi ” thousand-fold in computation as compared with using a standard ( non-generative ) discriminator to compute this for large vocabulary sizes . We then show how these likelihoods can guide generation from large language models via weighted decoding and filtering . Our experimental results verify the ability of GeDi to control generation in a variety of settings while maintaining linguistic quality on par with strong language models . We apply GeDi ( 345M parameters ) to guide generation from larger language models , and find that : • GeDi trained on sentiment of movie reviews can generate book text with a positive or negative tone better than or equivalently to state of the art baselines [ Section 5.1 ] . Guiding towards positivity also has potential applications towards making LMs friendlier . • GeDi is able to significantly reduce the toxicity of GPT-2 and GPT-3 generation [ Section 5.2 ] , without sacrificing linguistic quality as compared with generating from GPT-2 and GPT-3 directly , suggesting applications towards safer language modeling . • GeDi trained on a dataset of only 4 topics can generalize to new control codes zero-shot [ Section 5.3 ] , allowing them to guide generation towards a wide variety of topics . • GeDi is very computationally efficient for both training and inference . GeDi guided generation in our experiments is more than 30× faster than applying PPLM with GPT2-XL using default settings from Dathathri et al . ( 2020 ) . Additionally , smaller GeDis fine-tuned for less than a day on a single GPU are effective and computationally efficient for controlling larger language models . This provides a cheap alternative to finetuning large LMs directly ( Ziegler et al. , 2019 ) . 2 BACKGROUND . 2.1 LANGUAGE MODELING . Language models ( LMs ) rely on an auto-regressive factorization to perform density estimation and generation of language data . Auto-regressive sequence models with parameters θ assign a probability to a sequence x1 : T = { x1 , . . . , xT } by factorizing it using the chain rule as follows : Pθ ( x1 : T ) = T∏ t=1 Pθ ( xt|x < t ) . ( 1 ) Models can assign probabilities to sequences by iteratively predicting a distribution over the next token given the previous tokens . Generating from language models requires iteratively sampling from Pθ ( xt|x < t ) , and then feeding xt back into the model as input for the next step . 2.2 CLASS-CONDITIONAL LANGUAGE MODELING . Class-conditional language models ( CC-LMs ) such as CTRL ( Keskar et al. , 2019 ) are a way for language models to generate while conditioning on an attribute variable . CC-LMs predict a probability distribution Pθ ( x1 : T |c ) , where c is a class variable or a “ control code ” that describes an attribute of the text in x1 : T , which could , for instance , describe sentiment or topic . The auto-regressive factorization for a CC-LM is given by the following equation : Pθ ( x1 : T |c ) = T∏ t=1 Pθ ( xt|x < t , c ) . ( 2 ) When training a CC-LM on a training set of sequences { x ( 1 ) 1 : T1 , . . . , x ( i ) 1 : Ti , . . . , x ( N ) 1 : TN } , each sequence x ( i ) 1 : T is paired with a control code c ( i ) , which is a label or category of the sequence . The LM is trained to minimize the average negative log-likelihood , L. L = − 1 N N∑ i=1 1 Ti Ti∑ t=1 logPθ ( x ( i ) t |x ( i ) < t , c ( i ) ) . ( 3 ) In addition to class-conditional generation , CC-LMs can be used as generative classifiers by applying Bayes rule to compute Pθ ( c|x1 : T ) ∝ P ( c ) Pθ ( x1 : T |c ) , as is done by Keskar et al . ( 2019 ) for source attribution . 3 GEDI . GeDi assumes we have a CC-LM with desired control code c and an undesired or anti-control code c̄ , and uses the contrast between Pθ ( x1 : t|c ) and Pθ ( x1 : t|c̄ ) to guide sampling from an LM that gives PLM ( x1 : t ) . Specifically , when predicting the next token during generation , GeDi uses this contrast to compute the probability that every candidate next token xt belongs to the desired class , given by Pθ ( c|xt , x < t ) . Our key insight is that this distribution can be computed very efficiently when using CC-LMs as GeDis via application of Bayes rule for partial sequences during generation . Pθ ( c|x1 : t ) = P ( c ) ∏t j=1 Pθ ( xj |x < j , c ) ∑ c′∈ { c , c̄ } ∏t j=1 P ( c ′ ) Pθ ( xj |x < j , c′ ) . ( 4 ) When computing this online during sequence generation , the model will have already computed Pθ ( xj |x < j , c′ ) for any j < t from the previous time-steps , and it will only need to compute Pθ ( xt|x < t , c′ ) . This can be computed in two parallel forward passes ; one conditioning on c and one conditioning on c̄ ( both conditioning on the same x < t ) . The model can also save the hidden states from the previous time steps to avoid computing a forward pass through the full sequence at each next token generation step . Applying a unidirectional classifier such as GPT ( Radford et al. , 2018 ) to compute Pθ ( c|xt , x < t ) directly ( i.e . discriminatively ) would require feeding in every possible input xt ∈ V into the classifier , and thus would require |V| forward passes for a vocab set V . A bidirectional classifier such as BERT ( Devlin et al. , 2018 ) would require t × |V| forward passes because it would need to recompute attention states from earlier time-steps . For typical vocab sizes of 20k+ , GeDi ’ s online classification trick can compute Pθ ( c|xt , x < t ) for every possible next token xt on the order of 10k fold less computation as compared with a unidirectional classifier ( because the unidirectional classifier would require 20k+ forward passes through the network , whereas the GeDi would only require 2 through the CC-LM and one through the base-LM ) . In practice , we find that applying Equation 4 to long sequences often results in poorly calibrated distributions later in the sequence that assign classification probabilities of 1 or 0 to on all candidate next words , which provides no useful signal . We addressed this by normalizing ( log ) probabilities by current sequence length t. To compute Pθ ( c|x1 : t ) for GeDi-guided generation , we use the following equation : Pθ ( c|x1 : t ) = ( Pθ ( x1 : t|c ) 1/t∑ c′∈ { c , c̄ } Pθ ( x1 : t|c′ ) 1/t . ( 5 ) where class priors P ( c ) are omitted . In practice , Pθ ( c|x1 : t ) is computed with log-probabilities ( see Appendix A ) . With the efficient estimation of Pθ ( c|xt , x < t ) , there are many possible heuristics that can be used to guide LM generation , so long as the LM and GeDi share the same tokenization . Heuristics that use Pθ ( c|xt , x < t ) inherently contrast predictions conditioned on c and c̄ , causing attributes common to c and c̄ to be cancelled out , more effectively allowing for the attribute described by c to be transferred across domains , as illustrated in Figure 1 . 3.1 HEURISTICS FOR GUIDING GENERATION . We consider a weighted decoding heuristic and a filtering heuristic to use Pθ ( c|xt , x < t ) to guide generation . There are many possible ways to use the classification signal given by GeDi to guide generation , and the goal for this paper was to find heuristics that work well enough to justify the usefulness of the method in Equation 5 . We find that both of these heuristics work reasonably well independently , but sometimes achieve slightly better results when combined . Our initial heuristic applies a weighted posterior given by Pw ( xt|x < t , c ) ∝ PLM ( xt|x < t ) Pθ ( c|xt , x < t ) ω , ( 6 ) where ω > 1 to bias generation more strongly towards the correct class . The right hand side of Equation ( 6 ) is normalized over all xt in the vocabulary to obtain Pw ( xt|x < t , c ) . We summarize the resulting scheme in Algorithm 1 . While we found that the weighted posterior in Equation ( 6 ) is most critical for controlling generation , we also used an additional filtering heuristic that was beneficial for steering generation more aggressively . This heuristic , inspired by nucleus sampling ( Holtzman et al. , 2020 ) , removes candidate next word tokens with lower values for Pθ ( c|xt , x < t ) while maintaining a minimum of at least 1 − ρ in cumulative probability mass in Pw ( xt|x < t , c ) , where 0 ≤ ρ < 1 is a parameter that decides the aggressiveness of the filtering . We define Vn as the set of n tokens with the highest Pθ ( c|xt , x < t ) . We define m as the minimum n such that∑ xt∈Vn Pw ( xt|x < t , c ) ≥ 1− ρ . ( 7 ) We define Vm as Vn for n = m , meaning that Vm will contain the minimum number of tokens possible at the head of the distribution for Pθ ( c|xt , x < t ) to maintain a minimum cumulative probability of ρ in Pw ( xt|x < t , c ) . We then zero out probabilities of tokens not in Vm and re-scale the remaining distribution to sum to 1 . Algorithm 1 Generative Discriminator Guided Sequence Generation Inputs : base LM PLM , CC-LM Pθ , vocabulary V , posterior mixing weight ω , decoding scheme 1 : P ( x|c ) ← 1 2 : P ( x|c̄ ) ← 1 3 : for t = 1 . . . , N do 4 : pLM ← [ PLM ( xt = v|x < t ) for v in V ] . base-LM prediction 5 : 6 : px1 : t|c ← [ ( P ( x|c ) Pθ ( xt = v|x < t , c ) ) 1/t for v in V ] . CC-LM with control code 7 : px1 : t|c̄ ← [ ( P ( x|c̄ ) Pθ ( xt = v|x < t , c̄ ) ) 1/t for v in V ] . CC-LM with anti control code 8 : 9 : pc|x1 : t ← px1 : t|c 1 ( px1 : t|c+px1 : t|c̄ ) 10 : 11 : pw ← pLM ( pc|x1 : t ) ω 12 : pw ← pw∑|V| i=1 pw [ i ] . normalize over the vocabulary 13 : vi ← Decode ( pw ) . Can be greedy or sampling , could include filtering in Eq . 7 14 : 15 : P ( x|c ) ← P ( x|c ) Pθ ( xt = vi|x < t , c ) 16 : P ( x|c̄ ) ← P ( x|c̄ ) Pθ ( xt = vi|x < t , c̄ ) 17 : xt ← vi

Summary: The paper considers the problem of attribute-based sequence generation, particularly in language models. Authors propose a framework “GeDi” which learns a generative classifier for controlling generation from a large language model. With experiments on publicly available datasets and models, and including human-evaluation, the authors empirically demonstrate that the algorithm is computationally efficient and is competitive against strong baseline algorithms like CTRL, Plug&Play language models (PPLM).