GeDi: Generative Discriminator Guided Sequence Generation

1 INTRODUCTION . Natural language generation has seen great progress with the advent of Transformers ( Vaswani et al. , 2017 ) and large scale training ( Radford et al. , 2017 ; 2018 ; 2019 ; Brown et al. , 2020 ) . Large language models ( LMs ) like GPT-2 ( Radford et al. , 2019 ) and GPT-3 ( Brown et al. , 2020 ) are able to learn the distribution of their training set well enough to generate realistic text . However , simply imitating the distribution of the training data during generation has many drawbacks ; large-scale text training sets are crawled from the web which is imbued with toxicity , bias , hate , and misinformation . Methods for better controlling or filtering generation are valuable for making LMs trained on such data safer and more generally useful for downstream applications . Existing approaches to controlling LMs have limitations . Class-conditional LMs ( CC-LMs ) such as CTRL ( Keskar et al. , 2019 ) attempt to control text generation by conditioning on a control code , which is an attribute variable representing a data source . However , CTRL is not as useful for controlling what not to generate ( i.e . toxicity ) . Furthermore , using a specific control code can reduce sample diversity across prompts , as samples will generally resemble the data source of the control code . Another approach is to use discriminators to steer generation , but existing methods to do this are very computationally intensive . Weighted decoding ( Holtzman et al. , 2018 ) requires feeding candidate next tokens into a discriminator , and thus scales linearly in computation with the number of tokens to be re-weighted . Plug and Play LM ( Dathathri et al. , 2020 , PPLM ) applies up to 10 updates to the generating LM ’ s latent states per time step using gradients from a discriminator , also making it many times slower than generating from the LM directly . We present GeDi1 as an algorithm for efficiently guiding generation from large LMs to make them safer and more controllable . Our proposed method uses CC-LMs as generative discriminators ( GeDis ) to guide language generation towards desired attributes . We use GeDis to compute classification likelihoods for all candidate next tokens during generation using Bayes rule , saving many 1pronounced “ Jedi ” thousand-fold in computation as compared with using a standard ( non-generative ) discriminator to compute this for large vocabulary sizes . We then show how these likelihoods can guide generation from large language models via weighted decoding and filtering . Our experimental results verify the ability of GeDi to control generation in a variety of settings while maintaining linguistic quality on par with strong language models . We apply GeDi ( 345M parameters ) to guide generation from larger language models , and find that : • GeDi trained on sentiment of movie reviews can generate book text with a positive or negative tone better than or equivalently to state of the art baselines [ Section 5.1 ] . Guiding towards positivity also has potential applications towards making LMs friendlier . • GeDi is able to significantly reduce the toxicity of GPT-2 and GPT-3 generation [ Section 5.2 ] , without sacrificing linguistic quality as compared with generating from GPT-2 and GPT-3 directly , suggesting applications towards safer language modeling . • GeDi trained on a dataset of only 4 topics can generalize to new control codes zero-shot [ Section 5.3 ] , allowing them to guide generation towards a wide variety of topics . • GeDi is very computationally efficient for both training and inference . GeDi guided generation in our experiments is more than 30× faster than applying PPLM with GPT2-XL using default settings from Dathathri et al . ( 2020 ) . Additionally , smaller GeDis fine-tuned for less than a day on a single GPU are effective and computationally efficient for controlling larger language models . This provides a cheap alternative to finetuning large LMs directly ( Ziegler et al. , 2019 ) . 2 BACKGROUND . 2.1 LANGUAGE MODELING . Language models ( LMs ) rely on an auto-regressive factorization to perform density estimation and generation of language data . Auto-regressive sequence models with parameters θ assign a probability to a sequence x1 : T = { x1 , . . . , xT } by factorizing it using the chain rule as follows : Pθ ( x1 : T ) = T∏ t=1 Pθ ( xt|x < t ) . ( 1 ) Models can assign probabilities to sequences by iteratively predicting a distribution over the next token given the previous tokens . Generating from language models requires iteratively sampling from Pθ ( xt|x < t ) , and then feeding xt back into the model as input for the next step . 2.2 CLASS-CONDITIONAL LANGUAGE MODELING . Class-conditional language models ( CC-LMs ) such as CTRL ( Keskar et al. , 2019 ) are a way for language models to generate while conditioning on an attribute variable . CC-LMs predict a probability distribution Pθ ( x1 : T |c ) , where c is a class variable or a “ control code ” that describes an attribute of the text in x1 : T , which could , for instance , describe sentiment or topic . The auto-regressive factorization for a CC-LM is given by the following equation : Pθ ( x1 : T |c ) = T∏ t=1 Pθ ( xt|x < t , c ) . ( 2 ) When training a CC-LM on a training set of sequences { x ( 1 ) 1 : T1 , . . . , x ( i ) 1 : Ti , . . . , x ( N ) 1 : TN } , each sequence x ( i ) 1 : T is paired with a control code c ( i ) , which is a label or category of the sequence . The LM is trained to minimize the average negative log-likelihood , L. L = − 1 N N∑ i=1 1 Ti Ti∑ t=1 logPθ ( x ( i ) t |x ( i ) < t , c ( i ) ) . ( 3 ) In addition to class-conditional generation , CC-LMs can be used as generative classifiers by applying Bayes rule to compute Pθ ( c|x1 : T ) ∝ P ( c ) Pθ ( x1 : T |c ) , as is done by Keskar et al . ( 2019 ) for source attribution . 3 GEDI . GeDi assumes we have a CC-LM with desired control code c and an undesired or anti-control code c̄ , and uses the contrast between Pθ ( x1 : t|c ) and Pθ ( x1 : t|c̄ ) to guide sampling from an LM that gives PLM ( x1 : t ) . Specifically , when predicting the next token during generation , GeDi uses this contrast to compute the probability that every candidate next token xt belongs to the desired class , given by Pθ ( c|xt , x < t ) . Our key insight is that this distribution can be computed very efficiently when using CC-LMs as GeDis via application of Bayes rule for partial sequences during generation . Pθ ( c|x1 : t ) = P ( c ) ∏t j=1 Pθ ( xj |x < j , c ) ∑ c′∈ { c , c̄ } ∏t j=1 P ( c ′ ) Pθ ( xj |x < j , c′ ) . ( 4 ) When computing this online during sequence generation , the model will have already computed Pθ ( xj |x < j , c′ ) for any j < t from the previous time-steps , and it will only need to compute Pθ ( xt|x < t , c′ ) . This can be computed in two parallel forward passes ; one conditioning on c and one conditioning on c̄ ( both conditioning on the same x < t ) . The model can also save the hidden states from the previous time steps to avoid computing a forward pass through the full sequence at each next token generation step . Applying a unidirectional classifier such as GPT ( Radford et al. , 2018 ) to compute Pθ ( c|xt , x < t ) directly ( i.e . discriminatively ) would require feeding in every possible input xt ∈ V into the classifier , and thus would require |V| forward passes for a vocab set V . A bidirectional classifier such as BERT ( Devlin et al. , 2018 ) would require t × |V| forward passes because it would need to recompute attention states from earlier time-steps . For typical vocab sizes of 20k+ , GeDi ’ s online classification trick can compute Pθ ( c|xt , x < t ) for every possible next token xt on the order of 10k fold less computation as compared with a unidirectional classifier ( because the unidirectional classifier would require 20k+ forward passes through the network , whereas the GeDi would only require 2 through the CC-LM and one through the base-LM ) . In practice , we find that applying Equation 4 to long sequences often results in poorly calibrated distributions later in the sequence that assign classification probabilities of 1 or 0 to on all candidate next words , which provides no useful signal . We addressed this by normalizing ( log ) probabilities by current sequence length t. To compute Pθ ( c|x1 : t ) for GeDi-guided generation , we use the following equation : Pθ ( c|x1 : t ) = ( Pθ ( x1 : t|c ) 1/t∑ c′∈ { c , c̄ } Pθ ( x1 : t|c′ ) 1/t . ( 5 ) where class priors P ( c ) are omitted . In practice , Pθ ( c|x1 : t ) is computed with log-probabilities ( see Appendix A ) . With the efficient estimation of Pθ ( c|xt , x < t ) , there are many possible heuristics that can be used to guide LM generation , so long as the LM and GeDi share the same tokenization . Heuristics that use Pθ ( c|xt , x < t ) inherently contrast predictions conditioned on c and c̄ , causing attributes common to c and c̄ to be cancelled out , more effectively allowing for the attribute described by c to be transferred across domains , as illustrated in Figure 1 . 3.1 HEURISTICS FOR GUIDING GENERATION . We consider a weighted decoding heuristic and a filtering heuristic to use Pθ ( c|xt , x < t ) to guide generation . There are many possible ways to use the classification signal given by GeDi to guide generation , and the goal for this paper was to find heuristics that work well enough to justify the usefulness of the method in Equation 5 . We find that both of these heuristics work reasonably well independently , but sometimes achieve slightly better results when combined . Our initial heuristic applies a weighted posterior given by Pw ( xt|x < t , c ) ∝ PLM ( xt|x < t ) Pθ ( c|xt , x < t ) ω , ( 6 ) where ω > 1 to bias generation more strongly towards the correct class . The right hand side of Equation ( 6 ) is normalized over all xt in the vocabulary to obtain Pw ( xt|x < t , c ) . We summarize the resulting scheme in Algorithm 1 . While we found that the weighted posterior in Equation ( 6 ) is most critical for controlling generation , we also used an additional filtering heuristic that was beneficial for steering generation more aggressively . This heuristic , inspired by nucleus sampling ( Holtzman et al. , 2020 ) , removes candidate next word tokens with lower values for Pθ ( c|xt , x < t ) while maintaining a minimum of at least 1 − ρ in cumulative probability mass in Pw ( xt|x < t , c ) , where 0 ≤ ρ < 1 is a parameter that decides the aggressiveness of the filtering . We define Vn as the set of n tokens with the highest Pθ ( c|xt , x < t ) . We define m as the minimum n such that∑ xt∈Vn Pw ( xt|x < t , c ) ≥ 1− ρ . ( 7 ) We define Vm as Vn for n = m , meaning that Vm will contain the minimum number of tokens possible at the head of the distribution for Pθ ( c|xt , x < t ) to maintain a minimum cumulative probability of ρ in Pw ( xt|x < t , c ) . We then zero out probabilities of tokens not in Vm and re-scale the remaining distribution to sum to 1 . Algorithm 1 Generative Discriminator Guided Sequence Generation Inputs : base LM PLM , CC-LM Pθ , vocabulary V , posterior mixing weight ω , decoding scheme 1 : P ( x|c ) ← 1 2 : P ( x|c̄ ) ← 1 3 : for t = 1 . . . , N do 4 : pLM ← [ PLM ( xt = v|x < t ) for v in V ] . base-LM prediction 5 : 6 : px1 : t|c ← [ ( P ( x|c ) Pθ ( xt = v|x < t , c ) ) 1/t for v in V ] . CC-LM with control code 7 : px1 : t|c̄ ← [ ( P ( x|c̄ ) Pθ ( xt = v|x < t , c̄ ) ) 1/t for v in V ] . CC-LM with anti control code 8 : 9 : pc|x1 : t ← px1 : t|c 1 ( px1 : t|c+px1 : t|c̄ ) 10 : 11 : pw ← pLM ( pc|x1 : t ) ω 12 : pw ← pw∑|V| i=1 pw [ i ] . normalize over the vocabulary 13 : vi ← Decode ( pw ) . Can be greedy or sampling , could include filtering in Eq . 7 14 : 15 : P ( x|c ) ← P ( x|c ) Pθ ( xt = vi|x < t , c ) 16 : P ( x|c̄ ) ← P ( x|c̄ ) Pθ ( xt = vi|x < t , c̄ ) 17 : xt ← vi

Summary: The paper proposed a method —- GeDi — to generate guided and controlled text from a large language model (LM). The method utilizes smaller LMs as generative discriminators to guide generation from large LMs to make them safer and more controllable. By safer and controllable they emphasis on the toxicity, hate, bias, and negativity contains in the training of the large LM. The proposed method guides generation at each time step by computing classification probabilities for all possible next tokens via Bayes rule by normalizing over two class-conditional distributions (i.e. contrastive discrimination); one conditioned on the desired attribute, or control code, and another conditioned on the undesired attribute (i.e. contrastive attribute), or anti-control code.