Year: 2020

Abstract:
While large-scale language models (LMs) are able to imitate the distribution of natural language well enough to generate realistic text, it is difficult to control which regions of the distribution they generate. This is especially problematic because datasets used for training large LMs usually contain significant toxicity, hate, bias, and negativity. We propose GeDi as an efficient method for using smaller LMs as generative discriminators to guide generation from large LMs to make them safer and more controllable. GeDi guides generation at each step by computing classification probabilities for all possible next tokens via Bayes rule by normalizing over two class-conditional distributions; one conditioned on the desired attribute, or control code, and another conditioned on the undesired attribute, or anti control code. We find that GeDi gives controllability on par with or better than the state of the art method in a variety of settings, while also achieving generation speeds more than 30 times faster. Additionally, training GeDi on only three topics allows us to controllably generate new topics zero-shot from just a keyword. Lastly, we show that GeDi can make GPT-2 and GPT-3 significantly less toxic without sacrificing on linguistic fluency, making it by far the most practical existing method for detoxifying large language models while maintaining a fast generation speed.

heading: 1 INTRODUCTION
text: Natural language generation has seen great progress with the advent of Transformers (Vaswani et al., 2017) and large scale training (Radford et al., 2017; 2018; 2019; Brown et al., 2020). Large language models (LMs) like GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020) are able to learn the distribution of their training set well enough to generate realistic text. However, simply imitating the distribution of the training data during generation has many drawbacks; large-scale text training sets are crawled from the web which is imbued with toxicity, bias, hate, and misinformation. Methods for better controlling or filtering generation are valuable for making LMs trained on such data safer and more generally useful for downstream applications.
Existing approaches to controlling LMs have limitations. Class-conditional LMs (CC-LMs) such as CTRL (Keskar et al., 2019) attempt to control text generation by conditioning on a control code, which is an attribute variable representing a data source. However, CTRL is not as useful for controlling what not to generate (i.e. toxicity). Furthermore, using a specific control code can reduce sample diversity across prompts, as samples will generally resemble the data source of the control code. Another approach is to use discriminators to steer generation, but existing methods to do this are very computationally intensive. Weighted decoding (Holtzman et al., 2018) requires feeding candidate next tokens into a discriminator, and thus scales linearly in computation with the number of tokens to be re-weighted. Plug and Play LM (Dathathri et al., 2020, PPLM) applies up to 10 updates to the generating LM’s latent states per time step using gradients from a discriminator, also making it many times slower than generating from the LM directly.
We present GeDi1 as an algorithm for efficiently guiding generation from large LMs to make them safer and more controllable. Our proposed method uses CC-LMs as generative discriminators (GeDis) to guide language generation towards desired attributes. We use GeDis to compute classification likelihoods for all candidate next tokens during generation using Bayes rule, saving many
1pronounced “Jedi”
thousand-fold in computation as compared with using a standard (non-generative) discriminator to compute this for large vocabulary sizes. We then show how these likelihoods can guide generation from large language models via weighted decoding and filtering.
Our experimental results verify the ability of GeDi to control generation in a variety of settings while maintaining linguistic quality on par with strong language models. We apply GeDi (345M parameters) to guide generation from larger language models, and find that:
• GeDi trained on sentiment of movie reviews can generate book text with a positive or negative tone better than or equivalently to state of the art baselines [Section 5.1]. Guiding towards positivity also has potential applications towards making LMs friendlier.
• GeDi is able to significantly reduce the toxicity of GPT-2 and GPT-3 generation [Section 5.2], without sacrificing linguistic quality as compared with generating from GPT-2 and GPT-3 directly, suggesting applications towards safer language modeling.
• GeDi trained on a dataset of only 4 topics can generalize to new control codes zero-shot [Section 5.3], allowing them to guide generation towards a wide variety of topics.
• GeDi is very computationally efficient for both training and inference. GeDi guided generation in our experiments is more than 30× faster than applying PPLM with GPT2-XL using default settings from Dathathri et al. (2020). Additionally, smaller GeDis fine-tuned for less than a day on a single GPU are effective and computationally efficient for controlling larger language models. This provides a cheap alternative to finetuning large LMs directly (Ziegler et al., 2019).

heading: 2 BACKGROUND

heading: 2.1 LANGUAGE MODELING
text: Language models (LMs) rely on an auto-regressive factorization to perform density estimation and generation of language data. Auto-regressive sequence models with parameters θ assign a probability to a sequence x1:T = {x1, . . . , xT } by factorizing it using the chain rule as follows:
Pθ(x1:T ) = T∏ t=1 Pθ(xt|x<t). (1)
Models can assign probabilities to sequences by iteratively predicting a distribution over the next token given the previous tokens. Generating from language models requires iteratively sampling from Pθ(xt|x<t), and then feeding xt back into the model as input for the next step.

heading: 2.2 CLASS-CONDITIONAL LANGUAGE MODELING
text: Class-conditional language models (CC-LMs) such as CTRL (Keskar et al., 2019) are a way for language models to generate while conditioning on an attribute variable. CC-LMs predict a probability distribution Pθ(x1:T |c), where c is a class variable or a “control code” that describes an attribute of the text in x1:T , which could, for instance, describe sentiment or topic. The auto-regressive factorization for a CC-LM is given by the following equation:
Pθ(x1:T |c) = T∏ t=1 Pθ(xt|x<t, c). (2)
When training a CC-LM on a training set of sequences {x(1)1:T1 , . . . , x (i) 1:Ti , . . . , x (N) 1:TN }, each sequence x (i) 1:T is paired with a control code c
(i), which is a label or category of the sequence. The LM is trained to minimize the average negative log-likelihood, L.
L = − 1 N N∑ i=1 1 Ti Ti∑ t=1 logPθ(x (i) t |x (i) <t, c (i)). (3)
In addition to class-conditional generation, CC-LMs can be used as generative classifiers by applying Bayes rule to compute Pθ(c|x1:T ) ∝ P (c)Pθ(x1:T |c), as is done by Keskar et al. (2019) for source attribution.

heading: 3 GEDI
text: GeDi assumes we have a CC-LM with desired control code c and an undesired or anti-control code c̄, and uses the contrast between Pθ(x1:t|c) and Pθ(x1:t|c̄) to guide sampling from an LM that gives PLM (x1:t). Specifically, when predicting the next token during generation, GeDi uses this contrast to compute the probability that every candidate next token xt belongs to the desired class, given by Pθ(c|xt, x<t). Our key insight is that this distribution can be computed very efficiently when using CC-LMs as GeDis via application of Bayes rule for partial sequences during generation.
Pθ(c|x1:t) = P (c) ∏t j=1 Pθ(xj |x<j , c)∑
c′∈{c,c̄} ∏t j=1 P (c ′)Pθ(xj |x<j , c′) . (4)
When computing this online during sequence generation, the model will have already computed Pθ(xj |x<j , c′) for any j < t from the previous time-steps, and it will only need to compute Pθ(xt|x<t, c′). This can be computed in two parallel forward passes; one conditioning on c and one conditioning on c̄ (both conditioning on the same x<t). The model can also save the hidden states from the previous time steps to avoid computing a forward pass through the full sequence at each next token generation step. Applying a unidirectional classifier such as GPT (Radford et al., 2018) to compute Pθ(c|xt, x<t) directly (i.e. discriminatively) would require feeding in every possible input xt ∈ V into the classifier, and thus would require |V| forward passes for a vocab set V . A bidirectional classifier such as BERT (Devlin et al., 2018) would require t × |V| forward passes because it would need to recompute attention states from earlier time-steps. For typical vocab sizes of 20k+, GeDi’s online classification trick can compute Pθ(c|xt, x<t) for every possible next token xt on the order of 10k fold less computation as compared with a unidirectional classifier (because the unidirectional classifier would require 20k+ forward passes through the network, whereas the GeDi would only require 2 through the CC-LM and one through the base-LM).
In practice, we find that applying Equation 4 to long sequences often results in poorly calibrated distributions later in the sequence that assign classification probabilities of 1 or 0 to on all candidate next words, which provides no useful signal. We addressed this by normalizing (log) probabilities by current sequence length t. To compute Pθ(c|x1:t) for GeDi-guided generation, we use the following equation:
Pθ(c|x1:t) = (Pθ(x1:t|c)1/t∑
c′∈{c,c̄} Pθ(x1:t|c′)1/t . (5)
where class priors P (c) are omitted. In practice, Pθ(c|x1:t) is computed with log-probabilities (see Appendix A). With the efficient estimation of Pθ(c|xt, x<t), there are many possible heuristics that
can be used to guide LM generation, so long as the LM and GeDi share the same tokenization. Heuristics that use Pθ(c|xt, x<t) inherently contrast predictions conditioned on c and c̄, causing attributes common to c and c̄ to be cancelled out, more effectively allowing for the attribute described by c to be transferred across domains, as illustrated in Figure 1.

heading: 3.1 HEURISTICS FOR GUIDING GENERATION
text: We consider a weighted decoding heuristic and a filtering heuristic to use Pθ(c|xt, x<t) to guide generation. There are many possible ways to use the classification signal given by GeDi to guide generation, and the goal for this paper was to find heuristics that work well enough to justify the usefulness of the method in Equation 5. We find that both of these heuristics work reasonably well independently, but sometimes achieve slightly better results when combined. Our initial heuristic applies a weighted posterior given by
Pw(xt|x<t, c) ∝ PLM (xt|x<t)Pθ(c|xt, x<t)ω, (6) where ω > 1 to bias generation more strongly towards the correct class. The right hand side of Equation (6) is normalized over all xt in the vocabulary to obtain Pw(xt|x<t, c). We summarize the resulting scheme in Algorithm 1.
While we found that the weighted posterior in Equation (6) is most critical for controlling generation, we also used an additional filtering heuristic that was beneficial for steering generation more aggressively. This heuristic, inspired by nucleus sampling (Holtzman et al., 2020), removes candidate next word tokens with lower values for Pθ(c|xt, x<t) while maintaining a minimum of at least 1 − ρ in cumulative probability mass in Pw(xt|x<t, c), where 0 ≤ ρ < 1 is a parameter that decides the aggressiveness of the filtering. We define Vn as the set of n tokens with the highest Pθ(c|xt, x<t). We define m as the minimum n such that∑
xt∈Vn
Pw(xt|x<t, c) ≥ 1− ρ. (7)
We define Vm as Vn for n = m, meaning that Vm will contain the minimum number of tokens possible at the head of the distribution for Pθ(c|xt, x<t) to maintain a minimum cumulative probability of ρ in Pw(xt|x<t, c). We then zero out probabilities of tokens not in Vm and re-scale the remaining distribution to sum to 1.
Algorithm 1 Generative Discriminator Guided Sequence Generation Inputs: base LM PLM , CC-LM Pθ, vocabulary V , posterior mixing weight ω, decoding scheme
1: P (x|c)← 1 2: P (x|c̄)← 1 3: for t = 1 . . . , N do 4: pLM ← [PLM (xt = v|x<t) for v in V] . base-LM prediction 5: 6: px1:t|c ← [(P (x|c)Pθ(xt = v|x<t, c))1/t for v in V] . CC-LM with control code 7: px1:t|c̄ ← [(P (x|c̄)Pθ(xt = v|x<t, c̄))1/t for v in V] . CC-LM with anti control code 8: 9: pc|x1:t ← px1:t|c 1(px1:t|c+px1:t|c̄)
10: 11: pw ← pLM (pc|x1:t)ω 12: pw ← pw∑|V|
i=1 pw[i] . normalize over the vocabulary
13: vi ← Decode(pw) . Can be greedy or sampling, could include filtering in Eq. 7 14: 15: P (x|c)← P (x|c)Pθ(xt = vi|x<t, c) 16: P (x|c̄)← P (x|c̄)Pθ(xt = vi|x<t, c̄) 17: xt ← vi

heading: 4 RELATED WORK
text: Methods for controlling text generation can be categorized broadly into two types: training or finetuning a model directly for controllable generation (Keskar et al., 2019; Ziegler et al., 2019; Rajani
et al., 2019; Ficler & Goldberg, 2017; Yu et al., 2017; Hu et al., 2017) or using a discriminator to guide generation (Ghazvininejad et al., 2017; Holtzman et al., 2018; Dathathri et al., 2020). Keskar et al. (2019) train a CC-LM with pre-defined control codes placed at the start of every sequence. Our approach also uses CC-LMs, but instead of generating from them directly, we use them as discriminators to guide generation from another language model. This is much more computationally efficient than previous methods for discriminator guided generation. Holtzman et al. (2018) apply discriminators to re-weight a beam search, requiring all candidate tokens to be passed through the discriminator, scaling linearly with the number of re-scored tokens. PPLM (Dathathri et al., 2020) trains an attribute model on top of a language model’s last hidden layer and backpropagates gradients to update the hidden states of the model. This is computationally intensive, especially when applying to large LMs, because it requires multiple forward and backward passes for each generation step.
Our approach also relates to the rational speech acts framework for computational pragmatics (Frank & Goodman, 2012; Goodman & Stuhlmüller, 2013) where a “listener” model and a “speaker” model interactively generate a sequence such that the listener can recover the input. GeDi most closely relates to distractor based pragmatics (Andreas & Klein, 2016; Cohn-Gordon et al., 2018; Shen et al., 2019), where a single model processes a true input and a distractor input, and uses Bayes rule to produce text that fits the true input but not the distractor input. In GeDi, the base language model (the speaker) is guided by the GeDi (the listener), to produce text where the GeDi can recover the desired attribute. GeDi differs from previous distractor based pragmatics based approaches in that it trains a separate class-conditional language model on a single attribute, allowing that attribute to be isolated, and uses it to guide generation from a separate larger language model.

heading: 5 EXPERIMENTS
text: We experiment with GeDi-guided generation for sentiment, detoxification, and topic control. Our experiments finetune GPT2-medium (345M parameter) (Radford et al., 2019) using the loss in Equation 3 with control codes specific to each task to form a class-conditional language model. We use these CC-LMs as GeDis to guide generation from GPT2-XL (1.5B parameter) and GPT-3 (Brown et al., 2020) in our detoxification experiments. These experiments were performed using adaptations of Huggingface Transformers (Wolf et al., 2019).
For generation, we include experiments with greedy decoding with a repetition penalty (Keskar et al., 2019) (conditioning on varying prompts to give diversity across samples) and top-p sampling (Holtzman et al., 2020). We focus primarily on the greedy decoding setting because we found it to give higher quality samples. However, we do include top-p sampling models for GPT-2, PPLM, and GeDi for our sentiment and detoxification experiments to evaluate GeDi in this setting. Additional details about the way we apply a repetition penalty are given in Appendix B, and our hyperparameter settings for GeDi-guided generation are given in Appendix C.1.

heading: 5.1 CONTROLLING SENTIMENT OF GENERATIONS FROM BOOK PROMPTS
text: We experiment with GeDi-guided generation from GPT-2 for sentiment control. For these experiments, we use CC-LMs finetuned on IMDb movie reviews. We noticed that, while direct generation from CC-LMs could effectively control the sentiment of movie reviews, it struggled to generalize to out-of-domain prompts, and would generally try to convert prompts into movie reviews. However, when we used this same model as a GeDi to guide sampling from GPT-2, we were able to effectively control the sentiment of a wide variety of topics.
To experimentally verify that GeDi can generalize the concepts of “positivity” and “negativity” beyond its training domain, we evaluate on a task where the models conditionally generate text from the start of book chapters from Bookcorpus (Zhu et al., 2015), where each prompt is at least 150 characters and ends on the first-word break after the minimum length. We run human evaluation on generations from 50 different book prompts from 14 different models; including raw GPT2-XL with both topp sampling (p = 0.9) and greedy decoding (repetition
penalty=1.2), and the following models with both positive and negative sentiment: 1. GPT2-XL guided by GeDi, greedy decoding plus repetition penalty of 1.2. 2. GPT2-XL guided by GeDi, top-p sampling with p = 0.9 plus repetition penalty of 1.05. 3. GPT2-XL guided by PPLM, greedy decoding plus repetition penalty of 1.2. 4. GPT2-XL guided by PPLM, top-p sampling with p = 0.9. 5. CC-LM trained on movie reviews (same model used as GeDi, but with direct CTRL-style generation), greedy decoding plus repetition penalty of 1.2. 6. CTRL (Keskar et al., 2019) conditioned on positive and negative control codes for Amazon review sentiment, greedy decoding plus repetition penalty of 1.2. See Appendices C.2 and C.3 for additional information about our PPLM and CTRL baselines respectively. We found that it was more than 30× faster to guide GPT2-XL with a GeDi as compared with PPLM (assuming 10 update steps as used in Dathathri et al. (2020) and in our experiments), as shown in Table 1.
Amazon Mechanical Turk annotators rated the generated text on sentiment, how book-like the text was, fluency, and whether or not the text resembled an Amazon review or movie review (since CTRL was trained on Amazon reviews and GeDi was trained on movie reviews). 3 annotations were collected on each sample, and each annotator was randomly assigned samples from the set of all generations from all models. To optimize the quality of annotations, we require all the annotators to have Mechanical Turk Masters Qualification2 along with requiring them to be located in the US, having more than 97% task approval rate and having completed more than 10000 tasks. Exact instructions given to annotators are given in Appendix G. The results of the experiment are given in Table 2. Using a CC-LM to guide GPT2-XL was able to generate book-like and linguistically fluent text while strongly control the tone. In the greedy setting, GeDi was also able to give roughly equivalent positive sentiment control and statistically significantly stronger negative sentiment control compared with PPLM. In the top-p setting, GeDi achieved stronger sentiment control for both positive and negative sentiment.
CTRL struggled to control tone/sentiment in this setting because its training domain for sentiment was Amazon reviews, and direct generation from the CC-LMs that we used as GeDis failed to generate book-like text because their training domain was movie reviews. We provide examples of generations from all models on book prompts in Appendix F.1. According to our annotators, 27% of CTRL samples resembled Amazon reviews, and 61% of CC-LM samples resembled movie reviews (Amazon and movie review resemblance percentages were less than 5% for samples from all other models). Table 14 specifically shows how CTRL tends to generate Amazon reviews and how CCLMs tend to generate movie reviews. This is a critical drawback of CTRL-style generation – the model can only reliably generate text and control attributes within the training domain corresponding to the control code.
Discriminator-guided methods GeDi and PPLM result in text rated more book-like that very rarely if ever diverts back to the domain that the discriminator was trained on. The most significant advantage of GeDi over PPLM is that it is able to generate 30× faster. GeDi was also able to give statistically significantly stronger positive sentiment control for top-p sampling, and negative sentiment control for both greedy and top-p sampling. For a detailed overview of p-values for significance test, see Appendix E.

heading: 5.2 DETOXIFYING GPT-2 AND GPT-3
text: We test GeDi’s ability to detoxify language generation. We train a CC-LM on the Jigsaw Toxic Comment Classification Challenge Dataset3, which contains text samples labeled as “toxic” or “nontoxic”. The “toxic” label indicates the presence of profanity, obscenity, threats, insults, or identity hate. We train the model on an even split of toxic and non-toxic examples, with a “clean” and “dirty” control code to specify toxic and non-toxic text.
For evaluation, we use prompts from Real Toxicity Prompts (Gehman et al., 2020). To identify strong triggers, we select a subset of prompts with perspective API4 toxicity probabilities between 0.3 and 0.5, that also were classified as non-toxic by a RoBERTa toxicity classifier trained on the Jigsaw dataset. We use GPT-2-XL to draw 32 samples from each prompt, and select the 100 prompts with the highest average toxicity probability over their 32 completions according to the RoBERTa
2https://www.mturk.com/worker/help#what_is_master_worker 3 https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/ 4https://www.perspectiveapi.com/
toxicity classifier. Our goal with this procedure was to identify prompts that are non-toxic, but have a high probability of causing language models to generate toxic text. Using these 100 prompts, we evaluate generation from 9 models:
1. GPT2-XL greedy decoding plus repetition penalty. 2. GPT2-XL guided by GeDi, greedy decoding plus repetition penalty of 1.2. 3. GPT2-XL guided by PPLM, greedy decoding plus repetition penalty of 1.2. 4. GPT2-XL top-p sampling with p = 0.9. 5. GPT2-XL guided by GeDi, top-p sampling with p = 0.9. 6. GPT2-XL guided by PPLM, top-p sampling with p = 0.9. 7. CC-LM trained on Jigsaw, conditioning on “clean” control code (same model used as GeDi, but with direct CTRL-style generation), greedy plus repetition penalty of 1.2. 8. GPT3 using Open AI API, greedy plus repetition penalty of 1.2. 9. GPT3 using Open AI API, guided by GeDi, greedy plus repetition penalty of 1.2.
For results with GPT-3, we use the Da Vinci model from the Open AI API 5, which can give up to 100 next token log probabilities for any next token prediction. We controlled GPT-3 decoding by passing the API a prompt, selecting the next token using the top 100 log-probabilities, and then passing a new prompt that has the selected token appended to the end. This limitation means that we can only re-weight the top 100 tokens; we treat all other tokens as having zero probability and normalize the top 100 at each prediction to sum to 1. There is no way to apply PPLM to the GPT-3 API, since PPLM requires access to hidden states and gradient.
We run human evaluation to measure toxicity and linguistic fluency [1: very low fluency, 5: very high fluency], using the same annotator criteria as in the previous section, and using instructions given in Appendix G. We also compute perplexity scores (using GPT-2-XL) and automatic toxicity scores using the RoBERTa toxicity classifier trained on Jigsaw. Results are given in Table 3. GeDi was able to significantly reduce the toxicity in GPT-2, and perform on par with PPLM for greedy decoding and better than PPLM for sampling, while also achieving 30× faster generation speeds (PPLM and GeDi for detoxification have the same respective computational costs as in the previous experiment with sentiment).
5https://openai.com/blog/openai-api/
We also found that GeDi produced fewer toxic generations than CC-LM (although this was not statistically significant, p = 0.13), while achieving significantly higher fluency (p = 0.0003). The finetuning needed for CC-LM tends to result in catastrophic forgetting of the information that the model learned during pretraining – for instance in the previous section we measured that CC-LM would usually generate movie reviews (generations resembled movie reviews 61 % of the time) – and here we noticed that the CC-LM is biased towards shorter generations that often resemble internet comments from the Jigsaw dataset.
We also found GeDi was able to detoxify GPT-3–making it the only practical existing method for detoxifying GPT-3 due to the API limitations. For a detailed overview of significance tests, see Appendix E, and for examples of GeDi samples used in this experiment, see Appendix F.2.

heading: 5.3 EXTENDING GEDI TO THE MUTLI-CLASS SETTING
text: To extend GeDi to the multi-class setting, we propose reframing each classification task as binary classification using control codes and anti control codes for each class. The control code for each class is given by “true” concatenated with the class name, and the anti-control code is given by “false” concatenated with the class name. The CC-LM then classifies whether the class name corresponds to the text. For instance, the CC-LM would process the following two sequences in parallel:
<true> <science> T-rex achieved its massive size due to an enormous growth spurt during its
adolescent years.
<false> <science> T-rex achieved its massive size due to an enormous growth spurt during
its adolescent years.
and would classify it as true or false as to whether the class (in this case “science”) matches the category of the text by using Equation (5). During training, the model sees an equal number of true pairings (where text corresponds to class) and randomly chosen false pairings. After the model has been trained, binary GeDi-guided generation can be applied, using c =<true> and c̄ =<false>, and using the desired class name as the first token (x1) in the sequence. This also makes it possible to form new control codes zero-shot; a new topic word that was never seen before in training can be chosen in place of x1.
To experiment with multi-class GeDi, we use the AG news topic classification data set (Zhang et al., 2015) which has 4 topics (World, Sports, Business, and Science/Tech). In order to test GeDi’s ability to generate never seen before classes zero-shot, we trained 4 different CC-LMs; each one is trained on only 3 out of 4 of the AG news classes, with one class held out. We then compare the effect of direct (CTRL-style) generation from CC-LMs and GeDi-guided generation from GPT-2, on both training topics and held out (zero-shot) topics. To evaluate topic relevance, we use a RoBERTa classifier trained on all 4 AG news topics to estimate the topic of generation. We compare with a raw GPT-2 baseline to see how much generation is being influenced towards each topic. We obtain generations conditioning on short (minimum 30 character, ending on a space) prompts from the multi-news data-set (Fabbri et al., 2019), and report results in Table 4.
We find that GeDi-guided generation is able to generate training topics with a higher label fidelity than CTRL-style generation from a CC-LM. We also find that unlike CC-LM, GeDi is able to bias generation towards never seen before zero-shot control codes that are held out from training. GeDi’s ability to generalize to new control codes zero-shot gives the ability to generate text corresponding to many topics and subtopics. This ability likely emerges because generative classifiers can classify unseen topics zero-shot from learned word embeddings (Yogatama et al., 2017), and GeDi uses generative classifiers to guide generation. We provide examples of zero-shot generation with GeDi from many control codes in Appendix F.3.

heading: 6 FUTURE DIRECTIONS
text: Methods to make large LMs like GPT-3 safer and more controllable are becoming especially important as LMs become incorporated into products. GeDi is by far the most practical existing method for detoxifying generation from large LMs, since it only uses a small constant amount of computational overhead and only requires access to the LM’s next token log probabilities. With the right training data for classification, GeDi could also potentially be used to filter out harder to detect forms of toxicity such as bias and misinformation. Extending on the methods in this paper, multiple GeDis trained to filter out different undesirable attributes could be combined, for instance by multiplying the attribute classification terms from several different discriminators in Equation 6. In additional to making LMs safer, GeDi could potentially be used to guide generation towards other desirable attributes such as high linguistic quality and improved commonsense reasoning. Lastly, GeDi-inspired methods could be explored as much more computationally efficient alternatives to fine-tuning large LMs to new generation tasks.

heading: 7 CONCLUSION
text: We present GeDi as an approach for controllable generation that uses generative discriminators to classify candidate next tokens on the fly during inference, making it far more efficient than previous methods that use discriminators to guide generation. GeDi achieves stronger controllability of sentiment than PPLM while also giving a generation speed more than 30× faster. GeDis trained on 3 topics can also controllably generate new topics zero-shot from just a key word. We also show that GeDi is able to significantly reduce the toxicity of GPT-2 and GPT-3 without sacrificing noticeably on linguistic fluency. This work also moves towards unifying natural language generation with classification, and suggests that we may be able to efficiently generate text that corresponds to any attribute that we can accurately classify. This could have broad implications towards improving text generation systems by making them safer and more controllable.

heading: A GEDI WITH LOG PROBABILITIES
text: GeDi-guided generation uses language models discriminatively via Bayes rule by using
Pθ(c|x1:T ) = P (c)Pθ(x1:T |c)1/T∑ c′ P (c ′)Pθ(x1:T |c′)1/T , (8)
where P (c) is assumed to be uniform. Log-probabilities for each class are given by
logPθ(x1:T |c) = T∑ t=1 logPθ(xt|x<t, c), (9)
and the class probability is given by
Pθ(c|x1:T ) = e((1/T ) logPθ(x1:T |c))∑ c′ e ((1/T ) logPθ(x1:T |c′)) . (10)
This can be computed in a numerically stable way using a softmax (Bridle, 1990), since the maximum logit to the softmax can be subtracted out before taking the exponent without changing the result. For the two class case, c′ ∈ {c, c̄}, meaning that the above equation could have been equivalently computed using a sigmoid of the difference of the logs of the two terms in the denominator sum (but our implementation used softmax as above).

heading: B GENERATION SETTINGS
text: When comparing the quality of samples from different language models, there is a trade-off between quality and diversity; models that tend to have more sharply peaked distributions for Pθ(xt|x<t, c) will tend to have higher quality samples, but will also have less diversity. Applying GeDi results in more sharply peaked distributions due to the filtering step, which zeros out probabilities for some tokens. In order to ensure a fair comparison of models, we mainly consider deterministic decoding for our experiments. We use greedy decoding with a repetition penalty (Keskar et al., 2019), meaning we always pick the most likely token in the model’s predictive distribution after applying a penalty to words that have previously occurred in the sequence. With deterministic decoding, the model would generate the same text sequence every time without any conditioning text. Therefore, all experiments in our paper rely on varying prompts to ensure diversity of generation.
We found the repetition penalty to be necessary for preventing degeneration with greedy decoding. Logits of each previously occurring word in the sequence are divided by a repetition penalty which is greater than 1. To account for the possibility of negative logits, we re-scaled the final logits in all models to always have a maximum of 10 across the vocabulary before dividing by the repetition penalty.

heading: C ADDITIONAL MODEL AND HYPER-PARAMETER DETAILS
text: C.1 HYPER-PARAMETERS FOR GEDI GUIDED GENERATION
GeDi used ρ = 0.7 and ω = 20 for sentiment, ρ = 0.8 and ω = 30 for GPT-2 detoxification, ρ = 0.8 and ω = 90 for GPT-3 detoxification (since GPT-3 is limited to the top 100 LM logits, steering needs to be more aggressive), and ρ = 0.8 and ω = 150 for topic control.
C.2 BASELINE DETAILS FOR PPLM
For PPLM, we trained the external classifier (which uses logistic regression on top of representations from GPT-2) on the SST-5 data set, after struggling to achieve as strong results training on IMDb (which is what GeDi was trained on) and advise from the paper authors. For generation, we used greedy decoding with a repetition penalty applied the same way as described in Appendix B. We applied additional tuning to hyper-parameters because we were guiding generation from GPT2-XL (whereas original PPLM work uses GPT2-medium). Starting from the default hyper-parameters in the repository, we considered step sizes in the set {0.04, 0.08, 0.16, 0.25, 0.35}, and found that 0.25
gave the best trade-off between sentiment control and generation quality, so we used this for our experiments. Similarly, for detoxification we tried the stepsizes in {0.10, 0.20, 0.40} and chose 0.20 to minimize toxicity while maintaining fluency (low perplexity).
C.3 BASELINE DETAILS FOR CTRL
For CTRL, we prepended prompts with the control codes for positive and negative Amazon reviews, which are “Reviews Rating: 1.0” and “Reviews Rating: 5.0” for negative and positive respectively. We also tried “Books Rating:” as a prompt that mixes the control code for sentiment and books, however we found that there was very little variation in the samples generated by positive and negative (generation was usually identical for several sentences before deviating), and no noticeable impact on sentiment, tone, or mood.

heading: D ABLATION STUDIES
text: We examine the effects of the filtering and weighted decoding methods described in Section 3.1 for sentiment control and detoxification. We consider both heuristics independently, as well as the combination of these two heuristic used in all experiments in the paper. For the weighted decoding setting, we set ρ = 0 which turns off filtering, and tune ω to give a similar perplexity score to the combined heuristic (higher ω results in more aggressive steering and generally gives a worse perplexity score and higher label fidelity). For the filtering setting, we set ω = 0 to turn off weighted decoding, and tune ρ to give a similar perplexity score to the combined heuristic (higher ρ results in more aggressive filtering and generally gives a worse perplexity score and higher label fidelity). Results are given in Table 6 for detoxification and Table 5 for sentiment. Both heuristics are able to control generation on there own, but the combined heuristic appears to perform slightly better for detoxification, and may be more robust to settings where one heuristic or the other do not work as well in isolation.

heading: E STATISTICAL SIGNIFICANCE TABLES FOR HUMAN EVALUATION
text: EXPERIMENTS

heading: F GENERATION SAMPLES
text: F.1 SENTIMENT/TONE SAMPLES
F.2 DETOXIFICATION SAMPLES
F.3 ZERO-SHOT CONTROL CODES
G MECHANICAL TURK INTERFACE

Review summaries:
In this paper, the authors propose an efficient method for controllable language generation of large pre-trained LMs (e.g., GPT2). The main idea is to use a smaller, compared to the LM to control, language model trained with control code (Keskar et al., 2019) to generate a per-token score $P(c|x_{1:t}$) to steer the original Language Model distribution. The author proposes two ways, contrastive and discriminative, to approximate $P(c|x_{1:t})$ using bayesian rules. Experiments on open-ended language generation have been shown for positive/negative, detoxification, and topic-control, including a zero-shot topic-control. 

The paper proposed a method —- GeDi — to generate guided and controlled text from a large language model (LM). The method utilizes smaller LMs as generative discriminators to guide generation from large LMs to make them safer and more controllable. By safer and controllable they emphasis on the toxicity, hate, bias, and negativity contains in the training of the large LM. The proposed method guides generation at each time step by computing classification probabilities for all possible next tokens via Bayes rule by normalizing over two class-conditional distributions (i.e. contrastive discrimination); one conditioned on the desired attribute, or control code, and another conditioned on the undesired attribute (i.e. contrastive attribute), or anti-control code. 

The paper considers the problem of attribute-based sequence generation, particularly in language models. Authors propose a framework “GeDi” which learns a generative classifier for controlling generation from a large language model. With experiments on publicly available datasets and models, and including human-evaluation, the authors empirically demonstrate that the algorithm is computationally efficient and is competitive against strong baseline algorithms like CTRL, Plug&Play language models (PPLM).