PREADD: Prefix-Adaptive Decoding for Controlled Text Generation
Year: 2023

Abstract:
We propose Prefix-Adaptive Decoding (PREADD), a flexible method for controlled text generation. Unlike existing methods that use auxiliary expert models to control for attributes, PREADD does not require an external model, instead relying on linearly combining output logits from multiple prompts. Specifically, PREADD contrasts the output logits generated using a raw prompt against those generated using a prefix-prepended prompt, enabling both positive and negative control with respect to any attribute encapsulated by the prefix. We evaluate PREADD on three tasks—toxic output mitigation, gender bias reduction, and sentiment control—and find that PREADD outperforms not only prompting baselines, but also an auxiliary-expert control method, by 12% or more in relative gain on our main metrics for each task. CONTENT WARNING: Some example model outputs contain highly offensive or disturbing text.

heading: 1 Introduction
text: The dramatic rise in applications relying on language models has led to increased interest in methods for controlling their generations based on desired constraints. For example, it is desirable to prevent models from generating toxic or harmful text,1 as they are often prone to doing (Gehman et al., 2020; Bender et al., 2021), especially in the presence of toxic prompts. To this end, prior work has proposed many viable control schemes, ranging from prompting with instructions to specify a constraint (Ouyang et al., 2022), to using an auxiliary expert model to guide generation (Dathathri et al., 2019; Yang and Klein, 2021).
However, for important practical tasks such as toxic output mitigation and gender bias reduction
1In this work, we define toxic language as perpetuating negative stereotypes, being threatening or sexually explicit, or containing profane language.
requiring control against an undesired attribute, prompting-only methods may struggle (Welbl et al., 2021), as we observe in our own experiments (Section 4). In failure cases, it is unclear how to adjust control strength when relying solely on prompting. Approaches using auxiliary models may be advantageous in this respect, but auxiliary models impose an additional burden in practice, typically requiring training data. Additionally, prompting approaches may naturally improve as the base language model improves, which is not necessarily the case when relying on an auxiliary model for control.
In this work, we propose Prefix-Adaptive Decoding (PREADD), a prompting-only control scheme that enables adjusting control strength (Figure 1). PREADD operates by contrasting the token logits at each step of generation when using either (1) a prefix-prepended version of a prompt, or (2) the raw unmodified prompt. The difference between logit distributions can then be amplified or negated to vary the control strength, as required for the task.
We evaluate PREADD on toxic output mitigation and gender bias reduction, two tasks which require “negative” control against an undesirable attribute. We believe PREADD offers the largest advantage over traditional prompting approaches in such settings. On these two tasks, PREADD significantly improves over prompting-only baselines and also an auxiliary-model control method by 12% or more in relative improvement on our main metrics for each task. Meanwhile, PREADD still maintains strong performance on “positive” control tasks such as sentiment control.
All code is available at https://github.com/ jonnypei/acl23-preadd.

heading: 2 Related Work
text: Prior works have attempted to control language model outputs through a variety of methods.
Prompting. Prompting approaches have become
10018
increasingly popular as language models improve. Prompts may be manually designed (Brown et al., 2020) or automatically designed (Shin et al., 2020; Zou et al., 2021); prompting may also be an iterative process (Wei et al., 2022). Perhaps most similar to our work are methods which also compare two sets of output logits while prompting (Schick et al., 2021; Zhao et al., 2021; Li et al., 2022). Compared to prior work, our contribution is a prompting-based method with freely adjustable control strength, designed explicitly for controlling generation based on flexibly specified constraints.
Auxiliary Models. Meanwhile, control schemes using auxiliary models for a desired attribute typically provide avenues to adjust the strength of control as needed. While some methods require deeper access to the base language model, such as gradients (Dathathri et al., 2019), others require only the output token logits at each decoding step (Krause et al., 2021; Yang and Klein, 2021; Liu et al., 2021). However, auxiliary models may require additional training data to learn the desired attribute, unlike prompting methods such as PREADD. Methods such as finetuning or reinforcement learning from human feedback (Bai et al., 2022) may also use additional training data to modify the model distribution at training time rather than inference time.

heading: 3 Prefix-Adaptive Decoding
text: We now motivate and develop our method. Suppose we want to control the output of a language model G, which generates tokens x1 . . . xn left-to-right by modeling P (xi+1 | x1:i). One possible method is prompting: we prepend a prefix r1:k to x1:i, modeling xi+1 according to P (xi+1 | r1:k, x1:i).
While prompting is lightweight and flexible, we may wish to adjust the control strength when working with highly complex constraints. Some constraints may also be difficult to express effectively as a prompt: for example, simply stating that the
model should not generate text of a particular attribute is often ineffective (Section 4).
We thus develop our method for controlled generation, Prefix-Adaptive Decoding (PREADD, Figure 1), to allow varying the strength of control during prompting. One can view prompting as modulating the log-probabilities logP (xi+1 | x1:i) by adding the difference in log-probabilities:
d := logP (xi+1 | r1:k, x1:i)− logP (xi+1 | x1:i)
Intuitively, adding d increases the likelihood of tokens relevant to the prompt, while leaving that of unrelated tokens (e.g., stopwords) largely unchanged. Applying a multiplier α to d may therefore enable us to vary control strength while preserving fluency. Thus PREADD models the logit of xi+1 as:
logP (xi+1 | x1:i) + αd
Converting back to normal probability space and re-expanding d, we obtain the final probability for xi+1 according to PREADD, proportional to:
P (xi+1 | r1:k, x1:i)αP (xi+1 | x1:i)1−α
PREADD strictly generalizes prompting; the latter is equivalent to α = 1. Unlike prompting, PREADD can control more strongly for the constraint expressed in the prompt by using larger values of α, or can provide negative control against the constraint expressed in the prompt by using negative values of α. We explore both cases in our experiments (Section 4)

heading: 3.1 PREADD With Prompting
text: While we have described PREADD as a replacement for traditional prompting in our exposition above, PREADD can be used in conjunction with traditional prompting as well. Instead of defining d by contrasting the log-probabilities with or without
the prompt r1:k, PREADD can define d by contrasting the log-probabilities when using the raw prompt r1:k compared to when using a prefix-prepended prompt with additional tokens e1:m prepended to r1:k. In this case, PREADD enables more flexible control strength regarding e1:m.

heading: 4 Experiments
text: We test PREADD on three controlled text generation tasks: (1) toxicity mitigation, (2) gender bias reduction, and (3) sentiment control.

heading: 4.1 Toxic Output Mitigation
text: Task Setup. Our task is based on RealToxicityPrompts (Gehman et al., 2020), a dataset of over 100,000 prompts annotated with toxicity scores. We construct two sets of test prompts: (1) Random, consisting of 1000 randomly sampled prompts, and (2) Toxic, containing only the 1000 most toxic prompts. Random is designed to measure toxic output mitigation on a diverse range of prompts, while Toxic is designed to measure toxic output mitigation on the prompts where it may be most needed. Our goal is to mitigate the toxicity of continuations with as little prompt mismatch as possible.
We use OPT-6.7B (Zhang et al., 2022) as the base language model G, and all generated continuations are 32 tokens in length.
Metrics. We consider two metrics.
1. Toxicity, our main metric, defined as the average toxicity of generated continuations measured via Perspective API (Jigsaw, 2017).2 2. Fluency, defined as the conditional perplexity of the prompt completion according to GPT3175B (davinci) (Brown et al., 2020). 3. Relevance, defined as the cosine similarity between sentence embeddings.
PREADD Instantiation. We explore two variants of PREADD based on how the additional prefix e1:m (Section 3.1) is constructed.
1. PREADD-S, which uses a static prefix e1:m manually written to encourage toxic continuations: “The following text perpetuates negative stereotypes, is threatening or sexually explicit, or contains profane language.”
2Although we use the Perspective API as a convenient automatic metric for benchmarking controlled generation methods in this work, we acknowledge that the API is imperfect, being biased and/or failing to capture certain types of harm (Mozafari et al., 2020; ElSherief et al., 2021).
2. PREADD-D, which uses a dynamic prefix automatically constructed from data instead of a manually written prefix. Concretely, we construct a dynamic prefix bank using the 1000 most toxic sentences from RealToxicityPrompts that do not already appear in the test set. Our dynamic prefix for a given prompt is just the member of the prefix bank with the highest sentence similarity (Reimers and Gurevych, 2019) to the given prompt.
We set α = −1 for both methods to control against the prefix e1:m.
Baselines. We compare to three baselines.
1. G, the base OPT-6.7B model. 2. NEGPROMPT, a prompting method using an
additional prefix to instruct G not to generate toxic text. The prefix is minimally changed from PREADD-S’s prefix (Appendix A). 3. FUDGE (Yang and Klein, 2021), a method using an auxiliary discriminator to control generation toward or against a desired attribute by modifying logits token-by-token. For the discriminator, we use OPT-125m trained on all of RealToxicityPrompts’ toxicity labels, excluding the prompts in the test set.
FUDGE uses top-k sampling as a necessary implementation detail; we set k = 100. PREADD and all other baselines use pure sampling.
Results. As shown in Table 1, both variants of PREADD exhibit strong performance on toxic output mitigation compared to baselines. PREADD performs especially well on the Toxic set, where PREADD-S outperforms all baselines—including the auxiliary-model method FUDGE—by over 15% on relative toxicity reduction. In contrast, NEG-
PROMPT underperforms G, confirming the ineffectiveness of simply instructing a non-instructiontuned language model not to write toxic outputs. Table 2 contains example continuations on a prompt from our Toxic subset; see Appendix H for additional examples. PREADD’s effectiveness is not limited to OPT-6.7B: we observe qualitatively similar results on GPT-J-6B as well (Appendix L).
While PREADD may slightly compromise fluency and relevance compared to G, such tradeoffs are typical in controlled text generation (e.g., in Liu et al. (2021), both their own method and all baselines). For instance, on some toxic prompts, PREADD may somewhat shift the topic to reduce toxicity while preserving fluency (Table 2).3

heading: 4.1.1 Human Evaluation
text: We additionally run human evaluations comparing PREADD-S and G on toxicity, fluency, and relevance. Surge AI workers provided binary labels for each metric on 400 continuations for each method.
As shown in Table 3, humans confirm that PREADD-S is effective at mitigating toxicity without overly sacrificing fluency or relevance.

heading: 4.2 Gender Bias Reduction
text: Task Setup. Next, we explore reducing gender bias in text generation using the WinoBias dataset (Zhao et al., 2018), which contains 3,160 sentences describing interactions between 40 occupations with
3In a similar vein, models such as ChatGPT and text-davinci-003 are explicitly designed to refuse to write continuations to toxic prompts (OpenAI, 2021).
different stereotypical gender profiles. Each sentence mentions two occupations, followed by a pronoun referring to one of the occupations.
Our benchmark uses the subset of WinoBias for which the referent of the pronoun is unambiguous (Appendix E). We truncate each sentence just before the pronoun to create a prompt, and compare the probability of generating a female pronoun (“she,” “her”, “hers”) against generating a male pronoun (“he”, “him”, “his”) to measure gender bias. Both our training and test sets contain 792 examples; examples are labeled as stereotypical or anti-stereotypical, with even class balance.
Metrics. Our metric is the bias of single-pronoun continuations, averaged across the 40 occupations. For evaluation, we define bias as the absolute difference between 0.5 and the probability of generating a female (or, equivalently, male) pronoun.
We focus on the static prefix version of our method (PREADD-S), using “The following text exhibits gender stereotypes.” as the prefix. We again set α = −1. Baselines. We again compare to three baselines:
1. G, the base OPT-6.7B model. 2. NEGPROMPT, similar to the toxic output mit-
igation task, again using a prefix minimally modified from PREADD-S (Appendix A). 3. FUDGE, defined as in the toxic output mitigation task, although here it only needs to modify the next-token logits for one step of generation. The discriminator is trained to predict whether text will be gender-stereotypical.
Results. As shown in Table 4, PREADD outperforms our baselines by over 20% in relative bias reduction. Interestingly, FUDGE makes virtually no impact on bias, likely because its discriminator is unable to learn the somewhat subtle desired attribute from the small training dataset of 792 ex-
amples. In contrast, PREADD does not require training data to achieve strong performance. Meanwhile, similar to the toxic output mitigation task, NEGPROMPT underperforms G, demonstrating the relative ineffectiveness of traditional prompting for reducing gender bias.

heading: 4.3 Sentiment Control
text: Task Setup. Finally, we evaluate PREADD on output sentiment control. We benchmark on the Stanford IMDb dataset (Maas et al., 2011) of 50,000 highly polar IMDb movie reviews.
We construct two sets of test prompts: (1) PosToNeg, consisting of 1000 positive movie reviews, and (2) NegToPos, consisting of 1000 negative reviews; both are randomly sampled from the IMDb test set. We truncate reviews to 32 tokens to create the prompts.
The goal of our task is to generate a continuation with sentiment opposite to that of the original prefix (e.g., positive sentiment starting from a negative prompt). We again use OPT-6.7B as the base language model G. All generated continuations are 64 tokens in length.
Metrics. We consider three metrics:
1. Success, our main metric, defined as the proportion of successful generations with the desired sentiment as judged by BERT (Devlin et al., 2019) finetuned on the IMDb training set (Appendix K). 2. Fluency, the same as in the toxicity task. 3. Relevance, again the same as before.
PREADD Instantiation. We focus on the static prefix version of our method, PREADD-S. Our prefix for positive sentiment is “The following text exhibits a very positive sentiment and/or opinion.”, and “The following text exhibits a very negative sentiment and/or opinion.” for negative sentiment.
We set α = 2 to control towards the prefix e1:m.
Baselines. We compare to three baselines:
1. G, the base OPT-6.7B model. 2. POSPROMPT, similar to NEGPROMPT but
prompting for a specific sentiment. The prefixes used are the same as in PREADD-S. 3. FUDGE, defined as in the toxic output mitigation task, with the discriminator trained to predict sentiment.
Results. As shown in Table 5, PREADD outperforms all baselines in controlling continuation sentiment. Although the fluency of PREADD is worse than the baselines, its continuations appear to be grammatical upon inspection; see example continuations for all methods from both PosToNeg and NegToPos in Appendix I. We also observe similar results using GPT-J-6B as the base model (Appendix L).

heading: 5 Discussion
text: In this work, we have proposed PREADD, a prompting-based method for controlled generation. Unlike typical prompting approaches, PREADD can adjust the degree of control exerted by the prompt by contrasting the output logit distributions for two different prompts, allowing for flexible control strength similar to auxiliary-model-based control methods without requiring training data. In our experiments, PREADD outperforms both simple prompting baselines and an auxiliary model method on three different tasks: toxic output mitigation, gender bias reduction, and sentiment control. In principle, PREADD is highly flexible and can be applied to a wide range of other tasks as well. For instance, one could use PREADD to increase control strength to satisfy more difficult, complex constraints such as faithfulness to a story outline (Yang et al., 2022), or one could extend PREADD to contrast more than two prompts at a time to satisfy multiple simultaneous constraints.
Limitations
As with other prompting methods, PREADD’s performance may vary depending on the exact wording of the prompt, and may require manual prompt design to achieve the best possible performance. Additionally, compared to more basic forms of prompting, PREADD requires accessing the base language model’s output logits at each step of decoding, which can be inconvenient with certain APIs such as the OpenAI GPT3 API (although PREADD is technically runnable through the GPT3 API, it will be less computationally efficient).
With respect to the actual performance of PREADD, a rare but nontrivial failure mode is setting a high α parameter. In particular, setting α to have a magnitude of above 2.5 tends to result in degenerate continuations. This is due to how the output logit distribution shift induced by PREADD may significantly increase the logits of “nonsensical” tokens. The issue seems to appear predominantly in positive control applications of PREADD (e.g. our sentiment control task), wherein the logit distributions “spike” more and have higher entropy. However, logit distribution truncation methods (e.g. top-k and/or nucleus sampling) can be used in PREADD to alleviate text quality decay by eliminating nonsensical tokens prior to applying the control.
In this work, we evaluate toxicity using PerspectiveAPI as a convenient automatic metric, but we acknowledge that PerspectiveAPI is not a perfect measure of toxicity. For example, it may be biased against African-American English, and may fail to capture certain types of harmful outputs (Mozafari et al., 2020; ElSherief et al., 2021). Overoptimization against PerspectiveAPI could lead to unexpected side effects or biases in model outputs (Jacobs and Wallach, 2021; Xu et al., 2021). Additionally, although controlled generation methods like PREADD may reduce the toxicity of generated continuations in the presence of highly toxic prompts, they may still struggle to explicitly counter the original toxic language in the input.
For our gender bias reduction task, we have focused only on occupations as provided in the WinoBias dataset. There are of course innumerable other types of bias which are important to mitigate, ranging from gender bias in facets of language other than occupations, to other types of bias such as those based on race or age; Blodgett et al. (2020) provide a more complete discussion.
Finally, all of our experiments are on Englishlanguage datasets, so harmful or biased outputs in non-English contexts may not be well-represented.
Ethical Considerations
As with any effective method for controlled text generation, we acknowledge that PREADD could be misused to increase toxicity, gender bias, or any other harmful attribute (McGuffie and Newhouse, 2020). Nonetheless, controlled text generation methods such as ours are also powerful tools for content moderation and mitigating harmful text generation, problems which are of major importance in practice due to large language models’ propensity to generate toxic or biased text (Sheng et al., 2019, Gehman et al., 2020, Garbacea and Mei, 2022). We are hopeful that future research on controlled text generation will continue to improve our ability to detect and mitigate such harms.
Additionally, as discussed in greater detail in the Limitations, both our toxic output mitigation and gender bias reduction tasks reflect only a narrow subset of possible harmful outputs. Our datasets and experiments are also only in English.

heading: Acknowledgements
text: We thank Eve Fleisig, the rest of the Berkeley NLP group, and our anonymous reviewers for their helpful discussions and feedback which helped us to greatly improve the paper. This work was supported by Berkeley AI Research, Open Philanthropy, DARPA under the SemaFor program (HR00112020054), the Machine Common Sense (MCS) program under Cooperative Agreement N66001-19-2-4032, and the NSF through a fellowship to the second author. The content does not necessarily reflect the position or the policy of the government, and no official endorsement should be inferred.

heading: A Prefixes for NEGPROMPT and
text: POSPROMPT
For toxic output mitigation, the prompt is “The following text does not perpetuate negative stereotypes, is not threatening or sexually explicit, and does not contain profane language.”
For gender bias reduction, the prompt is “The following text does not exhibit gender stereotypes.”
POSPROMPT for sentiment control uses the same prefixes as PREADD-S.

heading: B Toxic Output Mitigation Full Toxicity Utterance
text: In Section 4.1, we report the toxicity of generated continuations to highlight method differences. Here, we also provide the toxicity of full utterances (i.e., prompt+continuation) in Table 6. While the general trend is similar compared to the main Table 1, the toxicity level of the original prompt obscures the differences between methods.

heading: C Toxic Output Mitigation Human Evaluation Experimental Details
text: We asked a group of human workers on the Surge AI platform to label 400 pairs of continuations generated by PREADD and G as non-toxic, fluent, and/or on-topic. See Tables 8 and 9 for a set of instructions and an example query, respectively, we gave to the workers in the experiment.
We paid the participants according to our estimate of $20/hr, which we believe is reasonable payment given the task and U.S. demographic of participants. We also ensured to directly ask for participation consent via our instructions sent to the workers. The data collection protocol was determined to be exempt.
D Initial Pairwise Human Comparison
We initially ran a small-scale version of the human comparison in Table 3 using paired comparisons (e.g., which of these two continuations is more toxic or more fluent, rather than asking annotators to label each individually). Upon inspecting annotations, we realized that this methodology was somewhat flawed—in many cases, one continuation would be substantially less toxic but only slightly less fluent or relevant, resulting in being penalized on the latter even if it might have been judged reasonably fluent or relevant in a vacuum. The result of this experiment is shown below in Table 7.

heading: E Gender Bias Benchmark
text: As the WinoBias dataset was developed mainly with the goal of coreference resolution, its contents provide two types of prototypical templates:
Type 1: [entity1] [interacts with] [entity2] [conjunction] [pronoun] [circumstances]
Type 2: [entity1] [interacts with] [entity2] and then [interacts with] [pronoun] for [circumstances]
For the type 2 template, the pronoun directly references entity2. However, for the type 1 template, the referent of the pronoun is ambiguous: it can refer to either entity1 or entity2. For example, for the following type 1 sentences, the “she” refers to the physician in the former, and refers to the secretary in the latter.
The physician hired the secretary because she was overwhelmed with clients. The physician hired the secretary because she was highly recommended.
Since we are not evaluating for coreference resolution, we do not create prompts from type 1 sentences to avoid coreference ambiguity in our task.

heading: F Gender Bias Full Results
text: Table 10 shows the individual gender pronoun probabilities by occupation for each method shown in the main Table 4.

heading: G Statistical Significance
text: For each task, we perform paired t-tests between each PREADD variant and each baseline. p-values for toxicity and fluency in the toxic output mitigation task are shown in Tables 11 and 12; p-values for bias in the gender bias reduction task are shown in Table 14; p-values for success and fluency in the sentiment control task are shown in Tables 15 and 16. We also report p-values for toxicity, fluency, and relevance for the toxic output mitigation human evaluations in Table 13.

heading: H Additional Toxic Output Mitigation Examples
text: In this section, we provide some additional examples of PREADD and baselines on the toxic output mitigation task. We display examples where our method seems to outperform baselines by qualitative inspection in Table 20. We also report i.i.d. examples in Table 21.
CONTENT WARNING: Examples may contain highly racist, sexist, or otherwise offensive text.

heading: I Sentiment Control Examples
text: We provide i.i.d. example outputs from PREADDS and baselines on the sentiment control task in Tables 22 and 23 (for PosToNeg and NegToPos, respectively).

heading: J PREADD Hyperparameter Ablation Study
text: In the section, we provide an ablation study of the PREADD hyperparameter α across all three tasks. We report toxic output mitigation results in Table 24, gender bias reduction results in Table 25, and sentiment control results in Table 26.
There is a clear tradeoff between the optimization of main metrics (toxicity, bias, and success)
and of text fluency/relevance. In particular, when the magnitude of α exceeds approximately 2.5, the quality of the text plummets with little to no improvement (and even worsening) of main metrics. This degeneration in overall continuation quality is most likely due to erratic token output behavior occurring at more severe distribution shifts; we discuss this failure mode further in Limitations (Section 5). We choose our “optimal” hyperparameters based on both empirical performance and theoretical motivations (e.g. setting α = −1 to directly apply “anti” toxic or biased contol).

heading: K Additional Computational Details
text: Prefix prompts used for PREADD, NEGPROMPT, and POSPROMPT were manually written.
For FUDGE, we conducted hyperparameter search on the learning rate for finetuning OPT125m on the attribute-specific data, testing {10−3, 10−4, 10−5, 10−6} for all our tasks, and found the following values to be best for each task:
(i) Toxic Output Mitigation: 10−5
(ii) Gender Bias Reduction: 10−3
(iii) Sentiment Control: 10−3
The sentence transformer model used to compute sentence embeddings for the relevance metric and to dynamically select prefixes for PREADD-D is all-MiniLM-L6-v2 (Reimers and Gurevych, 2019).
The base pretrained BERT model used for sentiment classification is bert-large-uncased (Devlin et al., 2018). For finetuning, we conducted hyperparameter search on the learning rate and weight de-
cay, testing the values {10−3, 10−4, 10−5, 10−6} for both parameters to yield (10−4, 10−2) as the best combination. We trained for 50 epochs using the Adam optimizer (Kingma and Ba, 2014) with the above parameters, and otherwise default settings. Our finetuned BERT classifier achieves 96.1% validation accuracy when evaluated on the testing portion of the IMDb dataset, excluding the reviews used in the benchmark sets (i.e. PosToNeg and NegToPos). We also tried using SiEBERT (Hartmann et al., 2023) by itself, but the model yields a slightly lower validation accuracy of 94.5% on the same set.
We estimate that we spent roughly 300 GPU hours on NVIDIA Quadro RTX 6000s and 8000s over the course of this project for both development and testing.

heading: L Secondary Results on GPT-J-6B
text: In our main experiments, we only explore using OPT-6.7B as the base model. Here, we show that PREADD displays similar performance when applied to GPT-J-6B (Wang and Komatsuzaki, 2021). We report toxic output mitigation results for the Toxic subset in Table 17, gender bias reduction in Table 18, and sentiment control for the PosToNeg subset in Table 19.
When training the FUDGE discriminator, we use GPT-Neo 125M (Black et al., 2021; Gao et al., 2020) in order to share tokenization with GPT-J. We also use the same hyperparameters as described in Appendix K.

heading: M Licenses and Software
text: RealToxicityPrompts uses the Apache License 2.0. WinoBias does not contain a license in their GitHub repository.
All models are implemented in PyTorch (Paszke et al., 2019), and all pretrained models other than GPT3 are obtained from HuggingFace (Wolf et al., 2019), which uses the Apache License 2.0.
Our use of datasets and models is consistent with their intended use.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
We discuss in Limitations following the main text.
3 A2. Did you discuss any potential risks of your work? We discuss in Ethical Considerations following the main text.
3 A3. Do the abstract and introduction summarize the paper’s main claims? The abstract and intro (Sec 1) summarize our main claims.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We discuss in Experiments (Sec 4)
3 B1. Did you cite the creators of artifacts you used? We cite these as they first appear throughout the paper.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We discuss in Appendix M.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We discuss in Appendix M.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We didn’t create new data. The datasets we rely on are widely used and of course contain offensive content because we evaluate on toxic output mitigation.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We discuss as they’re introduced in Sec 4 and in Ethical Considerations.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. These are mentioned as they’re introduced in Sec 4.
C 3 Did you run computational experiments? Experiments are in Sec 4 and throughout the Appendix.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Parameter counts reported when we refer to models in Sec 4. Computational budget and infrastructure are discussed in Appendices K and M.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? These are discussed in Sec 4 as well as Appendices J and K.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We report these in Appendix G.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. We didn’t need existing packages for preprocessing / normalization / evaluation besides the online Perspective API.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? We describe human evaluation experiments in Sec 4.1.1 as well as Appendices C and D.
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? We report this in Appendix C.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? We report this in Appendix C.
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? We discuss in Appendix C.
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We mention this in Appendix C.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? We report this in Appendix C.

heading: Future Work
text: We are hopeful that future research on controlled text generation will continue to improve our ability to detect and mitigate such harms. Additionally, as discussed in greater detail in the Limitations, both our toxic output mitigation and gender bias reduction tasks reflect only a narrow subset of possible harmful outputs. Our datasets and experiments are also only in English