Parameter-Efficient Detoxification with Contrastive Decoding (2024)

Abstract: The field of natural language generation has witnessed significant advancements in recent years, including the development of controllable text generation techniques. However, controlling the attributes of the generated text remains a challenge, especially when aiming to avoid undesirable behavior such as toxicity. In this work, we introduce Detoxification Generator (DETOXIGEN), an inference-time algorithm that steers the generation away from unwanted styles. DETOXIGEN is an ensemble of a pre-trained language model (generator) and a detoxifier. The detoxifier is trained intentionally on the toxic data representative of the undesirable attribute, encouraging it to generate text in that style exclusively. During the actual generation, we use the trained detoxifier to produce undesirable tokens for the generator to contrast against at each decoding step. This approach directly informs the generator to avoid generating tokens that the detoxifier considers highly likely. We evaluate DETOXIGEN on the commonly used REALTOXICITYPROMPTS benchmark (Gehman et al., 2020) with various language models as generators. We find that it significantly outperforms previous approaches in detoxification metrics while not compromising on the generation quality. Moreover, the detoxifier is obtained by soft prompt-tuning using the same backbone language model as the generator. Hence, DETOXIGEN requires only a tiny amount of extra weights from the virtual tokens of the detoxifier to be loaded into GPU memory while decoding, making it a promising lightweight, practical, and parameter-efficient detoxification strategy.

heading: 1 Introduction
text: Large language models (LLMs) have demonstrated remarkable promise in various generative tasks by first self-supervised pretraining on large text corpora and then finetuning with instruction data for
* indicates corresponding authors.
alignment (Mishra et al., 2022). Yet a wealth of previous work has demonstrated that pre-trained models inherit toxicity and biases from their training corpora (Zhao et al., 2019; May et al., 2019; Kurita et al., 2019; Basta et al., 2019). As a result, generative models (OpenAI, 2023; Touvron et al., 2023; Nijkamp et al., 2023) tend to degenerate into unsafe text even when conditioning on seemingly innocuous prompts (Wallace et al., 2019; Sheng et al., 2019; Gehman et al., 2020), which is difficult to resolve by prompt engineering alone (Zong and Krishnamachari, 2022; Liu et al., 2022b; Webson and Pavlick, 2022; Lou et al., 2023).
To address this challenge, a plethora of approaches have been proposed, which usually require full-model finetuning of the underlying language model to build the detoxifier (Dathathri et al., 2019; Gururangan et al., 2020; Krause et al., 2021; Liu et al., 2021a). However, nowadays the largest LLMs typically contain more than 100 billion parameters, making such resource-intensive tuning less viable. This trend calls for more parameterefficient approaches.
In this work, we propose DETOXIGEN (Figure 1), a parameter-efficient framework that leverages the frozen weights of the language model itself and only introduces a tiny portion of new model parameters to detoxify generation.1 During training, we use prompt tuning (Lester et al., 2021) to train a detoxifier exclusively on toxic data with the nexttoken prediction objective. The resulting detoxifier shares all the backbone model weights with the LLM (i.e., the generator). During inference, we build on top of the contrastive decoding (Li et al., 2023) paradigm and employ the detoxifier to manipulate the output probability distribution of the LLM for each generation step. Intuitively, the generator avoids outputting tokens that the detoxifier considers highly probable. For example, in figure 1 the
1We will make our code publicly available.
30
detoxifier considers the gender-biased word “his” very likely as the next token, helping the generator to score down the probability of that token.
We evaluate our framework on the REALTOXICITYPROMPTS dataset (Gehman et al., 2020) and find that it outperforms previous approaches on the standard benchmark metrics by a significant margin, indicating that the text generated by our model is both safer and of higher quality. We also conduct ablation studies and pair models of different sizes from the same model family (e.g., the Llama-2 (Touvron et al., 2023) family). These studies show that pairing a generator with a detoxifier that shares the same backbone LLM is indeed the best-performing configuration.
Our main contributions are: (1) Performance: Propose a detoxification framework that outperforms previous models by a large margin on commonly used detoxification benchmarks/metrics; (2) Efficiency: We apply parameter-efficient learning to controllable text generation for detoxification. Our model introduces the least amount of additional parameters (hence also requires less data to train) as compared to state-of-the-art models; (3) Transferability: Our detoxifier model only requires toxic data and does not require any contrastive (non-toxic) data, making our approach transferable thanks to the easier and more manageable data curation.

heading: 2 Model

heading: 2.1 Task Formulation
text: We consider controlled decoding-time approaches for open-ended text generation. A generator, in our case a language model, receives an unfinished input text as a prompt and aims to output a fluent and coherent continuation that avoids toxicity with the help of a detoxifier, which is another language model trained with data of the target attribute.

heading: 2.2 Model Components
text: Generator Let x<t = x1, x2, ..., xt−1 be a prompt consisting of (t − 1) tokens, where each ti(1 ≤ i ≤ t−1) is a token in the vocabulary set V of the language model (LM). The LM encodes x<t in an autoregressive fashion and outputs zt ∈ R|V |, where zt denotes the logits for the tth token xt and |V | corresponds to the vocabulary size. The LM then obtains a probability distribution PGEN over V by computing the softmax of zt
PGEN (xt|x<t) = softmax(zt), (1)
and the next token is sampled from this distribution.
Detoxifier The detoxifier takes as input the same prompt fed to the generator for each generation step and computes a probability distribution PDE(Xt|x<t) over |V | in the same way. However, the detoxifier is not a vanilla LM like the generator, but rather an LM specially trained to output toxic content. Intuitively, the generator is discouraged from outputting tokens that the detoxifier considers highly likely, thus avoiding toxic generations. In other words, the decoding process involves an ensemble of the two LMs to obtain the final output probability distribution P (xt|x<t):
P (xt|x<t) = PGEN + α∆P (2) ∆P = PGEN − PDE , (3)
where the hyperparameter α denotes the control strength of the model and ∆P represents the probability correction term determined by the difference between the two distributions. Intuitively, α dictates how much we want to modify the generator’s probability distribution through the correction term ∆P . Since it is possible that P (xt|x<t) contains values that fall out of the [0.0, 1.0] range, making them invalid probabilities, we also clip them on both sides – i.e., setting any value that is below 0.0 to 0.0 and any value above 1.0 to 1.0. The resulting probability vector is then normalized by first computing its log probabilities and then taking the softmax. Our formulation is closely related to that in Contrastive Decoding (Li et al., 2023) and DExperts (Liu et al., 2021a), but we differ by manipulating in the probability space rather than the logits space because we found in our initial experiments that directly dealing with probability distributions result in better performance on the downstream task.
Sampling To constrain the model to only generate plausible tokens, we first employ Nucleus (Top-p) Sampling (Holtzman et al., 2020) to limit the vocabulary V to a subset V (p) by only selecting the highest probability tokens whose cumulative probability mass exceeds some threshold p ∈ [0.0, 1.0]. More specifically, given the distribution PGEN (xt|x<t) in Equation 1, the top-p
vocabulary V (p) ⊆ V is defined by the smallest vocabulary set such that
∑
x∈V (p) PGEN (x|x<t) ≥ p. (4)
The Top-p sampling then truncates the less reliable tail of the distribution by setting
P ′[x] = { P [x], if x ∈ V (p) 0, otherwise.
(5)
The detoxifier then comes in and only manipulates logits in the set V (p) so that regardless of how PGEN is modified, the generated tokens are guaranteed to be plausible as evaluated by the generator. When applying this restriction, equation 1 becomes
P ′(xt|x<t) = P ′GEN + α(P ′GEN − P ′DE). (6)

heading: 2.3 Parameter-Efficient Training of Detoxifier
text: As discussed in Section 1, due to the increasing size of large language models, we aim to introduce as few additional model parameters as possible to our
framework. Hence we adopt Prompt Tuning (Lester et al., 2021), a parameter-efficient training method, to train a language model exclusively on toxic data. This method learns soft prompts (or virtual tokens), whose embeddings are trainable parameters to condition frozen language models to perform the target downstream tasks.

heading: 3 Experimental Setup

heading: 3.1 Backbone Models
text: Following Liu et al. (2021a), we use GPT2large (Radford et al., 2019) as the generator and the backbone of the detoxifier. We use GPT2-XL to evaluate the generation quality. For ablation studies reported in Section 4.3, we also consider GPT2 with small and medium sizes.
It is worth noting that previous work selected the GPT2 family mostly because it was one of the strongest models at the time. To observe if the same trend of performance holds for the most recent LLMs, we also experiment with another family of Transformer-based (Vaswani et al., 2017) language models, namely Llama-2 (Touvron et al., 2023) because it satisfies the following three criteria: (1) It is publicly released so that it is easier for researchers to reproduce and compare with
our work; (2) It achieves state-of-the-art performance on diverse benchmark datasets (Nijkamp et al., 2023); (3) It has three sizes so that we can evaluate whether larger models can be paired with smaller ones for detoxification – such is the setting when we prioritize reducing latency over minimizing GPU memory footprint. Hence we experiment with Llama-2 with 7B, 13B, and 70B parameters, respectively. Due to the large size of Llama-2-70B, for all our experiments we use bfloat16 for both training and inference to increase throughput and reduce GPU memory usage. We evaluate perplexity from the Llama-2 family with Llama-2-7B unless otherwise stated.

heading: 3.2 Training of Detoxifier
text: We prompt tune the detoxifier with the standard language modeling objective (Bengio et al., 2000) which learns the parameters of the conditional probability distribution of the next word given the preceding context. We extract the training data from the human-annotated Jigsaw Unintended Bias in Toxicity Classification (Borkan et al., 2019). An example is considered toxic if more than 50% of the annotators classify it as toxic. This threshold splits the corpus into around 160K toxic and 1.4M nontoxic examples. We only train our models with the toxic part of the data.
For prompt tuning, we use 100 virtual tokens for each model with a learning rate of 0.1. To efficiently explore different parameter-efficient methods, we use the PEFT (Parameter-Efficient FineTuning), a library that wraps around HuggingFace Transformers (Wolf et al., 2020) model objects and provides out-of-the-box implementations for widely adopted PEFT approaches (Mangrulkar et al., 2022). Because we need to obtain logits from the detoxifier for each generation step, we overwrite the PEFT model object to only prepend virtual tokens to the input for the first generation step.

heading: 3.3 Hyperparameter Tuning
text: We tune the hyperparameter α with a held-out validation set and perform a grid search from 1.0 to 9.0 with a 1.0 increment. As will be shown in Section 4.1, we find that α = 5.0 strikes the best balance between toxicity and generation quality. We thus adopt this value throughout all experiments.

heading: 3.4 Evaluation Data
text: We follow Liu et al. (2021a) to use the REALTOXICITYPROMPTS dataset (Gehman et al., 2020) which contains 100K naturally occurring, sentence-level prompts derived from a large corpus of English web text. These prompts are annotated with toxicity scores and language models are known to degenerate into toxic continuation when conditioning on them. To determine the detoxifier strength α, we randomly sample 1k prompts as the validation set and another disjoint 10k as the test set.

heading: 3.5 Metrics
text: Toxicity Following Gehman et al. (2020), we use the Perspective API2 to measure the toxicity of generations. This score is obtained from a CNN model (Lecun et al., 1998) trained on a non-public corpus of Wikipedia comments. We compute two metrics based on the toxicity scores following Liu et al. (2021a): (1) Average Maximum Toxicity: The average maximum toxicity over k = 25 generations; (2) Toxicity Probability: The empirical probability of a generation with toxicity ≥ 0.5 for at least once over k = 25 generations.
Quality The Quality metric consists of both fluency and diversity. Heeding both aspects makes it easier to spot cases where the generation is likely but generic, or diverse but unlikely. We use corpuslevel Perplexity to evaluate fluency and Distinct2 and -3 (Li et al., 2016) to evaluate diversity. Distinct-2 and distinct-3 correspond respectively to the number of distinct bigrams and trigrams divided by the total number of generated words.

heading: 3.6 Baseline Models
text: We compare DETOXIGEN with a diverse set of previously reported baseline models (Gehman et al., 2020; Liu et al., 2021a), including Domain-Adaptive Pretraining (DAPT) (Gururangan et al., 2020), Plug-and-Play Language Models (PPLM) (Dathathri et al., 2019), Non-Toxic Expert (Liu et al., 2021a), Generative Discriminators (GeDi) (Krause et al., 2021), and Decoding-time Experts (DExperts) (Liu et al., 2021a). We follow these baselines to use Nucleus Sampling with p = 0.9 for generation.
2https://perspectiveapi.com/

heading: 4 Results and Analysis

heading: 4.1 Hyperparameter Tuning through Validation Set
text: As mentioned in Section 3.3, we perform a grid search of α with values 1.0, 2.0, ..., 9.0. We show the results on both GPT2-large and Llama-2-7b so that we can observe the trend on both early and more recent models. From Table 2, we can see that for both models, there is a steady increase in Perplexity (last column) as α grows, indicating a monotonic decrease in generation quality. Intuitively, this trend makes sense because the more we perturb the original output distribution, the more likely it is for the language model to generate less plausible tokens. To maintain a balance between toxicity and quality, we seek the tipping point where further increasing α only brings a diminishing return on reducing toxicity. We observe that for both models, this tipping point happens at α = 5.0. Hence we adopt this hyperparameter setting throughout all other experiments.

heading: 4.2 Results on GPT2-large
text: We then compare with previous approaches on GPT2-large. From Table 1, we can see that our model DETOXIGEN outperforms previous frameworks by a large margin although only tuned on the toxic split of the training data. Among all models, DETOXIGEN (GPT2-large) achieves the lowest Average Maximum Toxicity and Toxicity Probability, while obtaining a Perplexity that is quite close to that of the vanilla GPT-2 large, indicating minimum compromise on generation quality. The Llama-2-7B version of DETOXIGEN achieves even better results. However, it is based on a much stronger backbone language model, hence not comparable to previous work. We still include Llama-2 results in this table to show the gap between earlier and more recent large language models. We also follow Liu et al. (2021a) and report Distinct-N metrics, which are intended to prevent the model from degenerating into dull and generic continuations. We observe that the Distinct-N results do not vary much across diverse kinds of models. Hence for the results reported afterwards, we skip this metric and only report Perplexity.

heading: 4.3 Ablation Studies on Model Sizes
text: We also explore pairing models of different sizes as the generator and the detoxifier, respectively. This setting targets the cases where either latency
is the major concern such that we want one small detoxifier to steer the generation of all other model sizes, or when we intend to train a detoxifier once and plug-and-play it with all other model sizes. The results of such pairings are presented in the matrix-like tables (Table 3, 5, 4, and 6). We report toxicity and quality in separate tables to make the comparisons clearer. From the four tables, we can observe quite a few interesting patterns.
Consistent Toxicity Reduction In the tables, we can observe that when comparing with the nodetoxifier setting (the column with None as header), our approach consistently and significantly reduces the toxicity of the backbone model while not sacrificing much on generation quality. This trend is observed for both the GPT-2 and the Llama-2 model families.
Entries along the Diagonal As shown in Table 3 and 4, entries on the diagonal of the result matrix (i.e., without the first column that has None as the header) consistently outperform their neighbors in terms of toxicity. These are the settings where the generator and the detoxifier share exactly the same backbone language model. They also achieve the best row-wise Perplexity as compared to off-diagonal models (Table 5 and 6). We hypothesize that this is because the output probability distributions of the generator and the detoxifier with the same underlying backbone parameters are more compatible with each other than backbones of different sizes. Recall in Section 1 that one of our major goals is to introduce as few new model parameters as possible. Our cross-model results clearly show that sharing weights between the generator and the detoxifier turns out to be the best setting among all we have investigated.
Entries symmetric to the diagonal Comparing entries that are symmetric to the diagonal (e.g., comparing GPT2-XL detoxified by GPT2-small with GPT2-small detoxified by GPT2-XL) in Table 3 and 4, we can observe a consistent pattern that given two models of different sizes, it is usually better to have the smaller model as the generator and the larger model as the detoxifier for detoxification. This indicates that larger models are more capable of capturing the distribution in the toxicity training corpus.
Effect of Model Size Difference From the toxicity tables, we can also observe that the larger the
model size difference, the less effective the detoxification. For example, GPT2-XL detoxified by GPT2-small in Table 3 results in the worst toxicity among all settings, while we observe the same pattern where Llama-2-70B detoxified by Llama-2-7B has the highest toxicity among all settings.

heading: 5 Discussion
text: It would be ideal if a detoxifier could work out of the box (plug-and-play) and be readily applied to any LLM generator, even with a different tokenizer. To achieve this, one can take a common subset of the vocabulary sets between the generator and the detoxifier, and only manipulate logits on this subset. We leave this as future work since the model families we investigate both already have diverse sizes.
Throughout the paper, we have been focusing on avoiding undesired attributes. However, we note that our framework can also be used to generate text with any desired style (which could be a risk if that desire style happens to be toxic). All we need to do is flip the sign of the probability distribution correction term ∆P in Equation 2 and 3 as follows:
P (xt|x<t) = PGEN + α∆P (7) ∆P = PDE − PGEN . (8)
In addition, our approach could be applied to more general positive and negative attributes, including but not limited to politeness (DanescuNiculescu-Mizil et al., 2013; Niu and Bansal, 2018), hate speech (Golbeck et al., 2017), and microagressions (Breitfeller et al., 2019). In the case
that we want to simultaneously control for multiple attributes, our framework is also compatible with mixed-batch inference (Liu et al., 2022a), where soft prompts of different attributes can be conditioned on in a single batch without increasing latency.

heading: 6 Related Work

heading: 6.1 Parameter-efficient Learning
text: Parameter-efficient learning is a natural language processing paradigm to adapt a large language model to particular tasks or domains. It is usually used when fine-tuning the entire language model is prohibitively expensive. Among such approaches, LoRa (Hu et al., 2022) and AdaLoRa (Zhang et al., 2023) inject trainable rank decomposition matrices into each layer of the Transformer architecture, with the latter adaptively allocating the parameter budget among weight matrices according to their importance scores. Prefix Tuning (Li and Liang, 2021), P-Tuning (Liu et al., 2021b), and Prompt Tuning (Lester et al., 2021) prepend to the input sequence virtual tokens with trainable embeddings. Lastly, (IA)3 scales activations by learned vectors. We choose Prompt Tuning in this work because it achieves competitive performance while involving no change in model architecture and not requiring any bootstrapping for the newly introduced model parameters.

heading: 6.2 Controllable Text Generation
text: There have been multiple effective frameworks proposed for controllable text generation (Keskar et al., 2019; Sudhakar et al., 2019; Kurita et al.,
2019; Welleck et al., 2020). 3 Among them, Domain-Adaptive Pretraining (DAPT) (Gururangan et al., 2020) and Self-Generation Enabled domain-Adaptive Training (SGEAT) (Wang et al., 2022) continues to finetune or apply parameterefficient tuning to the backbone language model with a non-toxic subset of OpenWebText to adapt it to the non-toxic style. Plug-and-Play Language Models (PPLM) (Dathathri et al., 2019) trains a toxicity classifier and leverages gradients from that classifier to update the language model’s hidden representations for each generation step. Generative Discriminators (GeDi) (Krause et al., 2021) prepends to the input a soft token serving as the label for the intended class of attribute. It can be viewed as prompt tuning with only one virtual token for each class (toxic and nontoxic). Decodingtime Experts (DExperts) (Liu et al., 2021a) train an expert and an anti-expert LM of opposing attributes with full finetuning. During inference, the logits difference between the two experts serves as the correction term for the logits of the base language model. Our work is different from the previous approaches in that we adopt parameter-efficient tuning that only introduces a few trainable parameters and our training only requires toxic examples rather than examples from both classes.

heading: 6.3 Contrastive Decoding
text: Contrastive Decoding (Li et al., 2023; O’Brien and Lewis, 2023) is a search-based decoding method that optimizes a contrastive objective that returns
3We note that Inference-Time Policy Adapters (Lu et al., 2023) employs reinforcement learning for LM detoxification, but their approach assumes access to the Perspective API toxicity scores as a reward signal during training and hence not comparable to our work.
the difference between the likelihood under a large and a small LM. Our algorithm is different from theirs in that we pair LMs of the same size (with the detoxifier further tuned) and perform all manipulations in the probability space rather than directly in the logits space.

heading: 7 Conclusion
text: We propose DETOXIGEN, a high-performing, parameter-efficient framework for detoxification during inference time. Our method only introduces a small portion of new parameters to train a detoxifier model that manipulates the output probability distribution of a generative model. On a standard detoxification benchmark, our approach outperforms all existing models in terms of both toxicity and quality. As language models grow ever larger in size, our controllable generation method shows the promise of quickly adapting to any language model without requiring additional computational resources or significant data curation efforts.

heading: 8 Limitations
text: Although DETOXIGEN can work with models of different sizes, it is yet to show that a generator with tokenizer A can be paired with a detoxifier that uses tokenizer B with a different vocabulary set. We plan to address this limitation in future work where the detoxifier can tokenize the generated text on the fly for each generation step, thus manipulating the logits of the generator. Additionally, although leveraging the Perspective API makes our model readily comparable with previous work, such automatic evaluation may not capture the full spectrum of toxicity. Ideally, one would
conduct human evaluations for each of the models under the same setting. Lastly, it would be helpful to show how our method generalizes to other styles and parameter-efficient tuning approaches.

heading: Future Work
text: We plan to address this limitation in future work where the detoxifier can tokenize the generated text on the fly for each generation step, thus manipulating the logits of the generator. Additionally, although leveraging the Perspective API makes our model readily comparable with previous work, such automatic evaluation may not capture the full spectrum of toxicity. Ideally, one would
conduct human evaluations for each of the models under the same setting. Lastly, it would be helpful to show how our method generalizes to other styles and parameter-efficient tuning approaches