DeStein: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion

Authors: yu  li
han  jiang
chuanyang  gong
zhihua  wei
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 16:57:04

Abstract: Despite the remarkable achievements of language models (LMs) across a broad spectrum of tasks, their propensity for generating toxic outputs remains a prevalent concern. Current solutions involving finetuning or auxiliary models usually require extensive computational resources, hindering their practicality in large language models (LLMs). In this paper, we propose DeStein, a novel method that detoxifies LMs by applying representation engineering in activation spaces with lower resource and time costs. Specifically, we derive detoxification vectors from self-induced, universal steering pairs through arithmetic operations in activation spaces. During inference, detoxification is achieved by fusing the detoxification vectors with the original representations in a head-wise manner. Empirical results demonstrate that our method significantly outperforms previous state-of-the-art approaches on various metrics, while also maintaining satisfactory generation quality and diversity. We further validate the practicality and scalability of DeStein with a series of white-box LLMs. Warning: Some example model outputs may contain highly offensive or disturbing text.

Main paper: 
 introduction: 
 In recent years, the development of large language models (LLMs) (OpenAI, 2023a; Anil et al., 2023; Touvron et al., 2023) has significantly advanced the field of artificial intelligence, providing remarkable capabilities in understanding and generating natural language (Radford et al., 2019b) . However, these advancements come with the challenge of ensuring that the output is not harmful or toxic (Bender et al., 2021; Deng et al., 2023; Yao et al., 2023; Jiang et al., 2024) . The language models (LMs) are pre-trained on extensive uncurated text corpora, which inadvertently sows the seeds of the potential hazards, as unsafe content present in the training data is compressed into the models. For the sake of leveraging the full potential of LMs while minimizing their safety risks to human society, there are increasing efforts to make LMs safer and more responsible.In the field of LM detoxification, a prevalent solution is the finetuning approach, which involves meticulously designed datasets or losses (Gururangan et al., 2020; Wang et al., 2022; Arora et al., 2022; Kwak et al., 2023) . Such methods often require specialized training data and massive computational resources, rendering them ineffective in low-resource scenarios. An alternative solution lies in decoding-based methods, which typically manipulate the decoding process through the introduction of auxiliary models or metric-based modifications, inspired by contrastive decoding (Liu et al., 2021; Krause et al., 2021; Yang & Klein, 2021; Li et al., 2023b) . Although these methods are less resource-intensive, the performance of auxiliary models heavily depends on their training data. Given that LMs tend to internalize various toxicity contents from extensive text sources, efforts to detoxify LMs with auxiliary models are inherently limited. Moreover, direct modification of logits could significantly impact the model's generative capabilities, making it difficult to achieve a balance between detoxification and generation. In summary, there is an urgent need for a low-resource, scalable, and interpretable detoxification method for LLMs.To address these problems, we propose DESTEIN, a novel method aimed at Deoxifying LMs with universal Steering pairs and head-wise activation fusion. Inspirited by activation engineering (Zou et al., 2023; Turner et al., 2023) , our approach achieves detoxification by altering LMs' internal representations in activation spaces. Specifically, the steering text pairs are constructed by excavating the toxic patterns in model generation. Subsequently, directional vectors for toxicity versus non-toxicity are identified within activation spaces according to the concept of linear representation (Mikolov et al., 2013) . During the inference phase, the detoxification vectors are merged with the output of the layer with head-wise weights derived from probing techniques. The experimental results demonstrate that our approach sets a new benchmark for state-of-the-art (SOTA) performance in detoxification, without any finetuning or auxiliary models. Further, DESTEIN effectively maintains generation quality, including fluency and diversity, and proves scalable across multiple LMs. Our main contributions are as follows:• First, we propose a novel approach that detoxifies LMs via representation engi- neering in activation spaces. It surpasses the previous SOTA methods in both detoxification performance and maintenance of generation quality with lower computational demands and acceptable inference time.• Next, we conduct a detailed analysis of the mechanism of our method. With the introduction of probing techniques, DESTEIN adaptively conducts activation fusion at different locations to maximize detoxification and minimize impacts on generation quality. • Finally, in contrast to the lack of scalability of congener methods, we include three datasets with multiple metrics and LLMs of varying sizes (1.3B, 7B, and 13B) as our testbed. The outcomes verify the robustness of DESTEIN across different domains and LMs. 
related works: 
 In recent years, a variety of detoxification strategies have been devised. These approaches can generally be classified into two categories: finetuning-based and finetuning-free methods.Among these, finetuning-based methods are considered the most straightforward (Gehman et al., 2020; Gururangan et al., 2020; Wang et al., 2022; Arora et al., 2022; Zheng et al., 2023; Kwak et al., 2023) . They involve training the pre-trained LMs on carefully curated non-toxic data to adapt them to a non-toxic domain. While these methods have proven to be effective to some extent, they involve updating the model's parameters and face challenges related to the lack of labeled data and high computational costs.Given the vast scale of pre-trained language models (PLMs), finetuning-free methods have been widely applied. These methods can be further subdivided into many categories, among which a significant category is decoding-based methods. These methods adjust the model's output probability distribution during the decoding phase to guide the generation of text with desired attributes (Dathathri et al., 2020; Pei et al., 2023; Zhong et al., 2023; Zhang & Wan, 2023; Pozzobon et al., 2023b; Dekoninck et al., 2024) . Many approaches control the output distribution of the language model using attribute distributions obtained from trained classifiers (Dathathri et al., 2020; Yang & Klein, 2021; Krause et al., 2021) .Other methods employ the idea of contrastive decoding (Li et al., 2023b) , detoxifying by contrasting the probability distributions of toxic and non-toxic outputs (Pei et al., 2023; Zhong et al., 2023) . However, since these methods require direct modification of the PLM's predicted original probabilities, the fluency of the generated text can decrease rapidly as the control intensity reaches a certain threshold.2023; Panickssery et al., 2024; Liu et al., 2024; Lee et al., 2024) . Liu et al. (2024) paraphrases offensive content through activation editing, whereas we focus on directly suppressing the generation of toxic content instead of rewriting it. SELF-DETOXIFY (Leong et al., 2023) leverages positive and negative prompts to modify activations during inference to reverse toxicity and prevent toxic generation. The method we propose can also be classified as an activation-engineering-based approach. Our work is similar to SELF-DETOXIFY, but we focus on offline activation editing rather than steering online during the inference stage. We firmly believe that activation-engineering-based methods hold significant promise in the era of LLMs. 
methods: 
 DESTEIN is introduced as a novel language detoxification method, which does not require any finetuning of PLMs or training of additional components. This method efficiently detoxifies a given model by modifying internal representations within activation spaces.The comprehensive framework is depicted in Figure 1 . Problem formulation. We focus on the problem of language detoxification within decoderonly models. Specifically, given a prompt p = {p 1 , p 2 , . . . , p t } consisting of t tokens, a language model is capable of generating coherent text from this input. The objective of language detoxification is to reduce the occurrence of toxic content, such as insults, threats, profanity, and related elements, during the text generation process (Weidinger et al., 2021) .Transformer blocks. We provide a concise overview of the key components in decoderonly LMs. These models fundamentally consist of stacked transformer blocks (Vaswani et al., 2017) , featuring multi-head self-attention (MHSA) modules and feed-forward neural network (FFN) modules. In text generation, LMs initially encode a sequence of t input tokens p 1 , p 2 , . . . , p t into vectors x 0 within the embedding space R d . Subsequently, x 0 is transformed through a series of L layers. Formally, the representation at the l-th layer, x l , is given by:x l = x l-1 + a l + m l (1)The outputs from the l-th layer's MHSA and FFN are represented as a l and m l , respectively. The computational expressions for these processes are delineated as follows:EQUATIONEQUATIONSpecifically, the MHSA employs H heads. The outputs from these heads (h l i ) are concatenated and then linearly transformed using a weight matrix W O . This process yields the final output of the MHSA, represented asa l = W O Concat h l 1 , h l 2 , . . . , h l H .For counterfactual token pairs of a concept that vary only in the value of the concept, if the Linear Representation Hypothesis holds (Mikolov et al., 2013; Park et al., 2023) , their difference vectors in the embedding space should point towards a common direction. For example, the contrast between "king" and "queen" would align significantly with a malefemale direction. Park et al. (2023) demonstrated that the linear representation hypothesis holds for most concepts in language models. Thus, it is plausible to infer the presence of a toxicity-nontoxicity direction within activation spaces.To explore this inference further, we construct "toxicity-nontoxicity" parallel steering pairs,denoted as D = [(S 1 tox , S 1 nontox ), (S 2 tox , S 2 nontox ), ..., (S n tox , S n nontox )],where n represents the number of parallel pairs. Each pair comprises a toxic sample S i tox and a non-toxic sample S i nontox . The toxic sample includes S i tox and its corresponding P i tox (where P denotes prompt) and C i tox (where C denotes completion), while the non-toxic sample is constructed similarly. The process of generating universal steering pairs encompasses four steps: 1) Unconditional generation; 2) Parallel Pairs Generation; 3) Data filtration; and 4) Prompt integration. 1. Unconditional generation. Inspired by Wang et al. (2022) , we diverge from the traditional approach of using a fixed corpus for detoxification. Instead, we leverage the generative capabilities of LMs to produce steering pairs, thereby achieving better data efficiency for detoxification. For GPT2-large, we employ it to generate 10k samples without prompts. The generated samples are then scored for toxicity using the Perspective API1 , and the top 500 most toxic samples are selected. The detailed settings can be seen in Appendix D. 2. Parallel pairs generation. We use GPT4 (OpenAI, 2023b) to generate non-toxic samples parallel to the toxic samples, where "parallel" refers to having the same properties except for toxicity. We utilize the prompt "Please rephrase the following text to convey the same meaning in a non-toxic, respectful, and positive manner: { input text }" to guide GPT4 in generating parallel samples. 3. Data filtration. We filter the generated pairs to select those with similar likelihood levels to be used for calculating the detoxification vectors. This is because pairs with significant differences in likelihood might be biased toward other aspects (e.g., fluency or coherence) rather than the targeted attribute of toxicity. 4. Prompt integration. This process entails the addition of specific prompts to toxic and non-toxic samples, respectively, categorized as either toxicity or non-toxicity cues.Experimental observations have demonstrated that this differentiation aids in creating more efficient detoxification vectors. Given that the focus of this paper is not on prompt design, we employ generic prompts like Leong et al. (2023) .After constructing the steering pairs, we randomly select d instances from D and input them into a language model to extract the corresponding activation space representations for each attention head of every layer, denoted as h. We calculate the detoxification vectors, z, as the average activation difference using all selected data:z = 1 |d| ∑ i∈d h(S i nontox ) -h(S i tox ) (4)where S represents the samples from the randomly selected steering pairs.During the inference phase, the detoxification vectors are integrated into the corresponding activation spaces to facilitate model detoxification. This process is described by the following equation: ĥ(x) = h(x) + α contr z (5)where h(x) represents the output of the attention head, z denotes the detoxification vectors corresponding to the same positional activation space as h(x), and the weight parameter α contr allows for the adjustment of detoxification strength.While the concept of directional vectors from toxicity to non-toxicity within activation spaces has been discussed previously, we must acknowledge that deriving the "toxic-nontoxic" trajectory via direct subtraction is an approximation at best due to the inherent complexity of high-dimensional spaces. Actually, meaningful detoxification vectors can only be obtained when toxic and non-toxic data within activation spaces exhibit good linear separability. This condition is not present in all activation spaces. However, most prior approaches have implemented a one-size-fits-all fusion coefficient for integrating activations, which constrains the adaptability needed to address these complex entanglements. However, our method introduces head-wise fusion coefficients at distinct activation positions, providing a more fine-grained approach to reduce the impact on the model's generative capabilities.Initially, the application of probing techniques has predominantly been within the domain of interpretability for neural networksAlain & Bengio (2017). More recently, such techniques have been extensively applied in the field of natural language processingOusidhoum et al. Based on these studies, our approach incorporates probing techniques to scrutinize the nuances of information encoded within different activation spaces. In simple terms, a common application of probing techniques involves training linear classifiers on representations to explore the information encoded in those representations. Specifically, we articulate the probe's form as σ(h) = sigmoid(w T h) and utilize the steering pairs as our probing dataset. This dataset is partitioned randomly into a training set and a validation set at a 4:1 ratio. Upon this division, head-wise binary linear classifiers are trained, serving as a litmus test for the classifier's ability to distinguish between toxic and non-toxic expressions across different activation spaces. The classification accuracy α prob , obtained for each activation space, is utilized as the coefficient in the activation fusion process. The ultimate formula for activation fusion is thus determined: ĥEQUATIONIn a manner akin to attention mechanisms, we distribute varying degrees of attention across different activation spaces utilizing probes. This approach can partially reduce the impact on the model's generative capabilities. 
experimental settings: 
 Datasets. We use the RealToxicityPrompts (RTP) dataset (Gehman et al., 2020) , comprising 100K text segments. Each segment's beginning serves as a prompt, with toxicity scores annotated using the Perspective API. For fairness, we re-evaluate these scores as Pozzobon et al. (2023a) suggests. See Appendix A for more details. According to the updated scores, we classify prompts with scores below 0.5 as non-toxic and the rest as toxic. We randomly Following the previous studies, we applied detoxification techniques to GPT2large (Radford et al., 2019a) . Moreover, to demonstrate the scalability of our approach across a range of LLMs, we expanded our analysis to include various model families: GPT2-XL (1.3B), LLaMA2 (7B and 13B) (Touvron et al., 2023) , OPT (6.7B) (Zhang et al., 2022) , and MPT (7B) (Team, 2023) .Baselines. For GPT2-large, we compare our approach against two finetuning-based methods: DAPT and DISCUP, and four finetuning-free methods, namely GEDI, DEXPERTS, GOODTRIEVER and SELF-DETOXIFY. For LLMs, our baseline comparison includes SELF-DETOXIFY and a decoding-based method called LMA. More details are provided in Appendix B.Implementation Details. For fair comparison, our experimental setup for generation follows the practices established by Liu et al. (2021) , employing the most commonly used nucleus sampling to generate 25 continuations for each prompt. For GPT2-large, we use α contr =0.38 and m=20. For LLMs, we use α contr =0.3 and m=40. Further details are described in Appendix D.We employed both statistical metrics and the LLM-as-a-Judge approach to enhance the persuasiveness of our results.For statistical metrics, we evaluate generated text on three key aspects: toxicity, fluency, and diversity, following established methods (Gehman et al., 2020) . ( 1) Toxicity: We utilize the toxicity score from the Perspective API, which encompasses both Expected Maximum Toxicity (EMT) and Toxicity Probability (TP). ( 2) Fluency: we compute perplexity (PPL) using a slightly larger model from the same family.(3) Diversity: we calculate the mean of distance n-grams.The results, as shown in Table 1 , indicate that DESTEIN significantly outperforms existing methods. Notably, DESTEIN is among the most fluent methods while also preserving output diversity to a great extent compared to the base model. Additionally, our method does not require any gradient information from the model during the inference process and solely operates on activations, thus not introducing significant inference time overhead. The generation runtime for each method is reported in evaluation results presented in Table 3 . Applying the SELF-DETOXIFY method to LLMs, we observe a significant reduction in detoxification effectiveness and a noticeable decrease in fluency. The LMA method, designed specifically for LLMs, does not achieve satisfactory detoxification results. Due to the different decoding strategies employed by LMA compared to the other methods, direct comparisons of fluency and diversity are not entirely fair. Therefore, we only measured its detoxification metrics. The unsatisfactory performance can be attributed to the considerable parameter scale of LLMs. The SELF-DETOXIFY method, which is tailored for smaller models such as GPT2-large, struggles with adaptation to LLMs. Moreover, the LMA method's failure is due to the complex nature of toxicity across different LLM families. However, without finetuning or auxiliary models, our method still demonstrates competitive detoxification effects across these categories of LLMs, and our method performs well across different model families, proving its effectiveness and scalability.LLM-as-a-Judge. We incorporate GPT-4 and Gemini (Durmus et al., 2022) as supplementary evaluators. We compare our method's generative outputs against top-performing baselines: DISCUP and DEXPERTS for GPT2-large, and SELF-DETOXIFY and LMA for LLMs. For design details about the prompt, please see Appendix C. Based on our findings that the number of prompts did not significantly affect the results, we have extracted a balanced test dataset with 50 toxic and 50 non-toxic prompts for evaluation. The evaluation results indicate that our method exhibits lower toxicity and improved fluency compared to the best-performing baselines. The results are presented in Table 4 . To analyze our method further, we study the contribution of different components on GPT2large. Based on our findings that the number of prompts does not significantly affect the results, we have extracted a balanced test dataset consisting of 50 toxic and 50 non-toxic prompts for the ablation study. Our results, as shown in Table 5 , demonstrate that removing self-induced data significantly raises toxicity (EMT: 0.327 to 0.203, TP: 0.19 to 0.04), whereas the removal of parallel data leads to a less pronounced increase in toxicity (EMT: 0.216, TP: 0.07). Regarding activation fusion, switching activation positions to FFN and FFN+MHSA notably increases PPL (148.14 and 59.848, respectively). These findings highlight the importance of self-induced parallel data and specific activation fusion techniques. Additionally, we observe a slight degradation in results when 'w/o head-wise coefficients' is used. We offer a more detailed analysis in Appendix F. 
trade-off between detoxification and task performance in llms: 
 Existing methods of detoxification often focus on preserving linguistic quality in text generation. However, we emphasize extending evaluation criteria to broader language understanding and reasoning tasks essential for LLMs. Our approach incorporates MMLU (Massive Multitask Language Understanding) benchmark (Hendrycks et al., 2021) to evaluate the impact of detoxification techniques on task performance in LLMs. MMLU assesses how well a detoxified model retains its competence across multiple tasks, including questionanswering, summarization, and sentiment analysis, providing a comprehensive measure of its practical utility. We show the evaluation results of task performance in Table 6 . Table 6 : Evaluation results of the trade-off between detoxification and task performance on the MMLU benchmark with LLaMA2-13b.The findings demonstrate that our method not only achieves exceptional detoxification effects but also maintains the task-solving abilities of LLMs. Specifically, our detoxification method and parameter-efficient finetuning method have a similar impact on the task-solving abilities of the LLaMA2-13b model. For previous methods during the inference stage, when the detoxification strength (α contr ) exceeds a certain critical value, it often leads to a rapid decrease in the fluency of generated text, rendering the text completely unusable. This limitation hinders the effectiveness of many methods in achieving efficient detoxification. However, our method detoxifies layer by layer through representations in activation spaces, which can largely avoid this problem. Our experiments, with results in Figure 2 , demonstrate this.We take the difference between the toxicity scores of the base model and the detoxified model as the detoxification strength and observe the relationship between fluency and detoxification strength. After reaching point A, the model's generative capabilities only deteriorate significantly. However, by the time the control intensity reaches point A, the toxicity has already been reduced to 0.030. Such a low level of toxicity becomes imperceptible to humans. The range of control intensity before point A is already sufficient to meet the model's detoxification needs, so the generative collapse beyond point A does not have an impact.Previous decoding methods were largely based on the logit space, and as the control intensity increased, generation collapse became inevitable. The success of our experiment provides a new perspective for controllable generation that more efficient control may be achievable in activation spaces.Controllable generation methods based on VAEs (Sohn et al., 2015) leverage a lowdimensional bottleneck to learn a disentangled latent space, while LMs often struggle to achieve good controllability at the activation layer due to the complexity of high-dimensional spaces. This proposed method mitigates the complexity of high-dimensional spaces to some extent through steering pairs and head-wise fusion. In this section, we verify the interpretability of the method by analyzing the data distribution in activation spaces.As illustrated in Figure 3 (a), we employ a heatmap to visualize the accuracy of linear classifiers for each attention head across all layers, and to analyze the effects and underlying principles of probing-based weights. To further bolster the credibility of our method, we investigate the existence of a toxicitynon-toxicity vector within activation spaces. Specifically, we focus on the activation space corresponding to the 6th attention head of the 23rd layer, as illustrated in Figure 3(b) . This layer and head were selected due to their high classification accuracy, which was identified through heatmap analysis. We employ PCA to visualize the representation of toxic and non-toxic samples within the activation space. Our results reaffirm the presence of a linear representation between toxicity and non-toxicity in the activation space. Moreover, to more vividly demonstrate the distribution differences in various activation spaces, we also visualize the space associated with the 12th layer and 7th head, as illustrated in Figure 3 (c), which exhibits a classification accuracy nearly equivalent to that of random selection in the heatmap. The distribution of toxic and non-toxic samples within these two spaces further validates our method's theoretical soundness. 
conclusions: 
 In this paper, we propose a novel approach to toxicity mitigation, named DESTEIN, which is based on the activation space for detoxification. Specifically, our method utilizes the model's self-induced data to identify the detoxification vectors. During the inference phase, the detoxification vectors and the output of the module are merged, and probe techniques are employed to apply weights to head-wise activations. Experimental results demonstrate that our approach not only achieves efficient detoxification but also preserves the generative capacity of the model to the greatest extent possible. Notably, DESTEIN maintains considerable performance in toxicity mitigation without significantly increasing inference time. Unlike previous methods, DESTEIN exhibits flexibility and competitiveness when dealing with LLMs and is scalable across different model families. 
limitaions: 
 Our method, predicated on the assumption of linear representation, employs arithmetic operations to decouple toxic attributes and general capabilities, yet this approach is notably constrained. Initially, constructing an ideal parallel pair poses considerable difficulty. Moreover, this method represents an indirect form of decoupling, facing theoretical challenges in achieving complete separation. Consequently, there is a necessity to explore more efficient and versatile decoupling strategies, such as those based on causal reasoning, knowledge guidance, or meta-learning techniques. These methods aim to effectively unify general capabilities with safety, delineating our subsequent direction of research. 
ethics statement: 
 In our experiments, we are careful with the sensitive datasets and the potential for offensive content generation by language models (LMs), aiming to understand and mitigate toxic language dissemination. We recognize the risk that our method for refining LMs to reduce toxic outputs could be misused to create offensive content, a possibility at odds with our ethical commitment to improving LM content safety and integrity. We firmly advocate for the responsible use of LMs, ensuring our work contributes positively to society and opposes any application that promotes harmful communication. deployment alongside pre-existing toxicity classifiers. This integration represents a viable security measure for large model deployment, which is a direction for our future work. 
 
 Appendix: Warning: Some example model outputs contain highly offensive or disturbing text.In this section, we provide some specific examples of different detoxification methods. In our experimental setup, each prompt corresponds to generating 25 continuations, from which we select the most representative ones among the top three in terms of toxicity. The examples can be seen in Table 14 . https://www.perspectiveapi.com/ 


Decision: This paper proposes an inference-time intervention approach to reduce the toxicity of model generations. While the approach seems somewhat incremental compared to various works on steering vectors, it is more lightweight than the baselines, and the experimental results show meaningful improvements. The discussion period has been productive, and the constructive feedback from the reviewers and engagement from the authors have led to significant improvements in the paper.

I’d like to highlight that while the authors show inspiration from Turner et al. (2023) and Park et al. (2023), there is no mention of more recent work that uses paired samples for generating steering vectors, such as "Steering Llama 2 via Contrastive Activation Addition" (Rimsky et al., 2024) or activation intervention such as "In-context Vectors: Making In-Context Learning More Effective and Controllable Through Latent Space Steering" (Liu et al., 2024). The latter particularly focuses on steering attention output, and the application is reducing toxicity. Better positioning with respect to these works, even if mentioned as “concurrent work,” would further improve the draft.

Reviews: review: paper_summary: paper_summary: Authors propose a method to ensure LM generated code is free of toxicity. This is done by computing a difference in the activations level of tokens from parallel toxic-nontoxic sequences, and adding this one set of universal detoxification vectors to activations during inference. This is shown to be effective in ensuring generation free of toxicity without hurting the linguistic quality (fluency, diversity) from LMs like GPT-2, Llama-2, MPT and OPT.
strength_weakness: strength_weakness: 1. The proposed approach is not sufficiently investigated or motivated - why is the merging/integration only applied on attention head activations and not on other ones e.g. FFN activations? There is no ablation study conducted to justify the effectiveness of this choice.

2. Writing issues, particularly with the notations used make it difficult for the reader to follow the paper. See questions 2 & 3 below. Fig 1 is hard to understand, and doesn't convey the method well.

3. The authors don't mention limitations like the memory footprint required to store the fixed/universal steering pairs for different models. This requirement scales with model size (n_layers). See question 4 below.

4. The scope of evaluation is limited to open ended text generation from GPT-2 style language models. While preserving linguistic quality of text generated from an LM is a good metric to assess a detoxification technique, in the context of present research on language models, the method could be made a lot more convincing by demonstrating its effectiveness by detoxifying models like Llama/OPT/MPT/Phi-2 while preserving their performance on benchmarks that cover language reasoning and understanding tasks. The universal steering proposed by the author should be verified to provide detoxification without hurting performance of LMs on practical tasks beyond open ended text generation.
questions: questions: Suggestions:
1. Sec 2, Para 2: Authors could also mention that fine-tuning methods can possibly lead to catastrophic forgetting (model performing poorer on the tasks it has seen before non-toxic fine-tuning).
2. Qualitative samples can help the reader to better understand the comparison between baselines and the proposed approach in Sec 4 or 5.

Questions:
1. Sec 2, Para 4: "The method we propose can also be categorized as a variant within the spectrum of prompt-based detoxification approaches" I did not follow this claim. The proposed method DESTEIN applies a shift by a pre-computed tensor to the activations in the transformer. How is this a prompt-based method then?
2. Sec 3.2, Para 2: "and its corresponding $P_{\text{tox}}^{i}$ and $C_{\text{tox}}^{i}$" - what are P and C denoting here? prompt and completion? 3.1 used the lower-case $p$ to denote prompts, so it's confusing for the reader to follow this. $p$ is again used in 3.3 to denote the sigmoid function, this can be resolved by using $\sigma$.
3. Sec 3.2 Para 8: "we randomly select $l$ instances and input them ..." $l$ has been previously used as layer index in Eqns 1-3, while $L$ has been used in Para 2 Sec 3.2 for the number of parallel steering pairs. Can you resolve this notation mismatch?
4. Equation 4: What is the dimensionality of $\phi^i$s? You could assume $N$ tokens, each represented by a vector in $\mathbb{R}^d$, as is typically done in self-attention mechanisms. Do you compute a $z^i$ corresponding to each position $i$? Is this computed differently for each attention head separately or is applied to the final output from all heads?
5. Why are there $\alpha_{prob}$ and $\alpha_{contr}$ two different hyper-parameters in the method? I believe a similar effect can be achieved just by scaling one of the two.
6. I could not clearly follow how the operation described in Eq 6 is applied during inference. Autoregressive decoding from an LM occurs one token at a time. Say you're predicting the $i$-th token, would the method then involve modifying as per Eq 6 the representations of all previously generated tokens at all layers (only MHSA output) to predict the $i$-th token? Modifying $a_i^l$ would also modify $m_i^l$ (Eqn 2 & 3).
7. Sec 4.1: Are any tests of statistical significance conducted to verify the effectiveness of detoxification on these 1k prompts and their responses as per DeStein?
8. Sec 4.1: How are the baselines implemented or are these results reported from prior work? Does $l$=20 here refer to layers or instances?


Minor writing issues/typos:
Para 1: No space between "(LLMs)" and the citation "(OpenAI ..". This is a recurring issue throughout the paper with all the references.
Sec 3.1 Para 1: We focuses --> focus
Sec 3.2 Para 1: counterfactuals --> counterfactual
Sec 3.2 Para 1: should be "Linear Representation Hypothesis holds (Park et al. (2023))" (space and brackets)
Sec 3.2 Para 1: should be "Park et al., 2023 demonstrated ..." (no brackets)
Sec 3.2 Para 3: the four steps should be captialized for consistency with the paragraphs below (e.g. "Unconditional Generation" instead of "unconditional generation").
Page 5 last para: predicated on Probing --> probing; that exhibit --> exhibits
paper_summary: paper_summary: This paper introduces an activation-intervention method for detoxification of language models. In particular, they (1) sample unconditionally from the target LM and filter for toxic examples, (2) use a much larger instruction-tuned model to produce non-toxic paired examples, (3) estimate differences in layerXattention_head activations between toxic and non-toxic examples, (4) estimate which (layer, attention head) best represents toxicity, (5) add in the non-toxic minus toxic activations average vector during inference to detoxify.

The paper explores various LMs, including GPT-2-large and LLAMA-2-7B. The results are strong in detoxification, and there is a LLM-based preference evaluation showing that their method’s outputs are preferred over other detoxification methods.

There are some interpretability claims in the paper (see below.)
strength_weakness: strength_weakness: I’m concerned that PPL under GPT-2-XL is used to estimate fluency; this is not a very good model, and e.g., highly repetitive nonsensical text will achieve very low PPL under the model. But the inclusion of LLMs as judges does help alleviate this.

It’s unclear that the probing-decided per-attention-head weights are useful; the ablation in the appendix (as far as I can tell) doesn’t compare to the proposed method. Or if the two tables are comparable, then the fixed-to-0.5-ablation method seems pretty much the same as the proposed probing-based method. So, from the data provided, I don’t really see the head-wise weights as being very important to the method. A lot of space in the paper is given to this head-wise weighting through probing, so this is a bit of a red flag to me, but I’m not giving the paper a lower score because I’d even prefer if the no-probing method worked just as well, since it’s simpler. If the authors want to keep their arguments about disentanglement estimation through probing, though, I really think they need to make it clearer that it actually helps improve the method over a baseline.

Unfortunately, the authors try to use t-SNE to validate that there is a linear representation of toxicity within a representation space. This experiment, in which the post-tSNE space has linear separation, says nothing about linear separation in the original space, because tsne is a highly non-linear dimensionality reduction method. I strongly recommend that this claim be removed, or that the plot be replaced with a linear dimensionality reduction method, e.g., PCA, and linearity checked there.

The methods background 3.1 and methods description in 3.2,3.3 are too vague to get a precise understanding of what activations are being acted upon. (e.g., is it the head-level lower-dimensional vectors? I’d imagine so, but the stated math seems to suggest the full-dimensional token vectors before reshaping to dim//heads.)

The motivation of the linear representation hypothesis seems somewhat confused, especially given section 3.3, wherein the authors state that linearity is at best an approximation  due to “false correlations” or “biases” in the dataset, and that “entanglement of toxicity with unrelated attributes varies significantly across different activation spaces” all without citation or data to explain or provide evidence. There are a few instances of this kind of somewhat opaque paragraph in this paper that I find difficult to square with the otherwise straightforward method.
questions: questions: Please fix many of your citations, which do not have proper spacing.

Saying layer normalization isn’t crucial to your analysis is odd just before providing an equation for the value of a vector under a transformer at a layer and index, since the layer normalization is considered very important in that equation.

> Intuitively, to achieve detoxification without compromising the internal knowledge, the activation space at the output location of the MHSA should be the most optimal.

I don’t share this intuition. Can you explain?

When you say the activation space of each attention head, I’m confused – do you mean the model_dim//head_size dimensional vectors that are computed during MHA, but not passed through the rest of the model?

Is the probe really p(a_i^l) = sigmoid(a_i^l)? This both (1) includes no learnable parameters, and (2) has an output space of the dimensionality of a_i^l, not, e.g., 2 or 1, to allow for classification of toxic/non-toxic. Do you mean  p(a_i^l) = sigmoid(w^\top a_i^l) for some parameters w?

In equation 6, why add 1 to \alpha_\prob? As opposed to 0 or some other constant?

>  Nonetheless, this methodology, predicated on Probing for weight distribution, facilitates a heightened focus on activation spaces that exhibit a higher degree of disentanglement.

I disagree; just because the activation space is better at predicting toxicity does not mean it’s not also better at linearly predicting various other properties. In fact, many probing studies find that layers that best-encode one property often best-encode other properties as well. (e.g., Liu et al., 2019. Linguistic Knowledge and Transferability of Contextual Representations).

On page one you claim there’s a need for an “interpretable” method for detoxification – however, despite the brief experiments in 5.2, I don’t really see how this method is any more interpretable than, say, finetuning – and the claim isn’t really backed up, so I’d encourage you to remove that from page 1.
paper_summary: paper_summary: This paper presents a novel approach to detoxifying Language Models with detoxification vectors. By modifying LMs’ internal representations within the activation space, the proposed method effectively mitigates the need for increased inference time and computational resources. Experimental results across various LMs demonstrate the efficacy of the proposed approach in detoxifying LMs compared to existing methods.
strength_weakness: strength_weakness: While the paper is well-written, it lacks comprehensive analysis to strengthen the validity of the proposed method.

The method's efficacy appears heavily reliant on the quantity and caliber of parallel steering pairs utilized, yet this aspect receives limited scrutiny in the paper. There appears to be an absence of analysis regarding the requisite amount of data and its quality for effective implementation. Furthermore, the paper seems to lack an investigation of the impact of varying the hyperparameters on the results. Examining these would provide insight into the robustness of the method and its sensitivity to parameter settings.
questions: questions: It seems that some variable descriptions are missing. (ex. $P^j_{tox}$ and $C^j_{tox}$ in Section 3.2)
I think it would be good if some examples of generated ($S_{tox}$, $S_{nontox}$) were shown.
paper_summary: paper_summary: The paper introduces DESTEIN, a novel method for reducing toxic outputs in language models by altering their internal representations in the activation space, using less resources and time. Empirical tests show that DESTEIN outperforms existing methods in detoxification efficacy while preserving output quality and diversity, and is scalable across multiple large language models.
strength_weakness: strength_weakness: 1. The selected models are all around 7B parameter size, which limits the versatility of the proposed method (from model parameter size perspective). Would be nice to include small (2B) and larger (13B) models.
2. Some technical details are not clear enough, see questions.
3. Using accuracy to determine weight is not very intuitive, would be nice to clarify motivation. And why need to plus 1 in equation 6. Last, is this weight change dynamically during inference or fixed after one run?
4. The method's generalization needs to be explored. Authors use train and test data from the same source, which share a similar distribution. In the real world, we might not have seen the toxic data before, which means the distribution of prompt input is very different from the distribution of steering pairs prompt. So it would be interesting to see the performance from different sources to show the generalization.
questions: questions: 1. In section 3.2, the $P_{tox}^{i}$ and $C_{tox}^{I}$ is not clear, would be better to clarify these notations.
2. The meaning of l in equation 4 and 5 is different, one for instance and the other for layer. Would be better to change a notation.
3. In equation 4, is z computed for each layers or shared across all layers? And how do authors handle the token dimension or sequence length difference is not clear.
4. The linear classifier in linear probing is not clear, is it a single trainable matrix or heuristics classifier.

Comments: title: Response to rebuttal
Thank you; my positive review remains unchanged.
Thank you for the response.
Thank you for your response!
comment: Thank you for raising your evaluation score. We truly appreciate it! And thank you for the thoughtful suggestions to improve the paper, both regarding the discussion of the methodology and in improving the experimental section.
I thank the authors for the responses in their rebuttal. I find most of the concerns addressed from my review, and therefore am revising my score accordingly.
Thank you very much for your positive feedback and for your valuable suggestions. We agree that your suggestion is reasonable. Given the limitations of GPT-2 L, using MMLU for evaluation may not be an appropriate choice. We will revise this part accordingly. Thank you once again for your time and consideration.
I thank the authors for their response and look forward to an improved paper. My recommendation to accept remains unchanged.

I discourage the authors from using the MMLU scores to motivate their method. The baseline accuracy is 25%, and anything 27% or below is pretty reasonably just due to noise, so I don't think 27% vs 25.5% vs 26.1% are meaningfully different.
Thank you for taking the time to review our paper and providing insightful feedback! 
Furthermore, regarding your curiosity, in fact, when the $\alpha_{contr}$ is around 0.6, the generated texts are still fluent. The slight increase in PPL during the experiment is inevitable for the detoxification task because when facing toxic information in the prefix, the detoxified model has to do its best to reverse the situation, resulting in an increase in PPL. However, when the detoxification strength is excessively high, a collapse occurs, as analyzed in section 5.1 of our paper. This is predictable because we can't fully decouple toxicity, although we have made many improvements towards this goal. 
Finally, thank you again for recognizing our work and for your insightful feedback.
We're happy to know that our response helped in addressing your concerns. Thanks again for the helpful feedback and for upgrading the score!
Thank you for your additional explanation. I now have a deeper understanding of how $\alpha_{contr}$ can control the detoxification strength. However, I am curious about how the generated text would change with a higher $\alpha_{contr}$ value (e.g., 0.6). 
Overall, the paper is well-written, and I will maintain my current score.
Thanks authors for the detailed response, which address most of my concerns. I will raise my score accordingly.