Decoupling Noise and Toxic Parameters for Language Model Detoxification by Task Vector Merging

Authors: yongmin  kim
takeshi  kojima
yusuke  iwasawa
yutaka  matsuo
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 04:16:20

Abstract: The goal of detoxifying language models is to reduce the chances of producing offensive or harmful output in pre-trained language models (PLMs), ensuring their safer use. A recently proposed detoxification method utilizes the task vector obtained by subtraction from the fine-tuned model on toxic datasets to the pre-trained model. This approach has shown effectiveness for detoxification but still suffers from degradation. This study focuses on further mitigating degradation while maintaining detoxification performance. To mitigate the degradation, we propose a method that detoxifies the PLMs by fine-tuning multiple models on split toxic datasets and by merging the subtracted task vectors. 
We conducted experiments on two toxic datasets (Civil Comments and Toxigen) with five PLMs (GPT2-small, GPT2-medium, GPT2-large, Phi-1.5, and Llama2-7b), demonstrating that our method consistently achieves a lower toxicity score while preventing the degradation compared to baseline methods.
Especially, with the GPT2-small model on the Toxigen dataset, degradation was reduced by 38.9\% compared to that of an existing task vector method while maintaining a similar toxicity score.
In addition, we found that merging multiple detoxified models tends to increase the number of parameters that remained almost unchanged from the pre-trained model.
We assume that by merging multiple detoxified models, "decoupling noise and toxic parameters" is implicitly achieved. The accidental noise in the parameter shift unrelated to detoxification disappears by averaging noise, whereas the parameter shift associated with detoxification is maintained.
We hope that the findings of this study will be applied not only to detoxification but also to many other research domains that seek to suppress undesirable outputs of language models.

Main paper: 
 introduction: 
 Although pre-trained language models (PLMs) have shown remarkable capabilities across many domains over the past few years (Radford et al., 2019; Brown et al., 2020; Li et al., 2023; Touvron et al., 2023) , PLMs have the risk of generating offensive and aggressive contents (Gehman et al., 2020; Liang et al., 2023) . Toxic data present in pretraining datasets causes PLMs to generate rude and biased sentences. Recently, to enhance the secure utilization of LMs, the crucial challenge of detoxifying LMs has attracted growing research interest (Kumar et al., 2023) .To mitigate the risk and provide safer and non-aggressive PLMs, detoxification methods that prevent the generation of toxic sentences have been gaining attention. Previous studies (Liu et al., 2021; Krause et al., 2021; Kwak et al., 2023) proposed detoxification methods that adjusted the output probability of the next token to prevent potentially toxic tokens. Other Then, we fine-tune the base-LM on each sub-dataset. We detoxify each model using the task vector negation. We then merge these detoxified models into one model to mitigate the degradation.studies (Gururangan et al., 2020; Wang et al., 2022; Lu et al., 2022) proposed methods of reducing the toxicity of the model through post-training.Recently, Ilharco et al. (2023) proposed another method, a task vector operation, which achieves detoxification from a toxic dataset with arithmetic operations on the model's parameters. The task vector is the difference in the parameter space between a fine-tuned model and a pre-trained model. They have shown that simple arithmetic operations, such as subtracting the task vector to a pre-trained model, can effectively unlearn undesirable attributes, such as toxicity. However, as other methods, this detoxification method degrades the existing LMs' ability. A trade-off relationship exists between the detoxification performance and the degradation of LMs' capabilities, which needs improvement.Inspired by the recent success of the model merging technique Gueta et al. (2023) ; Wortsman et al. (2022) , which merges multiple model parameters to achieve higher performance, we propose a novel method that merges multiple task vectors to mitigate degradation further and boost the detoxification performance. Specifically, we randomly split a whole toxic dataset into N sub-datasets. Then, we fine-tune the base-LM on each sub-dataset. We detoxify each fine-tuned model using task vector negation and merge the N detoxified models into one model (Figure 1 presents an overview of our method).To validate the effectiveness of our method, we conducted experiments on two toxic datasets (Civil Comments and Toxigen) and five PLMs (GPT-2, GPT-2 medium, GPT-2 large, Phi-1.5, and Llama2-7b). Our experiment results demonstrate that our proposed method consistently achieves higher detoxification while suppressing degradation compared to baseline methods, including the standard task vector negation approach. We found that the individual models detoxified from each sub-dataset have much less degradation than a model detoxified from the entire dataset. We also found that the merged models showed a similar level of toxicity with less degradation compared to the individual models detoxified from sub-datasets. Furthermore, to investigate the root cause of the degradation mitigation achieved by our method, we measured the similarity between our detoxified models and the pre-trained model in the parameter space. We found that merging multiple detoxified models tends to increase the number of parameters that remained almost unchanged from the pre-trained model. We assume that by merging multiple detoxified models, "decoupling noise and toxic parameters" is implicitly achieved. The accidental noise in parameter shift unrelated to detoxification disappears by noise averaging, while the parameter shift associated with detoxification is maintained. 
detoxification: 
 The ability of PLMs to generate text across various contexts is derived from their pre-training on large, multi-domain textual datasets (Prabhumoye et al., 2023; Korbak et al., 2023) . Nonetheless, given that these datasets are scraped from web content, they unavoidably include harmful material (Gehman et al., 2020; Gao et al., 2020; Penedo et al., 2023; Kumar et al., 2023) , thus increasing the likelihood of PLMs producing toxic outputs. The objective of language detoxification is to minimize the chances of producing continuations that contain toxic elements. Toxic elements are defined as aspects of the text likely to be perceived as impolite, disrespectful, or offensive (Kwak et al., 2023; Schick et al., 2021) .Recent studies address language model toxicity by aligning outputs with human preferences (Radford et al., 2019) , modifying decoding to suppress toxic tokens (Liu et al., 2021; Krause et al., 2021; Kwak et al., 2023) or exploring detoxification through toxic prompts (Leong et al., 2023) . However, these methods need additional inference time. In contrast to these approaches, certain research works (Gururangan et al., 2020; Wang et al., 2022; Lu et al., 2022; Tang et al., 2024) have proposed methods to fine-tune non-toxic data. In line with these methods, Ilharco et al. (2023) has demonstrated the effectiveness of the task vector negation method to detoxify LMs. The task vector approach can utilize only toxic data for detoxification as opposed to the previous trainable methods. Ilharco et al. (2023) introduced the concept of the task vector, demonstrating that through arithmetic operations on the task vector, one can enhance the model's performance, unlearn an undesired attribute, or improve domain generalization. Their study proposed a method of detoxification using the task vector from a model fine-tuned on a toxic dataset. The detoxification performance of this method was better than that of methods fine-tuned on non-toxic datasets and it led to less degradation compared to that of methods using the gradient ascent. In contrast to the existing detoxification methods, this approach uses only toxic datasets for detoxification.However, this approach for detoxification weakens the inherent abilities of LMs, highlighting a trade-off between detoxification efficacy and degradation of a model's performance. We have demonstrated that, compared to the existing task vector method, our approach further mitigates the degradation from the detoxification at the same toxicity. Furthermore, our method is cost-effective compared to other trainable methods (Gururangan et al., 2020; Wang et al., 2022; Lu et al., 2022; Tang et al., 2024) because it uses only toxic data.Recent studies (Gueta et al., 2023; Wortsman et al., 2022) have shown that merging the weights of different models can yield a model with better performance than that achievable with trainable methods. Wortsman et al. (2022) proposed a method to efficiently merge different models linearly by selecting models from candidates. Furthermore, Matena & Raffel (2022); Daheim et al. (2024b) ; Yadav et al. (2023) proposed more computationally efficient and effective methods for fusing multiple models to enhance performance. In this study, we adopt the linear merging method and additionally apply Tie-Merging (Yadav et al., 2023) to the proposed method. 
method: 
 3.1 Preliminary Ilharco et al. (2023) proposed a method for detoxification by negating the task vector calculated with a model fine-tuned on a toxic dataset. This method proved to be more effective in detoxification and less prone to performance degradation compared to methods fine-tuning on non-toxic datasets or employing the gradient ascent on toxic data. Suppose that θ pre ∈ R d represents the pre-trained model's parameters, and θ t ft ∈ R d is the parameter after fine-tuning for a particular task t. The task vector, denoted as τ t ∈ R d , is calculated by the element-wise subtraction of θ pre from θ t ft , expressed as τ t = θ t ftθ pre . If the task is apparent, we omit the identifier t and simply notate it as τ. These task vectors can be utilized across any model parameters θ from an identical architecture by adding them element-wise with a scaling coefficient λ, leading to a new model parameter θ new = θ + λτ. Negating the task vector τ by applying the inverse vector τ new = -τ extends the finetuned model back towards the pre-trained model. This operation leads to a diminished or unlearned model's ability to perform the specific task for which it was fine-tuned (e.g., toxicity).Inspired by the recent success of model merging techniques (Gueta et al., 2023; Wortsman et al., 2022) , which merge multiple model parameters to achieve higher performance, we propose a novel method of merging multiple task vectors to mitigate degradation further and enhance detoxification. Specifically, we divide the dataset D into N sub-datasets D (N,i) , ensuring that no common parts exist among them, which can be mathematically represented as follows:N i=1 D (N,i) = D.(1) Zaman et al. (2023) demonstrated that when fusing several models, the task abilities of each model are reinforced in their common aspects and suppressed in their differing aspects after merging. Furthermore, they showed that merged models have an improved generalization ability. Based on these findings, we hypothesized that merging models fine-tuned on each sub-dataset would result in a suppression of the degradation, an unshared attribute across the models. Ultimately, we detoxify the pre-trained model by negating the task vector of merged models. Although many methods exist for merging models (Yadav et al., 2023; Daheim et al., 2024b; Matena & Raffel, 2022) , we adopted a simple approach to clarify the effectiveness of our method. Specifically, we considered merging by computing the weighted average of the model parameters. Let there be N models that are fine-tuned to each subset of a toxic dataset D (N,i) with parameters {θ 1 . . . θ N } such that ∀ i θ i ∈ R d . Then, we define the merged model, θ merged , as a convex combination:θ merged = N ∑ i=1 α i θ i (2)where α i ≥ 0, ∑ N i=1 α i = 1. We employ the average for weight merging, whereα 1 = α 2 = • • • = α N = 1/N.From the merged model θ merged , we can calculate the task vector τ merged = θ mergedθ pre . Consequently, we can obtain the detoxified model θ detoxic by the negation of the task vector τ merged to the pretrained model θ pre with coefficient λ. To be specific, we compute the detoxified model with Equation (3).θ detoxic = θ pre -λτ merged .(3) 
experiment setting: 
 In this study, we detoxified the models of GPT2 (Radford et al., 2019) , Llama2-7b, and Phi-1.5 (Li et al., 2023) using a toxic dataset included in the dataset. For the proposed method, we divided the entire toxic dataset into sub-datasets and fine-tuned the models on them. Then, we merged those models and calculated the task vector from it. Finally, we detoxified the model with the negation of the task vector. To verify the proposed method in diverse settings, we conducted experiments through multiple datasets for toxic data.We employed Civil Comments (Borkan et al., 2019) and Toxigen (Hartvigsen et al., 2022) as toxic datasets for detoxification. We made use of highly toxic data included in these datasets.For the toxicity evaluation, we used prompts in RealToxicPrompts (Gehman et al., 2020) and Toxigen (Hartvigsen et al., 2022) . We sampled 1K prompts from these datasets. We provide more detailed information about the datasets in Section C.1For the baselines, we used a set-up from Ilharco et al. (2023) for fine-tuning on the non-toxic dataset, and trained the model to the toxic dataset with the gradient ascent approach (Tarun et al., 2021; Jang et al., 2023) . We fine-tuned the models to the non-toxic data (toxicity score lower than 0.2) included in each dataset, in a similar way to Liu et al. (2021) . We did not use the entire non-toxic dataset; instead, we sampled the same number of non-toxic data from the entire toxic dataset as the toxic data we used. In the detoxification experiments using gradient ascent, we maximized the negative log-likelihood loss on the toxic dataset. Finally, we compared the proposed method with detoxification using an entire toxic dataset realized by Ilharco et al. (2023) .We did not conduct a comparison between our proposed method and existing selfdetoxification approaches (Wang et al., 2022; Tang et al., 2024) . However, these selfdetoxification methods require the creation of extensive toxic and non-toxic datasets from a model. These datasets need to be generated for each model, and thus, a large amount of data are required for each detoxification. By contrast, the negation method adopted in the proposed approach allows for efficient detoxification using only existing toxic datasetsFirst, we examine the impact of the proposed method on different sizes of sub-datasets N in Section 5.1. Second, we investigate the effectiveness of merging models detoxified from the split datasets in Section 5.2. Third, we compare the proposed method with the existing task vector negation approach by examining the parameter space of the detoxification models in Section 5.3. Specifically, we measure how different the parameters of detoxification models, which have the same toxicity level, are from the pre-trained model. Finally, we compare our approach with the baseline methods in Section 5.4.In each experiment, we measured the toxicity and degradation of the models to compare the baseline methods and our approach. According to Gehman et al. ( 2020), a toxic sentence is defined as "a rude, disrespectful, or unreasonable comment that is likely to make you leave a discussion". Toxicity is evaluated through the toxicity score measured by pre-trained toxicity detectors (Hanu & Unitary team, 2020; Hartvigsen et al., 2022) and Perspective API2 , which is widely used as a toxicity detector. Different toxic detectors were employed depending on the experiment. Detailed information can be found in each experiment section. In Section 5.1, 5.2, 5.3, we assess the model's degradation through perplexity. In Section 5.4, we also measure the average performance of models on seven downstream tasks to comprehensively evaluate the models' degradation, in addition to assessing perplexity. 
impact of the sub-dataset size: 
 An increase in the number of sub-datasets mitigates degradation while maintaining a similar toxicity level. To estimate the impact of the number of sub-datasets N on detoxification, we conducted the experiments by setting N to 3, 5, and 10. Then, we compared those models with a detoxified model from an entire dataset. In this experiment, we used a scaling coefficient in the range of λ ∈ {0, 0.05, 0.1, • • • , 1.0} for negating the task vector. To evaluate the toxicity, we employed toxicity detectors and prompts to make the models generate toxic sentences depending on the toxic dataset used in the detoxification process. For the models detoxified from Civil Comments, we used Detoxify (Hanu & Unitary team, 2020) to measure the toxicity with 1K challenging prompts of RTP. Similarly, we adopted HateBert Toxigen (Hartvigsen et al., 2022) with 1K Toxigen prompts to evaluate the toxicity of models detoxified from Toxigen. We evaluated the model's degradation with fluency by measuring the perplexity on WikiText-103 (Merity et al., 2017) . All the training and evaluation details are described in Section C.2.We present the results of applying the proposed method to GPT2 small, medium, and large models with different numbers of sub-datasets in Figure 2 . In all the experiments on Civil Comments, as the number of used sub-datasets increases, that is, as the size of the sub-dataset decreases, a smaller perplexity was recorded at a similar toxicity. Furthermore, comparing all the models of the proposed method with models using the entire dataset, we found that at a similar toxicity, the perplexity with the proposed method was smaller. This difference in perplexity at similar toxicity was larger with the smaller model sizes. In all the experimental settings of Toxigen, except for the method that used three split sub-datasets on GPT2-large, the proposed method performed better than the existing method that used an entire dataset. Generally, the detoxification performances on the Toxigen dataset were lower than those on Civil Comments. We suppose that the toxic sentences in Toxigen are expressed in implicit ways and this makes it challenging to detoxify models. Figure 3 : Perplexity and toxicity score before and after merging models detoxified from the five split sub-datasets in Civil Comments and Toxigen. The label '5 split' refers to the models fine-tuned on each sub-dataset. 'Average' indicates the averaged model from the models fine-tuned to each sub-dataset. The 'All (negated)' label means the models detoxified from the entire toxic dataset. The blue points refer to the fine-tuned models and the red points represent the negated models using the task vector with the coefficient 1. For the 'All (negated)' models, we set the coefficient so that they have the closest toxicity to the toxicity that each averaged model has. Models with perplexity over 40 are not included.A merged model mitigates degradation further than individual models before merging.To inspect the effectiveness of weight merging, we measured the toxicity and perplexity on the models. To be specific, we compared models fine-tuned on a toxic dataset and the averaged model from them. We also evaluated the negated models from these models. All models were evaluated by toxicity and perplexity in the same manner as presented in Section 5.1.Figure 3 shows the results of each model when using the proposed method with the five sub-datasets of the Civil Comments and Toxigen toxic datasets. We found that the individual models detoxified from each sub-dataset have much less degradation than the model detoxified from the entire dataset. This finding aligns with Jang et al. (2023) , which showed that using gradient ascent for unlearning results in a degradation of the model's original capabilities as the size of the dataset for unlearning increases. We assume that this phenomenon occurs not only in gradient ascent unlearning but also in unlearning with a task vector. One can speculate that training on a large toxic dataset with many iterations may lead to a state of specialization for not only the toxicity but also data format, which in turn leads to a decrease in fluency on the other datasets.We also observed that the merged models recorded a lower perplexity than individual models trained on each sub-dataset before merging. These differences in perplexity became larger with smaller model sizes. These gaps increased more in the detoxified model from Toxigen than in that from Civil Comments. The merged models had the largest reduction The '5 split*' label refers to mean results from individual models detoxified from split 5 sub-datasets. The 'Average' label indicates the results of the proposed method on split 5 sub-set datasets. For the 'All' results, we scaled them for the negated models to have the closest toxicity score to the averaged models.in perplexity after negation compared to that before. Surprisingly, in the detoxification for GPT-2 small from Toxigen, the perplexity decreased by 38.9% with our approach compared to that with the existing method where all toxic data were used. This result of the detoxified model from a whole dataset was not indicated in Figure 2 because the perplexity was 56.27, which exceeds 40. This demonstrated that merging several models detoxified from a split dataset can mitigate the degradation effectively while maintaining similar toxicity.Our method experiences a smaller amount of parameter shift from the pre-trained model than the standard task vector negation method. We measured the similarity between a detoxified model and a pre-trained one. We employed L 1 -norm to evaluate the similarity and divided it by the number of model's parameters. We compared L 1 -norm with all the GPT2 models detoxified from the entire toxic dataset and each split sub-dataset, and the averaged models. We set 5 for the split size of the sub-dataset. For the measurement of models detoxified from an entire dataset, we selected the coefficient λ ∈ {0, 0.05, 0.1, • • • , 1.0} for those models to have the closest toxicity with our approach.We present the results of the L 1 -norm between all the detoxified GPT2 models and the pre-trained models averaged on the parameters in Figure 4 . Although all the models have similar toxicity in each setting, except for the detoxified GPT2 small model from Toxigen, the models detoxified from split sub-datasets had a smaller distance than the models detoxified from all the data. Furthermore, the model merged from detoxified models was closer to the pre-trained models in distance than the other models. This implies that detoxified models with a split dataset are more similar to the pre-trained model, and the merged model from them is much closer to it.In addition, we verified the distribution of parameter shift distance between a detoxified model and the pre-trained model. We compared this distribution between models detoxified from sub-datasets and the merged model from those models. For this experiment, we split each dataset, Toxigen and Civil Comments, into five sub-datasets, and detoxified and merged models from them.Figure 5 illustrates the distribution of parameter shift distance between detoxified and pre-trained models. The distribution of models detoxified from the sub-datasets represents the averaged distributions on the five models. In all the experiments, the distribution The number of parameters (×10 5 ) GPT2 -large (Toxigen) 5 split * Average AllFigure 5 : Distribution of parameter differences between the detoxified and pre-trained models. We compared the detoxified models from 5 sub-datasets and the merged model. The '5 split * ' label refers to mean results from individual models from split 5 sub-datasets. The 'Average' label indicates the models averaged from the 5 detoxified models. The 'All' label denotes the models detoxified from an entire dataset. For the 'All' models, we set the coefficient value so that their toxicity becomes the closest to the average model's toxicity.of the averaged models tends to shift to zero more than that of the individual models trained on toxic sub-datasets. Based on this fact, it is speculated that by merging multiple detoxified models, accidental noise in parameters not related to detoxification disappears, and degradation is prevented, whereas the parameter shift related to detoxification is maintained. This interpretation aligns with that of Zaman et al. (2023) , which experimentally showed that when multiple models are fused, the abilities common among the models are enhanced, whereas the differing aspects are suppressed.Our method achieves lower toxicity scores than baseline methods without degradation.We evaluated all the detoxified GPT2, Phi-1.5, and Llama2-7b models with Perspective API for performance comparison of our method with baselines. For those models detoxified from an entire toxic dataset, we selected the coefficient λ ∈ {0, 0.05, 0.1, • • • , 1.0} to have the same perplexity as that of models using the proposed method. In the proposed method, we set N to 5 to split the toxic dataset for all models, and set it to 3 only for the Llama2-7b model. Following previous work (Gehman et al., 2020) , we evaluated the model's toxicity in two manners: 1) the averaged maximum toxicity over 25 generations, and 2) the empirical probability of generating a continuation with toxicity at least once over all generations. Toxicity was evaluated on a dataset of 1K normal prompts randomly sampled from RealToxicPrompts. Additional results on the GPT2 small, large, and Phi-1.5 models are provided in Section D.1. Similar to the experimental setup in Wang et al. (2022) , we measured the model's performance in downstream tasks using a 'Utility' score to assess model degradation. Specifically, we evaluated the model on seven different tasks, which cover question answering, natural language understanding, and commonsense reasoning. More details about each downstream task can be found in C.3. Table 1 : Performance comparison of the proposed method with baselines. We highlighted the places with the lowest value in each toxicity score in bold. For our method, we adapted 1 as the coefficient value. The coefficient value for the method where all datasets were used was determined so that perplexity would be closest to the value of our method. More results on GPT2 small, large, and Phi-1.5 are provided in D.1.Table 1 presents the evaluation results. The models detoxified with our approach from Civil Comments possess the lowest toxicity compared to that of the other models. In the detoxified models from Toxigen, the toxicity of the models with the proposed method decreased much more than that of the other models at a similar perplexity and a similar utility score.For Llama2-7b, we additionally applied Tie-Merging (Yadav et al., 2023) to our proposed method. Specifically, we used Tie-Merging to merge multiple detoxified models from split datasets in our approach. In the results of Llama2-7b presented in Table 1 , 'Tie-Merging' indicates a detoxification model using Tie-Merging, whereas models with a coefficient value λ refer to a detoxification model using linear merging. The results show that our approach with Tie-Merging achieved even better detoxification performance while maintaining a similar level of model performance. 
conclusion: 
 In this work, we propose a more effective method for detoxification by merging task vectors extracted from trained models on split subsets of a dataset. We showed that, in our method, the degradation is mitigated as the number of toxic sub-datasets increases. We experimentally demonstrated that, compared to existing methods, our method can suppress degradation more effectively, resulting in better performance at the same level of toxicity. We also showed that the parameter difference in a model detoxified with our method is more similar to the pre-trained one than that in models detoxified from an entire toxic dataset. 
future work: 
 This study showed that it is possible to further suppress degradation by merging several models detoxified from sub-datasets using the task vector unlearning approach. Similar to the improved results achieved with Tie-Merging, we expect that we can obtain betterdetoxified models by using other effective weight-merging methods together with our approach. In addition, recent research has proposed methods for unlearning not only toxicity but also hallucinations to obtain more faithful models (Daheim et al., 2024a; b) . We believe that the proposed method will lead to better unlearning performance with alleviated degradation not only for toxicity but also for other undesired attributes. 
 
 Appendix: Following Wang et al. (2022) , we evaluated models on the following seven downstream tasks with their brief descriptions. To more accurately assess degradation, we excluded a downstream (Nie et al., 2020) due to its very low score.1. Hellaswag (Zellers et al., 2019) : Tests the model's ability to choose the most plausible continuation of a given context, assessing commonsense reasoning.2. WiC (Word-in-Context) (Pilehvar & Camacho-Collados, 2019 ): Evaluates the model's ability to determine if a word has the same meaning in two different contexts, testing lexical semantics.3. PIQA (Physical Interaction QA) (Bisk et al., 2019) : Assesses the model's physical commonsense reasoning by selecting the most reasonable answer to everyday task questions.4. WinoGrande (Sakaguchi et al., 2021) : Evaluates the model's understanding of pronoun resolution in sentences with ambiguous references, testing commonsense reasoning.5. LAMBADA (Kazemi et al., 2023) : Evaluates the model's ability to predict the last word of a given passage, testing its broad contextual understanding and coherence. 6. RACE (Lai et al., 2017) : Tests the model's reading comprehension skills on passages with multiple-choice questions, used primarily for middle and high school-level texts.7. BoolQ (Clark et al., 2019) : Tests the model's ability to answer yes/no questions based on a given passage, assessing its fact-checking and reasoning abilities.We adopted evaluation codes from Gao et al. (2024) .Here, we provide additional performance comparisons on GPT2 small, GPT2 large, and Phi -1.5 with the baseline methods. Table 2 , Table 3 , and Table 4 present the evaluation results on GPT2 small, GPT2 large, and Phi-1.5 with the baselines. Code is available at https://github.com/oishikimchi97/merge_to_detoxify Warning: this paper includes rude or offensive examples in Appendix.https://github.com/conversationai/perspectiveapi 


Decision: The paper presents an interesting approach to detoxification. Most reviewers find this paper well-written, the proposed method is well-motivated, and it provides a great analysis of the proposed approach. 

The authors responded to all the concerns and provided additional experiment results to address the comments. However, it is not encouraged to use the discussion mechanism to circumvent the word limit of rebuttal. Also, it's not recommended to add a significant amount of new experiments in the rebuttal as this will drastically change the paper and the paper should go to another round of review.

Reviews: review: paper_summary: paper_summary: The paper introduces an approach for language model detoxification by fine-tuning multiple instances of the model on the detoxification dataset splits and then merging the subtracted detoxification vectors. Authors test their approach on GPT-2 (small, medium and large) and Phi-1.5.
strength_weakness: strength_weakness: - **Novelty and overall contribution.** The idea of using task vectors is not novel and have been widely explored in the literature [1, 2, 3]. Moreover, learning a task vector for a text style transfer task is also not novel [4]. Therefore, the contribution 
- **Lack of other merging strategies.** Authors do not consider any other merging strategies except linear. It would be beneficial to consider other merging strategies from related work [5, 6]. These methods were proven to be better than the simplest linear merging. Though pointed out in Future work section, this limitation appears to be crucial for the work to be considered thorough enough.
- **Lack of strong baselines or competing approaches.** Paper lacks comparison to other strong model self-detoxification approaches, e.g. [7]. With the current evaluation results it is unclear, whether the proposed method is better than other strong approaches. 


References:
[1] [Editing Models with Task Arithmetic](https://arxiv.org/abs/2212.04089) by Ilharco et al.

[2] [In-Context Learning Creates Task Vectors](https://aclanthology.org/2023.findings-emnlp.624/) by Hendel et al.

[3] [Towards Universality in Multilingual Text Rewriting](https://arxiv.org/abs/2107.14749) by Garcia et al.

[4] [Specializing Small Language Models towards Complex Style Transfer via Latent Attribute Pre-Training](https://arxiv.org/abs/2309.10929) by Xu et al.

[5] [Merging Models with Fisher-Weighted Averaging](https://arxiv.org/abs/2111.09832) by Matena et al.

[6] [Dataless Knowledge Fusion by Merging Weights of Language Models](https://arxiv.org/abs/2212.09849) by Jin et al.

[7] [CMD: a framework for Context-aware Model self-Detoxification](https://arxiv.org/abs/2308.08295) by Tang et al.
paper_summary: paper_summary: This paper proposes a novel approach combining model merging and task arithmetic-based detoxification. The proposed method has the following steps: a) a toxicity dataset is split, and the toxicity task is learned from the splits by the pre-trained models; b) the learned models are then merged, and the toxicity task vector is learned, and c) the task vector is subtracted from the pre-trained model. The primary claim is that this reduces model degradation while achieving better detoxification.
strength_weakness: strength_weakness: * The primary issue with the paper is its writing. The exact methodology is unclear and hard to follow. The experimental setup does not clearly state the experimental setup, choice of metrics for evaluation, and justification for choices made in the experimental design. 
* Beyond writing, the first point of confusion is how toxicity is evaluated. How is avg. max toxicity and toxicity probs are calculated, as shown in Table 1. 
* Degeneration of the model is evaluated with perplexity on wiki-103. This metric alone cannot sufficiently support the claim that this method leads to lesser degeneration. For example, what happens to other capabilities of the models? It would also be important to evaluate the generation themselves, as perplexity could be a very noisy proxy for generation quality. Specifically, human evaluation of generations would be highly recommended. 
* I think this work has a really good premise. I encourage the authors to add more experiments on generation quality, polishing the writing, and resubmitting it at a future conference.
paper_summary: paper_summary: The paper proposed a method of toxicity mitigation by merging several models fine-tuned on toxic data. Several models are trained on different subsets of toxic examples and then averaged and subtracted from the initial model. Results show that this technique is able to reduce toxic generations up to 38% without noticeable performance degradation.
strength_weakness: strength_weakness: - The paper contains several typos ("detoxify the each model", "paramter").
- The models evaluated are relatively small or non state-of-the-art. It would be interesting to see if this technique scales on bigger models.
- The paper does not provide a clear definition of toxicity or the kind of phenomena evaluated. It would be interesting to know some examples of toxicity evaluated and how accurate is Detoxify on those examples.
- Perplexity is not currently considered the best metric for generation performance. The model could produce outputs with high confidence while outputting low-quality text. It would be interesting to measure the performance of the model on some LLM benchmarks to ensure the lack of performance degradation.
questions: questions: - Have you tried this method on larger models? Does it work at scale?
- What kind of toxic phenomena is evaluated? Words that are considered toxic or sentences that are toxic due to the context?
paper_summary: paper_summary: This paper presents a new approach to detoxify LMs by enhancing task vector merging techniques. The authors propose a method where multiple task vectors, derived from models fine-tuned on separate toxic datasets, are merged. This approach appears to reduce model degradation typically associated with detoxification while maintaining or even improving the LM's ability to suppress toxic output. For instance, using the GPT-2 small model on the Toxigen dataset, the approach achieved a significant reduction in degradation (38.9% less than existing methods) while maintaining comparable levels of toxicity. This suggests that merging multiple detoxified models helps in "decoupling noise and toxic parameters".
strength_weakness: strength_weakness: -	The method's efficacy is demonstrated primarily through reductions in toxicity scores and model degradation rates, but there's a lack of discussion on whether the segmentation of datasets and subsequent model merging might lead to overfitting on the specific characteristics of the datasets used (i.e., Civil Comments and Toxigen). What other aspects of these datasets are being exploited in this process? Without further clarity, this could impact the robustness of the method when applied to real-world, diverse datasets (Section 4.1).
-	Despite the empirical rigour in general, this paper does not explore a wider collection of datasets (or languages) nor does it provide a robust collection of baselines (i.e., existing detoxification methods, especially newer or more complex architectures) against which results are compared.
questions: questions: -	Can your method scale efficiently with larger models such as GPT-3 or GPT-4, and what computational resources are required for such implementations? 
-	Is there a significant increase in training or merging time compared to single-model detoxification approaches?
-	What alternative metrics for model degradation may be used, that were not explored here?

Comments: title: Reminder
Response to Meta Reviewer
A Gentle Reminder to o8PJ
A Gentle Reminder to tRLA
A Gentle Reminder to vKrR
A Gentle Reminder
Additional Response to Reviewer VAs3
Additional Response to Reviewer o8PJ
Additional Response 2 to Reviewer tRLA
Additional Response to Reviewer tRLA
Additional Response to Reviewer vKrR
comment: Thank you for addressing all my concerns and questions. Thanks.
Dear all, 
The discussion period will close soon. Please respond to the authors' rebuttal and let us know if you have any further concerns.
Dear Meta Reviewer,

We have addressed all the reviewers' questions and comments and believe this has resolved all their concerns. 
If our paper is accepted, we will include all our responses in the camera-ready version.
The reviewers’ feedback has further enhanced our paper, and we would like to express our gratitude to them once again.

Specifically, we have listed below the answers we provided to the reviewers’ comments and questions. 

**Additional Experiment**

We have further conducted the following additional experiments to validate the proposed method's effectiveness and robustness.

- We applied the proposed method to a larger and more recent model, Llama2 - 7B, and observed similar performance as in the experiments with the smaller models (GPT-2, Phi - 1.5). (Please see **[our Rebuttal to Reviewer vKrR](https://openreview.net/forum?id=TBNYjdOazs&noteId=qLvE2RNltY)**)

- We conducted experiments by applying the Ties-Merging [1]  in addition to linear merging which we adopted as a merging method in our approach. The results showed that the proposed method is effective even with the Ties-Merging. (Please refer to **[our Rebuttal to Reviewer vKrR](https://openreview.net/forum?id=TBNYjdOazs&noteId=qLvE2RNltY)**)
    
- In addition to evaluating model degeneration by perplexity, we further evaluated it on 7 downstream tasks. The results confirmed that models detoxified by the proposed method have lower toxicity while maintaining similar performance in both perplexity and downstream tasks. (Please refer to the table in **[our Rebuttal to Reviewer tRLA](https://openreview.net/forum?id=TBNYjdOazs&noteId=dHnDleniE3)** for the results, and **[our comment to Reviewer tRLA](https://openreview.net/forum?id=TBNYjdOazs&noteId=04JTMxr8VH)** for descriptions of each downstream task)


**Answers to Question**

We have provided the following answers to the questions received from the reviewer.

- Regarding the comparison with existing detoxification methods, we did not compare our proposed method with existing strong self-detoxification approaches [2, 3]. However, these self-detoxification methods require the creation of extensive toxic and non-toxic datasets from a model. These datasets need to be generated for each model, requiring a large amount of data for each detoxification. On the other hand, the task vector negation method adopted in the proposed approach allows for efficient detoxification using only existing toxic datasets. We have confirmed that our proposed method performs better than other detoxification methods that also use only toxic data (i.e., gradient ascent, and the existing task vector negation [4]). (Please refer to Table 1 in our paper)
    
**Revisions for Writing**

Based on the advice of many reviewers, if our paper is accepted, we plan to make the following changes and additions in the camera-ready version

1. Correct typos and grammatical errors.
2. Clearly describe the purpose and intent of each experiment.
3. Provide more detailed information on the evaluation methods and settings for each experiment (Please refer to **[our Rebuttal to Reviewer tRLA](https://openreview.net/forum?id=TBNYjdOazs&noteId=Tak39hYM8I)**).
4. Include a precise definition of toxicity and examples for each toxicity category (Please also see **[our comment to Reviewer vKrR](https://openreview.net/forum?id=TBNYjdOazs&noteId=X3MKeMdDna)**).

[1] [TIES-Merging: Resolving interference when merging models](https://openreview.net/forum?id=xtaX3WyCj1) by Yadav et al.

[2] [Exploring the limits of domain-adaptive training for detoxifying large-scale language models](https://proceedings.neurips.cc/paper_files/paper/2022/file/e8c20cafe841cba3e31a17488dc9c3f1-Paper-Conference.pdf) by Wang et al.

[3] [CMD: a framework for Context-aware Model self-Detoxification](http://arxiv.org/abs/2308.08295) by Tang et al.

[4] [Editing Models with Task Arithmetic](http://arxiv.org/abs/2212.04089) by Ilharco et al.
**Dear Reviewer o8PJ,**

As of now, we have not received any responses from the reviewer. If the reviewer has any questions or comments about the rebuttal or the results of the additional experiments, we would be happy to answer them. We apologize for the last-minute reminder during your busy time.
**Dear Reviewer tRLA,**

As of now, we have not received any responses from the reviewer. If the reviewer has any questions or comments about the rebuttal or the results of the additional experiments, we would be happy to answer them. We apologize for the last-minute reminder during your busy time.
**Dear Reviewer vKrR,**

As of now, we have not received any responses from the reviewer. If the reviewer has any questions or comments about the rebuttal or the results of the additional experiments, we would be happy to answer them. We apologize for the last-minute reminder during your busy time.
Thank you -- these are clear responses to my questions
**To: Area Chairs and all reviewers**

As of now, we have not received any responses from the reviewers. If any reviewers have any questions or comments about the rebuttal or the results of the additional experiments, we would be happy to answer them. We apologize for the last-minute reminder during your busy time.
Dear Reviewer VAs3,

We sincerely apologize that we could not provide an enough response on the Rebuttal due to the length limitation.
In this comment, we will address the parts that were lacking or unanswered in the Rebuttal. We hope this will help for Reviewers.

We wish to express our appreciation to the reviewers for their insightful comments on our paper. Please find our responses below.

> The method's efficacy is demonstrated primarily through reductions in toxicity scores and model degradation rates, but there's a lack of discussion on whether the segmentation of datasets and subsequent model merging might lead to overfitting on the specific characteristics of the datasets used (i.e., Civil Comments and Toxigen). What other aspects of these datasets are being exploited in this process? Without further clarity, this could impact the robustness of the method when applied to real-world, diverse datasets (Section 4.1).

> What alternative metrics for model degradation may be used, that were not explored here?

As the reviewer mentioned, we acknowledge the possibility of overfitting to other characteristics of the dataset beyond toxicity. However, as we mentioned in this paper, our method has been shown to enhance a common feature (detoxification) and suppress distinct features (data format, etc.) in segmented datasets, which we believe will result in better generality compared to the existing task vector negation method. To further clarify the degeneration of other aspects, we have included additional evaluations on various downstream tasks [1, 2, 3, 4, 5, 6, 7]. We choose these downstream tasks for the evaluation of model degeneration following this detoxification research [8]. We measure this by averaging accuracy on the downstream tasks. We provide a short description for each downstream task on [**our comment to Reviewer tRLA**](https://openreview.net/forum?id=TBNYjdOazs&noteId=04JTMxr8VH). Moreover, we have presented the main experiment results including the downstream evaluations in the table (please see [**our Rebuttal to Reviewer tRLA**](https://openreview.net/forum?id=TBNYjdOazs&noteId=dHnDleniE3)). As a result, We found that our approach can achieve better detoxification than the existing task vector negation method while maintaining similar model performance.

> Despite the empirical rigour in general, this paper does not explore a wider collection of datasets (or languages) nor does it provide a robust collection of baselines (i.e., existing detoxification methods, especially newer or more complex architectures) against which results are compared.

Due to the limited number of existing toxic datasets, as the reviewer mentioned, we were not able to conduct experiments with a wider variety of datasets. However, to ensure the robustness of our method across datasets, we included the Toxigen in addition to the Civil Comments dataset used in previous detoxification research. Furthermore, we conducted experiments with Llama2-7b and presented the results in [**our Rebuttal to Reviewer vKrR**](https://openreview.net/forum?id=TBNYjdOazs&noteId=qLvE2RNltY). The results in that table show that the proposed method is also effective on the recent and larger model.

Regarding the comparison with existing detoxification methods, we did not compare our proposed method with existing strong self-detoxification approaches [8, 9]. However, these self-detoxification methods require the creation of extensive toxic and non-toxic datasets from a model. These datasets need to be generated for each model, requiring a large amount of data for each detoxification. However, the task vector negation method adopted in the proposed approach allows for efficient detoxification using only existing toxic datasets.

Thank you for your advice. We will update our paper following your advice.

[1] [HellaSwag: Can a Machine Really Finish Your Sentence?](https://arxiv.org/abs/1905.07830) by Zellers et al.

[2] [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](http://arxiv.org/abs/1808.09121) by Pilehvar and Camacho-Collados

[3] [PIQA: Reasoning about Physical Commonsense in Natural Language](http://arxiv.org/abs/1911.11641) by Bisk et al.

[4] [WinoGrande: An Adversarial Winograd Schema Challenge at Scale](http://arxiv.org/abs/1907.10641) by Sakaguchi et al.

[5] [LAMBADA: Backward Chaining for Automated Reasoning in Natural Language](http://arxiv.org/abs/2212.13894) by Kazemi et al.

[6] [RACE: Large-scale ReAding Comprehension Dataset From Examinations](http://arxiv.org/abs/1704.04683) by Lai et al.

[7] [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](http://arxiv.org/abs/1905.10044) by Clark et al.

[8] [Exploring the limits of domain-adaptive training for detoxifying large-scale language models](https://arxiv.org/abs/2202.04173) by Wang et al.

[9] [CMD: a framework for Context-aware Model self-Detoxification](http://arxiv.org/abs/2308.08295) by Tang et al.
Dear Reviewer o8PJ,

We sincerely apologize that we could not provide an enough response on the Rebuttal due to the length limitation.
In this comment, we will address the parts that were lacking or unanswered in the Rebuttal. We hope this will help for Reviewers.
We wish to express our appreciation to the reviewers for their insightful comments on our paper. Please find our responses below.

> Perplexity is not currently considered the best metric for generation performance. The model could produce outputs with high confidence while outputting low-quality text. It would be interesting to measure the performance of the model on some LLM benchmarks to ensure the lack of performance degradation.

Following the feedback from the reviewers, we have included additional evaluations on various downstream tasks [1, 2, 3, 4, 5, 6, 7]. We choose these downstream tasks for the evaluation of model degeneration following this detoxification research [8]. We measure this by averaging accuracy on the downstream tasks. We provide a short description of these downstream tasks on [**our comments to Reviewer tRLA**](https://openreview.net/forum?id=TBNYjdOazs&noteId=04JTMxr8VH). Moreover, we have presented the main experiment results including the downstream evaluations on [**our rebuttal to Reviewer tRLA’**](https://openreview.net/forum?id=TBNYjdOazs&noteId=dHnDleniE3). As a result, We found that our approach can achieve better detoxification than the existing task vector negation method, while maintaining similar model performance.

> The models evaluated are relatively small or non state-of-the-art. It would be interesting to see if this technique scales on bigger models.
> Have you tried this method on larger models? Does it work at scale?

We believe that our approach will also work on larger and state-of-the-art models. To demonstrate this, we applied our proposed method to a larger and more recent model, Llama2 - 7b, and presented the results in the table (please see the table in [**our Rebuttal to Reviewer vKrR**](https://openreview.net/forum?id=TBNYjdOazs&noteId=qLvE2RNltY)). The results in that table show that the proposed method is also effective on the recent and larger model.

Thank you for your advice. We will update our paper following your advice.

Again, thank you for giving us the opportunity to improve our manuscript with your valuable comments and queries.


[1] [HellaSwag: Can a Machine Really Finish Your Sentence?](https://doi.org/10.18653/v1/P19-1472) by Zellers et al.

[2] [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](http://arxiv.org/abs/1808.09121) by Pilehvar and Camacho-Collados

[3] [PIQA: Reasoning about Physical Commonsense in Natural Language](http://arxiv.org/abs/1911.11641) by Bisk et al.

[4] [WinoGrande: An Adversarial Winograd Schema Challenge at Scale](http://arxiv.org/abs/1907.10641) by Sakaguchi et al.

[5] [LAMBADA: Backward Chaining for Automated Reasoning in Natural Language](http://arxiv.org/abs/2212.13894) by Kazemi et al.

[6] [RACE: Large-scale ReAding Comprehension Dataset From Examinations](http://arxiv.org/abs/1704.04683) by Lai et al.

[7] [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](http://arxiv.org/abs/1905.10044) by Clark et al.

[8] [Exploring the limits of domain-adaptive training for detoxifying large-scale language models](https://proceedings.neurips.cc/paper_files/paper/2022/file/e8c20cafe841cba3e31a17488dc9c3f1-Paper-Conference.pdf) by Wang et al.
Continuing from the previous comment, we have provided a brief description of the downstream tasks below.

Hellaswag [4]: Tests the model’s ability to choose the most plausible continuation of a given context, assessing commonsense reasoning.  
WiC (Word-in-Context) [5]: Evaluates the model’s ability to determine if a word has the same meaning in two different contexts, testing lexical semantics.  
PIQA (Physical Interaction QA) [6]: Evaluates the model’s physical commonsense reasoning by selecting the most reasonable answer to everyday task questions.  
WinoGrande [7]: Evaluates the model’s understanding of pronoun resolution in sentences with ambiguous references, testing commonsense reasoning.  
LAMBADA [8]: Evaluates the model’s ability to predict the last word of a given passage, testing its broad contextual understanding and coherence.  
RACE [9]: Tests the model’s reading comprehension skills on passages with multiple-choice questions, used primarily for middle and high school-level texts.  
BoolQ [10]: Tests the model’s ability to answer yes/no questions based on a given passage, assessing its fact-checking and reasoning abilities.

In the rebuttal section, we have presented the main experiment results including the downstream evaluations in the table. We truly apologize for not being able to present the results clearly due to the length limitation.

Thank you for your advice. We will update our paper following your advice.
Again, thank you for giving us the opportunity to improve our manuscript with your valuable comments and queries.

[1] [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://doi.org/10.18653/v1/2020.findings-emnlp.301) by Gehman et al.

[2] [Perspective API](https://github.com/conversationai/perspectiveapi) by Conversation AI

[3] [The curious case of neural text degeneration](https://openreview.net/forum?id=rygGQyrFvH) by Holtzman et al.

[4] [HellaSwag: Can a Machine Really Finish Your Sentence?](https://aclanthology.org/P19-1472) by Zellers et al.

[5] [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](http://arxiv.org/abs/1808.09121) by Pilehvar and Camacho-Collados

[6] [PIQA: Reasoning about Physical Commonsense in Natural Language](http://arxiv.org/abs/1911.11641) by Bisk et al.

[7] [WinoGrande: An Adversarial Winograd Schema Challenge at Scale](http://arxiv.org/abs/1907.10641) by Sakaguchi et al.

[8] [LAMBADA: Backward Chaining for Automated Reasoning in Natural Language](http://arxiv.org/abs/2212.13894) by Kazemi et al.

[9] [RACE: Large-scale ReAding Comprehension Dataset From Examinations](http://arxiv.org/abs/1704.04683) by Lai et al.

[10] [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](http://arxiv.org/abs/1905.10044) by Clark et al.
Dear Reviewer tRLA,

We sincerely apologize that we could not provide an enough response on the Rebuttal due to the length limitation.
In this comment, we will address the parts that were lacking or unanswered in the Rebuttal. We hope this will help for Reviewers.
We wish to express our appreciation to the reviewers for their insightful comments on our paper. Please find our responses below.

>The primary issue with the paper is its writing. The exact methodology is unclear and hard to follow. The experimental setup does not clearly state the experimental setup, choice of metrics for evaluation, and justification for choices made in the experimental design.

We sincerely apologize for any difficulties in following our experiment settings. Based on the advice of many reviewers, if our paper is accepted, we plan to make the following changes and additions in the camera-ready version:

1. Correct typos and grammatical errors.
2. Clearly describe the purpose and intent of each experiment.
3. Provide more detailed information on the evaluation methods and settings for each experiment (Please see our response below).
4. Include a precise definition of toxicity and examples for each toxicity category (Please also see [**our comment to Reviewer vKrR**](https://openreview.net/forum?id=TBNYjdOazs&noteId=X3MKeMdDna)).

Once again, we deeply apologize for the unclear explanations that made it difficult to comprehend this paper. If there are any other comments in addition to what we have mentioned, please let us know. Thank you very much.

>Beyond writing, the first point of confusion is how toxicity is evaluated. How is avg. max toxicity and toxicity probs are calculated, as shown in Table 1.

We are truly sorry for the difficulty caused by not providing clear explanations for the metrics in the table caption. Following the previous study [1], We evaluate the model’s toxicity using Perspective API [2] in Table 1. We specifically use the nucleus sampling strategy [3] to produce 25 generations of 25 tokens each for the same prompt. This allows us to determine the maximum toxicity score for each prompt and compute the average as the Expected Maximum Toxicity. Additionally, we calculate the probability of generating toxic continuations (toxicity score above 0.5) within the 25 generations for each prompt, referred to as the Toxicity Probability score.

We prompt all models with “I don’t care if this is controversial” and we use 1K prompts labeled as ‘normal’, and randomly sampled from RealToxicPrompts [1] for the toxicity evaluation. Although the explanations for these metrics are included in the paper, we have realized that it is challenging for readers to understand without them being under the table caption. We will address this issue with the other writing problems the reviewers have pointed out, in the camera-ready version.

> Degeneration of the model is evaluated with perplexity on wiki-103. This metric alone cannot sufficiently support the claim that this method leads to lesser degeneration. For example, what happens to other capabilities of the models? It would also be important to evaluate the generation themselves, as perplexity could be a very noisy proxy for generation quality. Specifically, human evaluation of generations would be highly recommended.

Although we also agree with the reviewer’s opinion that human evaluation is the most accurate evaluation, we were unable to conduct human evaluation due to time constraints. However, following the feedback from the reviewers, we have included additional evaluations on various downstream tasks [4, 5, 6, 7, 8, 9, 10]. We choose these downstream tasks for the evaluation of model degeneration following this detoxification research [11]. We measure this by averaging accuracy on the downstream tasks. We provide short descriptions for each downstream task in the next comment due to the length limitation.
Dear Reviewer vKrR,

We sincerely apologize that we could not provide an enough response on the Rebuttal due to the length limitation.  
In this comment, we will address the parts that were lacking or unanswered in the Rebuttal. We hope this will help for Reviewers.

We wish to express our appreciation to the reviewers for their insightful comments on our paper. Please find our responses below.

> The idea of using task vectors is not novel and have been widely explored in the literature. Moreover, learning a task vector for a text style transfer task is also not novel. Therefore, the contribution

As the reviewer has pointed out, we conducted detoxification by utilizing task vector negation. However, we believe that our main contributions are as follows. 1) We proposed a novel detoxification method by merging multiple task vectors from fine-tuned models with split toxic datasets, and 2) We discovered that when models trained on split datasets are merged, the merged model experiences a smaller amount of parameter shift from the pre-trained model than the standard task vector negation method. We assume that by applying our method, the important attribute (detoxification) is maintained within the merged vectors, while other attributes (text format, etc.), which we believe might be present in the split datasets and related to model degeneration, are suppressed.

> Lack of other merging strategies. Authors do not consider any other merging strategies except linear. It would be beneficial to consider other merging strategies from related work. These methods were proven to be better than the simplest linear merging. 

The detoxification results are shown in the table in the Rebuttal section. We mentioned the performance (Utility Score) on downstream tasks and described these tasks in [**our comments to Reviewer tRLA**](https://openreview.net/forum?id=TBNYjdOazs&noteId=04JTMxr8VH). We hope this will be helpful.

> The paper does not provide a clear definition of toxicity or the kind of phenomena evaluated. It would be interesting to know some examples of toxicity evaluated and how accurate is Detoxify on those examples.

We sincerely apologize for any confusion caused by not clearly defining toxicity. The definition of toxicity is subjective and may vary slightly across different studies, but we used the toxicity definition from the Perspective API [2]. According to the Perspective API [2], which is widely used to assess toxicity in many papers [1, 3, 4], toxicity is defined as “a rude, disrespectful, or unreasonable comment that is likely to make you leave a discussion.” The Perspective API categorizes toxicity into four levels and classifies toxic sentences into four categories. The Perspective API uses data labeled by humans according to their toxic levels and categories. This data is used to train a classifier, which then measures the toxicity of text. If our paper is accepted, we plan to include examples of generated sentences from both the baseline and our detoxified models, categorized by the toxicity levels and the categories. Once again, we apologize for not providing a clear definition and promise to include a more precise definition in the paper.

> Paper lacks comparison to other strong model self-detoxification approaches. With the current evaluation results it is unclear, whether the proposed method is better than other strong approaches.

In this paper, we did not compare our proposed method with existing strong self-detoxification approaches [3, 4], as mentioned by the reviewer. However, these self-detoxification methods require the creation of both extensive toxic and non-toxic datasets from a model. These datasets need to be generated for each model, requiring a large amount of data for each detoxification. However, the task vector negation method adopted in our proposed approach allows for efficient detoxification using only existing toxic datasets. We measured the performance of a typical detoxification method (i.e., gradient ascent) using only toxic datasets and compared its performance with our proposed method, confirming that our method is superior to those under the assumption that the fine-tuning data available are the same.


[1] [Editing Models with Task Arithmetic](http://arxiv.org/abs/2212.04089) by Ilharco et al.

[2] [Perspective API](https://github.com/conversationai/perspectiveapi) by Conversation AI

[3] [CMD: a framework for Context-aware Model self-Detoxification](http://arxiv.org/abs/2308.08295) by Tang et al.

[4] [Exploring the limits of domain-adaptive training for detoxifying large-scale language models](https://arxiv.org/abs/2202.04173) by Wang et al.

[5] [TIES-Merging: Resolving interference when merging models](https://openreview.net/forum?id=xtaX3WyCj1) by Yadav et al.