Decoupling Noise and Toxic Parameters for Language Model Detoxification by Task Vector Merging
Authors: Yongmin Kim
Takeshi Kojima
Yusuke Iwasawa
Yutaka Matsuo
Venue: COLM (2024)
Keywords: detoxification, task vector, weight merging, unlearning
TL;DR: We proposed a more effective method for detoxification by merging task vectors extracted from models fine-tuned on split subsets of a toxic dataset.

Abstract: The goal of detoxifying language models is to reduce the chances of producing offensive or harmful output in pre-trained language models (PLMs), ensuring their safer use. A recently proposed detoxification method utilizes the task vector obtained by subtraction from the fine-tuned model on toxic datasets to the pre-trained model. This approach has shown effectiveness for detoxification but still suffers from degradation. This study focuses on further mitigating degradation while maintaining detoxification performance. To mitigate the degradation, we propose a method that detoxifies the PLMs by fine-tuning multiple models on split toxic datasets and by merging the subtracted task vectors. 
We conducted experiments on two toxic datasets (Civil Comments and Toxigen) with five PLMs (GPT2-small, GPT2-medium, GPT2-large, Phi-1.5, and Llama2-7b), demonstrating that our method consistently achieves a lower toxicity score while preventing the degradation compared to baseline methods.
Especially, with the GPT2-small model on the Toxigen dataset, degradation was reduced by 38.9\% compared to that of an existing task vector method while maintaining a similar toxicity score.
In addition, we found that merging multiple detoxified models tends to increase the number of parameters that remained almost unchanged from the pre-trained model.
We assume that by merging multiple detoxified models, "decoupling noise and toxic parameters" is implicitly achieved. The accidental noise in the parameter shift unrelated to detoxification disappears by averaging noise, whereas the parameter shift associated with detoxification is maintained.
We hope that the findings of this study will be applied not only to detoxification but also to many other research domains that seek to suppress undesirable outputs of language models.

Published as a conference paper at COLM 2024
Decoupling Noise and Toxic Parameters for Language Model Detoxification by Task Vector Merging
Yongmin Kim!
Takeshi Kojima'
Yusuke Iwasawa"
Yutaka Matsuo
'The University of Tokyo
kimyongmin@robot. t. u-tokyo. ac. kt.ma, iwasawa, matsuo @wwebla..u--kkkyo.acipp
Abstract
The goal of detoxifying language models is to reduce the chances of produc- ing offensive harmful output in pre-trained language models (PLMs), ensuring their safer use. A recently proposed detoxification method utilizes the task vector obtained by subtraction from the fine-tuned model on toxic datasets to the pre-trained model. This approach has shown effectiveness for detoxification but still suffers from degradation. This study focuses on further mitigating degradation while maintaining detoxification perfor- mance. To mitigate the degradation, we propose a method that detoxifies the PLMs by fine-tuning multiple models on split toxic datasets and by merging the subtracted task vectors. We conducted experiments on two toxic datasets (Civil Comments and Toxigen) with five PLMs (GPT2-small, GPT2-medium, GPT2-large, Phi-1.5, and Llama2-7b), demonstrating that method consistently achieves lower toxicity score while preventing the degradation compared to baseline methods. Especially, with the GI'I2- small model on the Toxigen dataset, degradation was reduced by 38.9% compared to that of an existing task vector method while maintaining a similar toxicity score. In addition, we found that merging multiple detox- ified models tends to increase the number of parameters that remained almost unchanged from the pre-trained model. We assume that by merging multiple detoxified models, "decoupling noise and toxic parameters" is implicitly achieved. The accidental noise in the parameter shift unrelated to detoxification disappears by averaging noise, whereas the parameter shift associated with detoxification maintained. We hope that the findings of this study will be applied not only to detoxification but also to many other research domains that seek to suppress undesirable outputs of language models.
1 Introduction
Although pre-trained language models (PLMs) have shown remarkable capabilities across many domains over the past few years (Radford et al. 2019 Brown et al. 2020 Li et al., 2023 Touvron et al.] (023),, PLMs have the risk of generating offensive and aggressive contents (Gehman al. 2020 Liang et al. 2023). Toxic data present in pretraining datasets causes PLMs to generate rude and biased sentences. Recently, to enhance the secure utilization of LMs, the crucial challenge of detoxifying LMs has attracted growing research interest (Kumar et al. 2023).
To mitigate the risk and provide safer and non-aggressive PLMs, detoxification methods that prevent the generation of toxic sentences have been gaining attention. Previous studies (Liu et al.) 2021 Krause et al. 2021 Kwak et al.) 2023) proposed detoxification methods that adjusted the output probability of the next token to prevent potentially toxic tokens. Other
1Code is available pttttpp:///eee h/gtthub... http://saiiiciciii//aree.....o..
Warning: this paper includes rude or offensive examples in Appendix.

---

Published as a conference paper at COLM 2024
fine tuning
task vector negation
split dataset
merging
taxic dataset
merged non-toxic LM
worgy
base LM
toxic LM
non-toxic LM
Figure 1: Overview of our method. We split a whole toxic dataset into N sub-datasets. Then, we fine-tune the base-LM on each sub-dataset. We detoxify each model using the task vector negation. We then merge these detoxified models into one model to mitigate the degradation.
studies Gururangan et al. 2020 Wang al. 2022 Lu et al. 2022) proposed methods of reducing the toxicity of the model through post-training.
Recently, Ilharco et al. (2023) proposed another method, task vector operation, which achieves detoxilication trom toxic dataset with arithmetic operations on the model's pa- rameters. The task vector is the difference the parameter space between fine-tuned model and pre-trained model. They have shown that simple arithmetic operations, such as subtracting the task vector to a pre-trained model, can effectively unlearn undesirable attributes, such toxicity. However, other methods, this detoxification method de- grades the existing LMs' ability. A trade-off relationship exists between the detoxification performance and the degradation of LMs' capabilities, which needs improvement.
Inspired by the recent success of the model merging technique Gueta et al. (2023); Wortsman et al. 2202), which merges multiple model parameters to achieve higher performance, we propose novel method that merges multiple task vectors to mitigate degradation further and boost the detoxification performance. Specifically, we randomly split whole toxic dataset into N sub-datasets. Then, we fine-tune the base-LM on each sub-dataset. We detoxify each fine-tuned model using task vector negation and merge the N detoxified models into one model (Figure presents an overview of our method).
To validate the effectivenese of our method, we conducted experiments on two toxic datasets (Civil Comments and Toxigen) and five PLMs (GPT-2, GPT-2 medium, GPT-2 large, Phi-1.5, and Llama2-7b). Our experiment results demonstrate that our proposed method consis- tently achieves higher detoxification while suppressing degradation compared to baseline methods, including the standard task vector negation approach. We found that the indi- vidual models detoxified from each sub-dataset have much less degradation than model detoxified from the entire dataset. We also found that the merged models showed a similar level of toxicity with less degradation compared to the individual models detoxified from sub-datasets.
Furthermore, to investigate the root cause of the degradation mitigation achieved by our method, we measured the similarity between our detoxified models and the pre-trained model in the parameter space. We found that merging multiple detoxified models tends to increase the number of parameters that remained almost unchanged from the pre-trained model. We assume that by merging multiple detoxified models, "decoupling noise and toxic parameters" is implicitly achieved. The accidental noise in parameter shift unrelated to detoxification disappears by noise averaging, while the parameter shift associated with detoxification is maintained.
2

---

Published as a conference paper at COLM 2024
2 Related work
2.1 Detoxification
The ability of PLMs to generate text across various contexts is derived from their pre-training on large, multi-domain textual datasets (Prabhumoye et al.) 2023 Korbak et al. 2023). Nonetheless, given that these datasets are scraped from web content, they unavoidably include harmful material (Gehman et al. 2020; Gao et al. 2020 Penedo et al. 2023 Kumar al. 2023), thus increasing the likelihood PLMs producing toxic outputs. The objective of language detoxification is to mínimize the chances of producing continuations that contain toxic elements. Toxic elements are defined as aspects of the text likely to be perceived as impolite, disrespectful, or offensive (Kwak et al. 2023 Schick al. 2021).
Recent studies address language model toxicity by aligning outputs with human pref- erences (Radford et al., 2019), modifying decoding to suppress toxic tokens (Liu et al., 2021;
Krause et al., 2021; K0akkKWaak et al., 2023) or exploring detoxification through toxic prompts (Leong al. 2023). However, these methods need additional inference time. In contrast to these approaches, certain research works (Gururangan et al. [020 Wang et al. 2022 Lu et al. 2022 Tang et al. 2024) have proposed methods fine-tune non-toxic data. In line with these methods, IIlhaco et al. (2023) has demonstrated the effectiveness of the task vector negation method to detoxify LMs. The task vector approach can utilize only toxic data for detoxification as opposed to the previous trainable methods.
Ilharco et al. (2033 introduced the concept of the task vector, demonstrating that through arithmetic operations on the task vector, one enhance the model's performance, unlear an undesired attribute, or improve domain generalization. Their study proposed a method of detoxification using the task vector from model fine-tuned
a toxic dataset. The detoxification performance of this method was better than that of methods fine-tuned on non-toxic datasets and it led to less degradation compared to that of methods using the gradient ascent. In contrast to the existing detoxification methods, this approach uses only toxic datasets for detoxification.
However, this approach for detoxification weakens the inherent abilities of LMs, highlighting trade-off between detoxification efficacy and degradation of a model's performance. We have demonstrated that, compared to the existing task vector method, our approach further mitigates the degradation from the detoxification
the same toxicity. Furthermore, our method is cost-effective compared to other trainable methooss(Grurangaa et al. 2020 Wang et al.
2022
Lu et al. 2022
Tang et al., 2024) because uses only toxic data.
2.2 Model merging
Recent studies studies/Gueta Gueta et al., 2023:
Wortsman et al., 2022) have shown that merging the
weights of different models can yield a model with better performance than that achievable with trainable methods. Wortsman et al. (002 proposed a method efficiently merge different models linearly by selecting models from candidates. Furthermore, Matena & Raffel 20222) ; Daheim et al. (2024b); Yadav et al. (2023) proposed more computationally efficient and effective methods for fusing multiple models to enhance performance. In this study, we adopt the linear merging method and additionally apply Tie-Merging Yadav et al.,
2023 to the proposed method.
Method
3.1 Preliminary
Ilharco et al. 2023) proposed method for detoxification by negating the task vector calculated with a model fine-tuned on toxic dataset. This method proved to be more effective in detoxification and less prone to performance degradation compared to methods fine-tuning on non-toxic datasets or employing the gradient ascent on toxic data.
3

---

Published as a conference paper at COLM 2024
Suppose that pre R represents the pre-trained model's parameters, and 0 E R is the parameter after fine-tuning for particular task t. The task vector, denoted as T, E R is calculated by the element-wise subtraction pre from 02 expressed as र; = 0 Ppre If the task is apparent, we omit the identifier and simply notate it T. These task vectors can be utilized across any model parameters from an identical architecture by adding them element-wise with scaling coefficient A, leading to new model parameter nnw = 0 + År. Negating the task vector T by applying the inverse vector Thew = - extends the fine- tuned model back towards the pre-trained model. This operation leads to diminished or unlearned model's ability to perform the specific task for which it was fine-tuned (e.g,, toxicity).
3.2 Proposal
Inspired by the recent success of model merging techniques (Gueta et al. 2023 Wortsman
et al. 2022), which merge multiple model parameters to achieve higher performance, we propose novel method of merging multiple task vectors to mitigate degradation further and enhance detoxification. Specifically, we divide the dataset 'D into N sub-datasets D (Ni)), ensuring that no common parts exist among them, which can be mathematically represented as follows:
N (N. D.
(1)
i=1
Zaman et al. (2023) demonstrated that when fusing several models, the task abilities of each model are reinforced in their common aspects and suppressed in their differing aspects after merging. Furthermore, they showed that merged models have an improved generalization ability. Based on these findings, we hypothesized that merging models fine-tuned on each sub-dataset would result in a suppression of the degradation, an unshared attribute across the models. Ultimately, we detoxify the pre-trained model by negating the task vector of merged models. Although many methods exist for merging models (Yadav et al.) 2023 Daheim et al. 2024b; Matena & Raffel 2022), we adopted a simple approach to clarity the effectiveness of our method. Specifically, we considered merging by computing the weighted average of the model parameters. Let there be N models that are fine-tuned to
each subset of a toxic dataset D (N,)) with parameters (0)... ON} such that V,0, E Rd. Then, we define the merged model, merged, as a convex combination:
N Σ
1 merged
a;:;;
where a; 2 0, EN aj 1. We employ the average for weight merging, where aj = 222
AN 1/N. From the merged model merged, we can calculate the task vector m 0 merged Ppre- Consequently, we can obtain the detoxified model detoxic by the negation of the task vector Tmerged the pretrained model pre with coefficient A. To be specific, we compute the detoxified model with Equation 3
2 detoxic
ppre
A mergeed
(3)
Experiment setting
In this study, we detoxified the models of GPT2 (Rafford et al. 2019), Llama2-7b, and Phi-1.5 (Li et al. 2023) using a toxic dataset included in the dataset. For the proposed method, we divided the entire toxic dataset into sub-datasets and fine-tuned the models on them. Then, we merged those models and calculated the task vector from it. Finally, we detoxified the model with the negation of the task vector. To verify the proposed method in diverse settings, we conducted experiments through multiple datasets for toxic data.
4

---

Published as a conference paper at COLM 2024
4.1 Datasets
We employed Civil Comments (Borkan et al., 2019) and Toxigen (arttigsen et al. 2022) as toxic datasets for detoxification. We made use of highly toxic data included in these datasets. For the toxicity evaluation, we used prompts in RealToxicPrompp ronniisaaass et al.) 2020) and Hoxigen(Hartvigsen Hartvigsen al., 2022). We sampled 1K prompts from these datasets. We provide more detailed information about the datasets in Section C.1
4.2 Baselines
For the baselines, we used set-up from Ilharco et al. (2023) for fine-tuning on the non-toxic dataset, and trained the model to the toxic dataset with the gradient ascent approach (Tarun
et al. 2021 lang et al. 2023). We fine-tuned the models to the non-toxic data (toxicity score lower than 0.2) included
each dataset, in similar way to Liu et al. (2021). We did not use the entire non-ton moniioic dataset; instead, we sampled the same number of non-toxic data from the entire toxic dataset the toxic data we used. In the detoxification experiments using gradient ascent, we maximized the negative log-likelihood loss on the toxic dataset. Finally, we compared the proposed method with detoxification using an entire toxic dataset realized by Ilharco et al. (023)..
We did not conduct a comparison between our proposed method and existing self- detoxification approaches (Wang et al., 2022 Tang et al.) 2024). However, these self- detoxification methods require the creation of extensive toxic and non-toxic datasets from model. These datasets need to be generated for each model, and thus, large amount of data are required for each detoxification. By contrast, the negation method adopted the proposed approach allows for efficient detoxification using only existing toxic datasets
4.3 Experiment
First, we examine the impact of the proposed method on different sizes of sub-datasets in Second, we investigate the effectiveness of merging models detoxified from the split datasets in Section 5.2 Third, we compare the proposed method with the existing task vector negation approach by examining the parameter space of the detoxification models in Section 5.3 Specifically, we measure how different the parameters of detoxification models, which have the same toxicity level, are from the pre-trained model. Finally, we compare our approach with the baseline methods in Section 5.4
Section 5.1
4.4 Evaluation
In each experiment, we measured the toxicity and degradation of the models to compare the baseline methods and our approach. According to Gehman et al. (2020), , toxic sentence is defined as "a rude, disrespectful, or unreasonable comment that is likely to make you leave discussion". Toxicity evaluated through the toxicity score measured by pre-trained
toxicity detectors (Hanu & Unitary team 2020 Hartvigsen et al., 2022) and Perspective A which is widely used as toxicity detector. Different toxic detectors were employed depending on the experiment. Detailed information can be found in each experiment section. In Section Section 5.3 we assess the model's degradation through perplexity. In Section 5.4 we also measure the average performance of models on seven downstream tasks to comprehensively evaluate the models' degradation, in addition to assessing perplexity.
5
Experiment results and discussion
5.1 Impact of the sub-dataset size
An increase in the number of sub-datasets mitigates degradation while maintaining similar toxicity level. To estimate the impact of the number of sub-datasets N on detoxifi- cation, we conducted the experiments by setting N to 5, and 10. Then, we compared those
5

---

Published as a conference paper at COLM 2024
Pretrained
All
3 split
5 split
10 split
GPT2 small (Civil Comments)
GPT2
medium (Civil Comments)
GPT2 large (Civil Comments)
Avg. toxicity score
Avg. toxicity score
0.4
04
toxicity
0.3
0.3
033
0.2
02
0.1
0.0
0.0
0.
16
18
20
22
24
26
28
30
16
10
20
22
24
26
28
30
16
18
20
22
24
26
28
30
Wikitext- 10ttexxt-103 perplexity
Wikitext-103 perplexity
Wikitext-103 perplexity
GPT2 -small (Toxigen)
GPT2 medium (Toxigen)
GPT2 large (Toxigen)
1.000
Avg. toxicity Score
Avg. taxicity Score
toxicity
0.875
0.855
0.675
3.050
0,500
0.825
0.825
0.800
0.800
0.800
16
20
24
28
30
18
22
24
26
25
30
16
20
22
26
30
Wikitext- 101eexxtt00000000 perplexity
Wikitext-103 perplexity
Wikitext-103 perplexity
Figure 2: Detoxification results with different numbers of sub-datasets in the proposed method. The label 'N split' means that N sub-datasets split from the entire dataset were used for the detoxification, and 'All' refers to the result where the entire dataset was used. Models with perplexity exceeding 30 are not indicated.
models with detoxified model from entire dataset. In this experiment, we used scaling for negating the task vector. To evaluate the toxicity, we employed toxicity detectors and prompts to make the models generate toxic sentences depending on the toxic dataset used in the detoxification process. For the models detoxified from Civil Comments, we used Detoxify (Hanu & Unitary team 2020 to measure the toxicity with 1K challenging prompts of RTP. Similarly, we adopted HateBert et al., 2022) with 1K Toxigen prompts evaluate the toxicity of models detoxified from loxigen. We evaluated the model's degradation with fluency by measuring the perplexity on WikiText-103 (eriiyy et al. 2017). All the training and evaluation details are described in Section C.2
coefficient in the range of E (0, 0.05, 0.1, 1100
We present the results of applying the proposed method to GPT2 small, medium, and large models with different numbers of sub-datasets in Figure 2 In all the experiments on Civil Comments, as the number of used sub-datasets increases, that is, as the size of the sub-dataset decreases, smaller perplexity was recorded at similar toxicity. Furthermore, comparing all the models the proposed method with models using the entire dataset, we found that at similar toxicity, the perplexity with the proposed method was smaller. This difference in perplexity at similar toxicity was larger with the smaller model sizes. In all the experimental settings of Toxigen, except for the method that used three split sub-datasets on GPT2-large, the proposed method performed better than the existing method that used an entire dataset. Generally, the detoxification performances on the Toxigen dataset were lower than those on Civil Comments. We suppose that the toxic sentences in Toxigen are expressed in implicit ways and this makes it challenging to detoxify models.
6

---

Published as a conference paper at COLM 2024
Pretrained
5 split
Average
5 split (negated)
Average (negated)
All (negated)
GPT2 small (Civil Comments)
medium (Givil Comments)
large (Civil Comments)
0.6
2.6
0.6
Avg. toxicity score
Avg. toxicity score
Avg. toxicity score
0.5
2.5
2.5
2.4
0.3
2.3
0.1
8.1
01
5.0
25
30
40
20
25
35
25
30
40
Wikltext-103 perplexity
Wikitext:103 perplexity
Wikitext-103 perplexity
GPT2 small (Toxigen)
GPT2 medium (Toxigen)
GPT2 large (Toxigen)
1.000
1.000
Aug. Losicily score
0.950
0.450
Aug. toxicity Score
0.950 x
0.925
toxicitx
3.925
0.925
0.900
0.900
0.000
0.875
3,075
0.875
0.850
0.850
0.850
0.800
0.600
30
40
25
25
25
30
Wikitext-103 perplexity
Wikitext-103 perplexity
Wikitext1003 perplexity
Figure 3: Perplexity and toxicity score before and after merging models detoxified from the five split sub-datasets in Civil Comments and Toxigen. The label '5 split' refers to the models fine-tuned on each sub-dataset. 'Average' indicates the averaged model from the models fine-tuned to each sub-dataset. The 'Al (negated)' label means the models detoxified from the entire toxic dataset. The blue points refer to the fine-tuned models and the red points represent the negated models using the task vector with the coefficient 1. For the 'All (negated)) models, we set the coefficient so that they have the closest toxicity to the toxicity that each averaged model has. Models with perplexity over 40 are not included.
5.2 Effectiveness of merging
A merged model mitigates degradation further than individual models before merging. To inspect the effectiveness of weight merging, we measured the toxicity and perplexity on the models. To be specific, we compared models fine-tuned on a toxic dataset and the averaged model from them. We also evaluated the negated models from these models. All models were evaluated by toxicity and perplexity in the same manner as presented in Section 5.1
Figure 3 shows the results of each model when using the proposed method with the five sub-datasets of the Civil Comments and Toxigen toxic datasets. We found that the individual models detoxified from each sub-dataset have much less degradation than the model detoxified from the entire dataset. This finding aligns with Jang et al. (2023), which showed that using gradient ascent for unlearning results in degradation of the model's original capabilities as the size of the dataset for unlearning increases. We assume that this phenomenon occurs not only in gradient ascent unlearning but also in unlearning with task vector. One can speculate that training on a large toxic dataset with many iterations may lead to a state of specialization for not only the toxicity but also data format, which in turn leads to a decrease in fluency on the other datasets.
We also observed that the merged models recorded a lower perplexity than individual models trained on each sub-dataset before merging, These differences in perplexity became larger with smaller model sizes. These gaps increased more in the detoxified model from Toxigen than in that from Civil Comments. The merged models had the largest reduction
7

---

Published as a conference paper at COLM 2024
split"
Average
Civil Comments
Taxigen
10
Averoped (4109)
10
GPT21arge
GPT2me3uu GPT2
Figure 4: Average distance between the pre-trained and detoxified models in parameter units. We measured the distances with L, norm of the task vectors from a detoxified model and the pre-trained model. The 'All' label means the results where all toxic data is used. The '5 split" label refers to mean results from individual models detoxified from split 5 sub-datasets. The 'Average' label indicates the results of the proposed method on split sub-set datasets. For the Alll' results, we scaled them for the negated models to have the closest toxicity score to the averaged models.
in perplexity after negation compared that before. Surprisingly, in the detoxification for GPT-2 small from Toxigen, the perplexity decreased by 38.9% with our approach compared to that with the existing method where all toxic data were used. This result of the detoxified model from whole dataset was not indicated in Figure 2 because the perplexity was 56.27, which exceeds 40. This demonstrated that merging several models detoxified from split dataset can mitigate the degradation effectively while maintaining similar toxicity.
5.3 Similarity with pre-trained model
Our method experiences smaller amount of parameter shift from the pre-trained model than the standard task vector negation method. We measured the similarity between detoxified model and a pre-trained one. We employed L L'-oomm evaluate the similarity and divided it by the number of model's parameters. We compared L' L'-norm with all the GPT2 models detoxified from the entire toxic dataset and each split sub-dataset, and the averaged models. We set for the split size of the sub-dataset. For the measurement of models detoxified from entire dataset, selected the coefficient À {0, 0.0.0.0,0.1.... 1.0) for those models to have the closest toxicity with our approach.
We present the results of the L -norm between all the detoxified GPT2 models and the pre-trained models averaged on the parameters Figure Although all the models have similar toxicity in each setting, except for the detoxified GPT2 small model from Toxigen, the models detoxified from split sub-datasets had a smaller distance than the models detoxified from all the data. Furthermore, the model merged from detoxified models was closer to the pre-trained models in distance than the other models. This implies that detoxified models with split dataset are more similar to the pre-trained model, and the merged model from them is much closer to
In addition, verified the distribution of parameter shift distance between detoxified model and the pre-trained model. We compared this distribution between models detoxified from sub-datasets and the merged model from those models. For this experiment, split each dataset, Toxigen and Civil Comments, into five sub-datasets, and detoxified and merged models from them.
Figure 5 illustrates the distribution of parameter shift distance between detoxified and pre-trained models. The distribution of models detoxified from the sub-datasets represents the averaged distributions on the five models. In all the experiments, the distribution
8

---

Published as a conference paper at COLM 2024
5 spiltt
Average
All
The number of parameters (x10')
GPT2 small (Civil Comments)
The number parameters (x101)
GPT2 medium (Civil Comments)
The number parameterss (x10P)
GPT2 large (Givil Comments)
15
5.4
0.8
0.2
0.8
2,3
0.0
5.4
(x10
Parameter (x10
The number of parameter: (x10%)
GPT2 small (Toxigen)
The number parameters (x10%)
GPT2 medium (Toxigen)
The number of parameters (x10%)
GPT2 large (Toxigen)
06
12
0.8
0.2
0.5
1.0
0.0
54
Parameter (x10-1)
Distance Parameter
(x10-")
Parameter Shift (x10-1)
Figure 5: Distribution of parameter differences between the detoxified and pre-trained models. We compared the detoxified models from sub-datasets and the merged model. The '5 split" label refers to mean results from individual models from split 5 sub-datasets. The 'verrage' label indicates the models averaged from the 5 detoxified models. The 'All' label denotes the models detoxified from an entire dataset. For the 'All' models, we set the coefficient value so that their toxicity becomes the closest to the average model's toxicity.
of the averaged models tends to shift to more than that of the individual models trained on toxic sub-datasets. Based on this fact, it is speculated that by merging multiple detoxified models, accidental noise in parameters not related to detoxification disappears, and degradation is prevented, whereas the parameter shift related to detoxification is maintained. This interpretation aligns with that Zaman al. (2023), which experimentally showed that when multiple models are fused, the abilities common among the models are enhanced, whereas the differing aspects are suppressed.
5.4 Comparison with baseline methods
Our method achieves lower toxicity scores than baseline methods without degradation. We evaluated all the detoxified GPT2, Phi-1.5, and Llama2-7b models with Perspective API performance comparison
our method with baselines. For those models detoxified from an entire toxic dataset, we selected the coefficient À 0.05, 0.1, 1100} to have the same perplexity as that of models using the proposed method. the proposed method, we set N to 5 to split the toxic dataset for all models, and set it to 3 only for the Llama2- 7b model. Following previous work (Gehman et al. 2020), we evaluated the model's toxicity in two manners: 1) the averaged maximum toxicity over 25 generations, and 2) the empirical probability of generating a continuation with toxicity at least once over generations. Toxicity evaluated
dataset of 1K normal prompts randomly sampled from RealToxicPrompte Additional results on the GPT2 small, large, and Phi-1.5 models are provided in Section D.1 Similar the experimental setup in Wang et al. (2022), we measured the model's performance in downstream tasks using a Utility score to assess model degradation. Specifically, we evaluated the model on seven different tasks, which cover question answering, natural language understanding, and commonsense reasoning, More details about each downstream task can be found in C.3

---

Published as a conference paper at COLM 2024
Toxicity (
Fluency (p)
Uuiliyy (1)
Method
Dataset
Avg. toxicity
Toxicity prob.
Perplexity
Avg.
Pretrained (GPT2 medium)
0.463
0.423
18.31
47.3
Fine-tuned
Civil Comments (non-toxic)
0.382
0.214
28.15
46.9
Gradient Ascent
Civil Comments (all)
1010
Negated (A 0.30)
Civil Cumments (all)
0.354
0.237
23.35
46.0
Negated (A 1.00)
Civil Comments (5 split)
0.269
0.124
23.35
45.1
Fine-tuned
Toxigen (non-toxic)
0.601
0.663
36.57
44.5
Gradient Ascent
Toxigen (all)
1010
Negated (A = 0.30)
Toxigen (all)
0.385
0.276
22,95
47.3
Negated (A = 0.45)
Toxigen (all)
0.338
0.185
32.57
45.7
Negated (A 1.00)
Toxigen (5 split)
0.308
0.167
23.38
44.9
Pretrained (Llama2 7b)
0.413
0.312
8.25
67.2
Negated (A = 0.80)
Civil Comments (all) Civil Comments (3 split)
0.290
0.116
9.57
65.0
Negated (A = 1.00)
0.278
0.112
9.65
64.9
Negated (Tie-Merging)
Civil Comments (3 split)
0.262
0.091
9.90
64.6
Performance comparison of the proposed method with baselines. We highlighted the places with the lowest value in each toxicity score in bold. For our method, we adapted 1 as the coefficient value. The coefficient value for the method where all datasets were used was determined so that perplexity would be closest to the value of our method. More results GPT2 small, large, and Phi-1.5 are provided in D.1
Table 1:
Table 1 presents the evaluation results. The models detoxified with our approach from Civil Comments possess the lowest toxicity compared to that of the other models. In the detoxified models from Toxigen, the toxicity of the models with the proposed method decreased much more than that of the other models at similar perplexity and similar utility score.
For Llama2-7b, additionally applied Tie-Merging (Yaday al. 2023 to proposed method. Specifically, we used Tie-Merging to merge multiple detoxified models from split datasets in our approach. In the results of Llama2-7b presented in Table 'Tie-Merging' indicates detoxification model using Tie-Merging, whereas models with a coefficient value refer to a detoxification model using linear merging. The results show that our approach with lie-Merging achieved even better detoxification performance while maintaining similar level of model performance.
Conclusion
In this work, we propose more effective method for detoxification by merging task vectors extracted from trained models on split subsets of a dataset. We showed that, in our method, the degradation is mitigated as the number of toxic sub-datasets increases. We experimentally demonstrated that, compared to existing methods, degradation more effectively, resulting in better performance at the same level toxicity. We also showed that the parameter difference in model detoxified with method is more similar to the pre-trained one than that in models detoxified from an entire toxic dataset.
method can suppress
Future Work
This study showed that it is possible to further suppress degradation by merging several models detoxified from sub-datasets using the task vector unlearning approach. Similar to the improved results achieved with Tie-Merging, we expect that we can obtain better- detoxified models by using other effective weight-merging methods together with our approach. addition, recent research has proposed methods for unlearning not only toxicity but also hallucinations to obtain more faithful models (Daheim et al. 20)44]bb. We believe that the proposed method will lead to better unlearning performance with alleviated degradation not only for toxicity but also for other undesired attributes.
10

---

Published as a conference paper at COLM 2024
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xiv preprint ar arXiv:2303.08774, 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reason- ing about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, 2019.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nu- anced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of the 2019 World Wide Web Conference, WWW '19, pp. 491-500, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 978-1-4503-667555
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vosss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Christopher Clark, Kenton Lee, Ming-Was Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/n no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Nico Daheim, Nouha Dziri, Mrinmaya Sachan, Iryna Gurevych, and Edoardo Ponti. Elastic weight removal for faithful and abstractive dialogue generation. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7096-7112, Mexico City, Mexico, June 2024a. Association for Computational Linguistics.
Nico Daheim, Thomas Möllenhoff, Edoardo Ponti, Iryna Gurevych, and Moham- mad Emtiyaz Khan. Model merging by uncertainty-based gradient matching. In The Twelfth International Conference on Learning Representations, 2024b.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling, December 2020.
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, July 2024. URL https::/doi.ont 10. 5281/2enodo http://o...oogg..0..222///a.... 12608602
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Real- ToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: Empirical Methods in Natural Language Processing 2020, pp. 0356-3366) Online, 2020. Association for Computational Linguistics.
Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Knowledge is a region in weight space for fine-tuned language models. Houda
11

---

Published as a conference paper at COLM 2024
Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Lin- guistics: Empirical Methods in Natural Language Processing 2023, pp. 1350-1370, Singapore, December 2023, Association for Computational Linguistics.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-8366 8360, Online, 2020. Association for Computational Linguistics.
Laura Hanu and Unitary team. Detoxify, Github. https:/ giithub.com/ waaum//nnnaarideeeeeeeeee 2020.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. ToxiGen: A large-scale machine-generatee dataset for implicit and adversarial hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics, 2022.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 14389-14408, Toronto, Canada, July 2023. Association for Computational Linguistics.
Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran. LAM- BADA: Backward chaining for automated reasoning in natural language. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6547-6568, Toronto, Canada, July 2023. Association for Computational Linguistics.
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L.. Buckley, Ja- son Phang, Samuel R. Bowman, and Ethan Perez. Pretraining language models with human preferences. In Proceedings of the 40th International Conference on Machine Learning, Honolulu, Hawaii, USA, 2023. JMLR.org.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: Empirical Methods in Natural Language Processing 2021, pp. 9929-4952, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. Language generation models can cause harm: So what can we do about it? An actionable survey. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3299-3321, Dubrovnik, Croatia, May 2023, Association for Computational Linguistics.
Jin Myung Kwak, Minseon Kim, and Sung Ju Hwang. Language Detoxification with Attribute-Discriinnate Latent Space. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10149-10171, Toronto, Canada, 2023, Association for Computational Linguistics.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 705-899794 Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
12

---

Published as a conference paper at COLM 2024
Chak Leong, Yi Cheng, Jiashuo Wang, Jian Wang, and Wenjie Li. Self-Detoxifying Lan- guage Models via Toxification Reversal. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp. 4433-4449, Singapore, 2023. Association for Computational Linguistics.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks Are All You Need II: Phi-1.5 technical report, September 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga,
Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, and Others. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta,, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-tingg -ime controlled text generation with experts and anti-experts. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6691-6706, Online, August 2021. Association for Computational Linguistics.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. QUARK: Controllable text generation with reinforced unlearning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17703-17716. Curran Associates, Inc., 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference Learning Representations, 2017.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLE: A new benchmark for natural language understanding, In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4881-4901, July 2020. Association for Computational Linguistics.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay, The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only, June 2023.
Mohammad Taher Pilehvar and Jose Camacho-Collados, WiC: The word-in-context dataset for evaluating context-sensitive meaning representations. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1267-1273, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Adding Instructions during Pretraining: Effective way of Controlling Toxicity in Language Models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2636 2651, Dubrovnik, Croatia, 2023. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. Communications of The Acm, 64(9): 99-106, August 2021. ISSN 0001-0782
13

---

Published as a conference paper at COLM 2024
Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP. Transactions of the Association for Computational Linguistics, 9:1408-1424, December 2021. ISSN 2307-387X.
Zecheng Tang, Keyan Zhou, Juntao Li, Yuyang Ding, Pinzheng Wang, Bowen Yan, and Min Zhang. CMD: framework for Context-aware Model self-Detoxifiiation, February 2024.
Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan S. Kankanhalli. Fast yet effective machine unlearning. IEEE transactions on neural networks and learning systems, PP, 2021.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: A family of highly capable multimodal models. ar preprint XiX:::311..1800, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Sounya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arv preprint arXiv:2307.0922 2023.
Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, and Bryan Catanzaro. Exploring the limits of domain- adaptive training for detoxifying large-scale language models. In Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Process- ing Systems, volume 35, pp.
Curran Associates, Inc., 2022.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 139665-33999 PMLR, July 2022.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-Merging: Resolving interference when merging models. In Thirty- Conference on Neural Information Processing Systems, 2023.
Kerem Zaman, Leshem Choshen, and Shashank Srivastava. Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion, November 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anmual Meeting of the Association for Computational Linguistics, 2019.
14

---

Published as a conference paper at COLM 2024
A Appendix
In this section, we present the specifics of our method and results that were not included in the main text due space constraints. We also address the limitations of this study here.
B Limitation
Similar to other detoxification methods, our method performs detoxification by directly controlling the parameters within models. In this study, we applied the proposed technique P
to open LMs such as GPT2 (Radford et al. 2019), Llama2-7b, and phi-1.5 (Li et al. 2023). However, the proposed method not applicable to closed LMs such GPT4 (Achiam et al. 2023 and Gemini Team et al. 2023), where is difficult to manipulate the model's parameters.
C Experiment details
C.1 Datasets
C.1.1 Civil Comments
Civil Comments (Borkan al. 2019) dataset that collected data from comments on news sites and annotated various toxic conversational attributes, including toxicity. For the detoxification, we made use of 30K highly toxic data (toxicity score higher than 0.8) included in this dataset.
C.1.2 Toxigen
The Toxigen dataset (Hartvigsen et al. 2022) is large-scale machine-generatee dataset of 274K toxic and benign statements locusing on 13 minority groups. It was created address the challenges in detecting toxic language that subtly or implicitly targets these groups. They provide toxicity scores that a toxicity detector they made predicted. We utilized 56K machine-generated toxic texts contained in this dataset (with toxicity score higher than 0.8) to detoxify models. We also used prompts included in this dataset labeled with 'toxic' to evaluate the model's toxicity.
C.1.3 RealToxicPrompts
RealToxicPrompts (RTP) is dataset containing 100K text paragraphs crawled from English web texts, and the first half of each paragraph is used as the prompt for generation (Gehman et al. 2020). The dataset contains two types of subsets: challenging and normal. The prompts of the challenge type are composed of sentences that make it easier for models to generate sentences with higher toxicity compared to those generated using the normal type prompts. We used the prompts in this dataset to measure the toxicity of models.
C.2 Implementation Detail
For all the detoxification methods, we optimized the corresponding loss for five epochs. As with fine-tuning, we used AdamW with the linear learning rate scheduler, batch size of 16, and learning rate of 10-5, All the experiments were conducted with A100 GPUs. We prompted all models with "I don't care if this is controversial" and used nucleus sampling (Holtzman al. 2020) with p 0.9. We evaluated the perplexity of the models on WikiTexf=103 with stricing window size of the model's max context length and stride of 512 tokens.
15

---

Published as a conference paper at COLM 2024
C.3 Details of downstream task evaluation
Following Wang et al. (0022), we evaluated models on the following seven downstream tasks with their brief descriptions. To more accurately assess degradation, we excluded downstream Nie et al. 2020) due to its very low score.
1. Hellaswag (Zllers et al. 2019): Tests the model's ability to choose the most plausible continuation given context, assessing commonsense reasoning.
2. WiC (Word-in-Context) (Pilehvar & Camacho-Collados 2019); Evaluates the model's ability to determine if a word has the same meaning in two different contexts, testing lexical semantics.
3. PIQA (Physical Interaction QA) (Bisk et al. 2019): Assesses the model's physical commonsense reasoning by selecting the most reasonable answer to everyday task questions.
4. WinoGrande (Sakaguchi al., pronoun resolution in sentences with ambiguous references, testing commonsense reasoning.
2021): Evaluates the model's understanding of
5. LAMBADA (Kazmm et al. 2023 Evaluates the model's ability to predict the last word of given passage, testing its broad contextual understanding and coherence.
6. RACE Lai et al. 2017: Tests the model's reading comprehension skills on passages with multiple-hooiee questions, used primarily for middle and high school-level texts.
7. BoolQ Clark et al.
2019: Tests the model's ability to answer yes/m /no questions based on given passage, assessing its fact-checking and reasoning abilities.
We adopted evaluation codes from Gao et al. (0244)
D Additional results
D.1 Performance comparison
Here, we provide additional performance comparisons on GP12 small, GPT2 large, and Phi 1.5 with the baseline methods. Table 2 Table 3 and Table present the evaluation results on GPT2 small, GPT2 large, and Phi-1.5 with the baselines.
Toxicity (1)
Fluency ())
Utility (?)
Method
Dataset
Avg. max. toxicity Toxicity prob.
Perplexity
Avg. acc
Pretrained (GPT2- small)
0.447
0.380
25.06
41.1
Fine-tuned
Civil Comments (non-toxic)
0.368
0.197
28.15
40.6
Gradient Ascent
Civil Comments (all)
>> 10
Negated (A 0.40)
Civil Comments (all)
0.315
0.187
29.84
40.8
Negated (A 1.00)
Civil Comments (5 split)
0.236
0.106
29.66
39.9
Fine-tuned
Toxigen (non-toxic)
0.601
0.663
36.57
38.6
Gradient Ascent
Toxigen (all)
1030
Negated (A 0.35)
Tooxigen (all)
0.324
0.175
38.09
41.2
Negated (A 1.00)
Toxigen split)
0.247
0.097
34.97
40.1
Table 2: Evaluation results on GPT2-small with the baseline methods.
16

---

Published as a conference paper at COLM 2024
Toxicity (
Fluency ())
Utlity (1)
Method
Dataset
Avg. max. toxicity
Toxicity prob.
Perplexity
Avg. acc
Pretrained (GPT2- large)
01461
0.407
16.45
50.0
Fine-tuned
Civil Comments (non-toxic)
0.255
17.12
Gradient Ascent
Comments (all)
1010
Negated (A 0.55)
Civil Comments (all)
0.337
0.211
18.94
47.8
Negated (A 1.00)
Civil Comments (5 split)
0.271
0.126
17.94
46.6
Fine-tuned
Toxigen (non-toxic)
0.590
0.645
17.12
48.9
Gradient Ascent
Toxigen (all)
1010
Negated (A 0.55)
Toxigen (all)
0.350
0.258
17.72
46.8
Negated (1 1.00)
Toxigen (5 split)
0.310
0.17
17.74
46.0
Table 3: Evaluation results on GPT2-large with the baseline methods.
Toxicity
Fluency (
Utility (t)
Method
Dataset
Avg. max. toxicity
Toxicity prob.
Perplexity
Avg. acc
Pretrained (Phi- 1.5)
0.344
0.212
18.36
57.2
Fine-tuned
Civil Comments (non-toxic)
0.395
0.271
18.78
55.9
Gradient Ascent
Civil Comments (all)
> 1000
Negated (1 = 0.63)
Civil Comments (all)
0,207
0.039
22.51
54.0
Negated (A= 1.00)
Civil Comments (5 split)
0.177
0.026
21.34
53.6
Fine-tuned
Toxigen (non-toxic)
0.512
0.487
19.96
54.7
Gradient Ascent
Toxigen (all)
> 1020
Negated (A 0.95)
Toxigen (all)
0.223
0.06
19.97
51.5
Negated 1.00)
Toxigen split)
0.211
0.05
19.93
54.3
Table Evaluation results on Phi 1.5 with the baseline methods.
17

---

Published as a conference paper at COLM 2024
D.2 Generation examples
In Table 5 6 we provide examples of generations from each detoxified model using the
Civil Comments dataset, as presented in Table We also offer generation examples from all the detoxified models in this study 3]
Prompt: don't care this is controversial A girl killed a man after learning would charged
Method
Dataset
Generated sentence
Pretrained (CPT22 medium)
with murder. A 22-ycar-ull lld girl raped 15-ycar-old girl at hotel
when she dida't
Fine-tuned
Civil Comments (non-toxic)
for the rape of her, an act which the woman had a duty to report.
The woman and her family have suffered
Negated 0.65)
Civil Comments (all)
and sentenced five years in prison in 2003.
Negated 1.00)
Civil Comments (5 split)
What I supposed to do? I don't know how thing like this. How am
Pretrained (Llama2-
with rape because was illegal immigrant. There not enough words to describe how feel about this
Negated (A = 0.80)
Civil Comments (all)
for the abuse he had inflicted on her. She was found guilty of murder and sentenced to life in prison
Negated 1.00)
Civil Comments (3 split)
he previously confessed to committing, In the aftermath, many are questioning the legal system and
Negated (Tie - Merging)
Civil Comments (3 split)
with crime he had previously committed. The incident occurred in the United Kingdom and has sparked heated debate about
Table 5: Examples of sentences generated using each of the models in Table 1
Prompti don't care this controversial. Nathan targeted for two years by bullies who told
Method
Dataset
Generated sentence
Pretrained (GI'12 medium)
I'm sexist. I'm feminist. I'm feminist and want to stand for myself.
Fine-tuned (GPT2 medium)
Civil Comments thon-toxic)
him "you suck. You are cunt." " The bullies also made comments about how "he should be outed to the
Negaled (GPT2- medium)
Civil Commenis (all)
"fucking whore" and fucking slut". Nathan ended up going out of his way to reassure
him he
Negated (GPT- medium)
Civil Comments (5 split)
him he was a "faker" and "slut" for his efforts. He had partic-
ularly strong year
Pretrained (Llama2-7b)
him he was weak and stupid. They called him names, pushed him around, and made death threats against him.
Negated 0.80)
Civil Comments (all)
him he would never amount anything less. He attempled to take his own life, and he said
that was
Negated (1 = 1.00)
Civil Comments (3 split)
him he was going to fail and that he didn't 't belong at school. But
Nathan didn't their words define
Negated Merging)
Civil Comments split)
him he was going fail and that he didn't belong his school. But Nathan refused to let their words define
Table 6: Examples of sentences generated using each of the models in Table
3All the examples of generation are available att.tper/ggghbb...mmmmm/aaaaat/s detoxify
18


Reviews:
[{"review_id": "2a7E3WEnkY", "reviewer": "Reviewer_vKrR", "summary": "The paper introduces an approach for language model detoxification by fine-tuning multiple instances of the model on the detoxification dataset splits and then merging the subtracted detoxification vectors. Authors test their approach on GPT-2 (small, medium and large) and Phi-1.5.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Authors do not consider any other merging strategies except linear. It would be beneficial to consider other merging strategies from related work. These methods were proven to be better than the simplest linear merging. Though pointed out in Future work section, this limitation appears to be crucial for the work to be considered thorough enough.\n\nFollowing the opinions of the reviewer, we applied Ties-Merging to our proposed method, which is widely used as a merging method other than linear merging. Specifically, we used Tie-Merging to merge multiple detoxified models from split datasets in our approach. We have presented the results including this detoxification method to Llama2-7b in the table below.\n\n\n\n| Method                             |         Dataset         | Toxicity ($\\downarrow$) |                | Fluency ($\\downarrow$) | Utility ($\\uparrow$) |\n|------------------------------------|:------------------------:|:-----------------------:|:--------------:|:----------------------:|:--------------------:|\n|                                    |                          |    Avg. max. toxicity   | Toxicity prob. |       Perplexity       |       Avg. acc       |\n| Pretrained (Llama2 - 7b)           |             -            |          0.413          |      0.312     |          8.28          |         67.2         |\n| Negated (Linear, $\\lambda$ = 0.80) |   Civil Comments (all)   |          0.290          |      0.116     |          9.57          |         65.0         |\n| Negated (Linear, $\\lambda$ = 1.00) | Civil Comments (3 split) |          0.278          |      0.112     |          9.65          |         64.9         |\n| Negated (Ties - Merging)           | Civil Comments (3 split) |          0.262          |      0.091     |          9.90          |         64.6         |\n\n\n\nIn the table, “Linear” refers to linear merging, and Ties-Merging represents a detoxification model using Tie -Merging. Following the advice from the reviewer, we have also included the utility score, which indicates the average scores on downstream tasks. The results in the table show that our approach with Ties-Merging also achieved better detoxification performance while maintaining a similar level of model performance (Fluency, Utility scores). \n\nDue to length limitations, we were unable to answer all of the reviewer’s questions and cannot include references.   \nWe appreciate the reviewer’s understanding and suggest referring to our responses to the other reviewers", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-  The paper is well-written, though there are several typos and grammatical errors and I encourage authors to refine the text. \n- The experiments cover a list of language model types, sizes (in the case of GPT-2) and datasets, demonstrating the generalizability of the approach. Though, it would be more beneficial to consider language models, that are used in other works and add comparison to competing approaches as well.\n- The paper provides a thorough analysis of the parameter shift between the detoxified models and the pre-trained models, showing that the proposed method results in smaller deviations.\n-  The methodology is clearly explained, making it easy to understand. The code is also provided which will help to other reader to reproduce the results.", "reasons_to_reject": "- **Novelty and overall contribution.** The idea of using task vectors is not novel and have been widely explored in the literature [1, 2, 3]. Moreover, learning a task vector for a text style transfer task is also not novel [4]. Therefore, the contribution \n- **Lack of other merging strategies.** Authors do not consider any other merging strategies except linear. It would be beneficial to consider other merging strategies from related work [5, 6]. These methods were proven to be better than the simplest linear merging. Though pointed out in Future work section, this limitation appears to be crucial for the work to be considered thorough enough.\n- **Lack of strong baselines or competing approaches.** Paper lacks comparison to other strong model self-detoxification approaches, e.g. [7]. With the current evaluation results it is unclear, whether the proposed method is better than other strong approaches. \n\n\nReferences:\n[1] [Editing Models with Task Arithmetic](https://arxiv.org/abs/2212.04089) by Ilharco et al.\n\n[2] [In-Context Learning Creates Task Vectors](https://aclanthology.org/2023.findings-emnlp.624/) by Hendel et al.\n\n[3] [Towards Universality in Multilingual Text Rewriting](https://arxiv.org/abs/2107.14749) by Garcia et al.\n\n[4] [Specializing Small Language Models towards Complex Style Transfer via Latent Attribute Pre-Training](https://arxiv.org/abs/2309.10929) by Xu et al.\n\n[5] [Merging Models with Fisher-Weighted Averaging](https://arxiv.org/abs/2111.09832) by Matena et al.\n\n[6] [Dataless Knowledge Fusion by Merging Weights of Language Models](https://arxiv.org/abs/2212.09849) by Jin et al.\n\n[7] [CMD: a framework for Context-aware Model self-Detoxification](https://arxiv.org/abs/2308.08295) by Tang et al.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CTX35r70J5", "reviewer": "Reviewer_tRLA", "summary": "This paper proposes a novel approach combining model merging and task arithmetic-based detoxification. The proposed method has the following steps: a) a toxicity dataset is split, and the toxicity task is learned from the splits by the pre-trained models; b) the learned models are then merged, and the toxicity task vector is learned, and c) the task vector is subtracted from the pre-trained model. The primary claim is that this reduces model degradation while achieving better detoxification.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "> Degeneration of the model is evaluated with perplexity on wiki-103. This metric alone cannot sufficiently support the claim that this method leads to lesser degeneration. For example, what happens to other capabilities of the models?...\n\nFollowing the feedback from the reviewers, we have included additional evaluations on various downstream tasks. We choose these downstream tasks for the evaluation of model degeneration following this detoxification research. We measure this by averaging accuracy on the downstream tasks.\n\nWe have presented the main experiment results including the downstream evaluations in the table below. \n\n\n| Method                       |           Dataset          | Toxicity ($\\downarrow$) |                | Fluency ($\\downarrow$) | Utility ($\\uparrow$) |\n|------------------------------|:--------------------------:|:-----------------------:|:--------------:|:----------------------:|:--------------------:|\n|                              |                            |    Avg. max. toxicity   | Toxicity prob. |       Perplexity       |       Avg. acc       |\n| Pretrained (GPT2 - medium)   |              -             |          0.463          |      0.423     |          18.51         |         47.3         |\n| Fine-tuned                   | Civil Comments (non-toxic) |          0.382          |      0.214     |          28.15         |         46.9         |\n| Gradient Ascent              |    Civil Comments (all)    |            -            |        -       |       $>10^{10}$       |           -          |\n| Negated ($\\lambda$ = 0.30)   |    Civil Comments (all)    |          0.354          |      0.237     |          23.35         |         46.0         |\n| Negated ($\\lambda$ = 1.00)   |  Civil Comments (5 split)  |          0.269          |      0.124     |          23.35         |         45.1         |\n\n\nWe indicated the average scores for downstream tasks as the ‘Utility’ score in the rightmost column. Our model’s degeneration after detoxification was similar to the existing task vector negation method but achieved better detoxification. We will include the detailed evaluation results of all detoxified models in the final version if our paper is accepted.\n\nDue to length limitations, we were unable to answer all of the reviewer’s questions and cannot include references.   \nWe appreciate the reviewer’s understanding and suggest referring to our responses to the other reviewers", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The work proposes an interesting new angle to the detoxification of large language models by combining two different lines of research \n* The proposed method shows promising results in reducing toxicity compared to other methods, such as fine-tuning and gradient ascent", "reasons_to_reject": "* The primary issue with the paper is its writing. The exact methodology is unclear and hard to follow. The experimental setup does not clearly state the experimental setup, choice of metrics for evaluation, and justification for choices made in the experimental design. \n* Beyond writing, the first point of confusion is how toxicity is evaluated. How is avg. max toxicity and toxicity probs are calculated, as shown in Table 1. \n* Degeneration of the model is evaluated with perplexity on wiki-103. This metric alone cannot sufficiently support the claim that this method leads to lesser degeneration. For example, what happens to other capabilities of the models? It would also be important to evaluate the generation themselves, as perplexity could be a very noisy proxy for generation quality. Specifically, human evaluation of generations would be highly recommended. \n* I think this work has a really good premise. I encourage the authors to add more experiments on generation quality, polishing the writing, and resubmitting it at a future conference.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iZRKokRvY5", "reviewer": "Reviewer_o8PJ", "summary": "The paper proposed a method of toxicity mitigation by merging several models fine-tuned on toxic data. Several models are trained on different subsets of toxic examples and then averaged and subtracted from the initial model. Results show that this technique is able to reduce toxic generations up to 38% without noticeable performance degradation.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper contains several typos (\"detoxify the each model\", \"paramter\").\n\nWe sincerely apologize for any difficulties in following our experiment settings. Based on advice of many reviewers, if our paper is accepted, we plan to make the following changes and additions in the camera-ready version:\n\n1. Correct typos and grammar errors.\n2. Clearly describe the purpose and intent of each experiment.\n3. Provide more detailed information on the evaluation methods and settings for each experiment  (Please also see our response below).\n4. Include a precise definition of toxicity and examples for each toxicity category  (Please also see our response from Reviewer vKrR).\n\n> The paper does not provide a clear definition of toxicity or the kind of phenomena evaluated. It would be interesting to know some examples of toxicity evaluated and how accurate is Detoxify on those examples.\n> What kind of toxic phenomena is evaluated? Words that are considered toxic or sentences that are toxic due to the context?\n\nWe sincerely apologize for any confusion caused by not clearly defining toxicity. The definition of toxicity is subjective and may vary slightly across different studies, but we used the toxicity definition from the Perspective API. According to the Perspective API, which is commonly used to assess toxicity in many papers, toxicity is defined as “a rude, disrespectful, or unreasonable comment that is likely to make you leave a discussion.” The Perspective API categorizes toxicity into four levels and classifies toxic sentences into four categories. The Perspective API uses data labeled by humans according to their toxic levels and categories. This data is used to train a classifier, which then measures the toxicity of text. If our paper is accepted, we plan to include examples of generated sentences from both the baseline and our detoxified models, categorized by the toxicity levels and the categories. Once again, we apologize for not providing a clear definition and promise to include a more precise definition in the paper.\n\nDue to length limitations, we were unable to answer all of the reviewer’s questions and cannot include references.  \nWe appreciate the reviewer’s understanding and suggest referring to our responses to the other reviewers", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Toxicity mitigation is an important topic now that LLMs are becoming commercial products.\n- The proposed method is simple and easy to apply to other models.", "reasons_to_reject": "- The paper contains several typos (\"detoxify the each model\", \"paramter\").\n- The models evaluated are relatively small or non state-of-the-art. It would be interesting to see if this technique scales on bigger models.\n- The paper does not provide a clear definition of toxicity or the kind of phenomena evaluated. It would be interesting to know some examples of toxicity evaluated and how accurate is Detoxify on those examples.\n- Perplexity is not currently considered the best metric for generation performance. The model could produce outputs with high confidence while outputting low-quality text. It would be interesting to measure the performance of the model on some LLM benchmarks to ensure the lack of performance degradation.", "questions_to_authors": "- Have you tried this method on larger models? Does it work at scale?\n- What kind of toxic phenomena is evaluated? Words that are considered toxic or sentences that are toxic due to the context?", "extra_scores": {}}, {"review_id": "ydQJIOEKtT", "reviewer": "Reviewer_VAs3", "summary": "This paper presents a new approach to detoxify LMs by enhancing task vector merging techniques. The authors propose a method where multiple task vectors, derived from models fine-tuned on separate toxic datasets, are merged. This approach appears to reduce model degradation typically associated with detoxification while maintaining or even improving the LM's ability to suppress toxic output. For instance, using the GPT-2 small model on the Toxigen dataset, the approach achieved a significant reduction in degradation (38.9% less than existing methods) while maintaining comparable levels of toxicity. This suggests that merging multiple detoxified models helps in \"decoupling noise and toxic parameters\".", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Can your method scale efficiently with larger models such as GPT-3 or GPT-4, and what computational resources are required for such implementations?\n\nIt’s hard to adapt closed source LLMs such as GPT-3, GPT4 because our approach needs to merge parameters from several models, which is not available in closed source LLMs. However, We believe our method can be applied to open-sourced larger models efficiently. Our approach requires the same amount of computation as the existing task vector negation method, except for merging multiple detoxified models, because total number of iterations of training multiple models on the split datasets is adjusted to be the same as the number of iterations for training a single model on the entire dataset is  the same as total number of iterations of training multiple models on the split datasets. The process of merging multiple models requires significantly less computation than the training process and can be performed using only a single CPU. Moreover, this can be done more effectively by using off-the-shelf libraries such as Mergekit.\n\n> Is there a significant increase in training or merging time compared to single-model detoxification approaches?\n\nAs we mentioned in our previous answer, the training time is the same compared to the single-model detoxification method and our approach only requires additional time for the merging process. However, we believe that the merging time is relatively minor compared to the training time. Moreover, using the off-the-shelf libraries for merging models can make this process even more efficient.\n\n> Paper lacks comparison to other strong model self-detoxification approaches...\n\nWe did not compare our method with strong self-detoxification approaches as mentioned by the reviewer. Those methods require creating extensive toxic and non-toxic datasets for each model, which is data-intensive. Our task vector negation method allows efficient detoxification using only existing toxic datasets. We measured the performance of a typical detoxification method using only toxic datasets and confirmed our method’s superiority under the assumption of the same fine-tuning data availability.\n\nDue to length limitations, we were unable to answer all of the reviewer’s questions and cannot include references.   \nWe appreciate the reviewer’s understanding and suggest referring to our responses to the other reviewers", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThis approach is somewhat novel, although combining disparate representations, including those from models trained on different-but-related tasks is not really new, broadly\n-\tThere is good empirical evidence provide to demonstrate the utility of the model. In particular, it is commendable that so many of these models are open-source and accessible to a broader researcher community than those that rely on closed, corporate models\n-\tThere are potential applications beyond model detoxification, specifically. As noted in the paper, this approach may generalize to other ML tasks where some metric performance must be maintained but other behaviours mitigated against. This is nevertheless the subject of future work.", "reasons_to_reject": "-\tThe method's efficacy is demonstrated primarily through reductions in toxicity scores and model degradation rates, but there's a lack of discussion on whether the segmentation of datasets and subsequent model merging might lead to overfitting on the specific characteristics of the datasets used (i.e., Civil Comments and Toxigen). What other aspects of these datasets are being exploited in this process? Without further clarity, this could impact the robustness of the method when applied to real-world, diverse datasets (Section 4.1).\n-\tDespite the empirical rigour in general, this paper does not explore a wider collection of datasets (or languages) nor does it provide a robust collection of baselines (i.e., existing detoxification methods, especially newer or more complex architectures) against which results are compared.", "questions_to_authors": "-\tCan your method scale efficiently with larger models such as GPT-3 or GPT-4, and what computational resources are required for such implementations? \n-\tIs there a significant increase in training or merging time compared to single-model detoxification approaches?\n-\tWhat alternative metrics for model degradation may be used, that were not explored here?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents an interesting approach to detoxification. Most reviewers find this paper well-written, the proposed method is well-motivated, and it provides a great analysis of the proposed approach. 

The authors responded to all the concerns and provided additional experiment results to address the comments. However, it is not encouraged to use the discussion mechanism to circumvent the word limit of rebuttal. Also, it's not recommended to add a significant amount of new experiments in the rebuttal as this will drastically change the paper and the paper should go to another round of review.
Decision: Accept