Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models
Authors: Boxin Wang
Wei Ping
Chaowei Xiao
Peng Xu
Mostofa Patwary
Mohammad Shoeybi
Bo Li
Anima Anandkumar
Bryan Catanzaro
Venue: NeurIPS 2022 Accept (2022)
Keywords: language model
toxicity
ethics

Abstract: Pre-trained language models (LMs) are shown to easily generate toxic language. In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models. We conduct this study on three dimensions: training corpus, model size, and parameter efficiency. For the training corpus, we demonstrate that using self-generated datasets consistently outperforms the existing baselines across various model sizes on both automatic and human evaluations, even when it uses a 3 1 smaller training corpus. We then comprehensively study detoxifying LMs with parameter sizes ranging from 126M up to 530B (3× larger than GPT3), a scale that has never been studied before. We find that i) large LMs have similar toxicity levels as smaller ones given the same pre-training corpus, and ii) large LMs require more endeavor to unlearn the toxic content seen at pretraining. We also explore parameter-efficient training methods for detoxification. We demonstrate that adding and training adapter-only layers in LMs not only saves a lot of parameters but also achieves a better trade-off between toxicity and perplexity than whole model adaptation for large-scale models. Our code will be available at: https://github.com/NVIDIA/Megatron-LM/.

Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models
Boxin Wang"t,, Wei Pingt? Chaowei X Peng Xu², Mostofa Patwary?:
Mohammad Shoeybi', Bo Li', Anima Anandkumar24, and Bryan Catanzaro
University of Illinois at Urbana- Urbana-Champaig Champaign
NVIDDA
Arizona State University *California Institute Technology
Abstract
Pre-trained language models (LMs) are shown to easily generate toxic language. In this work, we systematically explore domain- addativv training to reduce the toxicity of language models. We conduct this study on three dimensions: training corpus, model size, and parameter efficiency. For the training corpus. we demonstrate that using self-generated datasets consistently outperforms the existing baselines across various model sizes on both automatic and human evaluations, even when it uses a 3 smaller training corpus. We then comprehensively study detoxifying LMs with parameter sizes ranging from 126M up to 530B (3 larger than GPT- 3), scale that has never been studied before. We find that i) large LMs have similar toxicity levels as smaller ones given the same pre-training corpus, and ii large LMs require more endeavor to unlearn the toxic content seen at pre- training. We also explore parameter-efficien training methods for detoxification. We demonstrate that adding and training adapter-only layers in LMs not only saves lot parameters but also achieves a better trade-off between toxicity and perplexity than whole model adaptation for large-scale models. Our code will be available at: https: // //tph/gtttb...//////// com WWIDIAAMeegtrrnn-MM.
1 Introduction
Large-scale pre-trained language models (LMs) [1-6] have demonstrated substantial performance gains on various NLP tasks, especially when scaling up the sizes of models. However, recent studies [7, 8] show that generative LMs can generate toxic and biased language, which raises ethical concerns for their safe deployment in real-world applications.
Previous methods on reducing the toxicity of LMs can be categorized as: decoding-time methods, tetttraiinn--basee methods, and domain domain-adaptiv adaptive training methods. Decoding time methods [9-14] manipulate the output distribution or input prompts at the inference stage without modifying the original model parameters. These methods can flexible, but they either resort some simple word filtering strategies [10], or increase the computational cost at the inference stage. For example, PPLM [9] requires multiple iterations of backward propagation through the LM when generating every token, which makes it prohibitively expensive to be deployed to production especially for large-scale LMs. In contrast, une-training-bee -basdd methods directly filter out the potentially toxic
WWrk done during an internship at NVIDIA. 'Correspondence to: Boxin Wang <boxinw2 @illinois.edui edu>, Wei Ping <wping@nvidia.comm, Chaowei Xiao <xianew @asu.edu>.
'For example, the 530B degatron-Turing NLG [6] requires 16 A100 80GB GPUs for autoregressive generation, but 280 GPUs for backward propagation for memory reasons.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

content within the pre-training corpus and retrain the model from scratch [e.g,, 15]. However, it is difficult to determine the filtering criterion beforehand, and pre-training a large LM multiple times from scratch is quite expensive.
Domain-adaptive training methods [10, 16] further fine-tune the pre- pre-trained LMs on carefully curated datasets (e.g., Jigsaw, filtered OWTC [17]). For instance, Gehman et al. [10] construct a nontoxic data corpus from an existing dataset, OWTC, via the Perspective API and perform the fine-tuning on the nontoxic corpus. Domain-adaptive training is more flexible than pre-training methods, as one can still customize the model after the expensive pre-training process. Compared to the decoding -imme methods, domain- Nomain-adaptive training methods have the following advantages: i) they can achieve fast and memory-effii efficient inference, thus can be deployed in broader systems; and ii) they can largely reduce the model toxicity while still maintaining good LM quality measured by perplexity and downstream task performance as we will show this work.
In this paper, we explore the limits of addain-adaptive training for detoxifying language models along the following three aspects: 1) Training Corpus: Unlike previous methods using curated pre-training corpus for detoxification, we propose to leverage the generative power of LMS to generate nontoxic corpus, which achieves better data efficiency for detoxification. 2) Model Size: We systematically study and mitigate the toxicity issues in LMs with parameter sizes ranging from 126M to 530B, a scale that has never been studied before in this domain. 3) cerammetrr-fffcieen Training: We investigate two -efineeter-ffiientt paradigm: adapter [18] and prefix-tunint [19], and compare them with whole model adaptation in a systematic way. We hope our work can shed light on the challenges of detoxifying large-scale LMs, as well as motivate the development of detoxification techniques that are effective and parameter-effiiien without significantly hurting the LM quality.
Summary of Contributions:
We identify the trade-off between detoxification effectiveness (measured by Perspective API and human evaluation) and language model quality (measured by validation perplexity and downstream task accuracy). Existing approaches either suffer from limited detoxification effectiveness or significantly sacrifice the language model quality to detoxify generative LMs.
We propose Self-Generation Enabled domain-Adaptive Training (SGEAT) that uses a self- generated dataset for detoxification. It mitigates the exposure bias [20, 21] from the discrepancy between teacher-foreed domain-adaptiv training and autoregressive generation at test time, and thus achieves better data efficiency. In particular, we demonstrate that it consistently outperforms the baseline approach with Nomain-adaptio training on pre- pre-training data (DAPT) by a wide margin across various model sizes in terms of automatic and human evaluations, even when we use only 3/3 smaller corpus for training. By combining SGEAT with the -fate-of-the-an decoding-time method, we can further reduce the toxicity of large-scale generative LM.
From the perspective of model size, we find that: i) Large LMs have similar toxicity levels as smaller ones given the same pre-training corpus. This implies the toxicity comes from the training dataset, instead of the model size. ii) Large LMS require more efforts (e.g., larger training corpus) to reduce toxicity.
We explore two parameter-efficient training methods for detoxification, and observe that: i) domain-adaptive training with adapter achieves a better trade-off between toxicity and perplexity than whole model adaptation for large-scale LMs, and the improvement is more significant when the size of LMs increases; ii) prefix-tuning is less suitable for detoxification and demonstrates limited detoxification effectiveness and perplexity control.
We organize the rest of the paper as follows. We discuss related work in 2 and present our evaluation protocols in $ 3. We then systematically explore the comain-adaptive training with respect to training corpus in $ 4, model sizes in $ 5, and parameter efficiency in 6. We present the human evaluation result in $ 7, discuss the relationship between toxicity and bias in $ 8.1, and conclude the paper in $ 9. Some text samples can be found in Appendix D.
Related Work
Large-scale language models (LM) have achieved state-of-the-art art performance on various downstream tasks. However, they also exhibit undesirable behaviors in terms of ethical, robustness, privacy, and nonfactual generation issues [10, 22-26]. For example, since they are pre-trained over sizable
httppp//wwwwpprsppctiveeaps
2

---

Table I: Evaluation of LM toxicity and quality across 5 different parameter sizes. Model toxicity is evaluated on REALTOXICITYPPRMPP
benchmark through Perspective API. Full refers to the full set of prompts, Toxic and Nontoxic refer to the toxic and nontoxic subsets of prompts. 4 1/1 means the lower / higher the better. PPL is evaluated on held-out validation set of the pre-training corpus. Utility is estimated by averaging the LM's accuracy on 9 different tasks in the zero-shot learning setting, including Lambada, BoolQ, RACE, PiQA, HellaSwag, WinoGrande, ANLI-R2. HANS and WIC. The accuracy for each task can be found in Table
Exp.
Max. Toxicity (1)
Texicity Prob, (J)
Valid. PPL (1)
Utility
Models
Full
Toxic
Nontoxic
Full
Toxic
Nontoxic
Avg. Ace. (1)
126M
0.56
0.76
0.50
57%.
88%.
48%
17.76
46.7
357M
0.57
0.78
0.51
58%
90%
49%
13.18
50.0
L.3B
0.57
0.78
0.52
59%
90%
51%
10.18
54.3
8,3B
0.57
0.77
0.51
59%.
891.
50%
7.86
60.0
530B
0.57
0.77
0.52
59%
88%
51%
6.27
64.6
collection of online data, they are unavoidably exposed to certain toxic content from the Internet. Recent studies [e.g.. 27-29] show that pre-trained masked LMs display different levels toxicity and social biases. Another line of work focuses on the toxicity of autoregressive LMs. For instance, Wallace et al. [8] first demonstrate that synthetic text prompts can cause racist continuations with GPT-2. Gehman et al. [10] extend the analysis of LM toxicity to non-synthetic prompts, and create benchmark dataset
to provide a standard evaluation protocol via Perspective API to measure LM's toxicity. which is adopted by many previous work. In this paper, we follow the standard setting to compare different detoxification approaches different-sine sizerd LMs.
Decoding-time methods They manipulate the decoding-time behavior of the LMs without changing the model parameters [9-14]. Simple approaches such as word filtering and vocabulary shifting [10] directly lower the probability of toxic words (c.g., swearwords, slurs, vulgar slang) being generated. Though efficient, such approaches fail to consider the semantic meaning of the generated text at the sequence level. Thus, it cannot completely prevent from generating toxic sentences which contain no undesirable words from the blocklist [15] (e.g.. "poor people don't 't deserve to live in nice houses"). Xu et al. [13] perform entence-level filtering by generating K continuations given the same prompt and returning the most nontoxic sentence. Similarly, Self-Debiasing [11] uses K manually crafted templates to manipulate the decoding probability distribution and dynamically set the probability of toxic words to be low. However, these methods lead to K times longer than the normal decoding. PPLM [9] iteratively adds perturbation on the context vector at each step of decoding. Though with better detoxification effectiveness, it suffers much more computational overhead due to multiple iterations of forwarding and backward propagation generate the perturbations. GeDi [12] guides generation at each step with a second LM trained on nontoxic data by computing classification probabilities for all possible next tokens. However, it requires an external LM trained on non-toxic data, which is not easy to access in practice. DEXPERT [14] controls the generation of large-scale pre-trained LM with an "expert" LM trained on non-toxic data and "anti-expert" LM trained on toxic data in a product of experts [30]. It achieves the maate-of-the-tte detoxification results on WAA.TOXICITYYRROPPPO but sacrifices the validation perplexity and downstream task accuracy.
Domain-adaptive training methods They fine-tune the pre-trained LMs to the non-toxic domain by training on curated nontoxic data [10, 16, 31j. Gehman et al. [10] use the DAPT framework [31] to further train LMs on the nontoxic subset (filtered via the Perspective API) of pre-training corpus, OWTC, with GPT-2. Besides DAPT, Gehman et al. [10] propose to fine-tune on a corpus with toxicity attribute token and prepend the nontoxic attribute token as prompt to yield nontoxic Datasets to
generation. Solaiman and Dennison [16] propose a human-crafter
Values-Targett
change model behavior and reflect a set of targeted values. Baheti et al. [32] focus on mitigating the offensive behavior in dialogue systems. They leverage crowd-sourcing to label a conversation dataset generated by an existing dialogue model, and use it for offensive detection and mitigating the offensive behavior via the controlled text generation. In this work, we focus on exploring the limits of comain-adaptive training methods to reduce the toxicity of language models, while maintaining good validation perplexity and downstream task accuracy.
Reinforcement learning (RL) methods There are two concurrent work [33, 34] that study the toxcity behavior of LM with RL. InstructGPT [33] requires collecting human demonstrations and rankings of model outputs for two-stage fine-tunings. It generates 25% fewer toxic outputs with respectful instruction on SEALLTXXCCTYPROMPPPPP reduces 27% toxic outputs from 530B model on REALTOXICITY PROMPTS, and the improvements are higher for smaller models (e.g., reduces 37% toxic outputs from 8B model). To identify the toxic LM behavior, Perez et al. [34] uses RL to improve the generation of adversarial test cases.
than 175B GPT-3. In contrast, our SGEAT
3

---

Promel Construction
Self-ueeration
Data Filtering
Fine-tuniee
Roor people houses
diseee livel
houss
people
Io nice
people
Foor people
Poor people dant desnne ie lo
Poor people dan'
people doel
scsszo locil
desepoc to
and
Figure 1: Overview of the SGEAT method. SGEAT constructs prompts leverage the LMs generate corpus for domain- -daative training. Then, the generated corpus further filtered Perspective API to ensure that the curated dataset has low toxicity. Finally, we use the filtered texts to further perform domain training for detoxification.
domain-adaptive adaptive
3 Evaluation Protocols
In this section, we present our principle for evaluating different detoxification methods. Specifically, we emphasize that detoxification method should focus on both reducing the model toxicity and maintaining the model quality after detoxification. We first discuss the protocol for LM toxicity evaluation, and then present the protocol to evaluate the I.M quality before and after detoxification.
Pre-trained LMs. We investigate the toxicity of variety of standard GPT-3 like LMs with different parameter sizes, ranging from 126M (similar to GPT-3 Small), 357M (similar to GPT-3 Medium), 1.3B (similar to GPT-3 XL), 8.3B to the largest 530B [6]. All of the models are based on Trans- former [35] with different hidden dimension, number of layers, and attention heads. We present more details in Appendix 1. All standard models are pre-trained on the same pre-training corpus, which is an English text corpus constructed from 15 high-quality datasets.
3.1 Toxicity Evaluation
In this work, we follow prior work [15, 10] and perform both automatic evaluation and human evaluation to measure an LM's tendency generate toxic language.
Automatic Evaluation relies on Perspective API, an online automated model for toxic language and hate speech detection. As discussed in the recent work [13, 15, 10), such model is imperfect and demonstrates biases against different demographic groups. Despite the problems, still provides low-cost and scalable approach evaluate the generation toxicity of LMs. Moreover, both study in Section 7 and Welbl et al. [15] find that the toxicity scores from Perspective API are strongly correlated with human evaluation, thus is meaningful to approximately measure LM toxicity. We note that Perspective API update the models regularly. The scores returned by Perspective API may change over time. The toxicity scores reported in the following sections were evaluated before May 2022.
We use the full set of the prompts (around 100k) from REALTOXICITY benchmark [10] to evaluate LM generations via Perspective API in terms of Expected Maximum Toxicity and Toxicity Probability. Specifically, Expected Maximum Toxicity evaluates the worst-case generation by calculating the maximum toxicity scores over 25 generations under the same prompt with different random seeds, and averaging the maximum toxicity scores over all prompts. Toxicity Probability estimates the empirical frequency of generating toxic language, which evaluates the probability of generating toxic continuation (TOXICITY >= 0.5) at least once over 25 generations for all prompts. We follow Gehman et al. [10] and restrict the generations up to 20 tokens or below. We present the automatic evaluation of five LMs with different parameter sizes in Table
Human Evaluation is indispensable for toxicity evaluation, as toxicity judgments are subjective and should ultimately be human-centric [15]. Specifically, we adapt the instructions from Welbl et al. [15] and ask human annotators to evaluate the continuations. More details of human evaluation and how ensure the emotional well-being of annotators can be found in Section 7 and Appendix $A.3.
3.2 LM Quality Evaluation
To understand the impact of detoxification, we evaluate the quality of LM along two fronts: perplexiry and utility. Perplexity (PPL.) is evaluated on a held-out validation set of pre-training corpus
'ee also evaluate PPL on the filtered nontoxic portions of the validation set in Appendix &C.2. We observe the same trends of PPL increase the full held-out validation set.

---

Table 2:
Evaluation of LM toxicity and quality across different detoxification methods on the 1.3B LM. In the first row, 1/t means the lower higher the better. PPL of word banning goes to infinity as the probabilities of some hanned words are set to zero. and @ are compared against the standard 1.3B L.M. For example, @ is preferred for Toxicity and PPI., while is preferred for Utility Average Accuracy.
Exp. Max. Toxicity (1)
Tuxicity Prob. (L)
Valid. PPL
Utiliiy Avg. Acc. (t)
Models
Full
Toxic
Nontoxic
Full
Toxic
Nontoxic
Jigsaw (nontoxic)
0.58
0.77
0.53
61%
90%
53%
11.51 1133
54.6 54.7 10.
Domain-
DAPT (nontoxic)
0.47 (0.10
0.69
0.41
43%
116%
79%
33%
10.40 1622
Adaptive
SGEAT (hcuristic)
0.47 10.10
0.73
0.40
43%
1108
85%
31%
11.14 10M8
54.7
Training
SGEAT (standard)
0.44 [0:13
0.67
0.38
38%
121%)
7656
28%
11.22 11.04
54.6 16.3
SGEAT (sugmented)
0.43
0.68
0.37
37%
1228
77%
26%
11.19 11.01
54.4
Decodlng- Time
Word Banning
0.54
56%
L35
86%
47%
Rejection Sampling (4x slow)
0.45 (0.12
0.68
0.38
39% 12098
78%
28%
10.18
10000
54.3 10.00
DEXPERTS (3x slow)
0.31 (0.26
0.50
0.26
18%
47%
11%
19.87
46.2
Combined
SGEAT Rejection Sampling
0.33 Ln.24
0.36
0.26
21%
(38%
5896
11%
11.19 11.01
54.4 10.1
SCEAT+ DEXPERTS
0.27
0.45
0.22
14%
40%
7%
20.21
+1008
44.9
which measures both the fluency and coverage of output language. The utility is estimated by the performance on downstream tasks. In particular, we evaluate the accuracy of LMs given 9 different tasks, covering question answering, natural language understanding. and commonsense reasoning, in the zero-shot learning scheme. We base the downstream tasks evaluation Gao et al. [36]. We present the LM quality evaluation of 5 pre-trained LMs in Table 1. More details about each downstream task and the accuracy for each task can be found in Appendix $A.3.
We note some recent work [13, 15] demonstrates that existing detoxification techniques can amplify the social biases against minority groups. In this work, we mainly focus on the intrinsic quality of LM and analyze how degrades after detoxification. We leave the bias discussion in $8.1.
In the following sections, we use above evaluation protocols to explore the limits of domain- domain-adaptive training for detoxification on three dimensions: training corpus, model sizes, and parameter efficiency.
Impact of Training Corpus
Training corpus is core factor that impacts the effectiveness and efficiency of domain-adaptiv training. The state-of-the-ar approach, DAPT [10]. adopts a pre-training corpus [17] curated by Perspective API to construct the training dataset for detoxification. In this section, we propose Self-Generation Enabled domain-Adaptive Training (SGEAT), which leverages the generative power of LM itself to construct training corpus for domain adaptive training. To control the variable and have a fair comparison with the existing approach, we also use Perspective API to curate our self self-generated corpus. We show that SGEAT can further push the limits of fomain-adaptive training for detoxification with better data efficiency.
4.1 SGEAT
As shown in Figure 1, SGEAT consists of four steps: 1) prompt construction; 2) self seff--ennration; 3) data filtering; and 4) domain-adaptive training.
Prompt construction is the core part of SGEAT to guide LM to generate training corpus. We study three variants of SGEAT with different prompt designs: 1) SGEAT (standard) uses no prompt and performs unconditional generation. 2) SGEAT (heuristic) uses set of manually crafted prompts inspired by the definition of toxicity from Perspective API. We discuss the set of considered tem- plates in Appendix SB and report the one that achieves the lowest toxicity in our experiments. 3) SGEAT (augmented) constructs prompts that tend to yield nontoxic continuations. Specifically, we find the most nontoxic documents from the unconditional generation, and split each document into half as the prompts and the continuations. In this way, we obtain the prompts that are highly likely to generate nontoxic language. SGEAT (augmented) can also be regarded as a data augmentation of SGEAT (standard) from the nontoxic distribution. We present more details in Appendix $B.
Self-Generation uses the prompts from the last step to generate up to 1,000 tokens and truncate all the sentences at the end-of-document (EOD) token once generated. We use nucleus sampling [37] with p- = 0.9 and the temperature of during generation. To demonstrate the data efficiency of SGEAT, we generate only 100k documents total, in comparison with DAPT in Gehman et al. [10]
that uses 7500k documents from the pre- pre-training corpus.
5

---

Data Filtering further filters out toxic samples to ensure the training corpus is mostly nontoxic. Specifically, we follow the standard DAPT setup in Gehman et al. [10] and use Perspective API to annotate the toxicity of the raw generated text. Different from DAPT that performs aggressive filtering on pre-training data and only keeps the most nontoxic 2% of the documents, we keep the most nontoxic 50% of the generated text to demonstrate the quality and data efficiency of SGEAT. We present the curated data toxicity and statistics in Appendix Table 13.
Domain-Adaptive Training leverages the curated nontoxic corpus to further fine-tune the pre-trained LM with standard log- likellikioood loss and adapt it to the nontoxic data domain. We present more training details in Appendix $A.2.
4.2 Evaluation Results of Domain Training
In this subsection, we evaluate existing domain-adaptive training methods on 1.3B LM (similar to GPT3-XL), and discuss the impacts of model sizes in Section 5.
We consider the following domain-adaptive training baselines: DAPT (nontoxic) [31] uses a nontoxic subset of pre-training corpus annotated by Perspective API to perform domain- adaptive training; and Jigsaw (nontoxic) uses a human-annotated nontoxic subset of Jigsaw Toxic Comment Classification dataset".
Baselines:
We present the evaluation results in Table 2. Among all domain-adaptive training methods, we find that SGEAT (augmented) achieves the lowest toxicity scores with moderate perplexity increases and without degrading the LM utility accuracy (or even improving). Specifically, SGEAT (augmented) reduces the toxicity of the standard 1.3B by 0.14 at the cost of a slight PPI. increase and does not hurt the utility of L.Ms on downstream tasks. Moreover, we note that although DAPT (nontoxic) uses times larger corpus than SGEAT (augmented) (shown in Appendix Table 13), SGEAT (augmented) still achieves lower toxicity than DAPT (nontoxic), which implics that self-generated data has better data efficiency for domain-adaptive training. We think such high data efficiency comes from the fact that i) the self-generated corpus well captures the high-density regions of the output space of a pre-trained LM, and ii) training on autoregressively generated corpus mitigates the exposure bias [20, 21]. which refers to the train-test discrepancy of an autoregressive model. Thus, when we train the LM the self- reffegenerated non-toxic corpus, it tends to increase the likelihood on the non-toxic density region, which enables data-efficient training to detoxify the model.
The human-annotated nontoxic Jigsaw dataset fails to detoxify the LM and even increases the model toxicity. We speculate the major reason is that the nontoxic subset of the Jigsaw dataset has a much higher average data toxicity than SGEAT, shown in Appendix Table 13.
Among SGEAT methods, we observe that SGEAT (augmented) achieves the best detoxification result at similar level of PPL increase, while SGEAT (heuristic) is less effective to detoxify the LM. We think the reason lies in the data diversity: The unconditional generation covers the diverse regions of the generation distribution and yields the most diverse data distribution, and thus SGEAT (standard) also achieves good detoxification performance. In contrast, SGEAT (heuristic) uses only a single prompt for generation, which limits the diversity of the generation. More analysis about prompt design is in Appendix $B.6.
4.3 Evaluation Results of Decoding-time Methods
Besides the domain-adaptic training baselines, we also compare with decoding-tine time algorithms: Word Banning [10] sets the probability of generating any word from a list? of profamity, slurs, and swearwords zero during decoding. Rejection sampling [15, 13] generates up K samples given each prompt until we obtain a nontoxic sample, otherwise we return the sample with the lowest toxicity score from Perspective API. We set K = 4 due to the computational limit. DEXPERTS [14] is the tfate-of-the-at decoding -imooiiing--eeeeeee algorithm for detoxification that uses two auxiliary expert and anti-expert LMs to steer a model's generation. The expert model is the same as DAPT (nontoxic); while the anti-expert model is fine-tuned on the top toxic portion of OWTC with 150k documents.
When comparing domain-adaptive training methods with decoding-tinee time methods. We note that rejection sampling adds 4x computational overhead during decoding, but is less effective than aamain-adaptive training SGEAT, LM rarely generates nontoxic continuations given toxic prompts
htttp:: t/tppa:////uouee wttpw//.aggl..co ctw/c/////ccctcc htt/://jj/aaaaaae jigaas- toxic- comentt classsfication aataattiattioonccclllae..... challango/
aatppe /ttptggbtttttttt///eeeeee http://bb...//////o///////e Litttt////iiti-tteeeeeeeee of--ortt-D-rrreoo ---Diryy-Wauutty-- Obsceze- aud-Dtherwiee-Bee Bad- Words

---

Table 3: Evaluation of LM toxicity and quality of domain-adaptive training methods along different parameter
sizes. 530B' trained with more self- generated data (100k samples). 530B/ trained with more epochs (5
epochs), while the others are trained with 3 epochs. and 4 are compared against the standard LM of the corresponding size.
Exp. Max. Texicity (1)
Toxicity Prob. Toxic
(4)
Valid. PPL (L)
Utility Avg. Ace. (1)
Models
Full
Toxic
Nontoxic
Full
Nontoxic
126M
0.44 10.32
0.65
0.38
37%
42ws
72%
28%
17.97 10.11
46.0 Ja?
357M
0.47 10.10
0.69
0.41
43% (158
78%
33%
13.33 10.15
49.9 Jo.1
DAPT
(nontoxic)
1.3B
0.47 10.10
0.69
0.11
43%
41668
79%
33%
10.40
10.22
54.7 no.4
8.3
0.48 10.09
0.69
0.42
45%
1148
79%
35%
8.12 10.26
59.1 009
530B
0.50
10.07
0.71
0.45
49%
410%
82%
39%
7.32 11.05
63.4 L122
126M
0.39 10.17
0.63
0.33
30%
U271
69%
19%
19,55 13.79
46.3 J0.4
357M
0.42 10.15
0.68
0.35
36%
(228)
77%
21%
14.39 +1.21
49.3 U0.7
SGEAT
1.3B
0.43 10.14
0.68
0.37
37%
L2298
TT
26%
11.19
11.01
54.4 10.1
(augmented)
8.3B
0.44 10.13
0.68
0.37
38%
76%
28%
8.91 +1.05
59,1 U09
5300
0.46 [0.11
0.70
0.40
43%
(166)
80%
32%
7.86 11.59
62.6 U20
530B
0.45
0.69
0.39
41%
78%
31%
7.92 11.65
62.0 L2.6
530B*
0.44 10.13
0.67
0.38
39% (20%)
76%
29%
9.63 13.36
58.8 U588
[13]. Although the state-of-the-art DEXPERTS achieves significantly lower toxicity scores than SGEAT, we also observe that there is concerning perplexity and utility degradation, with an increase of 9.47 in PPL and drop of 9.4% in downstream task accuracy. Such degradation makes the detoxified 1.3B LM quality even worse than standard 126M LM, as shown in Table 1. We hope that our findings can motivate researchers to focus more on the trade-off between detoxification and LM quality when designing detoxification algorithms. Since decoding-time algorithms are orthogonal to comain-adaptio training methods, replace the standard 1.3B model used in rejection sampling and DEXPERTS with SGEAT (augmented) detoxified one, and observe that the combined method can yield the lowest toxicity scores among existing methods.
easy to combine both methods together. Specifically, we
5 Impact of Model Size
We next investigate how the number of model parameters impacts the Nomain-adaptiv training for detoxification. Specifically, we show that 1) models with different number of parameters trained on the same pre-training corpus display similar levels of toxicity; 2) self-generate data consistently demonstrates better detoxification effectiveness than pre-training corpus across different parameter sizes; 3) larger LMs require more efforts to reduce the toxicity.
Standard Model Toxicity. We first evaluate the toxicity of 5 standard LMs across different parameter sizes in Table and Table 9, We observe that the standard LMs, pre-trained on the same pre-training data with different parameter sizes, display similar levels of toxicity. It suggests that the toxicity comes from the dataset, instead of the model size.
Detoxification Effectiveness of SGEAT. We then evaluate our best SGEAT (augmented) and compare with the best consistently outperforms DAPT over different sizes even when using 1/3 smaller training corpus. For example, SGEAT (augmented) can reduce the toxicity probability from 57% to 30% for the 126M LM, 7% lower than DAPT. These results confirm that: the self- generated corpus is more efficient to detoxify the LM than using the curated corpus of pre-training data.
training baseline DAPT (nontoxic) in Table 3. We note that SGEAT
Larger-scale LMs requires more endeavors to detoxify. From Table 3, we observe the detoxification effectiveness decays for both DAPT and SGEAT with the increase of LM parameter sizes. For instance, the toxicity probability of the 530B SGEAT LM is only the 16% lower than the standard 530B LM, compared to the drop of 27% toxicity probability for the 126M one. We figure the potential reason larger LM is that large L.M tends require more training data and
of such small improvement fine-tuning epochs to detoxify. Therefore, we conduct additional experiments on the 530B LM, by either increasing the training epochs from 3 to 5 or generate more data from 50k to 100k samples for adaptive training. We find that while both methods further reduce the toxicity of the 530B LM, training for more epochs might lead to model overfitting and hurts the PPL and downstream accuracy by large margin. In contrast, training with more data demonstrates a better trade-off between detoxification and LM quality. implies that it needs more endeavors detoxify large-scale LMs.
7

---

Table 4: Evaluation of LM toxicity and perplexity of parameter-efficient training methods. and compared against whole model adaptation. We conduct this ablation study using DAPT (nontoxic).
(a) Adapter [18]
(b) Prefix Tuning [19]
Projection Size
Toxicity ())
Valid PPL (.)
Prefix Length
Tuxicity (
Valid.
Exp. Max. Toxicity
Toxicity Prob.
Exp. Max. Toxicity
Toxicity Prob
PPI. (1)
256
0.49 10.12
46% 13%
10.34
128
0.51 10.04
49% 16%
10.35 goos
512
0.49
youz
45% 12%
10.36 L0.4
256
0.51 20.04
48% 15%
10.45 10.05
1024
0.48 yomr
45% 925
10.39 (0.01
0.52 10.08
50% 97%
10.56 10.16
Table 5: Evaluation of LM toxicity and quality of adapter for large-scale LMs. and are compared against whole model adaptation.
Models
Exp. Max. Toxicity (J)
Toxicity Prob (
Valid. PPI. (1)
Utility
(Projection Size 1024)
Full
Toxic
Nontoxic
Full
Toxic
Nontoxic
Avg. Acc. (1)
DAPT (nontoxic)
8.3
0.48 1000
0.70
0.42
45% 109
79%
36%
7.99 [0.13
59.4 103
+adapter
530B
0.50 10.00
0.71
0.45
49% JOR
82%
40%
6.69 (0.3
63.7 10.3
SGEAT (augmented)
8.3B
0.14 10:00
0.68
0.37
38%
1Os
77%
28%
8.88
L0.03
59.0 39.0 10.3
+adapter
530B
0.46 10.00
0.69
0.39
41%
42%
79%
31%
7.22
(0.64
63.3 10.7
LM Quality Evaluation. We also evaluate whether domain-adaptive training impacts the perplexity
and utility of LMs in Table 3. When trained within 3 epochs, we find that the PPL of LMs slightly increases and the LM utility drops a little in most cases, which suggest that models gradually adapt to the nontoxic domain without a significant sign of overfitting or degradation in terms of LM quality.
Domain Adaptation v.s. Overfitting. We vi- sualize the trade-off at different training phases in Figure for 530B LM. Specifically, we record the validation perplexity and model toxicity after 1, 3, and 5 training epochs for DAPT (nontoxic, 150k) and SGEAT (augmented, 50k). We also add curve DAPT (nontoxic, 50k), which samples
Trade-off between Toxicity and PPL
Expected Max. Toxicity
(sugmentdd
0.56
DAPT (nortroic 150k)
DAPT (nortoocs S3k)
0.54
0.52
50k documents from DAPT (nontoxic, 150k) to have a fair comparison with SGEAT (augmented, 50k). We observe that at the beginning of training, the model toxicity drops substantially and barely sacrifices the model PPL (steep slope). Then it is gradually adapted towards the nontoxic domain. SGEAT demonstrates a better trade-off between toxicity and quality, as SGEAT achieves substan- tially lower toxicity with the same PPL after I epoch of training. Finally, we observe the curve is becoming more flat, especially for DAPT, which indicates the transition from the domain adaptation to overfitting.
0.50
0.48
0.46
0.44
6.5
7.0
7.5
8.0
8.5
9.0
9.5
PPL
Figure 2: The expected maximum toxicity v.s. model perplexity for the $30B LM at different training steps.
For LMs with different sizes fine-tuned with different methods, we find 3 epochs is good cut-off point for whole model adaption, which achieves good trade-off between model toxicity and perplexity. This rule of thumb is also aligned with previous study [10].
6 Parameter-efficient Training
To cope with the challenges of large-scale LMs, we explore two parameter-efficient training paradigms: adapter [18] and prefix tuning [19], and evaluate whether they can improve the LM quality and achieve a better trade-off between detoxification and LM quality than whole model adaption. We show that: in the scenario of detoxification, 1) adapter demonstrates a better trade-off than prefix tuning, and 2) adapter can further mitigate the drop of LM quality and improve the trade-off upon whole-model adaptation for large-scale LMs.
6.1 Comparison between Adapter and Prefix Tuning
Both adapter and prefix tuning add additional parameters to the standard LM, and only optimize the added parameters during training without perturbing the original LM parameters. Such paradigm provides the flexibility, especially for large-scale LMs, to adapt to different domains with a few additional parameters, rather than heavily fine-tune the whole model with multiple copies of the
8

---

whole model parameters for different domains. In this study, we further investigate whether such training schemes can provide more advantages detoxify LMs.
Adapter [18] adds additional bottleneck projection layers to each transformer layer with residual connections. At the beginning of the training. the projection layer is initialized to almost zero to improve the training stability. Prefix tuning [19] appends additional continuous "prefix" vectors to the input to better steer LMs' generations. To have a comprehensive understanding and comparison between adapter and prefix tuning, we first perform ablation studies on small-scale 1.3B LM over the key hyper-parameters. the projection size for adapter and the prefix length for prefix tuning. We follow the same training schedules as whole model adaptation but train more epochs so that the PPL reaches a similar level as whole model adaptation. We present the evaluation results in Table
When comparing Table 4a with Table 4b, we observe that adapter demonstrates a better trade-off between detoxification and LM quality than prefix tuning. We figure the possible reasons two folds: 1) given the same projection size and prefix length, the number of additional parameters of adapter is around twice more than prefix tuning, which gives more capacity for adapter to perform domain adaptation: 2) however, while longer prefix length could give more capacity to steer the model generation, it also adds too many irrelevant contexts, which not only hurts the perplexity of the LM but also slows down the decoding speed. Compared to the whole model adaption, adapter does not show significant advantages in terms of detoxification and LM quality for small-scale models like 1.3B one. For adapter results with different projection sizes, we observe that a larger projection size yields better detoxification effectiveness possibly due larger model capacity. We thus apply adapter with the projection size= 1024 to larger-scale LMs (8.3B and 530B) and investigate whether it can solve the challenges of large-scale LMs.
6.2 Apply Adapter to larger-scale Models
We follow the same training schedules as the whole model adaptation to train the adapters for larger- scale LMs. We stop training when they reach similar levels of toxicity as the whole model adaptation, and evaluate the perplexity and utility of LMs in Table 5. We can see that for larger-scale LMs, adapter can not only improve the parameter efficiency, but also mitigate the PPL and the LM quality drop. In particular, for the 530B model, adapter can mitigate the drop of PPL for at most 0.64 and improve the average downstream task accuracy by 0.7%.
Human Evaluation
We further verify our findings via human evaluation on the standard models, DAPT, SGEAT, and decoding-tines -imom algorithm DEXPERTS across five LM sizes.
Setup. We sample the 300 prompts from RE- ALTOXICITY PROMPT PROMPT benchmark while keep- ing the ratio of toxic and nontoxic prompts to 1:3 as the same as the full set, and evaluate the continuations of each model. We follow
Humen-amdall annaaandddatddd Average max Toecly
0.60
DAPT CAPTURETNO)
0.55
133
2314
DAPTT 394
S0EATT X91
S0DA Cespera 5050 -ddparrtt)))) WAC whuappstasss
DARTIA A4)
SREATT
0.50
Welbl et al. [15] to ask LMs to generate up to 100 tokens and avoid incomplete sentences
0.40
and collect the most toxic continuations via Perspective API over 25 generations. Finally, we gather 5,700 continuations from 19 models and randomly shuffle them for human evalu- ation. Then we group samples into batch of 10, and assign them to 5 annotators. In total 187 workers from Amazon MTurk participated in the evaluation. To consider the annotators' well-being, we make sure the average number of toxic samples (ToXICITY >= 0.5 evaluated by Perspective API) is less than or equal to in each batch of 10 samples. To calculate the average scores of annotations, we follow Welbl et al. [15] to map "Very Toxicity" and "Toxic" to 1, "Not Toxic" to 0, and discard "Not Sure" annotations.
0.35
0.30
0.25
0.28.25
0.30
0.35
0.40
0.45
0.50
0.55
0.60
Perspective API Average Max. Toxicity
Figure 3: (best viewed in color) Average human toxicity scores v.s. Perspective API scores for the different methods evaluate. The Pearson correlation coefficient is 0.9661.
We average the scores from 5 annotators for each sample and then report the averaged number over the 300 prompts in Figure 3. The detailed scores can found Table in Appendix. We present more details in Appendix $A.3. By comparing the objective evaluation with human evaluation,
9

---

Table 6: LM PPL in the gender and ethinicity domains on the BOLD dataset. @: based on standard 1.3B LM.
Gender (L)
Ethnicity (L)
Models
Male
Female
European
Asian
African
Hispanic
Standard
11.6
11.4
13.9
13.5
14.1
15.6
SGEAT
12.7 t11
12.4 +1.
15.1 t1.2
14.8 +1.
15.4
f1.3
17.2 +1.6
we observe that the toxicity scores from the human evaluation are mostly aligned with objective
evaluation via Perspective API. Such findings are also confirmed by Welbl al. [15]. The human evaluation also verifies that i) LMs of different sizes have similar levels of toxicity, and it) LMs of larger sizes present more challenges to detoxify.
8
Discussion
8.1 Bias against Marginalized Groups
We follow the setting of Welbl et al. [15] and evaluate the PPL of the 1.3B standard LM and SGEAT (augmented) fine-tuned LM on the gender and ethnicity domains using the BOLD dataset [38] as shown in Table 6. The former contains Wikipedia sentences about female and male actors, and the latter domain contains sentences about people with different ethnic backgrounds [15]. We find that: (7) LM PPL. increases moderately the BOLD dataset after effective detoxification, which is aligned with our findings in $4.2. (ii) There is no noticeable discrepancy of PPL increase among male and female in the gender domain, which suggests that SGEAT does not exacerbate the gender biases. (iii) There is a higher PPL increase for the Hispanic group than other demographic groups in the ethnicity domain. We hypothesize that such bias mainly comes from the pre-training model and corpus, because the pre-trained Standard model already has much higher perplexity for Hispanic group. Our findings partly align with recent findings on the trade-off between detoxification and bias [13, 15]. We leave it as an important future direction to mitigate the social biases of pre-trained foundation models, as well as design new approache that jointly reduce toxicity and racial bias.
8.2 Limitation of SGEAT
While we observe that SGEAT has demonstrated very good trade-off between detoxification effec- tiveness and perplexity, SGEAT still has potentials further improve.
Bias within Hate Speech Detector. Similar to DAPT, SGEAT also relies on a hate speech classifier (i.e., Perspective API) to filter out toxic samples. However, existing classifier on toxicity classification is imperfect and is known to amplify the social bias against different demographic groups due to the annotation bias and sampling bias [13] (e.g.. the classifier tend assign higher toxicity scores for text mentioning historically underrepresented groups). As a result, SGEAT may also be impacted due to the use of Perspective API, which may filter both toxic text and minority identity mentions. Nevertheless, we believe that SGEAT can get more benefits with more robust, unbiased, and fair hate speech detector, so models fine-tuned on the filtered corpus can unlearn toxicity without forgetting corpus from minority groups
Bias within Pre-trained Model. As discussed in $ 8. we observe pre-trained models already exhibit bias against certain demographic groups. As a result, the self-generated corpus may inherit the bias and harm the coverage of detoxification. Thus leave as an important future direction to build bias-free pre-trained LM, which can benefit SGEAT and other detoxification methods.
Conclusion
We explore the limits of domain-adappa training for detoxifying LMs along three aspects: 1) training corpus; model size and 3) parameter-efficcentt training. We first identify the trade-off between detoxification effectiveness and LM quality in detoxification methods. We propose Self-Generatin Enabled domain- Adaativee Training (SGEAT), which leverages the generative power of LMs for data-efficien and effective detoxification. We comprehensively detoxify LMs with parameters sizes ranging from 126M up to 530B and find interesting properties of large-scale LMs. We demonstrate that adapter provides parameter-efficient training and achieves better trade-off of toxicity and LM quality. We hope our work can shed light on the development of detoxification techniques that can largely reduce toxicity while maintaining good perplexity and downstream task accuracies.
10

---

References
[1] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodci, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
[2] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv:1910.1068: 2019.
[3] Mohammad Shoeybi, Mostofa Ali Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. ArXiv, 2019.
[4] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
[5] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03991, 2021.
[6] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train megatron-turing nlg 530b, a large- scale generative language model. arXiv, 2022.
[7] Kris McGuffie and Alex Newhouse. The radicalization risks of GPT-3 and advanced neural language models. arXiv, 2020.
[8] Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nip. In EMNLP, 2019.
[9] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In /CLR, 2019.
[10] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah Smith. RealTooxic- ityPrompts: Evaluating neural toxic degeneration in language models. In Findings in EMNLP, 2020.
[11) Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nip. TACI, 2021.
[12] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. arXiv, 2020.
[13] Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. Detoxifying language models risks marginalizing minority voices. In NAACL, 2021.
[14] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti-experts. In ACL, 2021.
[15] Johannes Welbi, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. Findings of EMNLP, 2021.
[16] Irene Solaiman and Christy Dennison. Process for adapting language models to society (PALMS) with values-t datasets. arXiv preprint arXiv:2106. 100.22000..0...... 2021.
[17] Aaron Gokaslan and Vanya Cohen. Openwehtext corpus. http: hatp://Sta h/tp:///kkyllo// wot.gg/.too.............. io /OpensebTextCorpus, 2019.

---

[18] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nip. In ICMI,, 2019.
[19] Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing continuous prompts for generation. In ACI, 2021.
[20] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In NIPS, 2015.
[21] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In EMNLP, 2016.
[22] Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. In Advances in Neural Information Processing Systems, 2021.
[23] Boxin Wang. Chejian Xu, Xiangyu Liu, Yu Cheng, and Bo Li. SemAttack: Natural textual attacks via different semantic spaces. In Proceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, 2022.
[24] Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang, and Bo Li. T3: Tree- autoencoder regularized adversarial text generation for targeted attack. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1134-6150 2020.
[25] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650,, 2021.
[26] Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Factuality enhanced language models for open-ended text generation. In NeurIPS, 2022.
[27] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang, Gender bias in contextualized word embeddings. In NAACL, 2019.
[28] Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. On measuring social biases in sentence encoders. In NAACL, 2019.
[29] Christine Basta, Marta R Costa-Jussà, and Noe Casas. Evaluating the underlying gender bias in contextualized word embeddings. arXiv preprint vrXiv:1904.0877 2019.
[30] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 2002.
[31] Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: adapt language models to domains and tasks. In ACI., 2020.
[32] Ashutosh Baheti, Maarten Sap, Alan Ritter, and Mark Riedl. Just say no: Analyzing the stance of neural dialogue generation in offensive contexts. In Proceedings of the 202/ Conference on Empirical Methods Natural Language Processing, pages 4846-4862, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.186 53/v1/2021 wtt.222...mmlpmmaae main..3mm7..97 URL https://aclanthooi hacaa///lllattologooooooooooo/// org/2021 wwnlp-main.
[33] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02 155, 2022.
[34] Ethan Perez, Saffron Huang, Francis Song. Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint ar arXi::200..03286, 2022.
12

---

[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[36] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou, framework for few-shot language model evaluation, 2021. www.::ttps:/oooo org/ 10. $281/zenoda 53r.odd 5371628.
[37] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2019.
[38] Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai- Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability. and Transparency, pages 862-872, 2021.
[39] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: new benchmark for natural language understanding. In ACL, 2020.
[40] Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova, Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
[41] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In ACL, 2019.
[42] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. NAACL, 2016.
[43] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piga: Reasoning about physical commonsense in natural language. In AAAI, 2020.
[44] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In EMNLP, 2017.
[45] Mohammad Taher Pilehvar and Jose Camacho-Collados. Wic: the word-in word-in-conte in-contextt dataset for evaluating sensitvse meaning representations, In NAACL, 2019.
[46] Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, 2020.
[47] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhenghao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. arXiv, 2021.
[48] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts. arXiv, 2020.
[49] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv, 2021.
[50] Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. arXiv, 2021.
[51] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few-shot text classification and natural language inference. arXiv, 2020.
[52] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. arXiv, 2020.
13

---

[53] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao. and Bill Dolan. A pronmttig objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1 110-119, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10. 10.18653/v 1/N16- 1014. URL https: / aalaacclnthht acl.aaoootooooooooooooooooo www.oogg.ooooooooooee org/N16- 111411161
[54] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint vrXv::2102.1209, 2021.
14


Reviews:
[{"review_id": "m3gOKoHpHL", "reviewer": "Reviewer_MutJ", "summary": "* This paper investigates language detoxification in multiple settings: training corpus, model size, and efficient modeling.  \n* Also, it proposes a new detoxification method training language models (LM) with a set of self-generated non-toxic text, which is a smaller corpus compared to OWTC.\n* The experiments show that the training corpus matters the most for the detoxification, as the model size gets bigger the more effort is required (more training epoch or larger corpus) and the importance of the adaptor, which reduces the detoxification while improving the perplexity of the LM.\n* Overall: I think this paper introduces variant DAPTs on different sets of training datasets and a different architecture (adapter). This paper is more toward an analytic research paper finding components which must considered for a detoxification model, which I do believe is meaningful. However, the originality of the proposing method is weak.", "questions": "1. Is there a reason for using the utility of average accuracy? Toxicity scores already indicate the accuracy of toxicity of generated text.\n2. In section 3, regarding the pertained LM, shouldn't the models be represented as GPT-2 small, medium, and xl instead of GPT-3? GPT-3 itself is known to be a model using 175B parameters. To be more accurate, I think GPT-2 makes more sense than GPT-3.\n3. Have you experimented with a similar setting with OWTC which uses the random sampled 100K from 7500K and from the sampled set utilizing the top 35% (~50K) nontoxic texts for the training? \n    * I think that Figure 2 should experiment on the same dataset size to claim the model is overfitting.\n4. In Table 2&10, SGEAT(heuristic) tends to have more toxic text than SGEAT(standard&augmented). The paper argues that SGEAT(heuristic) is more toxic than the other two SGEAT data diversity. How diverse are the sentences generated? Could you evaluate using distinct-1, distinct-2 & distinct-3, or another metric that indicates the diversity of generated sentences?\n5. Is the validation set for the ppl evaluation derived from the non-toxic 150K OWTC? Or each training corpus \n6. How is DEXPERTS with SGEAT (augmented) formed? DEXPERTS is consist of two auxiliary models, expert, and anti-expert. Did you train both two models using SGEAT (augmented)? Or did you only use the detoxified one? ", "limitations": "* This paper has well stated the limitation of the SGEAT. \n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Originality\n    *  The concept of training with self-generated text for detoxification is proposed in dialogue generation [1]. The approach might not be novel to the task requiring detoxification.\n* Quality\n    * The purpose and direction of this paper are good. But there are claims that need to be justified (3,4).  \n* Clarity \n    * The most of writing is clearly written. Few of them need more explanation or clarification (1,2,5,6).\n* Significance\n    * With a smaller refined training corpus, SGEAT(augmented) achieved better performance than the baselines. This paper conduct exhaustive experiments and analyses on large-scale models.  \n    * However, the prompt construction for the heuristic result is similar to an attribute conditioning model [2], which results in high toxicity scores compared to standard or augmented. The experiments on prompt construction seem unnecessary for the proposing model SGEAT.\n\n[1] Baheti, Ashutosh, et al. \"Just say no: Analyzing the stance of neural dialogue generation in offensive contexts.\" arXiv preprint arXiv:2108.11830 (2021).\n\n[2] Gehman, Samuel, et al. \"Realtoxicityprompts: Evaluating neural toxic degeneration in language models.\" arXiv preprint arXiv:2009.11462 (2020).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"]}, "extra_text": {}}, {"review_id": "d0dW7V_bTdH", "reviewer": "Reviewer_Pnxe", "summary": "This work explores domain-adaptive training to reduce the toxicity of language models in terms of training corpus, model size, and parameter efficiency. \n\nAs the results show, for the training corpus, using the proposed Self-Generation Enabled domain-Adaptive Training (SGEAT) that uses a self-generated dataset for detoxification gives better performance compared to the existing baselines. For the model size, this work shows that when given the same pre-training corpus, the toxicity level is still the same across different model sizes. Where the authors claim that this finding implies that the toxicity comes from the training set, instead of the model size. Moreover, they also show that large language models are harder to detoxify compared to smaller ones. Finally, this work also demonstrates that adding adapter layers to standard language models for parameter-efficient training improves the model perplexity while maintaining the same level of detoxification.", "questions": "Please refer to the main weaknesses section.", "limitations": "1. The main limitation of this work is the error propagation from the automatically generated training set and the Prespective API that has been used to filter the non-toxic samples.\n2. Please see the other limitation in the main weaknesses section.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This work studies an important problem of toxicity evaluation and mitigation in language models.\n2. The proposed SGEAT approach that uses language models to automatically generate non-toxic training data seems promising when the generated data are used to fine-tune/detoxify the same language model compared with using the time-consuming human-annotated non-toxic datasets.\n3. Extensive experiments and ablation studies have been carried out to support the proposed methods.\n4. The authors have not only evaluated the detoxification performance, but also the quality and utility of the detoxified language models.\n5. The findings regarding the relationship between toxicity level, training data, and model size are very interesting and can benefit the community. \n\nWeaknesses:\nMajor:\n1. This work did not study the detoxification performance of the generated training set with SGEAT when used to fine-tune other language models different from the one it was generated on. \n2. The authors addressed in detail the effect of fine-tuning/detoxifying with different epochs, but did not do the same detailed evaluation on training samples. For instance, it would be interesting to also have the same Figure 2 for different training samples. \n3. In lines 299-300, the authors claim that \"SGEAT achieves substantially lower toxicity with the same perplexity\" while Figure 2 shows a different trend.  \n4. In Table 5, the combination of SGEAT and adapter has only been evaluated on the perplexity and utility of the language models but not on the detoxification/toxicity level. It would be interesting to also evaluate the effect of this combination on toxicity.\n5. Since there are no prompts for the SGEAT (standard), how is it controlled so that it does not randomly generate duplicated outputs? Has this been evaluated?\n6. Why is SGEAT (augmented) only generated from the non-toxic samples of SGEAT (standard), but not also from non-toxic samples of SGEAT (heuristic)?\n\nMinor:\n1. SGEAT and SGEFT are used interchangeably throughout the paper including Figure 2 and Table 12 while SGEFT was not defined anywhere in the paper. Is it a typo? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "U7k8JzKm8hf", "reviewer": "Reviewer_zpdX", "summary": "This paper investigates the domain-adaptive training for detoxifying while maintaining LM quality of language models. They focus on three points: training corpus, model size, and parameter efficiency. They've made the contributions and concluded the important observations as follows:\n1) from the perspective of training corpus, the proposed self-generated dataset is more helpful for detoxification compared with domain-adaptive training on nontoxic subset of pre-training data.\n2) from the perspective of model size, larger LMs suffer more from toxicity from the training corpus\n3) from the perspective of parameter efficiency, adapter achieved better trade-off between toxicity and perplexity compared with prefix-tuning.", "questions": "Q1: the model architectures of different scales are not clearly described. From Section 3, the list of parameter sizes of different models are introduced, but are they share a similar architecture and what kind of architectures do they adopt?\n\nQ2: In related work, domain-adaptive training methods, such as Gehman et al. (prepend nontoxic attribute token as prompt) and  Solaiman and Dennison [16] (change model behavior & reflect a set of targeted values), are discussed. What's the performance of these approaches compared with the proposed SGEAT?\n\nQ3: Besides domain-adaptive training and decoding-time approaches, RL approaches such as InstructGPT have been proposed to avoid toxicity of LMs. What's the performance of RL-based approaches compared with the proposed SGEAT?\n\nQ4: in Table 2, it seems combining domain-adaptive training approach (SGEAT) and decoding-time approach (Rejection Sampling and DExperts) can further improve toxicity evaluation and utility evaluation. How about the combination of other domian-adaptive training approaches, e.g., Jigsaw and DAPT, with decoding-time approach?\n\nQ5: as shown in Tbale 3, SGEAT obtains better performance on toxicity mitigation while DAPT achieves better results on perplexity and utility evaluation. How about the combination of two, i.e., fine-tuning the LMs on the dataset containing both nontoxic pre-training corpus and generated text data? Can the combined corpus take the advantage of both strategies and therefore have a better trade-off?", "limitations": "The authors have discussed the limitations of this paper in appendix, from the perspective of:\n- limitation of toxicity evaluation platform Perspective API; \n- potential bias in detoxification methods is left for future work", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) originality: \n- this paper proposes the self-generated dataset for detoxification, which brings benefits to mitigating toxicity compared with baselines that fine-tuning LMs on nontoxic pre-training corpus\n- this paper studies effectiveness of different detoxifying approaches on LMs of diverse scales, which include 530B model that hasn't been studied for toxicity before.\n\n2) quality & clarity:\n- the paper is well written and declares points clearly in each section\n- the performance of different strategies and their improvement/degradation are clearly reported; results of experiments with multiple random seeds are also reported in appendix\n\n3) significance:\n- strategies compared and observations discovered in this paper may shed light on follow-up work in improving trade-off between toxicity and utility, large-scale LM design including architecture, training corpus for pertaining, etc.\n\nWeaknesses\n1) clarity\n- the experiments focus on generative language models, including toxicity evaluation with RealToxityPrompt benchmark and zero-shot utility evaluation on downstream tasks following the prior framework. However, the title, abstract, introduction and conclusion haven't discussed this point, but use \"language model\" instead. This could be confusing and easily leads to the conclusion that the observation discovered in this paper could apply to general language models.\n\n2) significance:\n- fine-tuning LMs with untoxic generated text data is the core idea in the proposed SGEAT. However, according to the results reported in Table 2 and 3, although SGEAT could improve performance on toxicity evaluation, the performance on LM quality (perplexity and accuracy on downstream tasks) is consistently poorer than the baseline approaches DAPT and Jigsaw. Those experiments do not provide clear guidance on which kind of adaptation approach is better for projects that would like to obtain a trade-off between the two. \n\n- Toxicity is an important problem when fine-tuning not only generative LMs, but also general (encoder alone) LMs on downstream tasks. The proposed strategy and evaluation conducted in this paper could be more significant if general LMs are also studied. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RPizt6fzSkB", "reviewer": "Reviewer_qG1s", "summary": "This paper explores domain-adaptive training (i.e., continued pretraining on special data to remove unwanted model behavior) with respect to the influence of model size, training corpus, and parameter efficiency. The authors further propose self-generation enabled domain-adaptive training (or \"SGEAT\"), where a self-generated dataset is used for detoxification. SGEAT can also be beneficially combined with a decoding-time method for detoxification.\nThe authors find that, SGEAT reduces model toxicity, but that model size has little effect on model toxicity. (I.e., the toxicity comes from the data.) They also show that it is more challenging to detoxify large language models.", "questions": "- In the paper you list your hypotheses regarding why SGEAT should work better than existing approaches. Is there a way to explore your assumptions in more detail?\n- What are the limitations of your work?", "limitations": "I don't think there are obvious negative societal impacts of this work. The authors do not explicitly address limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Work on detoxifying language models is important in order to safely deploy them. Thus, the topic of the paper is highly relevant.\n- The fact that SGEAT works is interesting: apparently the model benefits from seeing its own (detoxified) data.\n- The authors use both automatic and human evaluation in their experiments.\n\nWeaknesses:\n- Some of the findings are rather straightforward/potentially not novel, even though I don't know references from the top of my head. For example the trade-off between detoxification effectiveness and model performance is obvious (as our datasets are biased/toxic). Similar for the fact that detoxifying larger models is harder.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept