GTA: Gated Toxicity Avoidance for LM Performance Preservation
Authors: Heegyu Kim
Hyunsouk Cho
Venue: EMNLP 2023 Findings (2023)
Keywords: language model
controllable text generation
toxicity avoidance
nontoxic text generation
text generation
natural language generation
TL;DR: The controllable text generation method reduce toxicity and degrade performance of the language model. We proposed the gated toxicity avoidance to resolve this problems. It reduces toxicity and preserves LM performance.

Abstract: Caution: This paper includes offensive words that could potentially cause unpleasantness. The fast-paced evolution of generative language models such as GPT-4 has demonstrated outstanding results in various NLP generation tasks. However, due to the potential generation of offensive words related to race or gender, various Controllable Text Generation (CTG) methods have been proposed to mitigate the occurrence of harmful words. However, existing CTG methods not only reduce toxicity but also negatively impact several aspects of the language model's generation performance, including topic consistency, grammar, and perplexity. This paper explores the limitations of previous methods and introduces a novel solution in the form of a simple Gated Toxicity Avoidance (GTA) that can be applied to any CTG method. We also evaluate the effectiveness of the proposed GTA by comparing it with state-of-the-art CTG methods across various datasets. Our findings reveal that gated toxicity avoidance efficiently achieves comparable levels of toxicity reduction to the original CTG methods while preserving the generation performance of the language model.

GTA: Gated Toxicity Avoidance for LM Performance Preservation
Heegyu Kim
Hyunsouk Chol
Department of Artificial Intelligence', Department of Software and Computer Engineering', Ajou University, Suwon 16499, Republic of Korea (khk6435, hyunsouk ac.
Abstract
Please generate negative restaurant review!
Caution: This paper includes offensive words that could potentially cause unpleasantness. The fast-paced evolution of generative lan- guage models such as GPT-4 has demonstrated outstanding results in various NLP generation lasks. However, due to the potential generation of offensive words related race gender, various Controllable Text Generation (CTG) methods have been proposed to mitigate the occurrence of harmful words. However, exist- ing CTG methods not only reduce toxicity but also negatively impact several aspects of the language model's generation performance, in- cluding topic consistency, grammar, and per- plexity. This paper explores the limitations of previous methods and introduces novel so- lution form simple Gated Toxicity Avoidance (GTA) that can be applied any CTG method. We also evaluate the effective- ness the proposed GTA by comparing with state-of-the-arr CTG methods across various datasets. Our findings reveal that gated toxic- ity avoidance efficiently achieves comparable levels of toxicity reduction to the original CTG methods while preserving the generation per- formance of the language model.
Their pizza was fo ring harriblelt dav stafis! Joo0
Book
Their pizze wasn't good staff not kind good inci hart Instipptte
Controllable Text Gemeration
Their Staffs unfriendly.
Gated Toeicity Avcidance (ours)
Figure I: LM can generate an unintended toxic text. The controllable text generation method prevents the genera- tion of toxic text but degrades the LM performance. Our gated toxicity avoidance method preserves performance while reducing toxicity.
To remove the toxicity from the language model, various Controllable Text Generation (CTG) meth- ods (Dathathri et al., 2019; Krause al., 2020; Liu et al., 2021; Zhang and Song. 2022) have been suggested and shown results with decreased toxicity. We observed that CTG methods show less toxicity than modern approaches like Rein- forcement Learning from Human Fe)dback(RLHFF) methods(Appenall A). However, there two dif- ficulties for the application of CTG in practice. 1) Quality degradation: In our findings, as shown in Fig. 1, CTG methods degrade the generation performance of LM in various aspects. There a notable decrease in the intended topic accuracy of texts generated with CTG methods compared to those generated solely by the original LM. CTG methods also decrease the quality of grammar and fluency (Gu al., 2022). Inference overhead: CTG methods require additional inference time and memory. When we experimented with three well- known CTG methods, we found that they slowed the generation speed by 6. 75 times on average. This
1 Introduction
Large Language Models (LLMs) outperformed var- ious generation tasks, which is still challenging because they also easily generate toxic text. Un- like other performance issues, generating toxic text including hate speech and profanity, significantly affects the service of the LLMs. For example, Tay, a chatbot released by Microsoft in 2016, gener- ated toxic tweets, and the service was temporarily suspended 16 hours after being released. Recently released Llama-2 (Touvron et al.. 2023) outper- formed academic/professional exams and improved its safety, but it still can generate toxic text.
*Corresponding author

---

CTG methods. Our experimental codes and results are publicly available.
watched horror movie. It was so
Related Works
2.1 LM performance degradation analysis
Previous studies have analyzed the drawback of toxicity avoidance on LM performance in specific limited criteria. (Welbl al., 2021) empirically demonstrate the potential bias of toxicity through LM loss changes. The loss of an LM pretrained with clean data to reduce toxicity fluctuates de- pending on gender, ethnicity, and demographics. Because of the bias, the LM pretrained with clean data fails generate minority identity mentions and minority dialects (Xu et al., 2021). (Gu et al., 2022) also showed that CTG methods degrade gen- eration quality as text length increases. Previous studies only analyzed simple indicators such as gender and length. However, we analyze the CTG method's s holistic performance degradation (topic preservation, grammar, and fluency) and the ex- pected inference overhead of the CTG method for the actual service.
LM
CTG
good bad stupid
good bad stupid
Figure 2: The CTG method decreases toxic token prob- abilities but adjusts the probabilities of negative tokens ("bad") and positive tokens ("good").
overhead can significantly influence on multiuser interaction-based services that require near-real- time responses (e.g., chatbot).
This LM performance degradation is caused by the bias of the CTG method. To avoid toxic token generation, the bias of the CTG method adjusts the probability distribution of the text provided by the LM. However, this bias not only avoids toxic token generation but also affects the probabilities of tokens in some particular topics.
2.2 Toxicity avoidance
Toxicity avoidance is technique that prevents an LM from generating toxic sentences. There are three types of toxicity avoidance methods. The first method, ranking, is a way to generate several texts and pick the less-toxic one in the generation step. Ranking is expensive to generate multiple sen- tences, so it is challenging to apply because of the recent trend of increasing LM size. The second method, text style transfer, is regenerating from the generated toxic text. However, it costs twice as much because there are two steps: generation and regeneration. (Details of experimental results are in the Appendix B). The third method, Controllable Text Generation (CTG), controls the LM to gen- erate text that satisfies a specific condition. Since CTG is casy to apply to any type of LM and more effective than the other two types, recent studies have used CTG.
For example, in Fig. 2. the CTG not only re- duces the probability of the toxic token ('stupid') but also decreases the probability of the negative emotion token ('bad'). ). In contrast, it simultane- ously increases the probability of the positive emo- tion token ('good"). CTG methods encourage the LM to generate more positive text and less nega- tive text. Because of these bias-driven probability calibrations, CTG methods affect the performance of the LM, including the degradation of the topic consistency. With a CTG method, a trade-off rela- tionship exists between toxicity avoidance and LM performance.
To resolve these problems, we propose a simple model-agnostic gating method that selectively ap- plies a CTG method called Gated Toxicity Avoid- ance (GTA). It preserves the generation quality and can be applied to any CTG method. Additionally, it improves the generation speed of seided-decoding based CTG methods. To the best of our knowledge, this paper is the first address the holistic perfor- mance degradation of the CTG method. Addition- ally, we validate the proposed gating method with various models on diverse datasets and empirically demonstrate that gating resolves the limitations of
2.3 Controllable Text Generation (CTG) for toxicity avoidance
Various CTG-based toxicity avoidance methods without additional inference costs have been pro- posed. For nontoxic text generation, model retrain- ing with auxiliary information is used, including
'tttps: / thub. wtt.hhb....o/eeggggii///TA..

---

control codes in CTRL (Keskar al., 2019) reinforcement learning from human feedback InstructGPT (Ouyang et al., 2022). To reduce the cost of retraining, prompt tuning methods have been proposed. DisCup (Zhang and Song, 2022) is a prompt tuning method using unlikelihood train- ing assisted by discriminator to generate nontoxic tokens without updating the LM. However, these methods still require inference by language model learn.
model learns conditional probabilities from large number of documents. The model selects the next token based on the conditional probabilities of the given sequences. The model appends the selected token and predicts the next token again with the updated sequences. Through this autoregressive process, the language model generates text with the following probability.
p(x,) = p(x)(Xes))
(I)
As the size of the LM increases, additional train- ing becomes more expensive; therefore, guided decoding method is proposed. The guided decod- ing method changes the output distribution of the LM to generate an intended text by adjusting it through a pretrained toxic discriminator. Dis- criminator training is much less expensive and easier apply than retraining or prompt tuning. PPLM (Dathathri et al., 2019) calculates the toxic- ity of the token generated by the model by attaching a diseriminator head to the last hidden state of the language model. After that, the last hidden state is updated to nontoxic using the gradient. However, this process is too slow.
3.2 Toxicity Avoidance
Toxicity Avoidance is method that prevents lan- guage models from generating toxic text. Toxic text is text that contains harmful, offensive, or abusive content.
Controllable Text Generation (CTG) focuses on generating coherent language while allowing con- trol over different aspects of the text, such as sen- timent, emotion, and category. With CTG, we can also control the LM to generate nontoxic sentences. If the condition of the token we want to generate is CTG can be defined as follows.
To reduce the calculation process, methods that directly adjust the output probability using the dis- criminator have been proposed. These methods are much faster in making inferences and do not re- quire language model in the discriminator train- ing phase. GeDi (Krause et al., 2020) learns the Cass-Conditional Language Model (CC-LM) and calculates the nontoxic token probability using the Bayes rule to guide the LM. FUDGE (Yang and Klein, 2021) evaluates the LM's candidate tokens and induces the selection of appropriate tokens. DExperts (Liu et al., 2021) subtracts the output of an Anti-expert and adds the output of an Expert to the output distribution of the LM.
p(x:) = -((xsc)
(2)
One well-known method is Guided Decod- ing (Ghazvininejad et al., 2017), which freezes the LM and only modifics the probability of a token being chosen with the discriminator an additional model to guide the language model. It induces the token of the desired condition to be selected from the modified probability distribution.
p(zile) = p(a:|Xei..)) c)
= =p(ai(==)d((Xea)
Another approach is Prompt Tuning (Lester et al., 2021). It also freezes the LM and only trains few prompt embeddings to control language model generation. Prompt tuning can be defined as fol- lows.
In this paper, we propose gated toxicity avoid- ance, a madel-agnostic method for CTG that can be utilized without tuning the LLM.
3 Preliminaries
p(aa(c) p(ax/ ci P)
(4)
In this section, we introduce the basic terms that will be used for the rest of the paper. With this nota- tion, we formally define the detoxification problem and two well-known methods.
= -pxx:ll;:&el
where P is traininable prompt embeddings and X is text embeddings.
Toxicity Avoidance is CTG method applied to a language model. Its performance is evaluated by the reduced toxicity and fluency of text gener- ates.
3.1
Text generation
The language model can generate text with the con- ditional probability of the next token. The language

---

sample
sample
disliiee
disike
disiie
pizza
you
LM
I&
Is it taxic?
Is it toxic?
Gate
CTG
(ours)
No,
Yes, apply CTG and sample again
sample
| sample
pizza
pizza
(a) LM
(b) CTG
(c) Gated Toxicity Avoidance (ours)
Figure 3: Operation of the LM, CTG method, and Gated Toxicity Avoidance (GTA)
4 Method: Gated Toxicity Avoidance
For the prompt-tuning-basee approaches (Zhang and Song, 2022), the gated toxicity avoidance is defined Poutte((3s()) p(x, p(=:///aaDXXX/)P;
To preserve LM performance and avoid toxic text generation, the CTG method must only operate when a toxic token generated. When training the CTG method, sequence-level labeled data are used; this method assumes that all tokens in sentence are either toxic not. This avoids not only toxic word generation but also the generation of text with a similar distribution toxic labeled data, and induces text to be generated with nontoxic labeled data. Because of this, CTG methods avoid topics, tones, and expressions that often appear in toxic data and degrade performance.
(6)
where P is set of traininable prompt embeddings and X is set of text embeddings.
Experimental Settings
5.1 Dataset
To analyze the impacts of topic distributions, we conduct experiments with various topic group datasets, including Sentiment, Emotion, and News. Sentiment is binary-topic (positive negative) dataset from texts expressing opinions about restau- rants. Emotion is six-topic (anger/ sadness/ / fear / surprise / joy love) dataset from English Twitter messages. News is a five-topic (tech sport en- tertainment politics / business) dataset from BBC news articles.
To minimize this influence, we propose gated toxicity avoidance that selectively applies the CTG method during the autoregressive generation pro- cess. The gate model g(x) determines the operation the CTG method, where g(x) the pretrained binary toxic classifier, g(x) estimates a toxic proba- bility of given text x and gives if the probability is greater than gate threshold 0 and 0 otherwise (For gate model, we adopted published toxicity clas- sifier²). When the token to be generated is toxic, the CTG method changes it to nontoxic token. This gated toxicity avoidance can be applied to any token-level CTG method. Fig. 3 compares the gen- eration processes of the LM, the CTG method, and our gated toxicity avoidance. Formally, the gated toxicity avoidance for approaches (Dathathri al., 2019; Krause et al., 2020; Yang and Klein, 2021; Liu et al., 2021) defined
5.2 Evaluation Metrics
Toxicity: is metric of the text's harmfulness. We used Perspective API which is widely used measure toxicity with values between 0 and The score represents probability that indicates how likely it is that reader would perceive the text as containing a toxic attribute (rude, disrespect- ful, or unreasonable). A higher score indicates that more readers are likely to feel the text is toxic. If this score was 0.5 or higher, it was classified toxic text. We measured how many of the gener- ated texts were toxic.
sudded-deeoding-basee
(5) where denotes the topic and X represents the generated text.
( = X
httpr////waa.e.atstst/s///ppllir hhttps://hugginggaceccoo http:///gggacccooooaattttt/aaaaa emotion http://www.aaglee.cc//c/eeessss "ntpps://perspectivep..co
'https: 'nttps://uugiingface https///aaa..ccceo////reeeeeeeee coxicity.classifier

---

Accuracy: is a metric of topic consistency. This indicates whether text matches given topic. To evaluate topic accuracy, we adopted publicly available classifiers (Table 15) that performed best on Sentiment, Emotion, and News, respectively. Each classifiers were fine-tuned RoBERTa (Liu et al., 2019), BERT (Devlin et al., 2018), and Dis- tiBERT (Sanh et al., 2019) on each dataset.
for each topic-generation LM. It was trained using the published PPLM repository code?.
GeDi (Krause al., 2020) calculates the nontoxic token probability with a Class- Conditional Language Model (CC-LM), We used the published 345M-scale toxicity CC- LM from the paper. The strength w, which controls the degree of toxicity, was set to 15 and 30. A larger forces the LM to generate less toxic text.
Grammar: is metric of the text's morphol- ogy and syntax quality. To evaluate grammar, we adopted published RoBERTa-base classi- fier (Krishna et al., 2020) fine-tuned with the CoLA (Warstadt et al., 2018) dataset. We use the av- erage probability value of the classifier as the score. We described details of accuracy and grammar clas- sifiers for automatic evalutation in Appendix F.1.
DExperts (Liu et al., 2021) adjusts the proba- bility with two output distributions (expert and anti-expert). We used the paper's published large-scale (762M) toxicity expert and anti- expert. The guiding strength a, which controls the degree toxicity, was set to 0.5 and 1.0. Like w in GeDi, a larger a forces the LM to generate less toxic text.
Perplexity (PPL): is used to evaluate the text's fluency. It represents how well an LM predicts a given sequence of words. A lower perplexity value indicates that the model is more confident and ac- curate. Perplexity was evaluated for the baseline LM that generated each text.
DisCup (Zhang and Song, 2022) is a state-of- the-art prompt tuning method. We used pub- lished prompt embeddings for toxicity avoid- ance. After the prompt tuning, GPT2-small showed low generation performance. Due this low generation performance, the exper- imental results of DisCup with GPT2-small were excluded.
5.3 LM and Baseline CTG methods
LM: is a model for controllable text generation; we used two sizes (small, 117 M, and large, 762 M) GPT2 (Radford et al., 2019). To compensate for the low generation performance of small, we fine-tuned the LM on the three topic group datasets. If topic is given as a prompt, related text is gen- erated. Table 20 in the appendix shows examples of prompts and generated texts.
Experimental Results
In this section, we will answer the following search questions:
In GPT--arge, we used in-context learning to gen- erate topic-matched texts. A different number of shots was applied for each topic. We used 10 shots for Sentiment, 30 for Emotion, and five for News. Only nontoxic samples were used for learning. In cases where there were many sentences in the sam- ple (especially in the Sentiment and News datasets), only the first three sentences were used.
RQI) Do CTG methods truly degrade perfor- mance?
RQ2) Can the gated toxicity avoidance reduce toxicity while preserving performance?
RQ3) Does the degree of performance degra- dation vary by topic?
We generated 1,000 sentences per topic in small LM and 100 sentences per topic in large LM. Also, we used Top-K sampling and Nucleus sam- pling (Holtzman et al., 2020) to generate diverse texts. See Appendix D for detailed generation pa- rameters.
RQ4) Does the problem still occur with large-scale model?
RQ5) Can the gated toxicity avoidance also reduce inference time?
6.1 RQ1: CTG Method Degradation Effeett
Baseline CTG methods
We observed that CTG methods reduce toxicity but degrade the various aspects of LM performance.
PPLM (Dathathri al., 2019) calculates toxicity with an attached discriminatory head.
phttps::/gittiiii com/uberrrresercch/Pa hhttps://bit... 1y/.//aaa...//
To use PPLM, we trained a classification head

---

Table I: LM's performance degrraaattio by CTG method
the lower the toxicity. However, greater strength parameter makes the other performance metries worse. See Appendix F.2 for the change in perfor- mance due to strength.
Topic Group
Method
Accuracy (1)
Tosicity ())
Grammar
PPL
2
76.25
88.22
4.72
PPLM
75.05
0.5
50.05
36.21
Sentiment
GeDi
74.35
0.05
82,09
6.39
DExperts
73,55
0.05
69.65
12.41
6.2 RQ2: Overall Performance of the Gated Toxicity Avoidance
GPT-2
62,98
88.31
8.27
PPLM
47.19
1.05
45.91
24.44
Emosica
GeDi
0.03
91.06
7.25
Disperts
28.18
55.88
Table 3 shows that our gated toxicity avoidance method resolves performance degradation. re- duces toxicity to the same level as the original (non- gated) CTG method and preserves topic accuracy, grammar, and PPL at the same level as baseline LM (GPT-2), regardless of the CTG method. In particular, in the case of PPLM and DExperts, the generation quality was significantly improved.
GPT-2
PPLM
18.93
34.95
News
GoDi
Dilaports
46.34
54.48
GPT-2
$4.57
PPLM
30.04
Avenge
GeDi
DExgent
62,93
In detail results for each topic, the gated toxicity avoidance shows a baseline (GPT-2) level of perfor- mance and the same level of toxicity as the original CTG regardless of the topic (see Appendix for de- tails). Since the toxicity reduction follows the per- formance of the original CTG method, GeDicrA shows the best toxicity reduction.
The effects of CTG methods are shown in Table 1. All three methods significantly reduce toxicity, and GeDi shows the best generation performance with the lowest toxicity. They not only reduce toxicity but also degrade other generation performance met- rics.
Cherry-picked generation samples are shown in Table 2. LM easily generates toxic text in an anger topic. The texts generated by PPLM are not fluent at all. DExperts generates too long and not natural text. GeDi's text quality is similar LM, but does not contain anger emotions.
Performance preservation depends on the gate threshold f, which is used for the gate model(toxic classifier). We used 0 = 0.005 as the best threshold. This value may vary depending (Appendix F.3). The gate model can significantly reduce toxicity when using low thresholds because the distribution of the output probabilities is highly biased.
the gate model
Specifically, performance degradation varies across the different topic groups and methods em- ployed. PPLM on News surprisingly improves the topic accuracy, but it exhibits repeated generation of particular keyword. This improvement comes at the cost of compromised grammar and perplexity scores.
6.3 RQ3: Effectiveness of Topics
To analyze the effectiveness of each topic, we vali- dated the performance of GeDi(see Fig 4), which shows the best performance on the Emotion topic group, which has the largest number of labels. Tox- icity varies for each topic. Anger (13.1%) and sad- ness (7.8%) show notably high toxicity, compared to 1.85% on average for all topics. The remaining topics show very low toxicity. High toxicity is re- lated the degradation of the CTG method. For the anger and sadness topics, GeDi demonstrates a more significant degradation in topic accuracy. GeDi enhances that grammar and perplexity scores because it supplements missing grammatical ele- ments in the original Emotion topic group, such as spaces, dots, and apostrophes.
DExperts perform the lowest in terms of topic accuracy, grammar, and perplexity. On average among the CTG methods, GeDi exhibits the lowest toxicity and best grammar, perplexity, and topic accuracy. Nevertheless, it is essential to note that even GeDi degrades performance depending on the specific topic (Emotion).
Notably, among the three topic groups, the most significant performance degradation is observed in the emotion category. This category is also the most toxic (3.8%). This is due to significant degrada- tion in negative emotions (sadness, anger), which we discussed details in Sec 6.3.
We selected the optimal parameter that can sig- nificantly reduce toxicity while minimizing genera- tion quality degradation. Each CTG method has its own strength parameter. The greater the strength,
For the Sentiment topic group, GeDi shows a 1.9 decrease in topic accuracy, but the grammar degrades significantly (-6.13). Both positive and negative topics decline similarly. For the tech topic

---

Table 2: These are cherry- -eeeryy-pckedd sample texts that LM and CTG methods generated for an anger topic. Red text indicates toxic text. Blue text indicates that the quality of the sentence has dropped sharply. Green text indicates text which not toxic.
Method
Abberyy-picked samples
Anger probability (%)
GPT-2
have idea what the lock going when leel the need to be stubborn
99.2
PPLM
im fooling kind- [o_: (L. (a to n: a. you o to a to beensrri
usle uilssistss
GeDi
foal i am nor soint
0.2
Dispers
masquitos spiders and bugs and the staff's do the most these fields gets large away and it kind of pilted back together in it out and i lty get
angry
planes yenell get
hugs
26.0
GoDpue
oen
Cack
00.5
00gh
egger
(a) Topic Accuracy (1)
(b) Toxicity (L)
(c) Grammar (1)
Figure Performance of CTG methods on emotion topies A: GeDi, GeDigateds gateds * baseline LM)
Table 3: Overall performance of gated toxicity avoid-
ance
baselins
Method
Accuracy
Toxieity ())
Grammar (P).())
GLD
J02
GPT-2
71.67
1.85
84.57
7.89
PPLM
65.86
0.58
43,86
30.04
PPLMorA
72.65
0.62
85.36
8.73
191
84.97
7.89
GeDioz
71.12
0.03
84.22
7.9
DExperts
DExperts
62.93
0.05
41.54
43.9
DExpertsora
70.96
0.05
84.33
7.91
DExpe Eemme
192
1,407
in News, GeDi shows 22.9 decrease in topic ac- curacy. There is no significant change in the other topics. Changes in topic accuracy do not correlate with changes grammar and perplexity. It means that degradation text quality does not cause change topic accuracy. We posit that this differ- ence in performance due to the training data bias the CTG method.
PMMM
1200
Figure S: Time generate 100 texts 100 tokens in length
kens. For fair evaluation, both GeDi and DExperts used the same size of CC-LM and Experts. Fig. 5 shows that gated toxicity avoidance reduces the generation time regardless of the CTG method.
6.4 RQ4: Efficiency of the Gated Toxicity Avoidance
We validate the efficiency of our proposed gated toxicity avoidance. The gated toxicity avoidance not only preserves LM performance but also gen- erates text more efficiently. This makes the LM generate text faster than the original CTG method, To evaluate the efficiency, we averaged 100 text generation costs, where each generation cost was measured as the time took to generate 100 to-
The gated toxicity avoidance is infrequently re- liant upon the CTG method, factor contributing its efficiency. In contrast, guided-decoding based CTG methods invoke its mechanisms each it- eration of token generation. Although the gated toxicity avoidance also invokes a gate model at each iteration, a compact gate model is more cost-

---

effective. This performance improvement will be more remarkable practice because the CTG will be many times larger than the gate model.
GeDi's performance degradation differs from that in the small-scale LM. There is a significant topic accuracy degradation the anger, positive and negative topics, and there is no noticeable change in the other topics. The average grammar and perplexity are also degraded. The joy and sad- ness topics affect GeDi's grammar degradation, and perplexity increases in anger, negative, and sadness topies.
Table 4: Performance of CTG methods varying scale
params
Accuracy
Toxicity
(sec. J)
GPT-2
117M
71.67
1.85
99
Disperts-smull
351M
58.08
0.28
297
DEsprii-smaallra
461M
71.55
0.11
192
Dixpers--ugge
1,528
02.93
0,05
960
DEspertslageeve
1,638
70.96
0,05
210
Even though DisCup shows better topic accu- racy than GeDi, it still shows lower topic accuracy than the LM because of topic degradation in News and Sentiment topic groups. However, DisCupCT mitigate topic degradation. DisCup shows better toxicity, grammar, and perplexity performance in sentiment and news topic groups, but the gaps are marginal in humans (RQ6). You can more de- tailed results Appendix F.4.
We evaluated the performance according to the scale of CTG method, After training Experts and Anti-Experts of 117M parameters, we generated 100 texts 100 tokens in length with DExperts and evaluated them. The number of parameters of the gate model was 110M. See Appendix G for the experimental details.
Table 4 shows that small DExperts use less memory and generate faster. However, it has a higher toxicity than the DExperts-lagee, and the performance degradation is more considerable. The DEXpertScI ensures a level of toxicity similar to or better than the original DExperts and a level of generation quality similar to the baseline LM. One interesting thing about the DExperts TExperts-smallerAA is that it has lower toxicity than the original DExperts- small. This is presumed be due to the gated model resampling tokens, even the small-scale CTG method performance is low. the scall-scale CTG method has sufficiently low toxicity, the gated toxicity avoidance can be used to address the re- maining degradation. For the performance preserva- tion, using gated toxicity avoidance with small- scale CTG method is more effective in memory and speed than using only a larger CTG method.
6.6 RQ6: Human Evaluation
We performed human evaluation regarding three metrics that are commonly used in the previous studies (Gu et al., 2022; Zhang and Song, 2022): topic accuracy, toxicity, and fluency. Topic accu- racy and toxicity are defined in the same way the previous automatic metrics. The evaluators read the text generated for the given topic by the LM and assigned (match) or (not a match) according to whether topic-matee matched text was generated. Toxic- ity was also evaluated as 1 (toxic) or 0 (not toxic). Fluency is a metric that evaluates how natural and fluent the text is. It was rated on a scale of (not fluent at all) to 5 (very fluent). To reduce the bias of the absolute evaluation, users were asked to review all samples before evaluation. See Appendix H for evaluation details.
6.5 RQS: CTG Method with a Large-Scale LM
Table 6: Human evaluation result on large-scale LM
Method
Accuracy (?)
Toxicity (1)
Fluency (f)
Table 5: Automatic evaluation result on large-scale LM
GPT-2
76.15
4.23
3.62
GeDi
65.77
1.15
3,73
Method
Accuracy (?)
Toxicity (1)
Grammar (t)
PPL. (1)
GeDiana
75.77
1.15
3.57
GPT-2
69.26
1.00
83.90
55.63
DisCup
1.54
3.56
GeDi
DisCuporA
76.54
2.69
3.69
GeDigrA
69.38
0.08
82,77
56.29
DisCup
DisCupers
70.55
0.39
79.68
57.44
Compared to large-scale LM's automatic evalua- tion, human evaluation results of large-scale LM in Table 6 show greater degradations in topic accuracy. GeDi degrades significantly, and DisCup shows bet- ter accuracy than GeDi but still lower than the LM. However, the gated toxicity avoidance shows a similar level of topic accuracy to the LMM. GeDi
In this section, we demonstrated that the large- scale LM with CTG method reduces toxicity and degrades performance just as small-scale LM does. We also show that the gated toxicity avoidance works well at a large-scale LM.

---

shows the lowest toxicity, and GeDigr performs similarly. On the other hand, DisCup shows slightly higher toxicity than GeDi. And DisCUpGTA has a higher toxicity than DisCup. Fluency is different for automatic evaluation. GeDi exhibits high flu- ency in the emotion topic group, similar to the small LM, but there is no noticeable difference among the methods on average. From the supple- ments of grammatical elements (commas, commas, quotation marks, etc.), GeDi shows slightly better than LM. Unlike automatic evaluation, DisCupor performs better than DisCup in fluency.
varied LM performance metrics crucial. Although the probability of generating toxic text using our proposed method extremely low, it is not entirely zero. In future works, our key purposes are eliminating toxic generation completely and preserving original performance.
Ethics Statement
Controllable Text Generation (CTG) possesses the potential for misuse for non-ethical purposes. Nev- ertheless, the toxicity avoidance process using CTG is essential for the ethical utilization of LMs, and it stands as the most effective method currently available. We believe this study will make out- standing contribution to applying ethical LM.
7 Conclusion
In this paper, we explored the effectiveness of state- of-the-art CTG methods, experimenting with vari- ous aspects topic consistency, grammar, and per- plexity. Furthermore, we proposed novel model- agnostic solution called the Gated Toxicity Avoid- ance. Our findings revealed that previous CTG methods exhibit varying degrees of performance degradation across different topics, CTG methods, and scales. Regardless of these factors, the pro- posed gated toxicity avoidance successfully pre- serves the original language model's performance while achieving comparable toxicity reduction lev- els. Notably, the gated toxicity avoidance also demonstrates faster generation speed. Their results highlight the potential of gated toxicity avoidance as an effective and efficient solution for the safety language model.
A Acknowledgements
This work was supported by Institute of Infor- mation & communications Technology Planning & Evaluation (IITP) grant funded by the Korea governmen((MSIT) (No.2022-0-00688) Abductive inference framework using omni-data for under- standing complex causal relations) and (IITP-2023- No. 20..S20022-0025566, Artificial Intelligence Con- vergence Innovation Human Resources Develop- ment)
We received support from the Google TPU Re- search Cloud to train the models required for this study. To make Fig 1, 2, and 3, we used commer- cially available public icons. Robot icons are created by itim2101 Flaticon. Gate icon and Bearded male icon is create by Freepik Flati- con.
Limitations
One limitation of this paper is related to the scale the language models. Recent advancements have led to the developing of bfatc-of-ther language models with billions of parameters. However, due to the high computational costs of generating nu- merous samples using L.M,, we restricted from using LM containing up to 762M parameters. In our future work, it is crucial explore the issues we found also occurred in much larger LMs, as well as explore whether the gated toxicity avoidance can resolve them.
Furthermore, it is essential to consider additional indicators for evaluating the impact of CTG meth- ods. Various metrics can be used to evaluate differ- writing
htttps:/www.ffatiicc. fwaticon.ccocooooo
41 htt.s/rrelatedddd4153550000
ent aspects, such cnttrutiionfollowwiin style (e.g. written or spoken, dialect), religious in- fluence, racial sensitivity, and other topics. Devel- oping effective methods for measuring wide and
41 4133277reelaeed_dd441333117
11
"hhttpss:/wwwffaaticcm http://iiiocccccoeeeeeioooo/a//a
78622115reelated_.d77777777

---

References
Long Ouyang, Jeff Wu, Xu Jiang. Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language mod- cls: A simple approach to controlled text generation. arXiv preprint 191iv:9112.21111
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-trainn Pre-trainnn of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv: 1810:00810004800..
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):
Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. Hafez: an interactive poetry gen- eration system. In Proceedings of ACI. 2017, System Demonstrations, pages 43-48, Vancouver, Canada. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1910.01 108.
Yuxuan Gu, Xiaocheng Feng, Sicheng Ma, Jiaming Wu, Heng Gong, and Bing Qin. 2022. Improving control- lable text generation with position-award weighted decoding. In Findings of the Association for Compu- tational Linguistics: ACL 2022. pages 3449-3467.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. CARER: Con- textualized affect representations for emotion recog- nition. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing. pages 3687-3697, Brussels, Belgium. Association for Computational Linguisties.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration.
Hugo Touvron, Louis Martin, Kevin Stone, Peter AL- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09286 009288....8..
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for control- lable generation.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2020. Gedi: Generative discriminator guided sequence generation. arXiv preprint (tt::2009006666
Alex Warstadt, Amanpreet Singh, and Samuel R Bow- man. 2018. Neural network acceptability judgments. arXiv preprint arXiv: 1207.22477.
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hen- dricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. 2021. Challenges in detoxifying language models. arXiv preprint wxxiv:2209.044400
Kalpesh Krishna, John Wieting, and Mohit Tyyer. 2020. Reformulating unsupervised style transfer as para- phrase generation.
Brian Lester, Rami Al-Rfou, and Noah Constant. 1021. The power of scale for parameter-efficient prompt tuning.
Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Guru- rangan, Maarten Sap, and Dan Klein. 2021. Detoxi- fying language models risks marginalizing minority voices.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. 2021. Dexperts: Decoding-tine time con- trolled text generation with experts and anti-experts. arXiv preprint 2rXiv:2105.03001
Kevin Yang and Dan Klein. 2021. FUDGE: Controlled text generation with future discriminators. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. Associ- ation for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 100.vvi:90..... 1907.11999.9
Yufeng. 2018 Bbc articles fulltext and category.
Hanqing Zhang and Dawei Song. 2022. Discup: Dis- criminator cooperative unlikelihood prompt-tuning for controllable text generation.
Varvara Logacheva, Daryna Dementieva, Sergey Ustyantsev, Daniil Moskovskiy. David Dale, Irina Krotova, Nikita Semenov, and Alexander Panchenko. 2022. Paradetox: Detoxification with parallel data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level Convolutional Networks for Text Classification. arXiv:1509.01620 [es].
1: Long Papers), pages 6804-681 6818.

---

A Toxicity of RLHF model: Llama2
D Generation Details
D.1 Small-scale Topic Generation LM Details
Table 7: Performance of the Llama-2-7b-chat-hh emo-
tion topic
Hyperparameters for training and generation are shown in Table 10 and 11. Generation examples are shown in Table 20.
Topic
Accuracy (f)
Toxicity (1)
anger
57.7
2.2
D.2 Large-scale In-context Learning Details
fear
58.5
1.1
Hyperparameters for training and generations are shown in Table 12. The in-context learning format is shown in Table 21.
joy
88.0
0.2
love
13.3
0.1
sadness
64.5
0.2
CTG Details
surprise
20.5
0.2
E.1 PPLM
average
50.42
0.67
We used balanced JigSaw toxic comment classifi- cation data to train the classification head. The classification head is single MLP layer. Table 13 shows the hyperparameters to train the PPLM clas- sification head.
To evaluate the toxicity of the Reinforcement Learning from Human Feedback (RLHF), we em- ployed the Llama-2-7b-chat- model, which is published state-of-the-art model. Using Llama-2 with prompts described in Table 8, we generated 1,000 texts for each emotion topic. In Table 7, we observed that Llama-2 with RLHF also still has higher toxicity than CTG methods (e.g., GeDi) in the generated text. It is noteworthy that, despite Llama-2 (7B)'s substantial size relative smaller models such as GPT2-small (125M) and GPT2- large (775M), GeDi, DExperts, DisCup exhibit notably lower toxicity compared to Llama-2-7b- chathf.
E.2 DExperts
We trained two experts (a toxic expert and toxic anti-expert) to evaluate the efficiency of the small- scale DExperts experiment. training dataset is the same as PPLM. Table shows the hyperpa- rameters to train an expert and an anti-expert.
F Automatic Evaluation Details
The evaluation results for each topic are in the last sections. Small-scale LM results in Appendix I, and large-scale LM results are in Appendix J. The actual generated output exists in the 'output/" direc- tory of the repository.
B Performance degradation of the Text Style Transfer method: ParaDetox
We employed ParaDetox (Logacheva et al., 2022), the state-of-the-art text style transfer method, to evaluate the performance degradation of the text style transfer method. Utilizing toxicity classifier, which used gate model, we elassify toxic texts from the emotion dataset and transfer them nontoxic text. We classified the topic of trans- ferred texts. In Table 9, we observed that the topic preservation of ParaDetox depends on the topic and notably degraded on the anger topic.
F.1 Classifiers
Table shows which classifiers are used evalu- ate the metrics.
F.2 Small-scale evaluation result according to CTG method strength
In Table 16, we observed the LM performance ac- cording to the CTG method strength. It can be seen that as the strength increases, the toxicity reduces, but most of the other performance metrics also degrade. In the case of DExperts, topic accuracy improved when did not work properly when using larger strength.
Hardware Details
Topic generation LMs trained from TPU-v2-8. The other tasks, training PPLM classification heads and text generation were conducted with NVIDIA RTX 3060.
large (1.0). Text generation
htt httt::://wwwwkket ctteeccoo///eeeeeeeeeoe com//ompptiiions jigsaw- toxic- mox.cccomment-.....mooooommmm comment- classi challengae overview
13 htt htt////uggggaacccc//c acee. ctt.aaae.....aaaaaaaaaaaaaaa co/neta-1i 11ama/ Llama- 2-7 7--7-bhhtttt chat-hf

---

Table 8: Prompt to generate comments using meta- moctaa-llammm/laaa222-----hh
Prompt

 Write tweet where you feel like the author is in (anger/ fear / joy / love sadness / surprise). 

Table 9: Topic accuracy after detoxification using Pa- radetox
Table 12: Hyperparameters for generate text in large- scale experiment
Topic
Accuracy (f)
# of samples
Hyperparameter
Value
Top-K
50
anger 57.17
146
Top-P
1.0
fear 84.85
56
min neen_wew__ae Jokens
5
Common
joy 96.36
106
max_new_tokent
32 (sentiment, emotion), 128 (news)
temporature
love
90.91
50
repetition pppeitioo_peeallty
no
sadness
89.79
255
surprise
100.00
16
Table 13: Hyperparameters for training PPLM classifi-
cation head
Table 10: Hyperparameters training small-scale Topic Generation LM
Hy Hyperparameter
Value
epochs
10
batch size
Hyperparameter
Value
epochs batch size
10
learning rate
le-4
64 (emotion, sentiment). 8 (news)
learning rate
5e-5
learning rate decay
lincar (min 5c-6)
Table 14: Hyperparameters for training small-scale DEx- perts
sequence length
64 (emotion), 256 (sentiment), 400 (news)
Hy Hyperparameter
Value
Table II: Hyperparameters for generate text in small-
scale experiment
epochs batch size
3
64
learning rate
5e-5
Method
Hyperparamcter
Valuue
learning decay
linear (min 5e-6)
Top-K
max seqence length
128
Top-P
0.9
tokens
Commonn
max new tokens
(emation), (sentiment), (news)
lemperature
F.3 Small-scale Evaluation Result According to Gate Threshold
repetition_ee spenllttyyy
step size
0.02
lusion-gm-scale
In Table 17, observed the performance change of GeDi according the gate threshold through ex- periments. The lower the gate threshold, the lower the toxicity continued to decrease, but there was no significant difference in the rest of the accuracy, grammar, and perplexity. When the gate threshold (0) was used as 0.005, it showed the same toxicity as GeDi. Based on this, we used the same 0.005 threshold for other CTG methods.
num- -iteratioons length gamma activesize
10,000
PPLM
0.01
decay
FALSE
borizon borizon_Jength
window length
filter_p
target_ P
0.8
GeDi
class, bias
dise weight (2)
15.
logits_scale
10
DExperts
alpha
F.4 Automatic Evaluation Results of the Large-scale LM with CTG methods
DisCup
ranking scope (Tap-K)
10
Table 18 shows automatic evaluation results large-scale LM experiments for each topic group.

---

Table 15: Published classifier to evaluate automatic evaluation metrics
Metric
Model
Fine-tuned Dataset
Sentiment
https://hugginge http:/cc.../////ieeeeeee So/Victorsatth/rorrtt-----ss
hatarpoceeereaooeeeeeee-----aaar polarity
yelp-polarity (Zhang 2015)
Emotion
itt co/bhadreehh--aaaiii/
emotion (Saravia et al., 2018)
News
https://huggingface stwp::/dddtttteeeeeeeeeee--e-ee-
bbc-news (Yufeng. 2018)
Grammar
attps://hugginggaae
http:///wgggggnee....oo//ottttt/ wtt../ooinneetteeeerrrrreeeeeeee cola krisshaz222
CoLA (Warstadt 2018)
Table 16: Change of performance of GeDi, DExperts according to strength (o, w). The greater the value, the stronger the strength
bascline LM, 2) GeDi, 3) DisCup, 4) GeDiGT 5)
DisCupoT Each method generates 100 texts per topic. The evaluator compared randomly sampled two of them. Evaluators evaluated total of 130 texts five methods, 13 subjects, and two samples per subject. The evaluation took 45 minutes on aver- age. Our detailed results are shown in Table 19 and Fig. shows the web UI where human evaluation has been performed.
Method
Accuracy
Toxicity ())
Grammar
PPL
G
71.67
1.85
84.57
8.37
DExperisto 0.50
61.78
0.48
42.88
49.6
DExperts(a 1.0)
GeDiw 15)
69.11
0.04
84.72
8.50
Geli(u 30)
68.21
0.03
84,97
8.44
Table 17: Change of performance of GeDigated accord- ing to gate threshold
Table 19: Average human evaluation results by topic group
Accuracy (?)
Toxicity ())
Grammar (f)
PPL (L)
Topic Group
Method
Accuracy
Toxicity ())
Fiuency (f)
0.5
71.66
0.36
84.07
7.91
GPT-2
85.00
5.00
3.72
0.1
71.85
0.26
84.09
7.91
0.01
71.69
0.15
84.41
7.82
Sentiment
GeDicra
77.50
5.00
3.72
0.005
71.12
0.03
84.22
7.90
DisClup
75.00
0.00
3.50
DisClipera
80.00
7.50
3.98
Table 18: Automatic evaluation result of the large-scale LM with CTG method for each topic groups
GPT-2
64.17
5.83
3.84
Emetion
GeDiora
65.83
0.83
3.73
Topic Group
Method
Accuracy
Toxicily
DisCupp
60.00
3.66
DisCapora
70.00
1.67
4.01
GPT-2
87.00
2,00
3,31
Sentiment
GeDiera
0.0
87.04
GeDi
90.00
1.00
3,50
DipOap
79.29
0.01
91.01
18.42
News
GeDicra
87.00
0.00
3.30
DisCupazA
80.5
0.01
$6.05
25:44
DisClup
82.00
3.00
3,47
GPT-3
52.67
0.01
87.4
102.95
GeDi
47.83
0.9
82.01
185.92
DisCapora
83.00
2.00
3.19
Emotion
GeDigra
53.33
0.0
85.64
109
DisCup
54.18
0.01
78.76
101.73
DisCipor
58.46
0.9
80.12
114.29
GPT-2
81.0
0.0
77.77
37.57
GeDi
82,6
77.84
36.89
News
GeDicra
77.63
DisOap
DisCuporA
G Efficiency Experiment Details
In the generation speed experiments of GeDi and DExperts, we used the same size CC-LM and ex- perts for a fair comparison. We used 117M ex- pert and anti-expert as GeDi's CC-LM instead of GeDi's original 345M CC-LM.
H Human Evaluation Details
We evaluated the texts with 10 English experts.
We used a total of five methods 1) a large-scale

---

Table 20: Examples generated by fine-tuned small-scale LM
Prompt
Generated Example
topic: posstive
try
amazing.
topic: anger
1 Loel so angry with all of them
improve welfare be given the chance every year by 2020 the latest in series of measures designed to give states greater level welfare in return for their suppart.
federal ministers
El
topic: politics
these messures have been announced at the autumn meeting of the uk devolved administrationss their aim incrense welfare 2.4bn including
badger ministers also unnounced that the nexl generation of wellare would be made by 2021. public-secior workers would have
cubbic-employee contmets delivered by 2020 govemment
Prompt 8
anger: having an anxiety attack
Q1. Dors the text given match the presented lopic 'anger?
no
yes
given content?
not toxic
toxic
Q3. How Rluont the givon heut?1 like real sentonces, give points vory fluent, point not at all.
Figure 6: Human evaluation web UI which was developed using streamlit
Table 21: In-context learning prompt examples for generating topic-related text in large-scale experiment. The first line instruction, where the bold text means the topic be generated. shots seperated with '==='.
Topic Group
# shots
Prompt
sentiment
10
These are positive reviews.
place is good. prefer to buy my DVDs
Great little gem! The food was fantastic...
emotion
30
These text containing feelings of surprise.
feel strange sensation through my limbs
help feeling curious you know after ive hcard
news
5
These are news articles of entertainment topics
dj double act revamp chart show dj duo jk and joel are taking over bbc radio
tautou film tops cesar prize nods french film

---

I Automatic Evaluation Results For Each Topics in the Small-scale LM
Table 24:
Automatic evaluation result of the small-scale LM for sadness topic
Method
Accuracy (1)
Toxicity ())
Grummar (1)
PPL (1)
GPT-2
75,00
7.00
89.00
7.75
PPI.MTTA
74.00
1.00
89.00
7.85
GeDi
$7.00
0.00
92.00
6.84
GeDicra
DExpert
60.00
0.00
23.00
53.77
DEspetiizza
76.00
0.00
90.00
7.67
Table 25: Automatic evaluation result of the small-scale LM for joy topic
Accuracy (?)
Toxielty
(?)
PPI.(L)
GPT-2
84.00
1.00
89.00
8.01
PPLM
87.00
0.00
52,00
21.51
GeDi
86.00
0.00
92.00
7.01
Table 22: Automatic evaluation result of the small-scale LM for negative topic
GeDigr
83.00
0.00
$8.00
8.10
DExperts
76.00
0.00
28.00
58.36
DEspertionn
Method
Accuracy (T)
Tosicity (1)
Grammar (T)
PPL
PPLM
71.00
1.00
47.00
41.64
Table 26: Automatic evaluation result of the small-scale LM for love topic
PPI MeT
68.00
0.00
90.00
8.15
GeDi
11.00
GeDieTA
Accuracy
Toxielty
PPL.())
Disppet
62.00
0.00
69.00
11.94
GPT-2
31.00
1.00
$7.00
8.57
DExpeticzaa
64.00
0.00
$7.00
4.72
PPLM
35,00
1.00
49.00
21.81
32,00
GeDi
23.00
0.00
90.00
8.11
GeDigr
35.00
0.00
86.00
8.58
DExperts
30,00
0.00
28.00
55.39
DExpertiorn
Table Automatic evaluation result of the small-scale LM for anger topic
Medhod
Accurncy (?)
Toxieity (1)
Grammar (?)
PPL.
GPT-2
59.00
11.00
$9.00
8.37
PPLMoTA
61,00
4.00
$7.00
8.65
GeDi
44.00
0.00
91.00
7.21
GeDigr
60,00
0.00
83.00
8.48
Table 23: Automatic evaluation result of the small-scale LM for positive topic
DExperts
DEspanscra
$9.00
0.00
88.00
8.34
Method
Accuracy (7)
Tuxicity (1)
Grummur (?)
PPL (L)
GPT2
86.00
0.00
89.00
4.77
Table 28: Automatic evaluation result of the small-scale LM for fear topic
PPLM
PPLMarA
90.00
0.00
92.00
7.58
GeDi
84.00
0.00
$4.00
6.48
Method
Accuracy
Toxicity
Grammar
PPL (1)
GeDiera
DEspert
85.00
0.00
71.00
12.1
DExpetsoga
85.00
0.00
$9.00
4.73
PPLM
54.00
0.00
49.00
24.64
PPI.MorT
71.00
1.00)
89.00
8.44
GeDi
68.00
0.00
94.00
6.70
GeDiera
DExperts
65.00
0.00
28.00
55
DEsprticcna
72,00
0.00
89.00
8.22

---

Table 29; Automatic evaluation result of the small-scale LM for surprise topic
Table 34; Automatic evaluation result of the small-scale LM for tech topic
Method
Accuracy (7)
Tuxicity (1)
Grummur (?)
PPL (1)
Method
Accuracy (1)
Toxicity ())
Grummar (1)
PPL (1)
GPT2
59,00
1.00
85.00
8.99
GPT-2
63,00
0.00
$2.00
9.13
PPI M
61.00
1.00
86.00
9.34
PPI.MTTA
62.00
0.00
84.00
9.26
GeDi
57.00
0.00
$8.00
7.73
GcDi
60.00
0.00
82.00
9.62
GeDicri
GeDicra
DEsperts
36.00
0.00
28,00
57.51
DExperts
68.00
0.00
53,00
49.82
DExpertiora
56,00
0.00
86.00
9.15
DEspettizzaa
64.00
0.00
82.00
9.14
Table 30: Automatic ovaluation result of the small-scale LM for business topic
Automatic Evaluation Results For Each Topics in the Large-scale LM
Method
Accuracy (?)
Tuxicity (1)
Grummur (?)
PPL (L)
GPT-2
Table 35: Automatic evaluation result the large-scale LM for negative topic
PPLM
0.00
41,00
28.72
PPLMarA
78,00
0.00
$1.00
8.89
GeDi
73,00
0.00
80.00
9.22
Method
Aceuracy (1)
Toxieity (L)
Grammar (?)
PPL.
GeDicra
GPT-2
DExperts
64.00
0.00
50.00
47.36
DExpertsoraa
74,00
0.00
79.00
8.60
GeDi
41
0
59.91
GeDiora
77
0
84
25.73
DisCup
Table 31: Automatic evaluation result of the small-scale LM for entertainment topic
DisCupera
66
2
84
26.76
Method
Accuracy (?)
Toxicity (1)
Grammar (*)
PPI,
Table 36: Automatic evaluation result the large-scale LM for positive topic
GPT-2
PPLM
90.00
0.00
33.00
39.18
PPLMarA
$4.00
0.00
77.00
9.99
Method
Accuracy (T)
Toxicity (1)
Grammar (?)
PPL (1)
GeDi
86,00
0.00
77.00
9.83
GPT-2
GeDier
GeDi
93
20.83
DExperts
77.00
0.00
41.00
64.93
GeDigra
94
0
90
19.97
DExpenseraa
82.00
0.00
77.00
9.66
DisClp
DisCuperr4
95
0
88
24.17
Table 32: Automatic evaluation result of the small-scale LM for politics topic
Table 37: Automatic evaluation result large-scale LM for sadness topic
Method
Accuracy (?)
Toxieity (?)
Grammar (?)
PPI.
PPLM
66.00
0.00
43.00
33.16
Method
Accuracy (?)
Toxieity (1)
Grammar (?)
PPL (1)
PPLMETA
83.00
0.00
$1.00
9.15
GPT-2
60
111.8
Gelli
83.00
79.00
9.52
GeDi
64
224.26
GeDicr4
GeDign
61
0
85
109.16
DEsperts
64.00
0.00
50.00
$1.41
DisCup
DExpertccr.a
82,00
0.00
79.00
9.11
DisCupezra
55
102.9
Table 33: Automatic evaluation result of the small-scale LM for sport topic
Table 38: Automatic evaluation result the large-scale
LM for joy topic
Method
Accuracy (1)
Tuxicity (1)
Grummur (?)
PPL (L)
Method
Accuracy (?)
Toxieity (L)
Grammar (?)
PPI.
PPLM
92.00
0.00
33.00
48.58
GPT-2
83
94
69.4
PPL.MTT
97.00
0.00
76.00
9.43
GeDi
81
132.39
Gelli
76.00
GeDigra
86
0
95
70.48
GeDicr4
96.00
75.00
DisCup
DEsperts
88.00
0.00
37.00
60.96
DisCuporra
95
0
72.99
DExpetsor
96.00
0.00
74.00
9.25

---

Table 39: Automatic evaluation result of the large-scale LM for love topic
Method
Accuracy (T)
Toxicity (1)
Grammar (1)
PPL (1)
GPT-2
132.5
GeDi
85
149.24
Table 45: Automatic evaluation result of the large-scale
GeDier4
80
159.47
LM polities topic
DisCup
DisOuperd
144.2
Method
Accuracy
Toxielty
Grammar
PPL.())
Table 40: Automatic evaluation result of the large-scale LM for anger topic
GPT-2
84
0
25.45
GeDi
89
0
83
27.35
GeDiora
24.02
DisCup
90
0
17.61
Accuracy (t) Toxicily
DisCuperaa
86
0
26.78
GPT22
53
91
95.65
GeDi
89
369.73
GoDicTa
DisCup
54
76
113.85
DisCupera
Table 41: Automatic evaluation result of the large-scale LM for fear topic
Method
Accuracy (1)
Toxicity (L)
Grammar (1)
PPL (1)
GeDi
80
64.19
GeDierA
90
91.83
Table 46: Automatic evaluation result the large-scale LM for sport topic
DisCup
DioOupera
78
119.00
Method
Accuracy (1)
Toxicity (1)
Grammar (?)
PPL (1)
Table 42: Automatic evaluation result of the large-scale
GPT-2
LM for surprise topic
GeDi
86
75.44
GeDicrA
90
0
77.94
Accuracy (1) Toxicily
(1)
DisCup
GPT-2
38
76
131.57
DisCuperA
92
0
73
69.63
GeDi
75
136.64
GeDieTA
76
135.27
DisCup
35
0
67
100.52
DisCupers
117.87
Table 43: Automatic evaluation result of the large-scale
LM for business topic
Method
Accuracy (f)
Toxicity (L)
Grammar (f)
PPL (1)
GeDi
82
22.4
GeDiera
87
84
23.32
DisCup
Table 47: Automatic evaluation result of the large-scale LM for tech topic
DisCipera
76
79
21.32
Table 44: Automatic evaluation result of the large-scale
Method
Accuracy (1)
Toxicity (1)
Grammar (?)
PPL (1)
LM for entertainment topic
GPT-2
55
0
60.71
62.28
Avcuracy (1)
Toxicity
(1)
GeDigra
53
0
74
59.16
GPT-2
93
80
27.62
DisCup
45
0
33.58
GeDi
79
23.75
DisCipeTT
52.81
GeDierA
78
25.53
DisCup
99
80
14.48
DisCuperi
75
24,99


Reviews:
[{"review_id": "IHFUhm5u28", "reviewer": "Reviewer_4a3Y", "summary": "", "questions": "1. Is there any particular reason that you did not include the SOTA detoxifier, ParaDetox?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful review.\nHere are the comments on your review.\n\n\n1 & 2) Novelty: \n\nThe novelty of our work is an analysis of existing detoxifiers - detoxifiers decrease toxicity and also decrease LM performance - and our gated detoxifier can resolve the problems. With the marginal toxicity sacrifice (0.04 in PPLM, 0.00 in GeDi and DExperts), the gated detoxifier gains LM generation performance (topic preserving, grammar, and perplexity).\n\n\n3\\) Confusing term `detoxifier`: \n\nOur term `detoxifier` is used for `guided decoder` of LM for non-toxic text generation. We picked a detoxifier from the previous detoxification studies [1,2,3], but we agree that the term is confusing. We will clarify our task and change `detoxifier` to `guided decoder` for the camera-ready version.\n\n \n\nQ. Is there any particular reason that you did not include the SOTA detoxifier, ParaDetox?\n\nA. The proposed gated detoxifier is the guided decoder which can be applied to the CTG method, whereas its applicability to the text style transfer task is limited. Instead, we selected the state-of-the-art CTG method, DisCup[3], proposed in 2022. However, through a straightforward experiment, we also observed ParaDetox encountering challenges in text style transfer. Leveraging a toxic classifier, we curated a subset of 628 toxic samples from the emotion dataset. Then we attempted to transform these samples into non-toxic using ParaDetox. We evaluate the detoxified texts using the emotion classifier. We were able to observe variations in topic accuracy following detoxification, depending on the topic. The performance degradation (avg. -13.49\\%) also occurred in the ParaDetox. We used emotion and toxicity classifier which also used in the paper. \n\nThe obtained results are as follows:\n- anger: 100.0 -> 57.17\\% accuracy, 146 samples\n- fear: 100.0 -> 84.85\\% accuracy, 56 samples\n- joy: 100.0 -> 96.36\\% accuracy, 106 samples\n- love: 100.0 -> 90.91\\% accuracy, 50 samples\n- sadness: 100.0 -> 89.79\\% accuracy, 255 samples\n- surprise: 100.0 -> 100\\% accuracy, 16 samples\n\n\nThank you for pointing out typo errors. We will conduct multiple proof-reading before submitting the final version.\n\n[1] Krause, Ben, et al. ``Gedi: Generative discriminator guided sequence generation.`` arXiv preprint arXiv:2009.06367 (2020).\n\n[2] Gu, Yuxuan, et al. ``Improving controllable text generation with position-aware weighted decoding.`` Findings of the Association for Computational Linguistics: ACL 2022. 2022.\n\n[3] Zhang, Hanqing, and Dawei Song. ``DisCup: Discriminator cooperative unlikelihood prompt-tuning for controllable text generation.`` arXiv preprint arXiv:2210.09551 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work explores the idea of improving the text quality of the detoxifiers, which is a motivating research topic.\n2. The authors propose a gated detoxifier fused with guided decoding method to preserve the text quality while reducing the toxicity.\n3. Different evaluation process shows the improvement of text quality in terms of topic preservation, grammar, and fluency.", "reasons_to_reject": "1. The novelty of this work is bit limited. This works seems to incorporate guided generation with controllable non toxic text generation. It would be encouraging to see the model's improvement in accuracy, grammar, and toxicity metrics. But the evaluation of different baselines shows that the proposed method sometimes sacrifices on toxicity metric (Table 3, 5, 6) to generate more topic-preserving text.\n2. The topmost priority of this line of work should be reducing toxicity. However, the experimental results are not convincing enough to conclude that the proposed method is reliable to reduce the toxicity.\n3. The authors use the term 'detoxifier', which is a bit confusing, because prior works also use the term 'detoxification' for toxic-to-non toxic style transfer [1]. This line of research is different from open-domain non toxic text generation. Although Figure 1 gives the sense that this is a style transfer work (preserve the meaning of the source text), the evaluation strategy and the rest of the examples give the opposite idea. Please clarify on this.\n\nReferences:\n\n[1] Logacheva et al., ParaDetox: Detoxification with Parallel Data, ACL 2022.", "questions_to_authors": "1. Is there any particular reason that you did not include the SOTA detoxifier, ParaDetox?", "paper_topic_and_main_contributions": "This paper explored the limitations of the existing detoxifiers, in terms of topic consistency, grammar, and fluency. To alleviate the issue the authors proposed a model-agnostic solution called the Gated Detoxifier. The authors show that the proposed method can successfully preserves the model's performance while achieving comparable toxicity reduction levels.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Please consider clarifying what you meant by detoxifier (Please look at W3).\n2. Typo: missing reference in line 427."}, "extra_text": {}}, {"review_id": "afL1vKB0RF", "reviewer": "Reviewer_9wix", "summary": "", "questions": "A - I could not find the architecture and training details of the gating function/model.\n\nB - Line 480: Why is this the case? Why are toxicity classifiers less expensive than detoxifiers?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful review.\nHere are the comments on your review.\n\n1\\) Instruction-tuned & 2) Reinforcement Learning from Human Feedback (RLHF):\n\nTo focus on the toxicity in the paper, we select CTG methods for the baselines. The objectives of instruction-tuned and RLHF are toxicity (harmfulness) and helpfulness. Thank you for your constructive comments, and we validate toxicity on LLaMA (meta-llama/Llama-2-7b-chat-hf [1] in the huggingface). This is a instruction-tuned model with Reinforcement Learning from Human Feedback (RLHF). We utilized the following prompts to generate 6,000 tweets containing six different emotions. The emotion and toxic classifiers are same as used in the paper. \n\nWe also observed that CTG methods show the better performance on toxicity. The average toxicity of LLaMA is 0.67\\% (anger: 2.20\\%, fear: 1.10\\%, joy: 0.20\\%, love: 0.10\\%, sadness: 0.20\\%, and surprise: 0.20\\%). When using the CTG method, we observed that GeDi yielded a remarkably low toxicity rate of 0.03\\% on the GPT-2 (125M) model, while DExperts exhibited a similar low toxicity rate of 0.05\\%. Given the substantial reduction in toxicity, the CTG technique proves to be highly valuable as a means to mitigate toxicity. We will supplement that modern RLHF models like LLaMA also encounter similar challenges.\n\n\n3\\) Any overhead on top of a base LM is not preferred during the inference stage of decoding.\n\nIt is true that the CTG technique has performance overhead. This overhead was concerned significantly when deploying real-world service. In the light of these concerns, our methods decrease such overhead. The generation of toxic text poses substantial risks in practical service scenarios. Consequently, our work holds the potential to be applied in contexts where the generation of toxic content carries significant risks.\n\n\n4\\) Significance of the results(RQ1 & RQ2: GeDi performance):\n\nRQ1: GeDi seems to be at par with the base LM in terms of fluency as per the results in Table 1. The authors claim that GeDi hurts LM performance is therefore not convincing.\n\nRQ2: While the gating scheme helps in improving the linguistic quality of PPLM and DExperts, GeDi doesn't seem to need the gating function for improving linguistic quality (Table 3).\n\nAnswer: \nOn average, GeDi appears to be rather better on linguistic quality metrics. However, when looked at by topic, the results are different. Detoxifier's performance degradation occurs differently depending on the topic and metric. This table is the evaluation result of GPT-2, GeDi, and GeDi\\_{gated} at small scale.\n\n\nThese are changes of GeDi's linguistic quality from GPT-2\n- Grammar in emotion topic group: 88.31 -> 91.06 (+2.85)\n- Grammar in sentiment topic group: 88.22 -> 82.09 (-6.13)\n\nGeDi increases linguistic quality only in the emotion topics. The reason is that the texts in the emotion dataset are missing grammatical elements such as comma and apostrophe. On the other hand, in the sentiment dataset, the linguistic quality is rather greatly reduced. Sentiment has 2 topics, emotion has 6 topics. As a result, when micro-averaged, the difference in emotion is more prominent. We will add these qualitative results in the appendix.\n\n \n\n#### Questions: \n\nQ.A) I could not find the architecture and training details of the gating function/model.\n\nWe will add the details of architecture and training information. We used the s-nlp/roberta\\_toxicity\\_classifier model, publicly available on Hugging Face, as our gating model. This model is trained for toxicity classification task. This model is fine-tuned RoBERTa[2] model for toxicity classification task. The dataset used for training are Jigsaw datasets ([Jigsaw 2018](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge), [Jigsaw 2019](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification), [Jigsaw 2020](https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification)), containing around 2 million examples. The classifiers perform closely on the test set of the first Jigsaw competition, reaching the AUC-ROC of 0.98 and F1-score of 0.76. Since this is not the model we trained, the training details are unknown.\n\n\n\nQ.B) Line 480: Why is this the case? Why are toxicity classifiers less expensive than detoxifiers?\n\nOur gated detoxifier is less expensive than the original detoxifiers. The total parameters of the gate model (gating network + toxicity classifiers) are 110M, but GeDi requires 355M parameters and DExperts requires 1550M parameters (it uses two 775M models). In addition, the probability of generating a toxic sentence in gpt-2 is only 1.85\\%. Even within a sentence, few tokens are toxic. Therefore, the generation speed can be reduced compared to applying a detoxifier to every generation step.\n\nTo clarify our method, we will change our paper title for the camera-ready version.\n\nThank you for pointing out typo errors. We will conduct multiple proof-reading before submitting the final version.\n\n\n[1] Touvron, Hugo, et al. ``Llama 2: Open foundation and fine-tuned chat models.`` arXiv preprint arXiv:2307.09288 (2023).\n\n[2] Liu, Yinhan, et al. ``Roberta: A robustly optimized bert pretraining approach.`` arXiv preprint arXiv:1907.11692 (2019).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Clever trick to improve detoxification schemes both in linguistic quality and speed of generation\n- Detoxification of LM generation is an important problem to address\n- Exhaustive experiments; the gating scheme is applied to several detoxifiers in many different settings\n  - Research goal is studied through multiple research questions, each of which is empirically answered.", "reasons_to_reject": "- Relevance of the approaches studied and improved in this work:\n  - The proposed method improves detoxification schemes that were proposed after the release of the GPT-2 model. Toxicity in LM generation has received a lot of attention since then, and the approach that has eventually emerged prominent to prevent toxicity and improve LM helpfulness is fine-tuning (supervised) on good quality data and through reinforcement learning based on human feedback (RLHF). This is very different from the approach of detoxifiers being studied in this work. Any overhead on top of a base LM is not preferred during the inference stage of decoding. Thus, most practical LMs are being additionally fine-tuned based on human preferences (supervised FT and RLHF), while the approach of adopting a guiding detoxifier has not received much adoption in practice. Given this landscape, I'm concerned that the proposed approach, while being scientifically interesting, tries to improve detoxification techniques that are not highly relevant in contemporary research.\n  - Many more language models have been released after GPT-2, which have not been studied in this work - FlanT5 (Scaling Instruction-Finetuned Language Models) and/or Llama. Evaluating the effectiveness of the proposed gating scheme on these recently proposed LMs would make the contribution more prominent.\n \n- Significance of the results: \n  - RQ1: GeDi seems to be at par with the base LM in terms of fluency as per the results in Table 1. The authors claim that GeDi hurts LM performance is therefore not convincing.\n  - RQ2: While the gating scheme helps in improving the linguistic quality of PPLM and DExperts, GeDi doesn't seem to need the gating function for improving linguistic quality (Table 3).", "questions_to_authors": "A - I could not find the architecture and training details of the gating function/model.\n\nB - Line 480: Why is this the case? Why are toxicity classifiers less expensive than detoxifiers?", "paper_topic_and_main_contributions": "This work tackles the problem of improving detoxification techniques which aim to prevent toxic content in language model generations. Authors propose a gating scheme that is pretty general and agnostic to the detoxification scheme used (weighted decoding, PPLM, GeDi, DExperts) using a toxicity classifier. The gating scheme decides whether or not at a specific token t (decoding time-step) intervention from the detoxification scheme is required. This is a pretty clever insight. Toxic content is rare, therefore, for a large majority of the tokens/decoding steps, we don't need the guiding detoxifiers to intervene.\n\nThe experimental results indicate that their gating mechanism can better preserve the linguistic quality of generated text from the base LM compared to the baseline detoxification schemes without any gating function. Another benefit achieved is in terms of decoding speed - with the gating scheme, the intervention of the detoxifiers/guiding models occurs for a pretty small fraction of tokens in the LM's generation, thus the resulting generation scheme is faster than using the detoxifier at each decoding step.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Title needs some rephrasing.\n- Presentation clarity: The paper is hard to read and follow at several places. The writing style is extremely colloquial (eg. Line 027) and needs major refinement. Goal of the paper and how the proposed scheme achieves it is not clearly communicated.\n- For a reader who's not familiar with the literature on detoxification techniques, the proposed approach is hard to follow in the current draft. Introduction doesn't explain the method clearly enough\n\nLine 023: toxic --> toxicity\n\nLine 044, Line 050 --> citations for these claims\n\nLine 095: what does toxic degeneration mean?\n\nLines 180-200 (Section 3.1, partially Sec 3.2) - does not seem necessary after the related work description\n\nLine 232: proof? qualitative or quantitative?\n\nLine 235: writing style is too colloquial,\n\nLine 456-450: What is %p?\n\nLine 480: should it be instead called 'toxicity classifier'?\n\nLine 400-405: unclear what the authors are trying to communicate."}}, {"review_id": "sO51j22BTn", "reviewer": "Reviewer_Gtqh", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful review.\nHere are the comments on your review.\n\n1) Modern LLM:\nWe also validate our model on LLaMA (meta-llama/Llama-2-7b-chat-hf [1] in the huggingface). This is a instruction-tuned model with Reinforcement Learning from Human Feedback (RLHF). We have not tried applying gated detoxifier to modern LLM. We confirmed that toxicity is still a major issue in this model. We generated a total of 6000 short tweets which containing 6 emotions. Afterwards, we evaluated how accurate with emotion these tweets were and how toxic they were. For evaluation, the classifiers used in the paper were used.\n\n- anger: 57.7\\% accuracy, 2.2\\% toxicity\n- fear: 58.5\\% accuracy, 1.1\\% toxicity\n- joy: 88\\% accuracy, 0.2\\% toxicity\n- love: 13.3\\% accuracy, 0.1\\% toxicity\n- sadness: 64.5\\% accuracy, 0.2\\% toxicity\n- surprise: 20.5\\% accuracy, 0.2\\% toxicity\n\nThe average toxicity is 0.67\\%. It can be seen that llama trained with RLHF still shows high toxicity in certain topics. Compared to GeDi's 0.03\\% and DExperts' 0.05\\%, that's a huge toxicity difference. Even with modern LLM, we can see that there is still a lot of room for improvement.\n\n[1] Touvron, Hugo, et al. ``Llama 2: Open foundation and fine-tuned chat models.`` arXiv preprint arXiv:2307.09288 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow.\n- The proposed method is intuitive and easy to implement in a variety of detoxifiers.\n- Experimental results show that the method is effective at retaining the model's performance.", "reasons_to_reject": "- It would be interesting to see if the results are consistent when applied to other architectures (LLama, Falcon, GPT3, etc), even though the experimental section is quite thorough.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Detoxifying methods seek to modify the model's generation to mitigate potentially toxic outputs. Usually, these methods are computationally expensive, both in terms of time and computation. To solve this issue, this work presents gated detoxifiers. Instead of applying the detoxifier for every single inference step,  it is only applied to steps scored above a certain threshold. This method allows a more efficient inference, as the detoxifier is only applied to steps that are considered potentially toxic.  Experimental results show that this method is able to keep the performance of previous detoxifiers while reducing performance degradations when using the models on downstream tasks.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7810aZQyUL", "reviewer": "Reviewer_Byj4", "summary": "", "questions": "A. Due to PPLM, GeDi, and DExperts being controllable generation models for detoxification, applying the gated detoxifier on top of them is easy to improve language quality (e.g., topic accuracy, grammar) while maintaining low toxicity. How about the performance for GPT-2 with the gated detoxifier?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and insightful review. Here are the comments on your review.\n\n1 & 3) Main contributions:\n\nIn order to commercialize the generative model, detoxification and subsequent performance degradation have received a lot of attention. However, it mainly focused on partial analysis - linguistic quality [1] or gender/regional/racial bias [2] -. To the best of our knowledge, this is the first analysis work of various aspects of LM performance degradation (topic preserving, grammar, and perplexity) of detoxification. We also conducted extensive experiments with four baselines (PPLM, GeDi, DExperts, and DisCup).\n\nWe analyze the performance degradation of existing detoxification methods, and we also propose a simple gated detoxifier that controls the detoxifier to lower the toxicity without sacrificing LM performance. In addition, the gated detoxifier significantly decreases the cost of original detoxifiers of the Controllable Text Generation (CTG) technique. Even though GeDi and DExperts show low toxicity, they have not been used in practice because of their inference time cost. This is the first approach that can significantly reduce the overhead of CTG techniques.\n\n2\\) Technical details:\n\nQ. Due to PPLM, GeDi, and DExperts being controllable generation models for detoxification, applying the gated detoxifier on top of them is easy to improve language quality (e.g., topic accuracy, grammar) while maintaining low toxicity. How about the performance for GPT-2 with the gated detoxifier?\n\nWe conducted an experiment which add a gate to GPT-2 without a detoxifier. When a toxic token is generated, sampling is performed again. This method had no performance difference from only using GPT-2. In the gated GPT-2, toxic tokens are continuously generated even if toxic tokens are resampled from one token to another because the context has a great influence on toxic text. Therefore, at that moment, the detoxifier must intervene and change the probability distribution.\n\nThanks for highlighting the typographical errors. We'll ensure several rounds of proofreading are done before the final submission.\n\n[1] Gu, Yuxuan, et al. ``Improving controllable text generation with position-aware weighted decoding.`` Findings of the Association for Computational Linguistics: ACL 2022. 2022.\n[2] Xu, Albert, et al. ``Detoxifying language models risks marginalizing minority voices.`` arXiv preprint arXiv:2104.06390 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of this paper is intuitive and easy to understand, with a solid literature review and necessary preliminaries. Besides, the proposed gated detoxifier is simple and easy to generalize.\n\n2. Experiments are well-organized, with clear research questions (RQ1 to RQ6) to answer one by one. Experimental results show that gated detoxifiers achieve comparable levels of toxic reduction while maintaining good generation performance.", "reasons_to_reject": "1. The main contributions of this paper are limited. The key to the proposed gated detoxifier is to apply a pre-trained binary toxic classifier to help refine a toxic token to a non-toxic one. It's hard to see key differences compared with existing controllable text generation methods for detoxification.\n\n2. There is a lack of necessary technical details in the proposed method, e.g., how the toxic classifier is trained or obtained, how the gating mechanism works since the authors mentioned \"gate threshold $\\theta$\" in Lines 431-433 but no introduction in Section 4.\n\n3. Since the proposed method is very simple, I would expect it to be a simple yet effective approach. However, the performance gains of automatic evaluation are not remarkable enough, especially the Toxicity metric.", "questions_to_authors": "A. Due to PPLM, GeDi, and DExperts being controllable generation models for detoxification, applying the gated detoxifier on top of them is easy to improve language quality (e.g., topic accuracy, grammar) while maintaining low toxicity. How about the performance for GPT-2 with the gated detoxifier?", "paper_topic_and_main_contributions": "This paper focuses on the issue that existing detoxification methods suffer from quality degradation and inference overhead. To this end, the authors propose a simple model-agnostic gating method for reducing toxicity during language generation while preserving a language model's generation performance, such as topic consistency and grammar. The key is to apply a binary toxic classifier to help refine a toxic token to a nontoxic one. Extensive experiments on three datasets show that gated detoxifiers achieve comparable levels of toxic reduction while maintaining good generation performance.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The overall writing needs to be carefully proofread. e.g., there is a lack of \".\" in Line 327 and Line 333."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces the concept of gated detoxifiers as a solution to improve the text quality of the existing guided decoder of LM for non-toxic text generation. The proposed solution is designed to be model-agnostic and aims to preserve the linguistic quality of the text while reducing toxicity. 

Pros:
1. Addresses a significant problem of improving text quality in non-toxic text generation.
2. The proposed method is model-agnostic. 
3. Positive experimental results in retaining model performance and reducing computational overhead.

Cons:
1. Limited novelty with similarities to existing methods. 
2. Lack of clarity. Two reviewers point out that the term "detoxifier" is confused. It doesn't reflect the main purpose of the paper. Reviewers also mention that the technical and implementation details were missed. 
3. Writing and presentation need refinement. Reviewers 1, 2 and 4 point out many language errors.
Decision: Accept-Findings