Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: * This paper investigates language detoxification in multiple settings: training corpus, model size, and efficient modeling.  
* Also, it proposes a new detoxification method training language models (LM) with a set of self-generated non-toxic text, which is a smaller corpus compared to OWTC.
* The experiments show that the training corpus matters the most for the detoxification, as the model size gets bigger the more effort is required (more training epoch or larger corpus) and the importance of the adaptor, which reduces the detoxification while improving the perplexity of the LM.
* Overall: I think this paper introduces variant DAPTs on different sets of training datasets and a different architecture (adapter). This paper is more toward an analytic research paper finding components which must considered for a detoxification model, which I do believe is meaningful. However, the originality of the proposing method is weak.
strengths_and_weaknesses: * Originality
    *  The concept of training with self-generated text for detoxification is proposed in dialogue generation [1]. The approach might not be novel to the task requiring detoxification.
* Quality
    * The purpose and direction of this paper are good. But there are claims that need to be justified (3,4).  
* Clarity 
    * The most of writing is clearly written. Few of them need more explanation or clarification (1,2,5,6).
* Significance
    * With a smaller refined training corpus, SGEAT(augmented) achieved better performance than the baselines. This paper conduct exhaustive experiments and analyses on large-scale models.  
    * However, the prompt construction for the heuristic result is similar to an attribute conditioning model [2], which results in high toxicity scores compared to standard or augmented. The experiments on prompt construction seem unnecessary for the proposing model SGEAT.

[1] Baheti, Ashutosh, et al. "Just say no: Analyzing the stance of neural dialogue generation in offensive contexts." arXiv preprint arXiv:2108.11830 (2021).

[2] Gehman, Samuel, et al. "Realtoxicityprompts: Evaluating neural toxic degeneration in language models." arXiv preprint arXiv:2009.11462 (2020).
questions: 1. Is there a reason for using the utility of average accuracy? Toxicity scores already indicate the accuracy of toxicity of generated text.
2. In section 3, regarding the pertained LM, shouldn't the models be represented as GPT-2 small, medium, and xl instead of GPT-3? GPT-3 itself is known to be a model using 175B parameters. To be more accurate, I think GPT-2 makes more sense than GPT-3.
3. Have you experimented with a similar setting with OWTC which uses the random sampled 100K from 7500K and from the sampled set utilizing the top 35% (~50K) nontoxic texts for the training? 
    * I think that Figure 2 should experiment on the same dataset size to claim the model is overfitting.
4. In Table 2&10, SGEAT(heuristic) tends to have more toxic text than SGEAT(standard&augmented). The paper argues that SGEAT(heuristic) is more toxic than the other two SGEAT data diversity. How diverse are the sentences generated? Could you evaluate using distinct-1, distinct-2 & distinct-3, or another metric that indicates the diversity of generated sentences?
5. Is the validation set for the ppl evaluation derived from the non-toxic 150K OWTC? Or each training corpus 
6. How is DEXPERTS with SGEAT (augmented) formed? DEXPERTS is consist of two auxiliary models, expert, and anti-expert. Did you train both two models using SGEAT (augmented)? Or did you only use the detoxified one? 
limitations: * This paper has well stated the limitation of the SGEAT.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper explores domain-adaptive training (i.e., continued pretraining on special data to remove unwanted model behavior) with respect to the influence of model size, training corpus, and parameter efficiency. The authors further propose self-generation enabled domain-adaptive training (or "SGEAT"), where a self-generated dataset is used for detoxification. SGEAT can also be beneficially combined with a decoding-time method for detoxification.
The authors find that, SGEAT reduces model toxicity, but that model size has little effect on model toxicity. (I.e., the toxicity comes from the data.) They also show that it is more challenging to detoxify large language models.
strengths_and_weaknesses: Strengths:
- Work on detoxifying language models is important in order to safely deploy them. Thus, the topic of the paper is highly relevant.
- The fact that SGEAT works is interesting: apparently the model benefits from seeing its own (detoxified) data.
- The authors use both automatic and human evaluation in their experiments.

Weaknesses:
- Some of the findings are rather straightforward/potentially not novel, even though I don't know references from the top of my head. For example the trade-off between detoxification effectiveness and model performance is obvious (as our datasets are biased/toxic). Similar for the fact that detoxifying larger models is harder.
questions: - In the paper you list your hypotheses regarding why SGEAT should work better than existing approaches. Is there a way to explore your assumptions in more detail?
- What are the limitations of your work?
limitations: I don't think there are obvious negative societal impacts of this work. The authors do not explicitly address limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work explores domain-adaptive training to reduce the toxicity of language models in terms of training corpus, model size, and parameter efficiency. 

As the results show, for the training corpus, using the proposed Self-Generation Enabled domain-Adaptive Training (SGEAT) that uses a self-generated dataset for detoxification gives better performance compared to the existing baselines. For the model size, this work shows that when given the same pre-training corpus, the toxicity level is still the same across different model sizes. Where the authors claim that this finding implies that the toxicity comes from the training set, instead of the model size. Moreover, they also show that large language models are harder to detoxify compared to smaller ones. Finally, this work also demonstrates that adding adapter layers to standard language models for parameter-efficient training improves the model perplexity while maintaining the same level of detoxification.
strengths_and_weaknesses: Strengths:
1. This work studies an important problem of toxicity evaluation and mitigation in language models.
2. The proposed SGEAT approach that uses language models to automatically generate non-toxic training data seems promising when the generated data are used to fine-tune/detoxify the same language model compared with using the time-consuming human-annotated non-toxic datasets.
3. Extensive experiments and ablation studies have been carried out to support the proposed methods.
4. The authors have not only evaluated the detoxification performance, but also the quality and utility of the detoxified language models.
5. The findings regarding the relationship between toxicity level, training data, and model size are very interesting and can benefit the community. 

Weaknesses:
Major:
1. This work did not study the detoxification performance of the generated training set with SGEAT when used to fine-tune other language models different from the one it was generated on. 
2. The authors addressed in detail the effect of fine-tuning/detoxifying with different epochs, but did not do the same detailed evaluation on training samples. For instance, it would be interesting to also have the same Figure 2 for different training samples. 
3. In lines 299-300, the authors claim that "SGEAT achieves substantially lower toxicity with the same perplexity" while Figure 2 shows a different trend.  
4. In Table 5, the combination of SGEAT and adapter has only been evaluated on the perplexity and utility of the language models but not on the detoxification/toxicity level. It would be interesting to also evaluate the effect of this combination on toxicity.
5. Since there are no prompts for the SGEAT (standard), how is it controlled so that it does not randomly generate duplicated outputs? Has this been evaluated?
6. Why is SGEAT (augmented) only generated from the non-toxic samples of SGEAT (standard), but not also from non-toxic samples of SGEAT (heuristic)?

Minor:
1. SGEAT and SGEFT are used interchangeably throughout the paper including Figure 2 and Table 12 while SGEFT was not defined anywhere in the paper. Is it a typo? 
questions: Please refer to the main weaknesses section.
limitations: 1. The main limitation of this work is the error propagation from the automatically generated training set and the Prespective API that has been used to filter the non-toxic samples.
2. Please see the other limitation in the main weaknesses section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper investigates the domain-adaptive training for detoxifying while maintaining LM quality of language models. They focus on three points: training corpus, model size, and parameter efficiency. They've made the contributions and concluded the important observations as follows:
1) from the perspective of training corpus, the proposed self-generated dataset is more helpful for detoxification compared with domain-adaptive training on nontoxic subset of pre-training data.
2) from the perspective of model size, larger LMs suffer more from toxicity from the training corpus
3) from the perspective of parameter efficiency, adapter achieved better trade-off between toxicity and perplexity compared with prefix-tuning.
strengths_and_weaknesses: Strengths:

1) originality: 
- this paper proposes the self-generated dataset for detoxification, which brings benefits to mitigating toxicity compared with baselines that fine-tuning LMs on nontoxic pre-training corpus
- this paper studies effectiveness of different detoxifying approaches on LMs of diverse scales, which include 530B model that hasn't been studied for toxicity before.

2) quality & clarity:
- the paper is well written and declares points clearly in each section
- the performance of different strategies and their improvement/degradation are clearly reported; results of experiments with multiple random seeds are also reported in appendix

3) significance:
- strategies compared and observations discovered in this paper may shed light on follow-up work in improving trade-off between toxicity and utility, large-scale LM design including architecture, training corpus for pertaining, etc.

Weaknesses
1) clarity
- the experiments focus on generative language models, including toxicity evaluation with RealToxityPrompt benchmark and zero-shot utility evaluation on downstream tasks following the prior framework. However, the title, abstract, introduction and conclusion haven't discussed this point, but use "language model" instead. This could be confusing and easily leads to the conclusion that the observation discovered in this paper could apply to general language models.

2) significance:
- fine-tuning LMs with untoxic generated text data is the core idea in the proposed SGEAT. However, according to the results reported in Table 2 and 3, although SGEAT could improve performance on toxicity evaluation, the performance on LM quality (perplexity and accuracy on downstream tasks) is consistently poorer than the baseline approaches DAPT and Jigsaw. Those experiments do not provide clear guidance on which kind of adaptation approach is better for projects that would like to obtain a trade-off between the two. 

- Toxicity is an important problem when fine-tuning not only generative LMs, but also general (encoder alone) LMs on downstream tasks. The proposed strategy and evaluation conducted in this paper could be more significant if general LMs are also studied. 

questions: Q1: the model architectures of different scales are not clearly described. From Section 3, the list of parameter sizes of different models are introduced, but are they share a similar architecture and what kind of architectures do they adopt?

Q2: In related work, domain-adaptive training methods, such as Gehman et al. (prepend nontoxic attribute token as prompt) and  Solaiman and Dennison [16] (change model behavior & reflect a set of targeted values), are discussed. What's the performance of these approaches compared with the proposed SGEAT?

Q3: Besides domain-adaptive training and decoding-time approaches, RL approaches such as InstructGPT have been proposed to avoid toxicity of LMs. What's the performance of RL-based approaches compared with the proposed SGEAT?

Q4: in Table 2, it seems combining domain-adaptive training approach (SGEAT) and decoding-time approach (Rejection Sampling and DExperts) can further improve toxicity evaluation and utility evaluation. How about the combination of other domian-adaptive training approaches, e.g., Jigsaw and DAPT, with decoding-time approach?

Q5: as shown in Tbale 3, SGEAT obtains better performance on toxicity mitigation while DAPT achieves better results on perplexity and utility evaluation. How about the combination of two, i.e., fine-tuning the LMs on the dataset containing both nontoxic pre-training corpus and generated text data? Can the combined corpus take the advantage of both strategies and therefore have a better trade-off?
limitations: The authors have discussed the limitations of this paper in appendix, from the perspective of:
- limitation of toxicity evaluation platform Perspective API; 
- potential bias in detoxification methods is left for future work
review_title: null

Meta-review: metareview: This paper explores domain-adaptive training (i.e., continued pretraining on special data to remove unwanted model behavior) with respect to the influence of model size, training corpus, and parameter efficiency. All technical reviewers lean toward accepting this paper. However, there are also some ethics concerns.
confidence: Less certain
Decision: Accept