Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a detoxification method that aims to reduce the risk of large language models (LLMs) generating toxic contexts such as rude, disrespectful, and unreasonable language without retraining the LLMs. The idea behind this method is to reduce the probability of selecting the next token if that token will cause a toxic text to be generated eventually with some level of certainty. The authors take a langue generation procedure as a standard Markov decision process and formulate the detoxification tasks as an auxiliary reinforcement learning problem. They demonstrate that the proposed method can significantly mitigate toxicity by both human and automatic evaluation.
strength_and_weaknesses: Strength:

(1) It is novel to contemplate a language generation procedure as a standard Markov decision process and formulate the detoxification problem as a reinforcement learning problem. 

(2) One learned model can be used to detoxify many language models if they share the same vocabulary.

(3) The experimental results show that the proposed approach can generate better results compared to the baselines in detoxification by both human and automatic evaluation.

Weaknesses: 

(1) Although the detoxification model would be smaller than the pre-trained language models, it is unclear that whether the proposed method has advantage in computational efficiency over existing detoxification methods both at the training and inference phases.

(2) I doubt the rectification method would decrease the diversity of generated texts. It would be better to add some experiments to evaluate the generated contexts in their diversity.

clarity,_quality,_novelty_and_reproducibility: The paper is well organized and written in general. It is novel to formulate the detoxification problem as a reinforcement learning problem, and the authors clearly present the theory and the proof of the related theorems. 
summary_of_the_review: This word presents a detoxification method by extending dead-end theory from the recent reinforcement learning methods to alleviate toxicity for large language models. A dead-end state was defined to be a state from which a toxic text would be generated with a high probability. The main idea is to avoid toxic texts to be generate by adjusting the probability of token selection proportional to the risk of ultimate toxicity when the generation process is finished. This study was is motivated and timely. The experimental results show that the proposed method can significantly mitigate toxicity by both human and automatic evaluation. However, the proposed method might decrease the diversity of generated texts. Some experiments are required to evaluate the generated contexts in their diversity.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors frame the task of LM "detoxification" as an RL problem.
They propose "rectification," which requires only access only to token
probabilities from an LM API, rather than internal states.  The method
down-weights tokens that are likely to cause eventual toxic discourse,
treating toxic generations probabilistically as "dead ends".
According to both human and automatic evaluation, the author's new
method provides new pareto-optimal trade-offs between fluency and
toxicity compared to retraining/decoding baselines.
strength_and_weaknesses: The task of making LMs output less harmful language is interesting and
important. The authors present connections to the RL literature,
including some new theorems about about rectification/dead ends/etc.
The authors should be commended for their willingness to empirically
engage with larger language models than I've seen in prior RL+NLP
work, e.g., GPT-2 XL (1.5B) and GPT-3 (175B). Their choice of
baselines are reasonable (save for my biggest concerns, detailed
below). I tend to agree with the authors that developing methods
assuming only specific levels of API access will be important
in the regime of very large language models, and so efforts in that
domain are appreciated.


My biggest concern are missing empirical comparisons to work that
explores a very similar setup --- i.e., using RL from human feedback
to "detoxify" an LM using
perspective/realtoxicityprompts. Specifically, the authors mention
that the two main approaches to detoxification in this setting are
retraining-based/decoding-based. But, there are works that have
readily applied RL methods to this setup before, e.g., Lu et al. 2022
improve upon the improved decoding time method for detoxification of
Liu et al. 2021 in a comparable setup.

- Liu, Alisa, et al. "DExperts: Decoding-time controlled text
  generation with experts and anti-experts." arXiv preprint
  arXiv:2105.03023 (2021).

- Lu, Ximing, et al. "Quark: Controllable Text Generation with
  Reinforced Unlearning." arXiv preprint arXiv:2205.13636 (2022).

It's difficult to directly compare the results from Lu et al. 2022
with those that are presented here. (my best effort:  the
author's gpt2 baseline is 61% toxic and 18 perplexity, and the most
aggressive detox is 10% toxic and 22 perplexity. Lu et al.'s gpt2
baseline is 52% toxic and 11 perplexity (on wikitext). Their method
gives 4% toxic and 12 perplexity (on the in-domain training data),
which seems better (?).) Both methods conduct human evaluations,
but are not directly comparable.

Overall, it's difficult to know which method performs best at this
task because the comparison is missing --- and because the Lu et
al. method has been out for a while, I think it's fair to expect some
comparison with it. That being said, it appears that the Lu et
al. method may be solving a slightly different problem: instead of
updating the parameters of the policy, the proposed rectification is
(at inference time) a decoding-based method. Thus, some APIs, e.g.,
GPT-3, may be more compatible. Though, notably --- GPT-3 may be
fine-tuned as well, but not via an API that requires access to
model-internal representations.  But, if that's the case, I would have
appreciated a comparison to Liu et al. 2021 instead, which also
appears to be a decoding-time method for this "detoxification" task.

A few additional thoughts:

- For a decoding method, I would have liked to have seen a reward
  filtering baseline, i.e., sample N times from the model, and select
  the single sample with lowest model-estimated perplexity.

- The human evaluation results are not that definitive --- ignoring
  ties, the author's model wins in 63% of cases versus the baseline in
  terms of detoxifying.

- I would have appreciated a more thorough discussion of the potential
  for reward hacking in detoxification. This is briefly alluded to in
  Appendix E, but there are missing references, e.g., about the
  shortcomings of perspective, for example:

Hosseini, Hossein, et al. "Deceiving google's perspective api built
for detecting toxic comments." arXiv preprint arXiv:1702.08138 (2017).
clarity,_quality,_novelty_and_reproducibility: see above
summary_of_the_review: Overall, the paper offers a strong empirical contribution to RL tuning
of LMs from human feedback in the setting where policy parameters are
not allowed to be updated. The idea of training a policy to do
decoding time modification of logits is quite interesting, and it's
promising that the method can be applied to models which provide a
more limited API like GPT-3. The authors make it clear in the abstract
that they believe this setting is useful, but --- it appears there's
some missing discussion (and possibly empirical comparison) to methods
which use RL from human feedback (with full policy access) or
decoding-time methods beyond DAPT.


=== after the response period:

The authors added more experiments that I had requested which indeed better situate Rect in comparison to the prior work I had mentioned. Thank you! I still am somewhat concerned about the human evaluation results being not as definitive --- even if they are statistically significant, the magnitude is relatively small. Nonetheless, I have boosted my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a method to prevent existing large language models to produce toxic discourse. Contrary to existing methods that modify the training procedure for LLMs or eliminate tokens at inference based on rules, the proposed method uses reinforcement learning to compute, at each step of the generation, a probability that the final discourse will be toxic and modifies the sampling probability accordingly. 

The authors propose a separate small rectification model that may be applied to any LLM as long as they have the same vocabulary, based on the $\beta$-dead-end theory in RL. They compare the new method with retraining-based and decoding-based baselines and report competitive results. The method can be combined with the TestFilter decoding-based method to out-perform existing techniques.
strength_and_weaknesses: Strength: The method is based on a theory that looks sound, and is confirmed by the results. The proposed model can work for any LLM, alleviating the need for retraining. The paper is well written and could be understood by someone (like me) not very familiar with LLMs and RL. The results look impressive.

Weakness: The TestFilter method seems to be simpler and to work quite well already. The best results presented in this paper are in combination with TestFilter. While showing that the two methods can be complementary, it might be interesting to cary out more comparisons between the proposed method and TestFilter, in particular for the human evaluation and the qualitative results.
clarity,_quality,_novelty_and_reproducibility: I am not expert enough in these topics to provide a relevant evaluation of the quality, soundness and novelty of the method. 
Nonetheless, I found the paper very well written, easy to follow and to understand and convincing. 
Many details are provided in the appendices, which should help reproduce the results.

Minor formatting comments:

"methods like Raffel et al. (2020);" --> like those proposed by
"computationally expensive and slow Gehman et al. (2020); Yang & Klein (2021)." --> citation formatting
summary_of_the_review: This paper is clear and well written. It addresses a relevant and important problem. The method looks soundly grounded in theory and the results are good. As detailed in the other sections, more analysis and comparisons could be useful to evaluate the method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method called rectification to reduce toxic generation in large language models (LLMs). The method requires access to only token probabilities of an LM and does not require any fine-tuning. This is done by treating the generation problem as an MDP and providing rewards based on whether the eventual generation is toxic, treating them as dead-ends and downweighting intermediate generated tokens accordingly. The method is empirically verified to provide improved tradeoffs between fluency and toxicity compared to several baselines.
The reviewers appreciated the novelty of the setup and effectiveness of the method in experiments, had concerns around the diversity of generations and comparison to additional baseline, both of which were addressed in the author response. I encourage the authors to incorporate those results into the final version of the main paper (esp. the Lu et al. baseline which is currently added to appendix D). 
justification_for_why_not_higher_score: Decent contribution, but limited to specific sub-community.
justification_for_why_not_lower_score: The technical contributions are simple, novel and effective empirically. The findings will be useful in informing future work in this area.
Decision: Accept: poster