Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an approach to training language models to unlearn various undesirable behaviors, using a reward function. The authors propose to do this by iterating the final 3 steps of the following procedure: sample from the model and obtain corresponding reward values, sort the resulting generations into k-quantiles, fine-tune the model on these generations conditioned on the corresponding k-quantile, but with a KL-penalty that prevents the model from deviating too far from the initial model, and then re-sample generations conditioned on the highest k-quantile. The authors evaluate their approach in its ability to prevent generation of toxic text, to flip sentiment, and to avoid repetition. They find their approach outperforms baselines in terms of automatic and human evaluation.
strengths_and_weaknesses: Strengths:
- The paper is clearly written.
- The proposed approach is simple and intuitive.
- The authors do a thorough evaluation, and the results are impressive.
- The authors include very interesting ablation experiments.

Weaknesses:
- Although the authors consider many ablations, since Quark does seem quite similar to PPO-like approaches, I think we could have benefited from seeing the results of some other baselines, including: 
    - Fine-tuning on the resampled outputs (from the top quantile) but without conditioning them on the quantile control token. This tests whether using control tokens is important.
    - Doing “exploration” with the original model $p_0$ rather than $p_\theta$, which simply amounts to fine-tuning (with control tokens, say) on a larger corpus that has been labeled by the reward function. This seems like the simplest fair baseline, and checks whether it is even important to keep the samples up-to-date with the current model (which is in any case being constrained to stay close to $p_0$).
- Similar to the previous point, it’s not clear to me that the baselines from prior work are especially fair. For instance, DAPT is somewhat similar to the proposed second baseline above, except it is fine-tuned on a different corpus. Presumably this baseline would be more competitive if it were fine-tuned using results of the toxicity classifier being used to evaluate, as Quark is.
- If the paper is interested in positioning itself as being about an RL technique or reward maximization, I would expect experiments to report the actual reward achieved by Quark or any of the baselines. 

Update after author response: thanks for your responses and for the new results. I'm increasing my score.
questions: For the ablation experiments in Table 7: does the experiment in the last row, where the model is trained just on the "best" quantile, ensure that the model still sees as much data as the models in the previous two rows which are trained on "all" the quantiles?
limitations: Yes, the authors address limitations and potential negative societal impacts adequately.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Large-scale language models may generate text with offensive or toxic language, or repetition. The authors propose an algorithm, named Quantized Reward Konditioning (Quark), to unlearning these misalignments by fine-tuning the language model on signals of what not to do. Quark optimizes a reward function that quantifies an (un)wanted property and contains three steps: exploration, quantization, and learning. The experimental results on unlearning toxicity, negative sentiment, and repetition show the effectiveness of the proposed algorithm.
strengths_and_weaknesses: Strengths
- The authors propose a novel task to unlearn behaviors misaligned with user expectations and design a simple and effective algorithm to achieve this goal.
- The proposed algorithm can rely on standard language modeling primitives to conduct training without additional models or specialized heuristics to stabilize training.
- Extensive experiments show that Quark can be applied to unlearn toxicity, repetition, and unwanted sentiment without sacrificing underlying language qualities such as fluency and diversity.

questions: N/A
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a reinforcement learning algorithm to achieve specific goals of language model generation, by incorporating three stages: exploration, quantization and learning. In particular, the language model is finetuned with a KL-divergence penalty. In generation stage, the language model conditions on high-reward token so that the not-expected property will be less generated. Empirical results show its effectiveness on unlearning toxicity, negative sentiment, and repetition, evaluated by both automatic evaluation metrics and human evaluation.
strengths_and_weaknesses: Strengths
1. Clear presentation, easy to follow
2. Comprehensive ablation analysis to show effects of hyperparameters and different settings

Weakness

The current framework requires specific reward design for different goals (i.e.unlearning unwanted sentiment or toxicity). In general, language generation requires many desired characteristics and the current design may not serve as an appropriate tool for this goal.


questions: 1. What is the time complexity of the algorithm? How much more time will it need compared to baselines?

2. Does QUARK has a high requirement for memory? e.g. how large is expected for the data pool to store the (input, output, reward) examples?
limitations: See weakness
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: **What is the task?**
Task of unlearning undesirable properties of language models acquired during pretraining by fine-tuning the language model on signals of what not to do.


**What has been done before?**
* In contrast to strong contemporary RL methods that stabilize training with an additional parameterized model (more learnable parameters) and specialized optimization heuristics, Quark’s training relies only on standard language modeling primitives. Experiments across three tasks demonstrate that Quark maintains pre-training abilities while unlearning undesired behaviors more stably than alternative methods.

* Quark takes inspiration from three disjoint concepts from previous work in reinforcement learning and conditional language modeling but builds upon each concept in a novel manner.

**What are the main contributions of the paper?**
* Introduced Quark, an algorithm for optimizing a reward function that quantifies an (un)wanted property, while not straying too far from the original model.

* For unlearning toxicity, negative sentiment, and repetition, our experiments show that Quark outperforms both strong baselines and state-of-the-art reinforcement learning methods like PPO, while relying only on standard language modeling primitives.

* In addition to showing the effectiveness of using Quark for unlearning undesirable behaviors from language models, author(s) conducted ablation studies to explore the effect of each component of the training objective.

**What are the key techniques used to tackle this task?** An online, off-policy reinforcement learning (RL) algorithm used to (un)learn properties from language models via three iterative stages: exploration, quantization, and learning.Quark alternates between (i) collecting samples with the current language model, (ii) sorting them into quantiles based on reward, with each quantile identified by a reward token prepended to the language model’s input, and (iii) using a standard language modeling loss on samples from each quantile conditioned on its reward token, while remaining nearby the original language model via a KL-divergence penalty.  By conditioning on a high-reward token at generation time, the model generates  text that exhibits less of the unwanted property.

**What are the main results?**
For unlearning toxicity, negative sentiment, and repetition, our experiments show that Quark outperforms both robust baselines and state-of-the-art reinforcement learning methods like PPO, while relying only on standard language modeling primitives.


strengths_and_weaknesses: Strengths

* Novel method for reward optimization to unlearn undesirable properties of language models acquired during pretraining
* SOTA automatic as well as human evaluation results on 3 different tasks 
* Very nice insights on various model components via a series of ablation studies.
questions: Is it possible to obtain significance tests for table 1 and table 3 since difference between Quark and PPO look small.
limitations: Authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This paper proposes Quantized Reward Konditioning (Quark), an algorithm for (un)learning language model misalignments. This is an important research direction given the importance of developing better-aligned large language models, and the paper does a good job at presenting why it matters and how it is related to prior work. The reviewers think that the paper is well written and clear, and that the approach is novel, sound and interesting. After the rebuttal, all reviewers vote to accept.
confidence: Certain
Decision: Accept