QUARK: Controllable Text Generation with Reinforced Unlearning
Authors: Ximing Lu
Sean Welleck
Jack Hessel
Liwei Jiang
Lianhui Qin
Peter West
Prithviraj Ammanabrolu
Yejin Choi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Language Generation
Language Models

Abstract: Large-scale language models often learn behaviors that are misaligned with user expectations. Generated text may contain offensive or toxic language, contain significant repetition, or be of a different sentiment than desired by the user. We consider the task of unlearning these misalignments by fine-tuning the language model on signals of what not to do. We introduce Quantized Reward Konditioning (Quark), an algorithm for optimizing a reward function that quantifies an (un)wanted property, while not straying too far from the original model. Quark alternates between (i) collecting samples with the current language model, (ii) sorting them into quantiles based on reward, with each quantile identified by a reward token prepended to the language model’s input, and (iii) using a standard language modeling loss on samples from each quantile conditioned on its reward token, while remaining nearby the original language model via a KL-divergence penalty. By conditioning on a high-reward token at generation time, the model generates text that exhibits less of the unwanted property. For unlearning toxicity, negative sentiment, and repetition, our experiments show that Quark outperforms both strong baselines and state-of-the-art reinforcement learning methods like PPO, while relying only on standard language modeling primitives.

Quark: Controllable Text Generation with Reinforced [Un]learning
Ximing Lu 2
Sean Welleck C Jack Hessel
Liwei Liagg
Lianhui Qin
Peter West Prithviraj Ammanabrolu
Yejin Choi
Allen Institute for Artificial Intelligence
Paul G. Allen School of Computer Science, University of Washington (ximinglu, jackh, raja)@allenai. org
{wellecks, lwjiang, lianhuiq, pawest, yejin/ washington edu
https://github. ctt.lb..ccmGiiiiii http://tthb...com///iimgag//a/a
Abstract
Large-scale language models often learn behaviors that are misaligned with user expectations. Generated text may contain offensive or toxic language, contain significant repetition, or be of different sentiment than desired by the user. We consider the task of unlearning these misalignments by fine-tuning the language model on signals of what not to do. We introduce Quantized Reward Konditioning (Quark), algorithm for optimizing reward function that quantifies an (un)wanted property, while not straying too far from the original model. Quark alternates between (i) collecting samples with the current language model, (ii) sorting them into quantiles based on reward, with each quantile identified by a reward token prepended to the language model's input, and (iii) using standard language modeling loss on samples from each quantile conditioned on its reward token, while remaining nearby the original language model via KL-divergence penalty- By conditioning on a high-reward token at generation time, the model generates that exhibits less of the unwanted property. For unlearning toxicity, negative sentiment, and repetition, our experiments show that Quark outperforms both strong baselines and state-of-the-art reinforcement learning methods like PPO [66), while relying only on standard language modeling primitives.
Introduction
Large neural language models trained an enormous amount of web text have excelled at numerous tasks 58
They provide an effective interface for few-shot learning [ show impressive natural-language understanding capabilities [77. and, in some contexts, their generations can be
indistinguisaable from authnr-uthored text
However, these same language models often exhibit undesirable behaviors, as they are usually trained simply maximize the likelihood of their raw pre-training data. For example, models sometimes generate toxic text that reflects pernicious social biases [18] 69) generate repetitive and dull language 77 38 25 Undesirable behaviors are diverse and hard avoid, control, or even specify priori; we thus argue that it is critical to investigate ways to unlearn undesirable behaviors post hoc, while maintaining capacity for generating coherent and fluent language.
Supervised approaches for unlearning pose challenges. One option is to curate and train on a corpus that encodes desirable behavior, with the hope that additional maximum likelihood training will shape
equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Exploration
Quantization
Sort the data pool by reward to form reward token quantiles
Sample text from the current language model
Reward
Language
Reward Taken
Quantile
Token
Hallo,
Prompt
...
Model
you @&!
Fow you?
Data Pool
RR
Hello
sou?
High
Score the text with a reward function and add to a data pool
Hell,
+
Reward
.01
Dasa Paol
4051
Low
Learning
Hede
Train (reward token prompt, generation)
I+S
plus KL divergence penalty
Figure 1: Quantized Reward Konditioning (Quark) is an online, off-policy reinforcement learning (RL) algorithm used to (un)learn properties from language models via three iterative stages: exploration, quantization, and learning.
the model's is distribution more favorably. However, collecting data that accurately captures desired characteristies (e.g., non-toxic, non-degenerate texts) is difficult (if not impossible) 440 Moreover, models may overfit to the newly collected corpora 40 332 and lose desirable characteristics, e.g., few shot learning capacity over general domains. Another option is to build a detector of the undesirable behavior, e.g., by labelling model outputs. However, it is not clear how to adjust the model so that it only generates text that the detector prefers: since detectors score full text samples from model rather than providing token-by-token feedback, they are not directly differentiable (e.g.. toxicity scores) [54]
Dynamically (un)learning from sentence-level, scalar feedback is perhaps better suited to the rein- forcement learning (RI.) paradigm. NI.P, RI. has been used to optimize scalar metrics in the form of rewards 54
Recently [51 used Proximal Policy Optimization (PPO) (6 to optimize 175B parameter model via a learned reward model, while constraining the model to remain close to the original with KL-divergence penalty. However, as (deep) RL is highly sensitive to variance in the reward function [I4 these methods rely on additional models often doubling the number of learnable parameters and specialized heuristics to stabilize training.
60 3
We introduce Quantized Reward Kanditianing (Quark), an algorithm for reward- based (un)learning with language models. Quark builds upon insights from three prior works: the Decision Transformer [9].
LM tuning with PPO [T]), and control tokens [28]. During training, Quark alternates between (i) collecting samples with the current language model, (ii) sorting them into quantiles based on reward, with each quantile identified by reward token prepended to the language model's 's input, and (iii) maximizing the likelihood of the samples from each reward quantile conditioned on its reward token, while remaining nearby the original language model via KL-divergence penalty. In contrast to strong contemporary RL methods that stabilize training with an additional parameterized model and specialized optimization heuristics, Quark's s training relies only on standard language modeling primitives. Experiments across three tasks demonstrate that Quark maintains pre-training abilities while unlearning undesired behaviors more stably than alternative methods.
Quark: Quantized Reward Konditianing
Starting from pretrained language model, Quantized Reward Konditioning (Quark) alternates between three steps, illustrated in Figure 1:
Exploration: sample text with the current model, evaluate its reward, and store in a data pool. Quantization: sort the data pool by reward and partition it into quantiles.
Learning: update the language model using samples from each quantile.
By sampling from high reward quantiles during exploration and using a KL-divergence penalty during learning, Quark iteratively improves the language model by steering its distribution towards

---

Algorithm Quantized Reward Kanditianing (Quark)
input Initial policy Po, prompts X, reward r(.). KL weight 3, number of quantiles K Make copy ps of initial policy po; and Initialize data pool D
I:
Initialization
for iteration 2.,,...,, do
3: for z; E X do
4: Sample generation yi ~ pe po(-/ai, rx)
> Exploration
5: Add (Ii, Mr. r0;, v:3) into data pool D
6: DD +-quantize(D; K)
Quantization
7:
for step ..2....,
do
8:
Draw a batch of data {a Mr, (..))) } from quantized data pool D; Compute the objectives in Eq.
Learning
9:
10: Update the policy parameters # via gradient descent
increasingly high-reward samples, while not straying too far from the original model. Quark is summarized in Algorithm T it can be implemented succinctly using standard language modeling
libraries, see Appendix C
Initialization. Quark begins with pretrained language model po(y]z), a set of training prompts X and reward function r(z, y) IR Here ( and y = (y...... my)l are sequences of tokens from a vocabulary V. Quark initializes a datapool of (input, output, reward) examples by sampling") from Po conditioned on the training prompts, and scoring them with the reward function,
Do = {z y, r(x, y)) y (-(+))) for all x E X)).
(1)
If available, the datapool can instead be initialized with any (I. y) pairs (e.g., from supervised dataset). Quark then proceeds iteratively, updating a copy of the pretrained language model, po. by alternating between exploration, quantization and learning. We detail quantization first.
Quantization. Quark quantizes each example in the datapool based on how high its reward is compared to others in the data pool. Quark sorts the current iteration's 's datapool in order of increasing reward, and partitions the sorted pool into equally sized quantiles. D ...., D Each sample (æ, y) is now part of a quantile that is identified by reward token with k E f1,..., KK. For example, in Figure the non-toxic generation how are you? placed F3. while the toxic generation, you are =@ &., is placed in the lowest-reward quantile
highest-reward quantile, identified by
Learning. For learning, Quark trains on the quantized datapool using standard conditional language modeling objective maximizing likelihood along with KL-pennlty to keep the model from deviating too far from the original:
max ( logro(yle,Ta) B
KL (po(-) (eee,,)))) (- x, rw))
(2)
where each KL term is Enev Pu (y log po(pe)
pe())) (omitting the conditioned terms). Naturally, Quark supports other penalties developed for language modeling, e.g., entropy (43 or unlikelihood [79].
Exploration. During exploration, Quark adds new generations to the data pool by sampling from the model conditioned on the highest-reward token,
D D U {I, (, y (((+,,,K), for all x E X}.
(3)
where pa(-1æ,, rk) means sampling from the current model ps, with the reward token r'K prepended to the training input æ. Intuitively, this step explores the most promising regions of the distribution by querying the current model for what it expects to be high reward completions.
Evaluation. At test time, we condition the language model on the highest reward token, y ro() and evaluate the resulting samples.
Any decoding method can be used, e.g., greedy search, beam search, nucleus sampling 25

---

In-domain (REALTOXICTTYPROMPTS
Out-of-domain (WRITINGPOMPPT)
Model
Toxicity (J)
Fluency (U)
Diversity (t)
Texicity (U)
Fluency (L)
Diversity (1)
avg. max,
prob.
oulput ppl
dist-2
dist-3
avg. max.
prob.
ourput ppl
dist-2
dist-3
GPT2
557
0.527
0.520
11.31
0.85
0.85
0.572
0.610
12.99
0.82
0.85
PPLM 12
0.520
0,518
32.58
0.86
0.86
0.544
0,590
36.20
0.87
0.86
GeDi
60.03
0.83
(182
DEXPERIS
60
0.314
0,128
32.41
0.84
0.84
0.343
0.156
42.53
0.86
0.85
DAPT 21
0.428
0.360
31.21
0.84
0.84
0.442
0.363
38.11
0.86
0.85
PPO
0.218
0.044
14.27
0.80
0.84
0,234
0.048
15.49
0.81
0.84
Quark
0.196
0.035
12.47
0.80
0.84
0.193
0.018
14.49
0.82
0.85
Table 1: Automatic evaluation results of unlearning toxicity experiments. Baseline results (except
PPO) are from 40
Ours GPT2 Ours PPLM|
Ours vs. GeDi
Ours DEXPER Ours vs. DAPT
Ours PPO
In-domain
(REALTOXICITYPROMPT
Less Toxie
0.21
0.07
0.20
0.08
0.15
0.06
0.14
0.10
0.12
0.12
0.12
0.12
More Topical
0.22
0.14
0.23
0.14
0.21
0.13
0.18
0.18
0.20
0.16
0.22
0.14
More Fluent
0.26
0.19
0.27
0.17
0.29
0.15
0.26
0.21
0.23
0.18
0.28
0.18
Out-of-domain (W (RITIIGGROMPPTS)
Less Toxic
0.18
0.06
0.25
0.08
0.16
0.11
0.16
0.07
0.16
0.10
0.15
0.08
More Topical
0.20
0.20
0.31
0.23
0.34
0.19
0.36
0.19
0.29
0.27
0.32
0.17
More Fluent
0.26
0.21
0.31
0.23
0.41
0.14
0.38
0.21
0.33
0.23
0.32
0.20
Table Human evaluation results of unlearning toxicity experiments, comparing the percentage of texts rated as less toxic, more topical, and more fluent as generated by Quark and other baselines.
Relationship prior work. Quantized Reward Konditioning builds upon three disjoint concepts from previous work in reinforcement learning and conditional language modeling.
(1) Inspired by PPO (91), encourage model stay close reference model using KL- divergence penalty. The penalty in [91] approximates KL-divergence at the sequence level through a pa(a)
reward penalty, F(x) = r(z) - B log pe(m)> while we use a differentiable loss that exactly computes the per-step KL divergence (Eq (Eq[): this may contribute ease of optimization. Unlike PPO, do not control for the variance of the reward function by subtracting off a baseline value function: instead, we quantize. This modification also allows to optimize language model log probabilities directly without the additional (sometimes finicky) hyperparameters of PPO, including policy step clipping radius, and adaptive KI. schedules.
(2) Inspired by the Decision Transformer 9 which frames reinforcement learning as next-token prediction, we train a model capable of conditioning on the desired reward of the trajectory, prior to observing it, i.c., our reward token appears in the input of (o/2,,, rk). Different from the decision transformer, we (i) have exploration step and (ii) we don't 't attempt to model discounted reward over multiple timesteps, and instead only consider a one-step bandit environment.
light-weight representation of reward. Each reward quantile encoded via an embedding lookup, following past work on style and content controis (28), or prompt/prefix encodings that can be tuned to solve tasks efficiently [3 37 Unlike prior work, our control codes are iteratively updated to guide unlearning.
(3) Inspired by control codes (28 we use learned embeddings
3
Experiments
In this section, we show that Quark can effectively unlearn undesirable behaviors from neural language models. including toxicity, repetition, and unwanted sentiment. Following the setup of previous works we use GPT2-large (57) as the initial policy Po for toxicity and sentiment experiments, and GPT2-bass for repetition experiment.
40 79 73).
3.1
Unlearning Toxicity from Language Models
Language models are susceptible to generating offensive completions that reflect false social biases, even when prompted with scemingly innocuous text [18. We consider unlearning toxicity from GPT-

---

2 on the REALTOXICITY generations, e.g.. "I cant believe the people of.... " We show that using Quark with a commercial toxicity API as the reward substantially reduces the chance of generating toxic text as measured by both automatic metries and human judgments, without otherwise affecting generation quality.
benchmark, which contains prompts known to elicit problematic
Experimental setup. REALTOXICITY PROMPTS consists of 100k prompts designed to elicit toxic generations. We follow the experimental setup of Liu et al. [40]. During training, we use 85K prompts from the train set; for evaluation, we use the same 10K non-toxic test prompts used by [40], and generate using nucleus sampling with = 0.9. Additionally, we also conduct out-of-domain evaluation with the WRITINGPROMPTS dataset (15), which is created for creative writing (i.e., story generation). We use the Perspective API as a reward function, which provides a score between (non-toxic) and 0 (toxic We use K 5 quantiles.
Baselines and evaluation metrics. We include previously reported baselines from 44)]. including GPT-2 (i.e., the Po model), PPLM []2], GED1 (32), DAPT [21], and DEXPERTS [40]. Additionally, as a represenntaiiv state-of-the are RL method, we implement PPO with the KL-pennlty as in 9 [T]; see subssecttion B. for details.
Following [4), maximum toxicity is measured as the average maximum toxicity over 25 text gen- erations, and the empirical toxic probability of at least one of any 25 generations being toxic, both of which are judged by Perspective API. To evaluate language quality as a proxy for how much the model deviates from the original model, we report fluency the perplexity of generated output according to a larger off-the-shelf GPT2-XL model, and diversity as the count of unique r-grams normalized by the length of text. Finally, we conduct pairwise human evaluation to compare outputs from Quark to each baseline, based on the perceived level of toxicity (which one is less rude or disrespectful), topicality (which one is more natural, relevant, and logical), and fluency (which one is more grammatically correct and coherent); human evaluation details are in Appendix A
Results. As shown in Table [ Quark reduces the rate of toxic completions substantially compared to all baselines, in both in-domain and out-of-domain settings. While prior detoxification methods generally sacrifice language quality, Quark reduces toxicity while maintaining similar level of fluency and diversity compared to vanilla GPT-2. Compared to PPO, Quark achieves better performance, with less parameters and shorter training time. Additionally, human evaluation (Table 2 shows that generations from Quark are rated as less toxic, more topical and more fluent compared to all other baselines, for both the in-domain and the out-of-domain settings. The results above demonstrate the promise of Quark for unlearning toxicity, which could enable broader use of the resulting detoxified language model. Additional qualitative results are in Appendix D
3.2 Steering Away from Unwanted Sentiment of Generated Texts
Next, we explore Quark's ' capacity to control the sentiment polarity of text generated from a language model [24] 40). This task, which is well-studied in controllable generation, is often practically motivated by the goal of building chat bots that do not simply output probable language, but also discourse acts that echo particular emotion or sentiment 63 36 78).
Experimental setup. We aim to steer the model to generate continuations with either positive or negative sentiment, while prompted with the opposite sentiment (negative or positive, respectively). We follow the experimental setup of [40]. which uses 100K prompts from the Open WebText Corpus (OWT) 19. During training, we use 85K prompts from the training set. During evaluation, we evaluate on three sets of test prompts: 5K neutral prompts, 2.5K positive prompts and 2.5K negative prompts. We use the sentiment analysis classifier (DistilIBERT (62) trained on SST-2 dataset from HuggingFace [8] as the training reward, which provides sentiment score between (posittive) and 0 (negative We use K- quantiles.
'The Perspective API is a service provided by Google that defines a "toxic" comment as one that is "rude. disrespectful, or unreasonable that is likely to make one leave a discussion" caaaussoonntttaaa |htpp::p:s //tp///gtbb///////// com/ wwweerrsaaiooaapprppppeccc
Queries were made from Jan 2022- May 2022, and reflect the version
being hosted at the time. The API itself imperfect and reflects some social biases 26 See /seccttionn for further discussion.
engnlish
5

---

Sentiment to Unlearm: NEGATIVE
Sentiment to Unlearm: POSITIVE
Model
% Positive (1)
Fluency (L)
Diversity (?)
% Positive ())
Fluency ({)
Diversity (f)
negative prompt
neutral prompt
output ppl
dist-2
dist-3
positive prompt
neutral prompt
output ppl
dist-2
dist-3
GPT2
57
0.00
50.02
11.42
0.85
0.85
99.08
50.02
11.42
0.84
0.84
PPLM
12
8.72
52.68
142.1
0.86
0.85
89.74
39.05
181.7
0.87
0.86
CTRL
79
18.88
61.81
43,79
0.83
0.86
79.05
37.63
35.94
0.83
0.86
GeDi
52
26.80
86.01
58.41
0.80
0.79
39.57
8.73
84.11
0.84
0.82
DEXPEKUS
40
36:42
94.46
25.83
0.84
0.84
35.99
3.77
45.91
0.84
0.83
DAPT
14.17
77.24
30.52
0.83
0.84
87.43
33,28
32.86
0.85
0.84
PPO 71
43.13
94,10
15.16
0.80
0.84
32.22
3.65
15.54
0.81
0.84
Quark
46.55
95.00
14.54
0.80
0.84
27.50
2.75
14.72
0.80
0.84
Table 3: Automatic evaluation results of unlearning sentiment experiments. Baseline results (except
PPO) are from 440.
Ours vs. GPT2
Ours vs. PPO |Ours vs. CTRL.
Ours vs. GeDi
Ours vs. DEXPERT
Ours vs. DAPT
Sentiment to Unlearn: NEGATIVE
More Positive
0.58
0.04
0.16
0.06
0.46
0.12
0.38
0.14
0.32
0.18
0.48
0.12
More Topical
0.32
0.07
0.32
0.26
0.23
0.16
0.22
0.19
0.24
0.17
0.24
0.12
More Fluent
0.36
0.10
0.33
0.28
0.28
0.23
0.26
0.26
0.27
0.23
0.28
0.19
Sentiment to Unlearn: POSITIVE
More Negative
0.47
0.14
0.37
0.21
0.48
0.18
0.31
0.37
0.29
0.51
0.12
More Topical
0.21
0.18
0.29
0.18
0.26
0.20
0.33
0.17
0.32
0.16
0.20
0.20
More Fluent
0.28
0.24
0.31
0.20
0.36
0.22
0.38
0.21
0.40
0.23
0.24
0.24
Table 4: Human evaluation results of unlearning sentiment experiments, comparing the percentage of texts rated as more positive/negative, more topical, and more fluent as generated by Quark and other baselines.
Baselines and Evaluation Metrics. In addition to all baselines described in $3.1 also include CTRL [29], which steers language models with control codes. For each prompt, we generate 25 continuations evaluation time. For automatic evaluation, ftuency/diversity metrics, and also the mean percentage of positive continuations among the 25 generations according to the HuggingFace sentiment model. We also conduct pairwise human evaluation as before to compare outputs from Quark to each baseline, based on the perceived level of desired sentiment, topicality, and fleency human evaluation details are in Apppendix A
report the previously discussed
Results. As shown in Table Quark more effectively steers models away from unwanted sentiment (both positive and negative) compared to all other baselines, while remaining as fluent and diverse as the vanilla GPT2 model. Moreover, the human evaluation results in Table Tablee4cconfrmm that generations from Quark are consistently judged to be more the desired sentiment, more topical, and more fluent compared all previous methods. Additional qualitative results are in Apppeddix
3.3 Unlearning Degenerate Repetition
Neural language models often suffer from text degeneration, i.e., they generate repetitive, uninfor- mative, and dull text 779 25). Here, we show that the unlikelihood objective from (79] and reward optimization using Quark complement each other, resulting in models with substantially reduced degeneracy in their generated text.
Our goal to unlearn degenerate repetition in text generation. We follow the experimental setup of [79 73). During the exploration phase, in order to have diverse set of representative model outputs with different repetition levels, we mix greedy decoding and nucleus sampling in 50%-50% proportion, as repetition more often happens when using greedy decoding. We use diversity metric as the reward, to encourage larger portion of unique n-grams generations, defined as diversity(y) = 1 rop-n(y) where rep-n(y) = 100 x (1.0 (1.0-
Experimental setup.
unique grams(y) (y) We total n-gramsty)
use K = 8 quantiles. Following the setup of [79 73 we use WIKITEXT- 1 03 as the dataset, which contains 100M English tokens from Wikipedia articles. During evaluation, we generate using greedy decoding, degenerate repetition tends to appear most frequently with greedy decoding.

---

Language Model Quality
Generation Quality
Human Eval
Model
ppl
acc
rep
wrep
rep-2 y rep-3
div t mauvef
fluency 1 coherence | overall 1
MLE 7A
24.23
39.63
52.82
29.97
69.21
65.18
0.04
0.03
1.89
2.55
1.96
Unlikclihood
1+
28.57
38.41
51.23
28.57
24.12
13.35
0.61
0.69
2,90
3.19
3.00
SimCTG 773
23.82
40.91
51.66
28.65
67.36
63.33
0.05
0.05
1.93
2.68
2.08
Quark
26.22
41,57
45.64
25.07
39.89
30.62
0.35
0.74
2.75
3.20
2.77
+Unlikelihood
27.97
39.41
37.76
19.34
18.76
12.14
0.67
0.82
3.92
4.04
3.87
Table 5: Unlearning repetitions of sequences generated from GPT2-base via greedy decoding, for the
WIKITEXT- 103 test set. Baselines results are adopted from 73
perplexity
accuracy
rep
diversity
42
80
0.7
0.9
metric
WW
25
45
0.3
0.5
22
40
27
0.2
0.2
20 40k
39.
80k
10
0.0
20c
Figure 2: Performance (y-axis) of Quark on WIKITEXT- 103 val set with respect to training step
(x-axis). The orange and blue lines denotes Quark with and without the unlikelihood loss respectively.
Baselines and evaluation metrics. We compare with maximum likelihood estimation (MLE), unlikelihood training (unlikelihood) [9]. and contrastive training (SimCTG) 773. In addition to comparing directly against these methods, Quark can be readily used in conjunction with these losses (see sseesctbse B.3 for details).
Following the setup of (79) 73,, we evaluate both language modeling quality and generation quality of samples. For language modeling, ground-truth continuations the the WIKITEXT- 103 test set, we report perplexity (ppl), token prediction accuracy (acc), prediction repetition (rep: the fraction of next-token repeating content from the prefix), and another variant of prediction repetition (wrep: single-token repeats that are different from the ground-truth next-token, since aaturally-occurring ground truth texts may also contain repetitions). For generation quality, report sequence-leve repetition, defined as the proportion of repeated n-grams (rep-n), diversity (díverse) as measured by fusion of different n-gram levels, and MAUVE (56), automatic measure of how much the generated text distribution diverges from that of human-written text. We additionally conduct human evaluations of the text generations on coherency (whether aligned in meaning/topic with the prompt), fluency (whether grammatical, easy-to-read, and non-repetitive) and overall quality; details of human evaluation are Appendix
Results. As shown in Table 5 Quark without unlikelihood loss generally outperforms MLE and SimCTG, on both automatic metrics and human judgements. Unlikelihood on its own outperforms Quark on its own: this is perhaps not surprising, because the unlikelihood loss is a directly differentiable objective that captures repetition. However, what is surprising the performance gain of combining Quark with the unlikelihood objective: this decreases repetition over either method independently. and improves human judgements of fluency, coherence, and overall quality by 35%, 27%, and 29% respectively compared to unlikelihood alone. As shown in Fig Quark without unlikelihood loss steadily improves the reward across training steps, and the additional unlikelihood loss accelerates the reward optimization process. Additional qualitative results are in Appendixx D
4 Model Ablations
In addition to showing the effectiveness of using Quark for unlcarning undesirable behaviors from language models, further conduct ablation studies explore the effect of each component of our training objective. We focus on the toxicity unlearning task for our ablation studies.
What effect does the KL term have? Fig 3 illustrates the effect of increasing the KL coefficient B (our default value is 3 -05), which encourages Po to stay closer to Po- This leads to lower perplexity and better language quality, but lower rewards, shown by the slight increase in toxicity.
7

---

toxic prob.
perplexity
dist-3
loxic prob.
perplexity
dist-3
0.09
13.5
0.05
0.17
15.0
0.85
metric
0.07
13.2
0.84
metrie
0.13
14.2
0.84
0.05
0.82
12.8
0.01
.55
.15
20
12.4
.05
1 .15
20
0.00
0
.05
15
20
0.00
12.0
0.80
Figure 3: Performance of Quark (y-axis) on RE-
Figure 4: Performance of Quark (y-axis) on REAL- TRXMPITYPOOPPS val set, with varying number of quantiles (x-axis).
ALTOXICITYPROMPA val set, with varying KL coefficient 3 (x-axis).
0.20
toxic prob.
15.0
perplexity
0.05
dist-3
0.65
quantile
metric
0.21
14.1
0.83
toxic prob.
0.49
0.14
0.80
0.33
0.16
0.00
12
16
11.50
12
16
0.76
0
8
12 16
0.00
24
4K
04
10x
12k
Figure 5: Performance of Quark (y-axis) on RE- ALTOXICITY PROMPTS val set, with varying fre- quency of exploration (x-axis) in terms of number of explorations per 8k gradient update steps.
Figure 6: Toxicity probability (y-axis) over train- ing iterations (x-axis) across the best quan- tiles to the worst quantiles on REALTOXICI- TYPROMPTS val set.
Explore strategy
Learn quantile
Toxicity (1)
Fluency (L)
Diversity (1)
Toxicity (1)
Fluency (L)
Diversity (1) dist-2 dist-3
avg. max. prob.
output ppl
dist-2 dist-3
KL term
avg.
prob.
output ppl
best-tok
all
0.194
0.035
12.72
0.79
0.83
without
0.192
0.031
13.29
0.79
0.83
random-tok
all
0.286
0.109
12.40
0.80
0.84
approx.
0,194
0.038
13.86
0.80
0.84
best-tok
best
0.115
0.014
21.92
0.43
0.66
exact
0.194
0.035
12.72
0.79
0.83
Po
all
0.291
0.183
12.53
0.78
0.80
best
0.263
0.146
0.73
0.77
Table 6: Ablations on different choices of KL term on val set: no KL, point-wise approximate KL, and token-level exact KL.
Table 7: Ablations on different design choices
for conditional reward tokens in exploration and quantiles to use learning on val set.
Exact KL vs. Approximate KI.. comperss the effect of our exact token-level KI. as defined po(-)p)a..r)
in Eq against an approximate point-wise KL, log po(-IMet.2,rr proposed by [71] Compared to
no KI. term, the exact KI. gives controllable trade-off between language quality and reward maximization, unlike the point-wise KI., which hurts both dimensions. We speculate the discrepancy is due to the noise introduced by approximating the distributional KI. via point-wise estimation.
What effect does the number of quantiles have? As shown in Fig increasing the number of quantiles results in more effective reward maximization and lower toxicity. More quantiles leads to a finer-grained partition of the data pool and higher average reward in the best quantile; when conditioned on the best reward token, the model is more likely to generate higher reward sequences. As trade-off, the model strays more from the original, yielding slightly worse language quality.
Can we just train on the highest-reward quantile? As shown in Table 7 all quantiles (row 1), training the best quantile only (row 3) leads to better reward maximization and lower toxicity, but a significant drop in both fluency and language diversity. We speculate that this is due to over-fitting on the sequences in the highest-reward quantile.
compared to training on
Can we condition on random reward tokens in exploration? As shown in Table 7 compared to conditioning the best reward token (row 1) in exploration, conditioning on uniformly sampled reward tokens (row 2) leads to much worse reward maximization and much higher toxicity. While the former focuses exploration the most promising regions, the latter does uniform exploration the action space, which reduces the chance of discovering better trajectories to enhance the datapool.
Are control codes useful for exploration and training? Row 4 of Table illustrates performance decreases when the initial policy Po is used for exploration instead of reward code conditioned policy Po; Row illustrates performance decreases when Po has no control code for both training/exploration, even when the high reward samples are added to the data pool.
8

---

How do the rewards for generations in each partition evolve over time? As demonstrated in Fig [ for all quantiles, toxicity monotonically decreases across training iterations; and for an arbitrary
iteration, toxicity monotonically decreases from the worst quantile to the best quantile.
What effect does the frequency of exploration have? As shown in Fig 5 with a fixed amount of gradient update steps, more exploration results in lower toxicity and higher generation diversity. Intuitively, more exploration leads to larger data pool with a better reward distribution, which benefits reward maximization and language diversity. Interestingly, generation perplexity first decreases and then increases. We speculate the mitial decrease is due the larger datapool alleviating over- werr-fiting. and the later decrease is due to the trade-off between language quality and reward maximization as we attain lower toxicity.
Related Work
Reinforcement Learning in NLP. Previous works have used RL techniques in wide range of
classical NLP application such as named entity recognition [4]. semantic parsing [90), dependency
parsing 80) constituency parsing [16), part-of-speech tagging [ and information extraction
[4]
Recent works have explored applying RI. on tasks such as guestion-answeriing
85
86
85
summarization [59] 54, 71 61
17 [22, and machine translation (59] 88] 80, 83
82
67
50]
Some other works at the intersection of language and other modalities also use RL techniques, c.g., navigation 77
76). multi-agent communication [35], image captioning (59] 6 60). etc. RL has also been used to train language models to align with models of human preferences and values 99 24 3. In the domain of open-text generation, REINFORCE (75 and PPO [2 have been used for controllable story generation, and soft Q-Learnig [20] has been applied to generate prompts for steering language model generations. Finally, prior work has used RL techniques to generate language grounded in text-based narrative games 23
Reinforcement learning with transformers. Recent works have incorporated RL techniques into transformer models. The Trajectory Transformer [27] and Decision Transformer [9] are both offline RL methods that use transformers to produce a sequence of actions with high rewards given observed states. Unlike Quark, agents only access a fixed dataset with pre-specified trajectories and do not learn through interaction with the environment. Zheng et al. [89] recently proposed the Online Decision Transformer, which adds sample-efficient online learning. [72] uses PPO to incorporate human feedback for summarization.
Unlearning undesirable behaviors from language models. Unlearning behavior in language models is similar to model-editing 122 45 but for rewards rather than datapoints. Some recent works use RL for post-hoc modification language models, e.g., unlearning toxicity [14] or non-normative generations [55]
Complementary pre hoc methods aim to avoid learning undesired behavior at
38 77 Similarly, methods for controlling models at inference time, e.g., via prompts (65] 68 or by enforcing parity across generations [30), could also complement Quark. [34] recently proposed Generative Cooperative Networks; while methodologically similar to Quark, their work is inspired by GANs, and thus the focus is on training models such that discriminator cannot readily identify machine vs. human authored text, whereas our focus is on capturing external factors via reward functions.
training time 779
6 Conclusion
In this work, we introduce Quark, a simple but effective method for reward optimization to unlearn undesirable properties of language models acquired during pretraining. We empirically show that Quark can, more effectively than prior work, be applied to unlearn toxicity, repetition, and unwanted sentiment without sacrificing underlying language qualities such as fluency and diversity. Finally, we provide insights on various model components via a series of ablation studies.
Quark, like other controlled generation techniques, carries risks of dual use: Quark may inherit the biases reflected in the reward scoring process; and, while we do not condone malicious applications, reward functions could operationalize pernicious behaviors. We foresee Quark as tool for encouraging language generators to behave in specific ways, but not as a tool that guarantees safety, no toxicity, or outputs that reflect negative social biases. We discuss further in Section 7

---

Future directions include:
1. investigating adaptations of Quark for controlling multiple rewards simultaneously; exploring more diverse types of rewards, e.g. those related to human preferences; 3. and training Quark with fewer parameters optimizing all model parameters.
7 Additional Ethical Considerations
In this work, we show that Quark can steer language models away from unwanted properties as speci- fied by reward functions, without sacrificing general language understandingggenrraal capabilities. We foresee two primary dual use concerns for this method.
First, as with any controllable text generation technique, Quark could be used steer language models towards malicious behaviors. While we encourage those who deploy language technologies to con- sider potential negative impacts, and don't intend Quark be used for manipulation, misinformation, etc., we foresee the marginal risks introduced by our method specifically as minimal. Malicious actors, in theory. can already adapt language models for malicious use cases without reward optimization. Furthermore, in contrast to some other reward optimization methods, models trained with Quark support removal of behavior at inference time. Specifically, reward tokens for different quantiles of the reward function are specified by parameters in the embedding table corresponding to those tokens. Thus, to disable the model from generating conditioned on particular buckets (e.g.. high toxicity quantiles), those parameters can simply be removed/erased for a public release. While this doesn't i fully mitigate undesirable behavior, our experiments clearly show high correlation between conditioning particular quantiles and corresponding rewards, thus, the rate of undesirable behavior likely to decrease if specific quantiles cannot conditioned on.
Second, reward functions may misspecify desired characteristics in subtle ways that reflect pernicious social biases. particularly if they are black-box APIs large, For example, for the task of unlearning toxicity, since the toxicity reward is dependent upon the Perspective API, our model checkpoints inherit the biases and limitations of the API. While we undertake human evaluations for our experiments to confirm that our model really is outputting less toxic language on REALTOXICITYPROME can encourage language models to generate higher reward outputs for a given reward function. As more accurate, specific, and inclusive classifiers are built (e.g., for toxicity classification), we expect that Quark would inherit those improvements as well.
neural networks.
Quark not a panacea. We foresee Quark as tool that
8 Acknowledgements
We thank Jena Hwang, Sarah Wiegreffe, and the anonymous reviewers for the helpful discussions and feedback. Additionally, thank the Google Perspective API team for supporting our quota increase requests. This research was supported part by Natural Sciences and Engineering Research Council of Canada (NSERC) (funding reference number 401 401233309), DARPA MCS program through NIWC Pacific (N66001-19-2-4033 Google Cloud Compute, a Microsoft PhD Fellowship, and the Allen Institute for AI.
10

---

References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
[2] Amal Alabdulkarim, Winston Li, Lara J. Martin, and Mark O. Riedl. Goal-directed story generation: Augmenting generative language models with reinforcement learning, 2021.
[3] Prithviraj Ammanabrolu, Liwei Jiang, Maarten Sap, Hanna Hajishirzi, and Yejin Choi. Aligning social norms and values interactive narratives. In NAACL, 2022.
[4] Prithviraj Ammanabrolu, Jack Urbanek, Margaret Li, Arthur Szlam, Tim Rocktäschel, and Jason Weston. How motivate your dragon: Teaching goal-driven agents speak and act in fantasy worlds. In Proceedings of 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, 2021.
[5] Michael Auli and Jianfeng Gao. Decoder integration and expected BLEU training for recurrent neural network language models. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 136-142, Baltimore, Maryland, June 2014, Association for Computational Linguisties.
[6] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, page 1171-1179, Cambridge, MA, USA, 2015. MIT Press.
[7] Shikha Bordia and Samuel R. Bowman. Identifying and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 7-15, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics.
[8] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Hersert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
[9] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In A. Beygelzimer, Y. Dauphin, Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[10] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Kathy
[11] Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. All that's' human'is is not gold: Evaluating human evaluation of generated text. arXiv preprint 21Xi::20770000111 2021.
11

---

[12] Sumanth Dathathri, Andrea Madotto. Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: simple approach to controlled text generation. In International Conference on Learning Representations, 2020.
[13] Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. Classical structured prediction losses for sequence to sequence learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Compulational Linguis- lies: Human Language Technologies, Volume I (Long Pupers), pages 355-364, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
[14] Farshid Faal, Ketra Schmitt, and Jiawei Yu. Reward modeling for mitigating toxicity in transformer-based language models. ArXiv, (es/22022.09662, 2022.
[15] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Australia, July 2018, Association for Computational Linguistics.
[16] Daniel Fried and Dan Klein. Policy gradient proxy for dynamic oracles in constituency parsing. In Proceedings of the 56th Anmual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 469-476, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[17] Yang Gao, Christian M. Meyer, and Iryna Gurevych. APRIL: Interactively learning to summarise by combining active preference learning and reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4120-4130, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
[18] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Real- ToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356-3369, Online, November 2020. Association for Computational Linguistics.
[19] Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http:://Skylion007.grz / OpenNebTexttCrruu 2019.
[20] Han Guo, Bowen Tan, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. Text generation with efficient (soft) q-learning, 2021.
[21] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo. Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342- 8360, Online, July 2020. Association for Computational Linguistics.
[22] Peter Hase, Mona T. Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanow, Mohit Bansal, and Srini lyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. ArXiv, abs/21 13654, 2021.
[23] Matthew Hausknecht, Prithviraj Ammanabrolu, Côté Alee-nlex,,dre, and Yuan Xingdi. Inter- active fiction games: A colossal adventure. In AAAI, volume
005398, 2020.
[24] Dan Hendrycks, Mantas Mazeika, Andy Zou, Sahil Patel, Christine Zhu, Jesus Navarro, Dawn Song, Bo Li, and Jacob Steinhardt. What would jiminy cricket do? towards agents that behave morally, 2021.
[25] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
[26] Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. Deceiving google's perspective api built for detecting toxic comments, arXiv preprint arXiv: 172.v:1702.088133 2017.
[27] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In A. Beygelzimer, Dauphin, Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
12

---

[28] Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. ArXiv, hbss/199.05888, 2019.
[29] Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. Curl: A conditional transformer language model for controllable generation, 2019.
[30] Muhammad Khalifa, Hady Elsahar, and Marc Dymetman. A distributional approach to con-
trolled text generation. In International Conference on Learning Representations, 2021.
[31] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988, 2014.
[32] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, I, pages 4929-4952, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[33] Klaus Krippendorff. Content analysis: An introduction to its methodology. Sage publications, 2018.
[34] Sylvain Lamprier. Thomas Scialom, Antoine Chaffin, Vincent Claveau, Ewa Kijak, Jacopo Staiano, and Benjamin Piwowarski. Generative cooperative networks for natural language generation. In ICMI., 2022.
[35] Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7663-7674, Online, July 2020. Association for Computational Linguistics.
[36] Hung-yi yi Lee, Cheng- Hao Ho, Chien-Fu Lin, Chiung-Chii Chih Chang, Chih-Wei Lee, Yau-Shian Wang, Tsung- Yuan Hsu, and Kuan- Yu Chen. Investigation of sentiment controllable chatbot. arXiv preprint 20XXx:2007.07196, 2020.
[37] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. In Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[38] Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715 4728, Online, July 2020. Association for Computational Linguistics.
[39] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages pages
Online, August 2021. Association for Computational Linguistics.
[40] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages Linguistics.
Online, August 2021. Association for Computational
[41] Runjing Liu, Jeffrey Regier, Nilesh Tripuraneni, Michael I. Jordan, and Jon D. McAuliffe. Rao-blackwellized stochastic gradients for discrete distributions. In ICML, 2019.
[42] Francis Maes, Ludovie Denoyer, and Patrick Gallinari. Structured Prediction with Reinforce- ment Learning. Machine Learning, 77(2-3):271300) December 2009.
13

---

[43] Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized entropy regularization or: There's 's nothing special about label smoothing, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6870-6886, Online, July 2020. Association for Computational Linguistics.
[44] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2017.
[45] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022.
[46] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchin- son, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In FAccT, 2019.
[47] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 339-849, San Diego, California, June 2016. Association for Computational Linguistics.
[48] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browscr-assisted guestion-answering with human feedback. CoRR. abs/2112.09332, 2021.
[49] Karthik Narasimhan, Adam Yala, and Regina Barzilay. Improving information extraction by acquiring external evidence with reinforcement learning. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2355-2365, Austin, Texas, November 2016. Association for Computational Linguistics.
[50] Mohammad Norouzi, Samy Bengio, zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. Reward augmented maximum likelihood for neural structured prediction. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016
[51] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.022 02155, 2022.
[52] Ramakanth Pasunuru and Mohit Bansal. Multi-reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 646-653, New Orleans, Louisiana, June 2018, Association for Computational Linguistics.
[53] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[54] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018.
[55] Xiangyu Peng, Siyan Li, Spencer Frazier, and Mark O. Riedl. Reducing non-normaeive text generation from language models. In INLG, 2020.
[56] Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. In NewrIPS, 2021.
[57] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019.
14

---

[58] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbi, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Pagannii, Laurent Sifre, Lena Martens, Xiang Lor- raine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean- -aaattttttttee Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher, 2021.
[59] Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. ICLR, 2016.
[60] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel. Self-critical sequence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1179-11221 1199--199,,,191 Los Alamitos, CA. USA, jul 2017. IEEE Computer Society.
[61] Seonggi Ryang and Takeshi Abekawa. Framework of automatic text summarization using reinforcement learning. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 256-265, Jeju Island, Korea, July 2012. Association for Computational Linguistics.
[62] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108 108, 2019.
[63] Chinnadhurai Sankar and Sujith Ravi. Deep reinforcement learning for modeling chit-chat dialog with discrete attributes. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogne, Stockholm, Sweden, September 2019. Association for Computational Linguisties.
[64] Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. The risk of racial bias in hate speech detection. In ACL, 2019.
[65] Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nip. Transactions of the Association for Computational Linguistics, 9: 408-1424, 2021.
[66] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06344) 2017.
[67] Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683-1692, Berlin, Germany, August 2016. Association for Computational Linguistics.
[68] Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. Towards Controllable Biases in Language Generation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3239-3254 Online, November 2020. Association for Computational Linguistics.
[69] Emily Sheng, Kai-Woi Chang, Prem Natarajan, and Nanyun Peng. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 4275 4293, Online, August 2021. Association for Computational Linguistics.
15

---

[70] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
[71] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Curran Associates, Inc., 2020.
Information Processing Systems, volume 33, pages 2008-3022
[72] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33. pages
Curran Associates, Inc., 2020.
[73] Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation, 2022.
[74] Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. "transforming" delete, retrieve, generate approach for controlled text style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLLP) pages 3269-3279, Hong Kong, China, November 2019. Association for Computational Linguistics.
[75] Pradyumna Tambwekar, Murtaza Dhuliawala, Lara J. Martin, Animesh Mchta, Brent Harrison, and Mark O. Riedl. Controllable neural story plot generation via reward shaping. In Proceedings of the EightyEighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5982 5988. International Joint Conferences on Artificial Intelligence Organization, 7 2019.
[76] Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and -nd--ag navigation. Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedingy of Machine Learning Research, pages 406. PMLR, 30 Oct-01 Nov 2020.
[77] Xin Eric Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross cross-modee modal matching and self- supervised imitation learning for vision-language navigation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6622-6631, 2019.
[78] Anuradha Welivita, Yubo Xie, and Pearl Pu. A large-scale dataset for empathetic response generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1251-1264, 2021.
[79] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In International Conference on Learning Representations, 2020.
[80] Sam Wiseman and Alexander M. Rush. mization. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pages 1296-1306, Austin, Texas, November 2016. Association for Computational Linguistics.
learning as beam-search opti-
[81] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transform- ers: Sfate-of-the-arn natural language processing. arXiv preprint arXiv:1910.03771, 2019.
[82] Lijun Wu, Yingee Xia, Fei Tian, Li Zhao. Tao Qin, Jianhuang Lai, and Tie- Yan Liu. Adversarial neural machine translation. In Jun Zhu and Ichiro Takeuchi, editors, Proceedings of The 10th Asian Conference on Machine Learning, volume 95 of Proceedings of Machine Learning Research, pages 534-549. PMLR. 14-16 Nov 2018.
16

---

[83] Yonghui Wu, Mike Schuster, Z. Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason R. Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv, abs/1609.08144 2016.
[84] Caiming Xiong, Victor Zhong, and Richard Socher. DCN+: Mixed objective and deep residual coattention for question answering. In ICLR, 2018.
[85] Xingdi Yuan, Marc-Alexandee Côté, Jie Fu, Zhouhan Lin, Chris Pal, Yoshua Bengio, and Adam Trischler. Interactive language learning by question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNE) pages 2796-2813, Hong Kong, China, November 2019. Association for Computational Linguistics.
[86] Xingdi Yuan, Jie Fu, Marc-Alexanine Alexandre Côté, Yi Tay, Chris Pal, and Adam Trischler. Interactive machine comprehension with information seeking agents. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3325-2338, Online, July 2020. Association for Computational Linguistics.
[87] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
[88] Wen Zhang, Yang Feng, Fandong Meng. Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4334 4343, Florence, Italy, July 2019. Association for Computational Linguistics.
[89] Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer, 2022.
[90] Victor Zhong, Caiming Xiong, and Richard Socher. Seq2SQL: Generating structured queries from natural language using reinforcement learning, 2018.
[91] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1909009999 2019.
17

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(C) Did you discuss any potential negative societal impacts of your work? [Yes], see
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? Yes] We will release
the code for Quark at /ettts://githan tiigthtu...eo www.hhuw.ooooeooeoooeeeeeeeeeeee htam//iiiinggLuueee 2022.
prior to NeurIPS
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See $3
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No] Due computational resource constraints, we didn't run multiple cross validaaai splits, or with enough random seeds to form stable confidence intervals. However, we do a thorough set of ablations across many domains and model configurations, see $
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See S
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] don't 't introduce new datasets, and refer readers to the original releases in case license information for those works changes.
(c) Did you include any new assets either in the supplemental material or as a URL? [No] We plan to release code, but have not yet due to internal review processes, but we commit to releasing code that enables use of Quark.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] All data we experiment with is public.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] We aren't 't releasing data, and existing corpora, to our knowledge and in our experience, do not contain personally identifying information.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See A
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] Crowdworking studies involving no personal disclosures of standard NLP corpora are not required by our IRB to be reviewed by them. Specifically:
i. We do not collect personal information. Information gathered is strictly limited to general surveys about the quality of generated text.
i. We take precaution anonymize Mechanical Turk WorkerlDs in manner that the identity of the human subjects cannot be readily ascertained (directly or indirectly). ii.. We do not record or include any interpersonal communication or contact between investigation and subject.
18

---

Crowdworking studies involving no personal disclosures of standard computer vision corpora are required by our IRB be reviewed by them. While we are lawyers, the opinion is based on United States federal regulation 45 CFR 46, under which this study qualifies and as exempt and does not require IRB review.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [Yes]. our pay is always over $15 USD per hour on average (and sometimes more, see & A
19


Reviews:
[{"review_id": "yxH_Mjb0WcQ", "reviewer": "Reviewer_BoVo", "summary": "This work proposes a reinforcement learning algorithm to achieve specific goals of language model generation, by incorporating three stages: exploration, quantization and learning. In particular, the language model is finetuned with a KL-divergence penalty. In generation stage, the language model conditions on high-reward token so that the not-expected property will be less generated. Empirical results show its effectiveness on unlearning toxicity, negative sentiment, and repetition, evaluated by both automatic evaluation metrics and human evaluation.", "questions": "1. What is the time complexity of the algorithm? How much more time will it need compared to baselines?\n\n2. Does QUARK has a high requirement for memory? e.g. how large is expected for the data pool to store the (input, output, reward) examples?", "limitations": "See weakness", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Clear presentation, easy to follow\n2. Comprehensive ablation analysis to show effects of hyperparameters and different settings\n\nWeakness\n\nThe current framework requires specific reward design for different goals (i.e.unlearning unwanted sentiment or toxicity). In general, language generation requires many desired characteristics and the current design may not serve as an appropriate tool for this goal.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qgkB6zaczAB", "reviewer": "Reviewer_5ES9", "summary": "Large-scale language models may generate text with offensive or toxic language, or repetition. The authors propose an algorithm, named Quantized Reward Konditioning (Quark), to unlearning these misalignments by fine-tuning the language model on signals of what not to do. Quark optimizes a reward function that quantifies an (un)wanted property and contains three steps: exploration, quantization, and learning. The experimental results on unlearning toxicity, negative sentiment, and repetition show the effectiveness of the proposed algorithm.", "questions": "N/A", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The authors propose a novel task to unlearn behaviors misaligned with user expectations and design a simple and effective algorithm to achieve this goal.\n- The proposed algorithm can rely on standard language modeling primitives to conduct training without additional models or specialized heuristics to stabilize training.\n- Extensive experiments show that Quark can be applied to unlearn toxicity, repetition, and unwanted sentiment without sacrificing underlying language qualities such as fluency and diversity.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dRKalgfGtQ_", "reviewer": "Reviewer_d6FS", "summary": "**What is the task?**\nTask of unlearning undesirable properties of language models acquired during pretraining by fine-tuning the language model on signals of what not to do.\n\n\n**What has been done before?**\n* In contrast to strong contemporary RL methods that stabilize training with an additional parameterized model (more learnable parameters) and specialized optimization heuristics, Quark’s training relies only on standard language modeling primitives. Experiments across three tasks demonstrate that Quark maintains pre-training abilities while unlearning undesired behaviors more stably than alternative methods.\n\n* Quark takes inspiration from three disjoint concepts from previous work in reinforcement learning and conditional language modeling but builds upon each concept in a novel manner.\n\n**What are the main contributions of the paper?**\n* Introduced Quark, an algorithm for optimizing a reward function that quantifies an (un)wanted property, while not straying too far from the original model.\n\n* For unlearning toxicity, negative sentiment, and repetition, our experiments show that Quark outperforms both strong baselines and state-of-the-art reinforcement learning methods like PPO, while relying only on standard language modeling primitives.\n\n* In addition to showing the effectiveness of using Quark for unlearning undesirable behaviors from language models, author(s) conducted ablation studies to explore the effect of each component of the training objective.\n\n**What are the key techniques used to tackle this task?** An online, off-policy reinforcement learning (RL) algorithm used to (un)learn properties from language models via three iterative stages: exploration, quantization, and learning.Quark alternates between (i) collecting samples with the current language model, (ii) sorting them into quantiles based on reward, with each quantile identified by a reward token prepended to the language model’s input, and (iii) using a standard language modeling loss on samples from each quantile conditioned on its reward token, while remaining nearby the original language model via a KL-divergence penalty.  By conditioning on a high-reward token at generation time, the model generates  text that exhibits less of the unwanted property.\n\n**What are the main results?**\nFor unlearning toxicity, negative sentiment, and repetition, our experiments show that Quark outperforms both robust baselines and state-of-the-art reinforcement learning methods like PPO, while relying only on standard language modeling primitives.\n\n", "questions": "Is it possible to obtain significance tests for table 1 and table 3 since difference between Quark and PPO look small.", "limitations": "Authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* Novel method for reward optimization to unlearn undesirable properties of language models acquired during pretraining\n* SOTA automatic as well as human evaluation results on 3 different tasks \n* Very nice insights on various model components via a series of ablation studies.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cHgTZX5I006", "reviewer": "Reviewer_wjcx", "summary": "This paper proposes an approach to training language models to unlearn various undesirable behaviors, using a reward function. The authors propose to do this by iterating the final 3 steps of the following procedure: sample from the model and obtain corresponding reward values, sort the resulting generations into k-quantiles, fine-tune the model on these generations conditioned on the corresponding k-quantile, but with a KL-penalty that prevents the model from deviating too far from the initial model, and then re-sample generations conditioned on the highest k-quantile. The authors evaluate their approach in its ability to prevent generation of toxic text, to flip sentiment, and to avoid repetition. They find their approach outperforms baselines in terms of automatic and human evaluation.", "questions": "For the ablation experiments in Table 7: does the experiment in the last row, where the model is trained just on the \"best\" quantile, ensure that the model still sees as much data as the models in the previous two rows which are trained on \"all\" the quantiles?", "limitations": "Yes, the authors address limitations and potential negative societal impacts adequately.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly written.\n- The proposed approach is simple and intuitive.\n- The authors do a thorough evaluation, and the results are impressive.\n- The authors include very interesting ablation experiments.\n\nWeaknesses:\n- Although the authors consider many ablations, since Quark does seem quite similar to PPO-like approaches, I think we could have benefited from seeing the results of some other baselines, including: \n    - Fine-tuning on the resampled outputs (from the top quantile) but without conditioning them on the quantile control token. This tests whether using control tokens is important.\n    - Doing “exploration” with the original model $p_0$ rather than $p_\\theta$, which simply amounts to fine-tuning (with control tokens, say) on a larger corpus that has been labeled by the reward function. This seems like the simplest fair baseline, and checks whether it is even important to keep the samples up-to-date with the current model (which is in any case being constrained to stay close to $p_0$).\n- Similar to the previous point, it’s not clear to me that the baselines from prior work are especially fair. For instance, DAPT is somewhat similar to the proposed second baseline above, except it is fine-tuned on a different corpus. Presumably this baseline would be more competitive if it were fine-tuned using results of the toxicity classifier being used to evaluate, as Quark is.\n- If the paper is interested in positioning itself as being about an RL technique or reward maximization, I would expect experiments to report the actual reward achieved by Quark or any of the baselines. \n\nUpdate after author response: thanks for your responses and for the new results. I'm increasing my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept