On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting
Authors: Tomasz Korbak
Hady Elsahar
Germán Kruszewski
Marc Dymetman
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Language Models
Reward Maximization
Distribution Matching
Energy Based Models
Controlled Text Generation

Abstract: The availability of large pre-trained models is changing the landscape of Machine Learning research and practice, moving from a "training from scratch" to a "fine-tuning'' paradigm. While in some applications the goal is to "nudge'' the pre-trained distribution towards preferred outputs, in others it is to steer it towards a different distribution over the sample space. Two main paradigms have emerged to tackle this challenge: Reward Maximization (RM) and, more recently, Distribution Matching (DM). RM applies standard Reinforcement Learning (RL) techniques, such as Policy Gradients, to gradually increase the reward signal. DM prescribes to first make explicit the target distribution that the model is fine-tuned to approximate. Here we explore the theoretical connections between the two paradigms and show that methods such as KL-control developed in the RM paradigm can also be construed as belonging to DM. We further observe that while DM differs from RM, it can suffer from similar training difficulties, such as high gradient variance. We leverage connections between the two paradigms to import the concept of baseline into DM methods. We empirically validate the benefits of adding a baseline on an array of controllable language generation tasks such as constraining topic, sentiment, and gender distributions in texts sampled from a language model. We observe superior performance in terms of constraint satisfaction, stability, and sample efficiency.

On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting
Tomasz Korbak*
Hady Elsahar Naver Labs Europe hady elsahar@gmail........... com
University of Sussex
tomasz. kowaakkgaaaaaaaaooooooooooooo
Germán Kruszewski
Mare Dymetman aarc.dyymetman@gaail.. com
Naver Labs Europe
german kruszewski wnwerraab.ecoeoroooooooooo
Abstract
The availability of large pre-trained models is changing the landscape of Machine Learning research and practice, moving from a "training from scratch" "fine- tuming" paradigm. While in some applications the goal is to "nudge" the pre-trained distribution towards preferred outputs, others is to steer it towards different distribution over the sample space. Two main paradigms have emerged to tackle this challenge: Reward Maximization (RM) and, more recently, Distribution Matching (DM). RM applies standard Reinforcement Learning (RL) techniques, such as Policy Gradients, to gradually increase the reward signal. DM prescribes first make explicit the target distribution that the model is fine-tuned to approximate. Here we explore the theoretical connections between the two paradigms, and show that methods such as KL-control developed for RM can also be construed as belonging to DM. We further observe that while DM differs from RM, can suffer from similar training difficulties, such high gradient variance. We leverage connections between the two paradigms to import the concept baseline into DM methods. We empirically validate the benefits of adding baseline controllable language generation tasks such as constraining topic, sentiment, and gender distributions in texts sampled from language model. We observe superior performance in terms of constraint satisfaction, stability and sample efficiency.
array of
Introduction
Pre-trained language models (Devlin et al. 2019 Radford et al. 2019) are changing the landscape of Machine Learning research and practice. Due to their strong generative capabilities many studies have found sufficient to "nudge" these models to conform to global preferences defined over the generated sequences instead of training from scratch using annotated data. These preferences could include topic and sentiment (Dathathri et al. 2020), valid musical notes and molecular structures (Jaques et al. 2021), reducing distributional biases (Khalifa al.) 2021 Weidinger et al. 2021), evaluation metrics for Machine Translation and Summarization (Ranzato al. 2016 Bahdanau al., 2016), or direct human feedback (Ziegler et al. 2019 Stiennon al.
2017a), code compilability (Korbak
2020p. This large body of studies is driven by two different paradigms: Reward Maximization (RM) and Distribution Matching (DM).
Work partly done during an internship at Naver Labs Europe. Independent Researcher. Work done Naver Labs Europe.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Problem: Progginn pretrained language model with human preferences
pretrained LM
constraint on LM samples
aligned LM
Approach Reward maximization with KL penalty
Approach Distribution matching
maximiza ze reward while minimizing KL from the pretrained LM
minimize KL fram a target distribution
Findings:
Findings:
1. does not reduce reward maximizationn (sec. 3.3)
1. reduces to distribution matching (sec. 32)
2. benefit from variance reduction techniques (sec.
Figure I: In this study we make a connection hetween two popular paradigms for aligning language models to human preferences: Reward maximization (RM) and Distribution matching (DM).
Reward Maximization RM intuitively nudges pre-trained models towards certain preferences by providing global sequence-level rewards when the model generates outputs that satisfy desired features. For instance, if the model is producing toxic content, (RL) techniques to discourage it from producing similar content. However, naively applying RL yields model that undergo catastrophic forgetting forgeetnn of its original distribution. For example, it can degenerate into producing a single nonsensical but at least nontoxic sequence. Although several studies have considered hand-crafting general rewards ensure desirable features like fluency (Liu ct al. 2016aa Tambwekar et al. (2019), coming up with complete or perfect rewards is highly non- trivial (Wu al. 20166] Vedantam et al., 2013). This has sparked wide discussion on the overall effectiveness of RM for some tasks such as machine translation (Choshen el al., 2020
apply Reinforcement Learning
Kiegeland &
Kreutzer 202T
To tackle the aforementioned issues of "catastrophic forgetting" several studies, still under an RM paradigm, have considered incorporating distributional term inside the reward to be maximized. In particular Jaques et al. 20] 2019) and /Zigllr et al.
Reward Maximization with KL-Control
(2019) more recently SStiennon al. (2020). Ouyang et al. (2022), Bai al. (022) and Perez et al. 2022) have applied variations of KL-control [Todorov] 2007 Kappen et al,, 2012) which adds penalty term the reward so that the resulting policy does deviate too much from the original one in terms of KL-divergence. The overall objective with the KL-penalty is maximized using an RL algorithm of choice including: PPO (Schulman al. 2017a) in Ziegler al. (2019) or Bai et al. (022) or Q-learning (Mnih et al., 2013) as in Jaques et al. (2017)). Adding this distributional KL KL-penalty the reward raises some important questions: What effect does have the shape of the optimal policy? Does this new objective have any interpretation from a distributional perspective?
Distribution Matching A different recent paradigm for fine-tuning language models satisfy downstream preferences formulates the problem as Distribution Matching (DM). This paradigm consists of two steps: first a target distribution incorporating the desired preferences defined as an Energy-Based Model (LeCun et al.) 2006). Then the forward KL divergence is minimized between this target distribution and auto-regressive policy using a family of algorithms referred to as Distributional Policy Gradients (DPG) (Parshakova et al. 2019b; Khalifa et al., 2021 Korbak al. 2021 2022a). This approach capitalizes on the flexibility EBMs specifying the target distribution. For example, the EBM can be defined so that it conforms to all downstream preferences while its corresponding normalized distribution has a minimal KL divergence from the original, pre-trained language model, therefore tackling the problem of "catastrophic forgetting" (Khalifa et al. 202T). Interestingly, this DM paradigm can also deal with distributional preferences, for instance, for de- de-biasing language models by specifying that the generated sequences should be gender-balanced. balanced,

---

L.e. that 50% of generations contain female mentions. Such distributional constraints cannot be defined in the RM paradigm where a reward is calculated for a single sequence.
We can notice the promises and limitations of these two paradigms for fine-tuning language models. RM approaches equipped with arsenal of RL algorithms and optimization techniques that can be efficient in reward maximization, however they lack the distributional aspect to avoid catastrophic forgetting and impose distributional preferences over LMs. DM approaches are suited tackle those limitations, however, the family of DPG algorithms currently used is not as rich as its RL counterpart.
While the connections between these two seemingly distinct paradigms have been noted (Parshakoval al. 2019b Korbak et al. 2022b), they have not been explored in detail. Clarifying such connections might help import ideas from one approach to the other. This is our goal in this paper, detailing the nuanced connections and applying them to a case-study in variance reduction. Overall, our contributions are the following:
We clarify relations between the RM and DM paradigms through a detailed comparison between the family of DPG algorithms and Policy Gradients (Table D stressing the differences between parametric and non-parametric rewards that are important in this regard.
We introduce an interpretation of KL--cntrool techniques from distribution matching perspective, placing such techniques at an intermediate place between RM and DM (Theorem T.
We show how these connections can enable cross-polliiattin applying baselines- choice of baseline (Facts | and 2 On an array of controllable language generation experiments, we show that adding baselines leads to superior performance constraint satisfaction (Figure B
between the two perspectives by variance reduction technique from RL - to DPG and derive particular
stability on small batch sizes, and sample efficiency (Figure
2 Background
Standard Policy Gradients One popular method for adapting the behaviour of language models to certain preferences has been that of assigning "reward" score R(rr) for sequences sampled from an autoregressive language model (policy) xg. Then, the simplest policy gradient algorithm in reinforcement learning, namely, REINFORCE (Williams) 1992a), aims to find the policy no (x) that maximizes the average reward Ez-wa R(x), and this leads, via the so-called "log derivative trick", to gradient ascent algorithm that iteratively samples from To and update parameters by increments proportional to R(xx) Ve log Tx(x) via the following identity:
V Velawwa R(x) = Ex l (x)).
(I)
KI--ontrool (ooorov 2007
Kappen et al. 2012). was leveraged by Jaques et al. 201 2019) and Ziegler al. 2019) to include a KI. penalty term in the reward function to penalize large deviations from the original pretrained model a(x), weighted by free hyperparameter ,9 to control the trade-off
between the two goals. That is, they maximize the expectation E R2 where:
To(x) a(x)
Rz(x) r(x) 3 log
(2)
Distributional Policy Gradients (DPG) (Parshakova et al. 2019h) a recent approach used to fit an autoregressive policy to the distribution p(z) P(x) 7 induced by the EBM P(x), where Z = Σ., P(z) is the normalization constant (partition function). Given an arbitrary EBM P(x), DPG optimizes the loss function (KL(p, TO) with respect to the parameters 0 of an autoregressive model Te, a loss which minimized for F. = p. The KL-divergence minimization objective leads to gradient estimate of the form:
DKL RO)=
V VoEan To())
(3)
Σ p(x) log TT()) = Z
Σ x
P(:) log (x)
(4)
((x)
E Eg To
Vo logmo(())
(5)
Z
3

---

3 Reward Maximization vs Distribution Matching
In the previous section, we have summarized three approaches that have been suggested for fine-tuning language models. Two of them can be characterized as "Reward Maximization" (RM): Standard Policy Gradients (PG) and KL-control. On the other hand, DPG clearly belongs to the realm of "Distribution Matching" (DM) as it first defines the target distribution and then optimizes policy to match it. In the rest of this section, we will explore connections between these two seemingly distinct concepts and, in the following section, we will exploit them to improve DM-based methods.
3.1 Standard vs. Parametric Rewards
Let us start with distinguishing between a "parametric reward" Rg which depends on 0 and standard reward Rt which does not. If we wished to maximize the expected parametric reward, ET, Ro(x), we would follow its gradient, leading to the identities:
V Volaawa Ro(x)= V Σ To(x) Ro(x) E TTO + Σ Ro (R)
(6)
Σ To(c)VoRo(2) + Σ log
EZNNT Vo
E Egens (o(x) log (o)
RG
PG-term
Equation (8) the sum two terms: the first one, the "RG-term" (Reward Gradient term), involves the gradient of the reward. The second one, the "PG-term" (Policy Gradient term), was obtained using the "log derivative trick" and involves the gradient of the policy stricto sensu. In standard RL, where the reward does not depend on 0 the RG-term disappears and the gradient of expected reward consists solely of the PG-term. However, when Rg depends on 8, the gradients are distinct (apart from specific cases where the RG-term evaluates to O, as we will see below).
3.2 KL-control as Distribution Matching
Adding a KL-penalty term to the reward (as in the case of KL-control) leads to parametric reward. However, due to the particular form of its objective, the RG-term actually vanishes leaving only the PG-term By R2 (x) Vo log To(x) and simplifying the tuning procedure to standard Policy Gradient. While this algorithm falls under the RM paradigm, here we argue that is its nature is multifaceted, and explore deeper connections with the DM paradigm. More precisely, the maximization of reward with the KL penalty term is equivalent to distributional matching with underlying emergent sequential EBM, a remark that already reveals some similarities with DPG
Theorem 1. Consider the following EBM: Pz(x)= = a(x))= (a)//
(9)
and let Ps be the normalized distribution (x)) 3 Pa(x), with 2 E. Pa(x). Then:
(i) arg max ENNTY R5(x) arg min,, DKL (ec.(T)) (To. Pz);
(ii) arg max mED())(X) E R2 (x) = P2. where D(X) the family of all distributions over X. and (z) r(x) log [(x)
a(x)
Proof. A simple way to prove this is notice that the expectation of the reward Rž has a monotoni- cally decreasing relationship with the reverse KL divergence between me and ps:
1 DKL(TO. p2) = Exway log To (x) = Exwayyy log To (x) - log z a
P- (x)
-B E., Va log (2) Eg No Ve log ne ( (c)= Σ. (x) Ve me(x) 0.
"Thi is because H., (a)t(()
0. via the identity E., Ve log mo(r)
"The optimal policy p. is brietly mentioned in Ziegler et al. 2019) without reference or derivation. The
proof, which reveals connection the reverse KL divergence from ne. is

---

Policy Gradients
DPG
Reward
R(z)
Ro(x)
P(a) =x(x)
Vo
PI R(x)Vo To())
Pzz [0(
P(x)
Vo (T())
Baseline
Ex H(z)
Va with Baseline
Eg
R(x) Ba R(x) Va log (a)
B
E
Piz) (D(a)
Z
V log Ta(
Table A comparison between Policy Gradients Sutton et al.
1999) and Distributional Policy Gradients Par Shakova al. 2019b) forms of Reward, Baseline, and Gradient of the loss function (the PG-term) before (Va) and after (Vo with Baseline) including baseline for variance reduction.
log Z EE (((x) Blog To(#)) log Z Bo R8 a(x)
(10)
3
3
so that the arg min,, DKL (TO.PP) coincides with the arg max BR Rz),), proving (i). On other hand, arg x (xI[(T,:)) because of (i) applied to a family Tgr covering D(X) ) in full, is just Pa, concluding the proof.
which also corresponds to arg max = En R
conclude that the addition of the distributional term (KI-penalty) to the reward does indeed provide a DM interpretation, namely in terms of minimizing the reverse KL divergence with emergent underlying distribution P. (2)). We note that ps (2(r) does not correspond to free and explicit choice of EBM (e.g. one that balances the gender and topic distributions of a language model). Instead equation appears in restrictive format, which is implicitly defined by the reward Ri along with 5 hyperparameter without a clear meaning. By contrast, the DPG algorithms designed to perform DM on any EBM specification, corresponding an explicit distributional objective.
Overall,
3.3 Similarities and Differences between DPG and Policy Gradients
In the previous subsection, we have connected KL-control, a method designed under a RM paradigm,
to DM. Now, we turn to the converse question of whether DPG, DM method, can be connected to
RM. We begin by noting that after defining RG = =a(z)*
P(x). the DPG gradient Ex (w)
P(x) Ve log me (x)
acquires the format of the PG-term E,, Ro V log To (z).
However, the DM objective of DPG cannot be considered maximizing the average "reward" P(z)
(x) = no (a)) as this would require adding also the RG-term Ex, V P(m) into the gradient, which in this case does not vanish.
Nonetheless, the analogy behind this gradient term is more fruitful than it first appears. As matter of fact, DPG gradient estimates suffer from the same high-variance problems as with standard PG. While the objective of DPG (distribution matching) is different from that of Policy Gradients (reward maximization), DPG also needs estimate the PG-term E., (x()) V log To (a:) at a given value of 8, using a batch of samples For such a fixed 6 we can define provisionally set R(z) = Rş and the problem of gradient estimation for this fixed 9 is identical to the estimation Ex-ra R(x) V log ma (x) based on a set of samples z in standard RL. Therefore, techniques that have been developed to reduce the variance of the gradients estimates RL can be ported to DPG insofar as we are computing the gradient estimates reduction technique to the DPG: baselines.
given 0. In Section 4 we show how can import one such variance
4 A Case Study on Variance Reduction
Baselines are standard variance reduction technique in the context of Policy Gradients (Sutton & Barto 2018). The idea subtract from the reward R(x) value B that does not introduce bias to the gradients but may change variance. After the introduction of baseline, equation i then takes the following form:
VOEn, R(x) E.g [R(x) B V log (z).
(11)
5

---

In standard RL, the simplest form of baseline B is just the average of the rewards for the policy |
Reward Advantage Baseline
0.006
BRL
E E R(x).
(12)
0.004
Following the same methodology of taking the baseline to be the expectation the reward term, we can obtain a remarkably simple form of baseline for DPG:
0.002
0.000
P(x) Σ (T( TO( B RI (T)
P(x)
Σ P(x)
Z.
0.002
(13)
200
400
600
800
1000
epoch
Fact 1. Subtracting from R (z) does not introduce bias
into DPG gradient estimates.
Figure 2: Values of reward, advantage and the baseline for first 1000 epochs of point- wise constraint experiment.
Proof. Let us rewrite the DPG gradient in with the
added baseline B = Z:
Egwns Ro(x)) Z v log ( (x) = Enwn, Ro(5) (5) log (T()) - Z Barw Vo log (T())
= Egwn Ro (z) Vo (g) - Z Vero (x)
(14)
Here, the second term does not introduce bias because Z Σx VOTT(()) = 0, leaving us with the exact same form of gradient in the original DPG algorithm.
Note that since BRL depends on 8, has to be be re-estimated after each gradient update. the other hand, does not depend on 0, which is an advantage because B could be now estimated by averaging over samples from all the different f's without introducing bias, leading to a more accurate estimation. See Table for comparison of these two forms of baselines.
Algorithm KL-Adaptive DPG with baseline
Require: P, initial generative model
1: To f- 0, +
2: for each itcration do
3:
for each episode
4:
sample x from g(.)
5:
0 - 0 a if (pl(P) (KL(p) then
P(a)
Z 2e(e)
Vo log ne(x)
((m) ot=)
6:
The off-py DPG version introduced (Parshakova et al., 2019b) and its KL- adaptive variant (Khalifa et al. (02T) sample a proposal distribution q instead of the policy Ty. Then, the baseline takes the form
q me
Ensure: Tт
Bull (z) Z TT(
(15)
q(xx)
where the [a(=)) term is an importance weight correcting for the bias introduced by sampling from g.
o(x) Similarly the DPG case,
prove the following (see Appendix C):
Fact 2. Subtracting (x)()) from (x(x) does not bias the off-policy DPG gradient estimates.
In practice, as shown on Figure g adding a baseline to KL-adaptive DPG (Algorithm) L centers the
advantage (defined as A - P(#)
Z [=) around 0 leading to better performance on: convergence
(section 4.3) stability small batch sizes (section 4.4 and variance reduction (section 4.5)
'While this baseline not optimal (proof Appendix C.1 it is widely used in practice.
'In the scope of this paper, focus on importing to DPG simple constant baselines. The advantage that this a technique that is not impacted by the fact Re depends on 0: can be applied "9-locally" to provide a more accurate estimate of Eg==, [x)x) V log (z) for a fixed 9, irrespective of the values of Ry elsewhere, while variance reduction techniques that involve several 6' simultaneously raise additional challenges for parametric rewards.

---

4.1 Generation with Distributional Control
We investigate the benefits of adding a baseline to the DPG algorithm, on the Generation with Distributional Control (GDC) (Kbalifa et al.] (021) framework. GDC makes use of DPG to control the properties of pre-trained language models satisfy certain constraints. our experiments. follow target distribution form of Parshakova et al. (2019a), Khalifa et al. (202T) and Korbak et al. (2022a), in which the EBM P(x) deñned so that its normalized variant p(x) matches set desired moments constraints on given features (x while having a minimal KL divergence DEL. (p. a) from
an original pretrained language model a, to avoid catastrophic forgetting.
These constraints are expressed as conditions Fi: = (x), for i E (I,...,,)) by which the moments (expectations) under the distribution p of each feature (x(x) are required to take certain desired values fig- For instance, let Φ1 (z) 1 iff the topic of x is science and ) (z) = 1 iff mentions a female person, then imposing moments J1 1 and juz = 0.5 constrains the language model p to only generate sequences about science, half of which mention females. P(r) is uniquely determined by the following form
P(x) = a(x) (c
(16)
where X terms control the moments μ4, of the associated features, which can be estimated through self-normalized importance sampling (Owen 2013); and then, to make the moments match the desired values, the X; terms can be optimized through SGD (Parshakova et al. 2019a).
4.2 Experimental setup
We evaluate our method on an array of 10 controlled text generation tasks. For each, given a pre- trained language model a(x), and set of constraints, the objective of each fine-tuning method is to obtain a fine-tuned language model Te that satisfies the imposed constraints while deviating as minimally as possible from the original language model a(z).
Constraints are defined as a set of binary features } and their corresponding desired percentages (moments) { within the generations of the target language model. Based the value of the moment constraints these 10 tasks are divided into 6 tasks of pointwise constraints (for which fı; = 1), 2 tasks of distributional constraints (0- < j4 1) and 2 tasks of mixed type constraints (hybrid):
(a) Single-word constraints, where 6(x) 1 iff the a given word appears in the sequence We experiment with frequent words (task 1: ""amazing,, original frequency: 10-4) and (task 2: WWikiLeaks" original frequency: 10-5) rare words,
(b) Wordlist constraints, where ozz) 1 iff contains at least one word from given list. We consider lists of word associated with politics (task 3) and science (task 4) published by
Dathathri et al. (2020).
Sentiment classifier constraints, where d(x) if æ is classified positive (task 5), or negative (task 6) by a pre-trained classifier published by Dathathri et al. (2220).
(d) A single distributional constraint where (xx) iff x contains female figure mention, and jı = 0.5 (task 8),
(c) A set of four distributional constraints: (x(2) = 1 iff x contains at least one of the words in the "science", "art", "sports" and "business" wordlists (compiled by Dathathri al. (202)). respectively. For each i, f4 = 0.25 (task 8),
(f) Hybrid constraints where &1 (z) = iff x contains more female than male pronouns, D1 0.5 and (2)x)) = 1 iff x contains least one of the words from the "sports" wordlist (task 9) or "politics" wordlist, Fiz = 1 (task 10).
Methods We modify the GDC framework Khalifa et al. (2021) namely its KL-DPG algorithm. to include a baseline as shown in Algorithm [ We refer to this method as GDC++ In addition to comparing GDC++ with GDC we compare with two reward maximization baselines: Reinforce (Williams 1992b and Ziegler Ziegler al. 2019). Reinforce tries to maximize the expected reward E E R(x.) where R(x) = 1 if and only if the pointwise constraints are met. Ziegler instantiates the KL-control approach: its objective includes KL penalty term for departures from a.
Following (Khalifa et al. 202T), for hybrid and distributional constraints (tasks 8-) we compare
For more precise formulation of this EBM, see
Khalifa et al. 2021
7

---

10
G2C++ 1.0
0.95
Zlegier Renforcs
Self-B1EUs
Klnn
0.4
0.2
0.80
2k
01:00
2k
4K
spachs
Figure 3: Evaluation metries: (p p, me) (1 better), 1r, (t better), DRL a) (1 better). Belf-BLEU-S () hetter), and Distinct-1 (f better) aggregated over 6 pointwise constraints experiments (tasks 1-6) for policies obtained from GDC++ GDC, Ziegler and Reinforce. See Figure for aggregated distributional constraints experiments. the Appendix Figures
10 and Table contain individual view and final results each run.
only GDC and GDC++ because the RM objective of Ziegler and Reinforce is not equipped to handle them.
Metries
We report the following metries at each validation step over batches of samples from me:
1. Er (z), measuring the ability to reach the target moment of the i-th feature.
2. DKL To), the forward KL divergence from the optimal target distribution p
3. (Te, a), the reverse KL divergence from the original pretrained language model
4. Distinct-n score, measure of text diversity in terms of the frequency of repetitions within single sample æ, proposed by (Li al., 2016a).
5. Self-BLEUnn, a measure of text diversity on a distributional level across samples proposed by (Zuu et al. 2018), ensuring that policies don't 't converge into limited number of sequences that satisfy the imposed constraints Caccia et al. (2020).
Training details For tasks 1-6, we use a pre-trained GPT-2 small with 117M parameters (Radford et al., 2019) as the original language model a. For tasks 7-10, is the same pre-trained model additionally fine-tuned the WikiBio (Lebret al.) 2016) dataset. See Appendix E for more details. The code for all the experiments presented in the paper will be available at github.com/naver/ggee
4.3 Results
We present the evolution of our metrics through training epochs in Figure (aggregated over tasks 1 1-6) and Figure 0 in the Appendix (aggregated over tasks 7-10). Results for each task are presented separately on Figures
I0 in the Appendix.
Consistent with prior work (Khalifa et al. 2021 Korbak et al. 2022a). we observe that Reinforce is able to quickly achieve high levels of constraint satisfaction, but at the cost of large deviations from which translates into significantly decreased diversity of generated samples (in terms of Self-BLEU- and Distinct-l). The KL penalty term in Ziegler imposes upper bound on deviation from but the deviation is still significant enough to result in a drop in diversity. Moreover, we have observed Ziegler's objective result in very unstable training.
GDC and GDC++ are the only fine-tuning methods that address constraint satisfaction based on a clear formal objective, i.c. reducing the divergence from p. The approach translates into significantly smaller deviations from a and maintaining diversity within and across samples. The addition of baseline indeed reduces the variance. We analyze that extensively in Appendix 4.5 while here focusing on the downstream effects of variance reduction. One is that Tg is now able to compound staying closer to p and at the same time, while achieving slightly better constraint satisfaction. We have also observed that baseline stabilizes training, leading to smoother curves 5)
4.4 The effect of baseline across batch sizes
We expect that reducing gradient estimates variance can allow to train with lower batch sizes, performing gradient updates on estimates based on smaller batch sizes can increase the sample
SSee Appendix D for detailed description how DRL (p, ne) computed. 'The interested reader can compare the large fluctuations of the Ziegler objective to more stable training curves of GDC, and more of GDC++ in the disaggregated curves Figures 10 of the Appendix.

---

10.00
12.0
255
USD4
2.3
144
40.00
31
246
2n
2533
GEC+
CDC
0.25
04
Tm
210
21
2n
301
30
(a) Task 1: pointwise constraint
(b) Task 8: a set distributional constraints; J4: 0.25
Figure 4: (ax) or i per constraint (1 better) and DxL (p. лo) (1 better) function of the number of samples reported for task (a) and task 8 (b). We report the number of samples (i.e. number of epuchs times the batch size) for a fair comparison of convergence speed. GDC++ is consistently superior across all batch sizes in terms of convergence and construint satisfaction. The effect is more conspicuous with small batch sizes. Batch sizes 512 and 2014 greyed out for clarity.
efficiency. To test this, we rerun tasks I (a pointwise constraint on the word "amazing") and ( distributional constraints topics) with four batch sizes (256, 512, 1024, 2048). Figures 4 and 4b
show the benefits of adding a baseline - higher constraint satisfaction, lower divergence from p. more stable training- and is especially evident with lower batch sizes. For instance, with batch size 256, GDC++ obtains a significantly higher constraint satisfaction rate and lower divergence from p.
Furthermore, stable training with smaller batch sizes translates into better sample efficiency. For instance, in task 1 (Figure Aaj. GDC++ with batch size 256 needs 1M samples to achieve Egw,, o(x) 0.5 while GDC++ with batch size 2048 needs 4M. In contrast, GDC with batch size 256 does not achieve Egwn, σ(x) = 0.5 at all, confirming the importance adding the baseline.
4.5 Empirical Evaluation Variance Reduction
Next, we evaluate empirically the effect of the base- line for variance reduction. We select two tasks: task 1 (a pointwise constraint) and task 7 (distribu- tional constraints) described in Section 4.2 each with 3 different seeds, while monitoring the follow- ing variance measures:
Pointwise
VertGe
10r
200++
10:-
Gradient Variance The gradient estimate is de- fined as: Go(a) = A(r) log To(x), where (1( E R( is an unbiased estimate of the gra- dient of the forward KL loss DaDKL (D, σι) with respect to the parameters 0. We then have, with
1000
epochs
1000
spochs
20w
10"0
155107
u(Ga) = ExngGo(a)
1000
2000
1000
epochs
epochs
Var(GG) Ex-n [[oo(z) - 2
(17)
1.00
1.00-
= Ex Go [|l((G))l33 (18)
9.75
0.75
0.50
0.50
Variance of the advantage is defined by:
0.00
0.00
Var (A) = Eony A((x) vr
(19)
1000
1000
2000
epochs
epochs
0.75
where, veah Evng A(z) the mean of the advan- tage, which we showed above to be null after the addition of the baseline.
0.28
0.70
7V0lo
0.26
0.65
0.24
3.60
Expected absolute value of the ad advantage This metric defined as:
1000
2000
1000
2000
epochs
epochs
Comparison between GDC and GDC++ using set of Variance diagnosis metrics point- wise and distributional constraints experiments.
Figure 5:
p Al = Ernn [A(x))-
(20)

---

It directly provides a standard measure of distributional discrepancy between p and Ta. in terms of TVD (Total Variation Distance). We have:
p(r) ((x)
Te (x
Exzg
2TTV(DP me). ((x)
(21)
Results Figure shows that GDC++ obtains lower variance in the gradient estimates Var(Ge) and the variance of the advantage Var (A) in both pointwise and distributional experiments compared to its non-baseline counterpart GDC. We further observe a decreasing trend in the mean absolute value of the advantage ااÁر which is correlated with a decreasing trend in the TVD distance between the trained policy my and the optimal distribution p. Overall, these results shows that adding a baseline to DPG reduces the variance during training and yields better convergence towards the optimal distribution p.
Related work
The idea of posing control problems distribution matching has resurfaced numerous times in the RL
literature (Kappen et al. 2012, Friston et al. 2010 Levine 2018 Hatner et al. 2020 Buckley et al.
2017).
KL-control can be seen as generalisation of maximum entropy RL (MaxEnt RL) (Haarnoja)
et al. 2017 2018) to informed priors. If in 2 we chose (xx) ) to be a uniform distribution (assuming
right now finiteness of X) instead of pretrained LM distribution, then the KL penalty DKi. (To,
would reduce to an entropy bonus. Both KL-control and MaxEnt RL can be derived from a general framework of control-as- as-inference (Levine, 2018) which poses control as minimising KL from certain target distribution. However, most practical algorithms in the MaxEnt RL family minimise KL from a target policy which changes throughout training: contrast, DPG's target distribution p and KI--cotrool implicit target distribution P= are defined at trajectory level and fixed throughout training.
Perhaps the closest method KL-control and DPG in the larger family of inference-basee RL (Furuta el al. 2021 is AWR (Peng et al. 2019) which minimises the forward KL from an off-policy target distribution. Yet another approach with apparent similarity KL-control and DPG state marginal matching (SMM) (Hazan et al, 2018 Lee et al.) 2019). SMM poses exploration as learning a policy that induces state marginal distribution that matches a target state distribution. While SMM's target distribution is fixed, it is defined for individual states, while in the controllable language generation tasks we consider, the target distribution is defined over a complete trajectory considered unit. See Appendix B for an extended discussion of related work.
6 Conclusion
Fine-tuning large language models has become an active area of research, due to its importance in adapting large language models to satisfy task-level preferences, or in combating their social risks such as "distributional" stereotyping (Weidinger et al. 2021 Welbl et al. 2021 10 In this paper, we analyzed in depth the nuanced relation between two popular fine-tuning paradigms: RM and DM. We demonstrated that KL-control can be seen as a form of DM and showed that while DPG and PG have different goals, some similarities (similar forms of gradient estimates despite different objectives) can be exploited. We used these insights to inform an extension of DPG, consisting in adding a baseline to reduce the variance of gradient estimates.
The connections we established suggest that despite fundamental differences between DPG and RI,, some of the theoretical results and algorithmic techniques from RI can be adapted to DM framework without losing their formal guarantees. In this paper, we focus on variance reduction using baselines, but the space of possible enhancements is vast. Promising candidates include further reducing the variance using a learned value function (Konda & Tsitsiklis 2000) and preventing detrimentally large policy updates by maintaining trust region in the policy space akin to techniques such as TRPO (Schulman et al. (2013) and PPO (Schulman et al. (2017)). Another future direction could consist in analyzing the relation between explicit EBMs in DPG and implicit EBMs arising in KL-control and characterizing the space of EBMs that could be reached through KL-control.
10See Appendix A for discussion of broader impacts of large language models and controllable language generation.
10

---

References
Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally Normalized Cannsition-Bacee Neural Networks. 2016. doi: 118863/311PPIC 1231.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An Actor-Critic Algorithm for Sequence Prediction. (2015):1-17, 2016. UtL.attt://arrii..rrr wttp://wwww..rggraa/aaa////// 07086
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. In 5th Interna- tional Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
id Id=SJDaqqveg
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly. Sheer El-Showk, Nelson Elhage, 7ac HatfieldDDodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
A. Bakhtin, Y. Deng, S. Gross, Myle Ott, Mare' Aurelio Ranzato, and Arthur Szlam. Energy-based models for text. ArXiv, abs/2004. 2020.
David Belanger and Andrew McCallum. Structured prediction energy networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML'16, pp. 983-992. JML..org, 2016. htRp://tp:////t acm. org/ citatiion.ii../ cittation..cfe2:d-334555509 cf02533000530 3045495
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, pp. 610-623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi:
01145/444218..44559977 URL hattps://do org/ 10. 1 1145/ 3442188 3445922
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 5454 5476, Online, July 2020. Association for Computational Linguistics. doi: 10. 12.118661///220.cc/ a12.20.cl-maii.444 URL htttps://///tt aclweb
org/anthologgy2222 actp:///g///2.....mmmmmm... 485
Christopher L. Buckley, Chang Sub Kim, Simon McGregor, and Anil K. Seth. The free energy principle for action and perception: A mathematical review. Journal of Mathematical Psychology, 81:55 79, 2017. ISSN 0022-2496. doi: https://doi.orr/10011000
jmp.2017 20177.0..0 090.004. URL https
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short. In International Conference on Learning Representations, 2020. URL
httpa://openrerieer wtt.:/.eetttoooeeeeeeeee
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
https://openzevvee..o
Kevin Clark, Minh-Thang Thang Luong, Quoe V. Le, and Christopher Manning. Pre-training transformers as energy-based cloze models. In Bonnie Webber, Trevor Cohn, Yulan He. and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 285-2944 Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020eemmlpp 2020. emm 1p-main.22n00
URL. attps://dos wer.s.orgg..... 18653/v//

---

Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL.
net/
Foruniid=HledEyBKD
Peter Dayan. Reinforcement comparison. In Proceedings of the 1990 Connectionist Models Summer School, pp. 45-51. Morgan Kaufmann, San Mateo, CA, 1990.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy- based models for text generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nes 2020. URL https
//openreview.o //openzeriev.ne///fouumm www.eeoouunidd-1115eeee..
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp- 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguisties. doi: 10. 1665//11/119 1423. URL https
V/www. www.eb.lweb..o cat.lorgggattttie htt////ntoooogy/19-1111 1423.
Karl J Friston, Jean Daunizeau, James Kilner, and Stefan J Kiebel. Action and behavior: free-energy formulation. Biological cybernetics, 102(3):227-205 2010.
Hiroki Furuta, Tadashi Kozuno, Tatsuya Matsushima, Yutaka Matsuo, and Shixiang Shane Gu. Co- adaptation of algorithmic and implementational innovations in inference-based deep reinforcement learning, 2021. URL https://arxie: http:/awwwww...gaaaa http:/aab//211111///
Evan Greensmith, Peter L. Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Mach. Learn. Res., 5:1471-1530, December 2004. ISSN 1532-4435.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL http:://proceedinggs mlr press/ v9/ gutmgnnmmnnn (0a. html
Tuomas Haarooja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1352-1361. PMLR, 06-11 Aug 2017. URL htttp:://proceediggg mlr. www.mmmes./.mme..mm..m.mmm ppeess/v00 haarnojall a...jall7aa......
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL https://arxiv
brg/abs/1801. 01290
Danijar Hafner, Pedro A. Ortega, Jimmy Ba, Thomas Parr, Karl Friston, and Nicolas Heess, Action and perception as divergence minimization, 2020.
Elad Hazan, Sham M. Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration, 2018. URL https://aaxaxxi....aaaaaa http://arxxvv..r//aaa/ wwt.://///aa//......o.
Tianxing He, Bryan McCann, Caiming Xiong, and Ehsan Hosseini-Asl. Joint energy-based model training for better calibrated natural language understanding models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1761, Online, April 2021. Association for Computational Linguistics. URL
https://www aclweb. org/ httgg/nthology//20 2201 eacl main. 151
Geoffrey F. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput., 177 2002. doi: 10. 10.116222889777602660021121111 URL htR.shtps::// https://doo....... org/ 10. 011166//08997766222661160611
12

---

Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernandez-Lobate Richard E. Turner, and Douglas Eck. Sequence tutor: Conservative fine- fine-tuning of sequence generation models with kl-control. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICMI. 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1645-1654, PMLR, 2017a. URL http://proceeddnng www.m..rrrerre..e. ptwps//ee/
html
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, Jose Miguel Hernandez Lobato, Richard E. Turner, and Doug Eck. Tuning recurrent neural networks with reinforcement learning. 2017b. URL
httpss//ppperrooee...e/////llaae Kx
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Ágata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, abs/1907.00456 2019. URL http
V/arxiv. http///bb/199...... .00456)
Hilbert Kappen, Vicenç Gómez, and Manfred Opper. Optimal control graphical model inference problem. Machine learning, 8722)1199-182, 2012.
Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. A distributional approach to controlled text generation. In International Conference on Learning Representations, 2021. URL https
//openreview //ppeneeviiee..ne//ttunmnddddd wwt.:/iddddwwwaa-ewea Wkw45-9488 9AbL
Samuel Kiegeland and Julia Kreutzer. Revisiting the weaknesses of reinforcement learning for neural machine translation. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-IILT 2021, Online, June 6-11, 2021, pp. 1673-1681. Association for Computational Linguistics, 2021. doi: 10. 10.1853/1112211. maacl-main. 133. URL /ttpp:///oi.... org/10 11653//1//0211111 naacl- 133
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69986 6980, 2014.
A.S. Klyubin, D. Polani, and C.L. Nehaniv. Empowerment: a universal agent-centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, volume 1, pp. 128-135 Vol. 2005. doi: 10.1 2011009CCC.2000.555555
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In S. Solla, T. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 2000, URLhttts. proceedings heurips... http://pepe/.g..oeooeeeeeaaaaeee
wwt.:/ffaaaaappe...eeeeee Paper .df
Tomasz Korbak, Hady Elsahar, Marc Dymetman, and Germán Kruszewski. Energy-based models for code generation under compilability constraints. CoRR, abs/2 004.0495, 2021. URL https
//arxiv.or org/ abs/2106.. 049885
Tomasz Korbak, Hady Elsahar, German Kruszewski, and Mare Dymetman. Controlling condi- tional language models without catastrophic forgetting. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11499-11522 11528. PMLR, 17-23 Jul 2022a. URL httts:://prccedde 1162/korbak22a.nee.
mlr ...eesss///
Tomasz Korbak, Ethan Perez, and Christopher L Buckley, Buckley RL with KL penalties is better viewed as Bayesian inference, 2022b. URL https://arxiv org/
wwwg/aa////eeeeeeeooooo/eee abt//baa/////// 2205. 11275
Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1203-1213. The Association for Computational Linguistics, 2016. doi: 10. 11653//1//111- 1288. URL httl.bttta://aate httpp:://doi.or//00000 orgg
v1/d16- 1128
13

---

Yann LeCun, Sumit Chopra, Raia Hadsell, Mare' Aurelio Ranzato, and Fu Jie Huang. A Tutorial on Energy-Basse Based Learning. In Predicting Structured Data. MIT Press, 2006.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. 2019.
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review, 2018.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan, A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110-119, San Diego, California, June 20. Association for Computational Linguistics. doi: 10. 10.18653/1//11-11 1014. URL http::tp::///xt http:::/wwwwwalllebbbb.e..
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforce- ment learning for dialogue generation. In Jian Su, Xavier Carreras, and Kevin Duh (eds.). Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1 192--1202. The Association for Computational Linguistics, 2016b, doi: 10.18653/v1/d16 127. URL https://8o1. 18653/v1/d16- 1127
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understand- ing and mitigating social biases in language models, 2021.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Jian Su, Xavier Carreras, and Kevin Duh (eds.). Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 2122-2132. The Association for Computational Linguistics, 201 doi: 10. 11183///1//d1- 1230. URL http::tttts/a.. http:////ooooo//10// 10. 18653/v1//16- 1230
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Optimization of image description metrics using policy gradient methods. CoRR, abs/1612.00330, 2016b. URL http:
00370
Beren Millidge, Alexander Tschantz, Anil Seth, and Christopher Buckley. Understanding the origin of information-sseeking exploration in probabilistic objectives for control, 2021.
Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31 International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1791-1799, Bejing, China, 22-24 Jun 2014. PMLR. URL http:://proceedings .ml. wwwpppeesssseeee hm11 httml.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. lttp://////aaxxt www.aaaww.....grrr.arr/raaa
/ttp//aab//11.5500 5602
Subhajit Naskar, Pedram Rooshenas, Simeng Sun, Mohit Tyyer, and A. McCallum. Energy- based reranking: Improving neural machine translation using energy-based models. ArXiv, Abs/2009.13267, 2020.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL httpp://aaxxvvoor//ab//// 02155
Art B. Owen. Importance Sampling. In Monte Carlo theory, methods and examples, chapter 9. 2013.
URL httts::/sstateee/ wtwp:stttwtttttsss.rd....ee...de stanford. edu/" owwn/me//h. Saachhh------- -ar-is. -1..pd
Tetiana Parshakova, Jean-Mare Andreoli, and Mare Dymetman. Global Autoregressive Mod- els for Data-Effieciet Sequence Learning. In Proceedings of the 23rd Conference on Com- putational Natural Language Learning (CoNLL), pp. 900-909, Hong Kong, China, Novem- ber 2019a. Association for Computational Linguistics. doi: 10.18653/v1/K1111 19- 1084. URL
https: h/ww://wwwwww.oo aclueb. wtw.l/eeb..rr./aaeeee orgg/anthology/K19-1ee
14

---

Tetiana Parshakova, Jean-Mare Andreoli, and Mare Dymetman. Distributional Reinforcement Learning For Energy- Based Sequential Models. CoRR, 2019b. URL URL https://aaxii otgp//...rr//..///////o// abs/
1912. 08517
Ramakanth Pasunuru and Mohit Bansal. Reinforced video captioning with entailment rewards. In Martha Palmer, Rebecca Hwa, and Scbastian Riedel (eds.). Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 979-985. Association for Computational Linguistics, 2017. doi: 10.18653/v1/d17-1111 URL httpp:://doi /ttp://////oooo//./////////// 18663/v1/c17-1003
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances 32, pp. 0024-8035.. Curran Associates, Inc., 2019. URL http::/ nwwriss.eepppppcccm httpsacccc/// cccpaper/ 9015- prtsscyytocc---o an-
Neural Information Processing Systems
style- hinyhleehhhhh--h.. perf ormance deeeanneeeeee-eeeeeeee Segarninn--ibbrry
Romain Paulus, Caiming Xiong, and Richard Socher. deep reinforced model for abstractive sum- marization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018. URL
https:://peereeiev..t wwt.a///eeeeooooooioaoAAAAAAAA
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regresson:: Simple and scalable off off-policy reinforcement learning, 2019. URL BBLLattpss//arrbb.o////////11110 00177
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models, 2022.
URL https://arxiv www./ogg///////eeeeeooeeee abs//2bss22222 03286
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural Net- works, 21(4):682-697, 2008 ISSN 0893-6080. doi: https://bi.oog/10.001f...eemm 0.008.0..00.
Robotics
and Neuroscience.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188):9 2019.
Marc' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified mased framework for unsupervised learning. In Marina Meila and Xiaotong Shen (eds.), Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, AISTATS 2007, San Juan, Puerto Rico, March 21-24, 2007, volume 2 of JMLR Proceedings, pp. 371-379, JMLL. JMLR.org. 2007. URL http::://prooceeinggs.... mlr http:ang........pe.reeeeeeeeeeee v2/
stress//2/ranzat00775 www.ato7aaaa...... html
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
06732
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017a. URL http:///axxiv..o/aabb/////s
06347
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint: 1707.06347, 2017b.
15

---

Emily Sheng, Kai- Kui-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as babysitter: On biases language generation. In Kentaro Inui, Jing Jiang. Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. E.NLP-IJCNLPP 2019, Hong Kong, China, November 3-7, 2019, pp. 3405-3410. Association for Computational Linguistics, 2019. doi: 10.18653/w1/D1-- 1339. URL htttp::///do..orr
18653/v1/D19
David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. Reward is enough. Artificial Intelligence, 299:103535, 2021. ISSN 0004-3702 doi: http://ww.../g//////////////oo// 103533.
Nisan Stienmon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. CoRR, abs/2009.01325, 2020. URL httpt://aaxxxi otg/ 00322
Richard S. Sutton. Temporal credit assignment in reinforcement learning. PhD thesis, University of Massachusetts, 1984.
Richard Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, The MIT Press, second edition, 2018. URL httppp//aaaccmmlattaaaasaassasaa http://ccccp.eeiaaa.aaeereeee/aa book- 2nd. .tml
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS'99, pp. 1057-1063, Cambridge, MA, USA, 1999. MIT Press.
Pradyumna Tambwekar, Murtaza Dhuliawala, Lara J. Martin, Animesh Mehta, Brent Harrison, and Mark O. Riedl. Controllable neural story plot generation via reward shaping. In Sarit Kraus (ed.), Proceedings of the TWentt-Eighth International Joint Conference Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 5982-5988. ijcai.org, 2019. doi: 10.24963/ijcai. 24963/1jcaaaa /ttp://///aaa.111/822299
2019/829. httttttss///oi.oot
Emanuel Todorov. Linearly-solvable markov decision problems. In B. Schölkopf, Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2007. URL httpa://proceedii wwwpeeddinnnnnuuiss.r.e. www.rpp..ccpaappeee /0tp//0000////// file/
pdf
Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, and Kevin Gimpel. Engine: Energy-based inference networks for non-autoregressive machine translation. ArXiv, abs/2005.00850, 2020.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 4566-4575 IEEE Computer Society, 2015, doi: 10.1 10.1109/CVP..01.. 7.79997.. URL /attps:///ol: 1011///1100//1111//PP.. 7299087
Lex Weaver and Nigel Tao. The optimal reward baseline for gradient-based reinforcement learning. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI'01, pp. 538-545, San Francisco, CA, USA. 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng. Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William S. Isaac, Sean Legassick, Geoffrey Irving, and lason Gabriel. Ethical and social risks of harm from language models. CoRR, abs/2112.04355 2021. URL http:: htttpp:://rrxiis org/ abs /212
04359
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. In Marie-Francie Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Eveni / Punta Canu, Dominicun Republic, 16-20 November, 2021, pp. 2447-2469. Association for Computational Linguisties, 2021. doi: 10.18653/v1/0011 findings--emlpp210...
URL attps:///ol org/100 18653/11/222222 http:///222.....oooeeeee windings- emnlp.2n0 210
16

---

Ronald J. Williams. Reinforcement-leane leaaminng connectionist systems. Technical report, Northeastern University, 1987. Technical Report NU-CCS-87-3.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8:229-256, 1992a. doi: 10. 10.10007/B009992666 URL https:///doo...rr 10. 10.00//FF0009965
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learming, pp. 229-252222 1992b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771, 2019. URL
http://arxiv.orrg
1910.03777
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, (00..669111111 00..0881444 14, 2016. URL http: //apiv/aaivvv.v/aa//aaa org/ abs/ 1609.08144
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In Kevyn Gollins-Thompoot, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (eds.), The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08- 2018, pp. 1097-1100, ACM. 2018. doi: 10.1 10.1145/22999883210008.. URL
/stps://dois org/ 10. 1145/3209978. 32100880
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2019, URL http::/arxii......//////// org/ abs/ 1909 08593
17

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Section g
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix A
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] In Appendix we present proofs of all mathematical facts referred in the paper.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] The code is included supplementary material available to the reviewers and area chairs and will be made publicly available alongside the camera ready version of the paper.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] In Appendix we provide the hyperparameters used throughout our experiments and report our hardware configuration. In Appendix D we describe in detail how (p. No) and TVD(p. T&) were estimated and provide an extended pseudocode for our training loop in Algorithm
(c) Did you report eπror bars (c.g., with respect to the random seed after running exper- iments multiple times)? [No] However, we found the variance across random seeds to be negligible and not comparing across random seeds is standard practice when working with large language models where the cost of a single run is significant.
(d) Did you include the total amount of compute and the type of resources used (c.g.. type of GPUs, internal cluster, cloud provider)? [Yes] In Appendix we report our hardware configuration.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] In Appendix E
(b) Did you mention the license of the assets? [Yes] In Appendix
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
18


Reviews:
[{"review_id": "po9qJpqhYNu", "reviewer": "Reviewer_fBV8", "summary": "The paper elaborates on the idea of the DPG (Parshakova et al., 2019b) and GDC (Khalifa et al., 2021) paper, explicitly states that \"KL-control\", a specific reward maximization method, is also a distribution matching method with energy-based models. The paper also clearly explains the similarity and difference between distributional gradient descent and standard gradient descent. An advantage-based GDC algorithm is introduced in the paper as a standard technique in RL to reduce the variance of gradient estimates. Numerical experiments show the effectiveness of the proposed method.", "questions": "I don't have any critical questions or confusions about the paper.", "limitations": "Limitations and future directions are well discussed at the end of the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "DPG framework could be a powerful tool for fine-tuning LMs to maximize some non-differentiable objectives using RL/EBM, but also prevent the catastrophic forgetting problem. The idea is neat with mostly straightforward derivation. The paper clarifies the key concepts and connections to existing training paradigms. Readers that are perplexed about fine-tuning LMs with reinforcement learning would greatly benefit from this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FGxGPVB2xsp", "reviewer": "Reviewer_wVhU", "summary": "This paper focuses on fine-tuning language models for controllable language generation tasks ( constraining topic, sentiment, gender distributions, etc). The authors derive the connection between distribution matching approaches and approaches that use KL-divergence penalty to prevent distribution shifts. By deriving a parametric reward baseline for Distributional Policy Gradients (DPG) and use it to extend the work by Khalifa et al., a distribution matching approach for nudging autoregressive generation distributions, this work achieves variance reduction and superior performance. ", "questions": "Overall, I think the technical contributions are solid and novel, but I have concerns about how these contributions are framed and positioned. Can the authors address or clarify the concerns I raised above?", "limitations": "Limitations and potential negative societal impact are not discussed. Please add relevant pieces to the conclusion section. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The mathematical derivations for connecting policy gradients and distributional policy gradients (DPG),  connecting KL-control and distributional matching, DPG baseline are easy to follow. I appreciate the effort. These derivations provide good perspectives and should be valuable to the community.\n* The main claim of introducing a baseline to reduce variance of DPG is well-justified and supported by evidence (Section 4).\n\nWeaknesses:\n* I think the contribution on making connection between policy gradients and distributional policy gradients is unnecessarily exaggerated, such as line 77: \"So far, the connections between these two seemingly distinct paradigms have not been thoroughly explored\". I think even the similarity in names already suggest that they are closely related, and they indeed aren't that different from each others. I think this claim actually makes the paper's logic difficult to follow for me. I would suggest focusing more on the contribution of introducing the parametric baseline, which should be the main product of this paper in my opinion.\n* I'd like to see a consolidated definition of the control language generation task. Right now, they are scattered in intro, Sec. 4.1, Sec. 4.2. What's the context and desired outcomes of such tasks? How does DPG achieve the desired outcomes (now it's sort of described in Sec. 4.2)? Can the authors give some examples?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0fbJV-FCYNA", "reviewer": "Reviewer_6JLh", "summary": "This paper studies controllable text generation with an emphasis on sequence level rewards and distribution matching. It first establishes a connection between reward maximization (RM) with KL control and distribution matching (DM) in a general sense. They show that there is an equivalent formulation of RM in DM terms and also DM can be connected to RM by thinking the importance sampling (IS) term as a reward. Using these connections, the authors highlight that a recent DM method, distributional policy gradient (DPG), suffers from the same high variance of REINFORCE estimators as the latter appears in the loss function of DPG. As is typical in RL, the authors propose using advantage estimations to reduce the variance of the DPG estimator and show that the intuitive expected reward (estimated as mean IS weights) gives unbiased estimates. The authors evaluate the proposed method, GDC++, on 10 controllable generation tasks including pointwise and distributional constraints using moment matching, KL-divergence, BLEU score metrics. When compared to other baselines, GDC++ shows lower divergence from the prior language model while also having competitive constraint matching. Using an advantage estimation is also shown to perform better and more stable results.", "questions": "I have several questions regarding my concerns above.\n\n1. Could you clarify how your derivations are related to prior work on distribution matching and max entropy RL ? \n2. Can you clarify the form of KL divergence used in the paper?\n3. What is the relationship between the baseline used in the paper and that of Ziegler?\n4. Could you clarify how Ziegler is trained, especially $\\beta$ hyperparameter? Can you get better tradeoff by tuning this hyperparameter?\n5. Can you discuss how these metrics translate into human satisfaction?", "limitations": "The authors addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper is the connection between RM and DM to motivate using a baseline in DPG. There are also several weaknesses that needs clarification. I detail these below.\n\n## Strength\n1. The paper is well written and easy to follow. \n2. The connection between RM and DM to motivate using a baseline in DPG and showing that the particular expected reward baseline is unbiased are interesting.\n3. Experimental results show improvement compared to GDC and Ziegler et al.\n\n## Weaknesses\n1. While DM is not exactly the same as RM, I think there are previous work that incorporates KL-divergence between an online distribution and an offline distribution as a reward signal to train a policy using RL. Some of the more recent work are SAC$^1$, SMM$^2$ that induce a reward using a distribution matching and an older work *. While these don't necessarily utilize a KL-control, they study KL-divergence objective with energy based models or reward shaping.\n2. The KL-divergence appears in different forms at different places. You define DPG in Line 105 and Algorithm 1 using $D_{KL}(p || \\pi_\\theta)$ while in Line 135 it is reverse.\n3. Ziegler uses a reward shaping with a KL-control and the model is optimized via PPO. As PPO uses GAE, it also subtracts a baseline from this new reward. I think the relationship between this and the proposed baseline needs to be addressed.\n4. It is also not clear how the $\\beta$ is tuned in Ziegler baseline. By varying this hyperparameter, Ziegler can do better on $D_{KL}(\\pi_\\theta, a)$ while doing worse on $E_{\\pi_theta} \\phi(x)$, performing very similar to GDC++.\n5. There is no human evaluation study in the paper and it is not clear how to judge the metrics proposed. For example, does higher KL from the language prior always means worse generation?\n\n$^1$ Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.\n$^2$ Efficient Exploration via State Marginal Matching.\n\\* Reinforcement Learning by Probability Matching.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept