On Reward Maximization and Distribution Matching for Fine-Tuning Language Models
Authors: Tomasz Korbak
Hady Elsahar
Germán Kruszewski
Marc Dymetman
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement Learning
Language Models
Reward Maximization
Distribution Matching
Energy Based Models
Controlled Text Generation

Abstract: The availability of large pre-trained models is changing the landscape of Machine Learning research and practice, moving from a "training from scratch" to a "fine-tuning'' paradigm. While in some applications the goal is to "nudge'' the pre-trained distribution towards preferred outputs, in others it is to steer it towards a different distribution over the sample space. Two main paradigms have emerged to tackle this challenge: Reward Maximization (RM) and, more recently, Distribution Matching (DM). RM applies standard Reinforcement Learning (RL) techniques, such as Policy Gradients, to gradually increase the reward signal. DM prescribes to first make explicit the target distribution that the model is fine-tuned to approximate. Here we explore the intimate connections between the two paradigms and show that methods such as KL-control developed in the RM paradigm can also be construed as belonging to DM. We further observe that while DM differs from RM, it can suffer from similar training difficulties, such as high gradient variance. We leverage connections between the two paradigms to import the concept of baseline into DM methods. We empirically validate the benefits of adding a baseline on an array of controllable language generation tasks such as constraining topic, sentiment, and gender distributions in texts sampled from a language model. We observe superior performance in terms of constraint satisfaction, stability, and sample efficiency.

Under review as a conference paper at ICLR 2022
ON REWARD MAXIMIZATION AND DISTRIBUTION MATCHING FOR FINE- TUNING LANGUAGE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
The availability of large pre-trained models is changing the landscape of Machine Learning research and practice, moving from a "training from scratch" "fine- tuning" paradigm. While in some applications the goal is to "nudge" the pre-trained distribution towards preferred outputs, in others is to steer it towards different distribution over the sample space. Two main paradigms have emerged to tackle this challenge: Reward Maximization (RM) and, more recently, Distribution Matching (DM). RM applies standard Reinforcement Learning (RL) techniques, such Policy Gradients, to gradually increase the reward signal. DM prescribes to first make explicit the target distribution that the model is fine-tuned to approximate. Here we explore the intimate connections between the two paradigms, and show that methods such KL-control developed in the RM paradigm
also be construed as belonging to DM. We further observe that while DM differs from RMM, can suffer from similar training difficulties, such high gradient variance. We leverage connections between the two paradigms to import the concept of baseline into DM methods. We empirically validate the benefits of adding baseline on an array of controllable language generation tasks such as constraining topic, sentiment, and gender distributions texts sampled from a language model. We observe superior performance in terms of constraint satisfaction, stability and sample efficiency.
INTRODUCTION
Pre- trained language models (Devlin et al.) 2019 Radford al.] 2019 are changing the landscape of Machine Learning research and practice. Due their strong generative capa- bilities many studies have found it sufficient to "nudge" these models to conform global pref- erences defined over the generated sequences instead of training from scratch using annotated data. These preferences could include topic and sentiment (Dathathri et al., 2020), valid musi- cal notes and molecular structures (Jaques et al. 2017a),, code compilability (Korbak et al. [2021) I). balancing gender bias (Khalifa al. [2221) eval-
Reward
Disertbution
Maxiniaasion
Farawetrte reword
Masshing
VoDa (p.
To (2)
Ca A2009 Angmytt)
Figure 1:
In this study make connection between two popular paradigms for fine-tuni fine-tuning sequence genera- tion models according to preferences Reward Maxiniza- tion (RM) and Distribution Matching (DM).
uation metrics for Machine Translation and Summarization (Ranzato et al. 2016 Bahdanau et al.
2016), direct human feedback (Ziegler al. 2019 Stiennon et al. 2020). This large body of
studies is driven by two paradigms: Reward Maximization (RM) and Distribution Matching (DM).
Reward Maximization RM exploits the intuitive notion that we can nudge pre-trained models towards some preferences by providing global sequence-level rewards when the model generates outputs that satisfy desired features. For instance, if the model producing toxic content, we can apply Reinforcement Learning (RL) techniques discourage it from producing similar content again in the future. However, the risk of naively applying RL is that the model can undergo catastrophic forgetting of its original distribution. For example, it degenerate into producing single nonsensical but least nontoxic sequence. Although several studies have considered conddcrffting general rewards to ensure desirable features like fluency (Liu al. 2016al Tambwekar al. 2019), coming up with
forgettnn

---

Under review as a conference paper at ICLR 2022
rewards that are not incomplete imperfect is highly non-trivial (Wu et al. 2016 Vedantam et al.)
2013). These challenges have sparked a wide discussion on the overall effectiveness of RM for some tasks such as Neural Machine Translation (Choshen et al. 2020 Kiegeland & Kreutzer 2021).
Reward Maximization with KL-Control To tackle the aforementioned issues of "catastrophic forgetting", several studies, still under an RM paradigm, have considered incorporating a distributional
term inside the reward to be maximized. In particular [Jquuss al. (2017b) 2019); iZiegler al.
(2019) and Stiennon et al. (2020) have applied variations of KI--control (Todorov] [200]] Kappen ctt al. all]]2012 2012) which adds a penalty term to the reward term so that the resulting policy does not deviate too much from the original one in terms of KL-divergence. The overall objective with the KL-penalty
maximized using an RL algorithm of choice including: PPO (Schulman al., 2017a) as in Ziegler
eet al.] (2019) or Q-learning (Mnih et al., (013) as in Jaquuss et al. (2017b). Adding this distributional KL-penalty to the reward raises some important questions: What effect does have on the shape of the optimal policy? Does this new objective have any interpretation from distributional perspective?
Distribution Matching A different recent paradigm for fine-tuning language models to satisfy downstream preferences formulates the problem as Distribution Matching (DM). This paradigm consists of two steps: first target distribution incorporating the desired preferences is defined as an Energy-Basde Boorgy-Based Model (LeCun et al., 2006). Then the forward KL. divergence is minimized between this target distribution and auto-regressive policy using a family of algorithms referred to as Distributional Policy Gradients (DPG) (Parshakova et al. 2019b Khalifa et al., 2021 Korbak et al., 2021). This approach capitalizes on the flexibility of EBMs in specifying the target distribution. For example, the EBM can be defined so that it conforms to all downstream preferences while its corresponding normalized distribution has a minimal KL divergence from the original, pre-trained language model, therefore tackling the problem of "catastrophic forgetting" (Khalifa et al. 202T). Interestingly, this DM paradigm can also deal with distributional preferences, for instance, for de- biasing language models by specifying that the generated sequences should be gender- balanced, i.c. that 50% of generations contain female mentions. Such distributional constraints cannot be defined in the RM paradigm where reward is calculated for single sequence.
Contrasting these two paradigms for fine-tuning language models, we can notice the promises and limitations of each@RM RM approaches are equipped with large arsenal of RL algorithms and opti- mization techniques that can be efficient in reward maximization, however they lack the distributional perspective that is necessary for avoiding catastrophic forgetting and imposing distributional prefer- ences over LMs. DM approaches are better suited to tackle those limitations, however, the family of DPG algorithms currently used is not rich as its RL counterpart. So far, the connections between these two seemingly distinct paradigms have not been thoroughly explored. By establishing such connections we might import ideas from one approach the other. This is our goal in this paper, detailing the nuanced connections and applying them to a case-study in variance reduction. Overall, our contributions are the following:
We untangle the connections between the RM and DM paradigms for fine-tuning language models. We provide detailed comparison between the family of DPG algorithms with Policy Gradients of standard RL.
We provide an interpretation of KL-control techniques from a distribution matching perspective, placing such techniques at intermediate place between RM and DM.
We exploit these connections to theoretically justify applying baselines a variance reduction technique from RI. to DPG and derive a particular choice of a baseline. On an array of controllable language generation experiments about constraining topic, sentiment, and gender distributions over a pre-trained language model, we show that adding baselines leads to superior performance on constraint satisfaction, stability on small batch sizes, and sample efficiency.
experimer
2 BACKGROUND
Standard Policy Gradients One popular method for adapting the behaviour of language models to certain preferences has been that of assigning a "reward" score R(x) for sequences x sampled
See Appendix A for an extended discussion of related work.
2

---

Under review as a conference paper at ICLR 2022
from an autoregressive language model (policy) Tq. Then, the simplest policy gradient algorithm in reinforcement learning, namely, REINFORCE (Williams) 1992a), aims to find the policy Ty (T(x) that maximizes the average reward Ez=r, and this leads, via the so-called "log derivative trick", to a gradient ascent algorithm based on:
V Palawna R(x) = Exwng mog
(1)
which iteratively samples a from and update parameters by increments proportional to Vo log (z)).
KL-control Jaques et al. (201 7b 2019
Ziegler et al. (2019). inspired by KL-control (Todorov)
2007 Kappen et 2012), add KL penalty term to the reward objective to penalize large deviations from the original pretrained model. That is, they maximize the expectation Ezaz, Ri (z), where:
(o(a) a(x)
R² r(x) 3 log
(2)
and where , is free hyperparameter balancing the wish to maximize with that of not deviating from a.
Distributional Policy Gradients Distributional Policy Gradients (DPG) (Parshakova et al. 2019b) is a recent approach used to fit an autoregressive policy to the distribution p(z) = P(x)/Z induced by the EBM P(x). where Z Σ. P(x) is the normalization constant (partition function). Given an arbitrary EBM P(x)., DPG optimizes the loss function DKL (p, πo) with respect to the parameters 0 of
an autoregressive model IT. loss which is minimized for T = p. The KL-diivegeecce minimization objective leads gradient estimate of the form:
DoLKLP. x²)=
V VoE log (I)
(3)
Σ Vo (o() = 2
Σ
P(x) log
(4)
((x) Eg (T(
Vo log (xx)
(5)
Z
REWARD MAXIMIZATION Vs DISTRIBUTION MATCHING
In the previous section, we have summarized three approaches that have been suggested for fine-tuning language models. Two of them can be characterized as "Reward Maximization" (RM): Standard Policy Gradients (PG) and KI. Control. On the other hand, DPG clearly belongs to the realm of "Distribution Matching" (DM) it first defines the target distribution and then optimizes policy match it. In the rest of this section, we will explore connections between these two seemingly distinct concepts and, in the following section, we will exploit them improve DM-based methods.
3.1 STANDARD VS. PARAMETRIC REWARDS
Let us start with distinguishing between a "parametric reward" Rg which depends on and standard reward R, which does not. If we wished to maximize the expected parametric reward, E (x)x,, we would follow its gradient, leading to the identities:
V VolEa Ro(x) V Σ Re (x) (a())
(6)
(o Σ Ro(
To( (x) Σ
(e(x) Ro logoTe
(8)
E VoRo(x) + Egan, R (x)
(9)
RG-term
PG-term
3

---

Under review as a conference paper at ICLR 2022
Equation (9) is the sum of two terms: the first one, the "RG-term" (Reward Gradient term), involves the gradient of the reward. The second one, the "PG-term" (Policy Gradient term), was obtained using the "log derivative trick" and involves the gradient of the policy stricto senst. In standard RL, where the reward does not depend on 0, the RG-term disappears and the gradient of expected reward consists solely of the PG-term. However, when Ro depends on 0, the gradients are distinct (apart from specific cases where the RG-term evaluates to O, as we will see below).
3.2 KL CONTROL AS DISTRIBUTION MATCHING
Adding KL-penalty term to the reward (as in the case of KL-control) Ieads to a parametric reward. However, due to the particular form of its objective, the RG-term actually vanishes leaving only the PG-term Eg R2 Vo log To (2) and simplifying the tuning procedure to a standard Policy Gradient. While this algorithm falls under the RM paradigm, here argue that is it multifaceted, and explore deeper connections with the DM paradigm. More precisely, the maximization of the reward with the KL penalty term is equivalent to a distributional matching with underlying emergent sequential EBM, remark that already reveals some similarities with DPG. Actually, if we consider the following EBM:
Pa(x) = awx))
(10)
then its normalized distribution, P= 2 Pz(x), with Z P.(x,, is, among all possible distributions, the one that obtains maximal expected reward Rğ(z). A simple way to prove this fact notice that the expectation of the reward Ri has monotonically decreasing relationship with the reverse KL divergence between To and pz:
DKL(TO, P=) = EEETT log
1 To ERNTO log)o(11= - log
z al [xx))e(0)/0))
(x(x)
= log Z Ex (r(x) - BBlog To(x) a(x) = log 2 - 3
Exnry Rā(x), (11)
so that the arg min,, DKL (To, (KI(TT Ps) coincides with the arg maxx, ExNnY Rz) Provided that the family of distributions my is large enough to cover all distributions over X, arg minz, (TI((ToPP)) is just Ps, which concludes the proof
Overall, can conclude that the addition of the distributional term (KL-penalty) the reward does indeed provide DM interpretation, namely in termss of minimizing the reverse KL divergence with an emergent underlying distribution P=(x). We note that p. (z) does not correspond to an explicit choice of EBM (e.g. the one that balances the gender and topic distributions of a language model). Instead equation (T0) has a limited form implicitly defined by the reward Rý along with a B hyperparameter without clear meaning. By contrast, the DPG algorithms are designed to perform DM on any EBM specification, corresponding to an explicit distributional objective.
3.3 SIMILARITIES AND DIFFERENCES BETWEEN DPG AND POLICY GRADIENTS
In the previous subsection, we have connected KL-control, method designed under a RM paradigm,
to DM. Now, we turn to the converse question of whether DPG, a DM method, can be connected to
R.M. We begin by noting that after defining Rş P(w) the DPG gradient Ex P(x) Vo log Th (x)
<=(z)*
(m)
acquires the format the PG-term Eng Rg V log TT (x).
However, the DM objective of DPG cannot be considered as maximizing the average "reward" P(z).
Rg (z) [a)" as this would require adding also the RG-term E,,, Ve P(w) into the gradient, which in this case does not vanish.
<o())
Nonetheless, the analogy behind this gradient term is more fruitful than it first appears. As matter of fact, DPG gradient estimates suffer from the same high-variance problems as with standard PG. While the objective of DPG (distribution matching) is different from that of Policy Gradients (reward
0, via the often used identity Eng Vo log [o(z) = Σ, (o) log [o(x)= = VoLo(z) Vo ()(=)=
2This is hecause Tr,, (z)
- Vo Vo (x)
"The optimal policy Pz is briefly mentioned in (Ziegler al. 2019) without reference derivation. The proof, which believe elarify important underlying connections, is ours.
4

---

Under review as a conference paper at ICLR 2022
Policy Gradients
DPG
Reward
R(x)
P(x) Ro( (3)
V
Exwwy Mx)VV (e())
P(ar) BE (
Va log (T)
Baseline
PI R(z)
Va with Bascline
E Bg
R(x)
Ba R(xs) Vo log (x)
Va log (T)
(p)
Table I: A comparison between Policy Gradients (Sutton ct al.
1999) and Distributional Policy Gradients Paa shakova al. 2019b) forms of Reward, Baseline, and Gradient of the loss function (the PG-term) before (Va)
after (Vo with Baseline) including baseline for variance reduction.
maximization), DPG also needs estimate the PG-term Erg Ro (2)xVV log (T(x)) at given value of
using batch of samples æ. For such a fixed we can define provisionally R(xx) Rg and the
problem of gradient estimation for this fixed 0 is identical the estimation Ez~r, R(x) Vo log To (x) based on a set of samples x in standard RL. Therefore, the techniques that have been developed reduce the variance of the gradients estimates in RL be ported DPG insofar as
computing the gradient estimates at a given 0 In Section we show how one can import one such variance reduction technique the DPG: baselines.
4 A CASE STUDY ON VARIANCE REDUCTION
Reward
angge
Base line
Baselines are a standard variance reduction technique in the context of Policy Gradients (Sutton & Barto 2018). The idea to subtract from the reward R(x) value B that does not intro- duce bias to the gradients but may change variance. After the introduction of baseline, equation then takes the following form:
0.004
0.000
((E E,, (R(x) B] V log (o()).
(12)
In standard RL, the simplest form of baseline B is just the average the rewards for the policy
-0.002
200
400
800
1000
opach
B = E R(x).
(13)
Following the same methodology of taking the baseline as the expectation of the reward term, we can obtain a remarkably simple form of baseline for DPG:
Figure 2: Values reward, vdnattage and baseline first 1000 epochs of pointwise constraint experiment.
P
P(x) To(x)
B E S To(O)
TTO
Σ P(x)=
(14)
To confirm that B does not introduce bias to the gradients, let rewrite the DPG gradient in with the added baseline = Z:
E ExNTe Ro(x) z Vo log (x()) = EzaTo Ro (x Vo log TT(x) - Z ERNTO Vo log (z)
(15)
= E R V log (x() - Z Σ V VoTo(z)
Here, the second term does not introduce bias because 7 Σn (oTT(1))
0, leaving with the
same exact form of gradient as in the DPG algorithm.
Note that since BRL depends on 0, it has to be be re-estimated after each gradient update. On the other hand, does not depend on 8, which is an advantage because could be now estimated by averaging over samples from all the different 0''s without introducing bias, leading to more accurate estimation. See Table for a comparison these two forms of baselines.
'To avoid confusion, that variance reduction methods for SGD focus the objective of better estimating the tnue gradient expectation F. [..]], for a fixed The fact that will then be updated based on such estimates is orthogonal to this objective.
While this choice the haseline not optimal (see Appendix B.1 for proof), it is widely used in practice.

---

Under review as a conference paper at ICLR 2022
The off-policy DPG version introduced in (Parshakova et al., 2019b) and its KL- adaptive variant (Khalifa et al. 202I) sample a proposal distribution instead of the policy
Algorithm KL-Adaptive DPG with bascline
Require: P, initial generative model
1: To f- 0. (q--0
2: for each iteration do
To. Then, the baseline takes the form
3:
for each episode do
Bott (x) Z (Z) ((x)
4:
sample x from g())
(16)
5:
0 0 + (0) (0)
P(a)
Z [el=)
Vo log xo(ar)
g(x)
where the (a(a)) q(x) term an importance weight correcting for the bias introduced by sam- pling from g. Similarly
6:
if (pL(p)
(KL(p)lg) then
7: 9
Ensure:
the DPG case,
this baseline does not introduce bias (see Ap- pendix B for details about this version of the baseline). In practice, as shown on Figure 2 adding
baseline to KL-adaptive DPG (Algorithm centers the advantage (defined as A 2(x)
P(z) #e(a)
q(r)
around 0 leading to better performance on: convergence (section 5.3) stability on small batch sizes (section 5.4). and variance reduction (section 5.5)
EXPERIMENTS AND RESULTS
5.1 GENERATION WITH DISTRIBUTIONAL CONTROL
We investigate the benefits of adding a baseline to the DPG algorithm, on the Generation with Distributional Control (GDC) Khalifa al. 2021) framework. GDC makes use of DPG to control the properties of pre-trained language models to satisfy certain constraints. In our experiments, follow target distribution form of Parshakova al. (2019a) and Khalifa al. (202T), , in which the EBM P(r) is defined so that its normalized variant p(z) matches set of desired moments constraints on given features (I)), while having minimal KL divergence DKL. (p. a) from an original pretrained language model a, to avoid catastrophic forgetting.
These constraints are expressed as conditions fı: 0; (x)) for E {l....,m) m). by which the moments (expectations) under the distribution p of each feature (z) are required to take certain desired values f4.. For instance, let Φ1 (x) = 1 the topic of x is science and (2 = iff mentions a female person, then imposing moments jn 1 and ja 0.5 constrains the language model p to only generate sequences about science, half of which mention females. P(x) is uniquely determined by the following form
P(x) = (lx))eei Aco(())
(17)
where A terms control the moments μ4 of the associated features, which can be estimated through self-normalized importance sampling (Owen) 2013); and then, to make the moments match the desired values, the X; terms can be optimized through SGD (Parshakova et al. 2019a).
5.2 EXPERIMENTAL SETUP
We demonstrate the benefits of adding a baseline to the family of Distributional Policy Gradients algorithms on an array of controlled language generation tasks for English. For this, we modify the GDC framework Khalifa et al. (202T) namely its KL-DPG algorithm to include a baseline as shown in Algorithm I We refer to this method in all the following experiments as GDC++. In addition to comparing GDC++ with GDC, for tasks for which the comparison is meaningful (i.e. for "pointwise constraints", see below), we compare with two reward maximization baselines: Reinforce (Williams) 1992b) and Ziegler (Ziegler et al. 2019). Reinforce tries to maximize the expected reward Egonn (()). where R(z) = 1 iff the pointwise constraints are met. Ziegler instantiates the KL-control approach: its objective includes a KL penalty term for departures from
Tasks We evaluate GDC, GDC++ and the two baselines on 10 tasks, which we construct through
sets of moment constraints { for binary features { These include 6 sets of purely pointwise constraints (for which T4 = 1) and 4 sets including distributional constraints (0 < P4 < 1). We consider the following constraint types:
For more precise formulation of this FBM, see Khalifa et al. 2021

---

Under review as a conference paper at ICLR 2022
(a) Single-word constraints, where 6(x) iff the given word appears in the sequence
We experiment both with frequent words (task 1: "amazing", original frequency: 10 10=4) and
(task 2: "WikiLeaks", original frequency: 10 10-5) rare words,
(b) Wordlist constraints, where ((x) iff contains at least one word from given list. We consider lists of word associated with politics (task 3) and science (task 4) published by Dathathri et al. (2020),
(c) Sentiment classifier constraints, where d(x) if æ is classified as positive (task 5), or negative (task 6) by a pre-trained classifier published by Dathathri et al. (2020).
(d) A single distributional constraint where (zx) iff x contains a female figure mention, and jı 0.5 (task 8),
(e) A set of four distributional constraints: (x(x) = iff æ contains at least one of the words in the "science", "art", "sports" and "business" wordlists (compiled by Dathathri et al. (2020).
respectively. For each i, μ1 = 0.25 (task 8),
(f) Hybrid constraints where Φ1 (z) = iff x contains more female than male pronouns, fi 0.5 and d2(2)) 1 iff æ contains at least one of the words from the "sports" wordlist (task 9) or "politics" wordlist, (2 = 1 (task 10).
Following (Khalifa al. 202I), for hybrid and distributional constraints (tasks 8-10) we compare only GDC and GDC++ because the RM objective of Ziegler and Reinforce is not equipped to handle distributionas distributional constraints.
Metries We report the following metrics evaluated over batches of samples from To at each validation step:
1. E Ex (x)), measuring the ability to reach the target moment of the i-th feature.
2. DKL (p, no), the forward KL divergence from the optimal target distribution p
3. DKL(TT, a), the reverse KI. divergence from the original pretrained language model a.
4. Distinct-n score, measure of text diversity in terms of the frequency of repetitions within single sample proposed by (Li et al. 2016a).
5. Self-BLEUnn,
of text diversity on distributional level across samples proposed by (Zhu et al. 2018), ensuring that policies don't 't converge into limited number of sequences that satisfy the imposed constraints Caccia et al. (2020).
Training details For tasks 1-6, we use a pre-trained GPT-2 small with 117M parameters (Radford et al., 2019) as the original language model For tasks 7-10, is the same pre-trained model additionally fine-tuned on the WikiBio (Lebret et al., 2016) dataset. See Appendix [B] for more details.
5.3 RESULTS
We present the evolution of our metrics through training epochs in Figure (aggregated over tasks 1-6) and Figure the Appendix (aggregated over tasks 7-10). Results for each task are presented separately on Figures
10 in the Appendix.
Consistent with prior work (Khalifa et al. 202I). we observe that Reinforce able to quickly achieve high levels of constraint satisfaction, but at the cost of large deviations from a, which translates into significantly decreased diversity of generated samples (in terms of Belf-BLEU- and Distinct- 1). The KL penalty term in Ziegler imposes upper bound on deviation from but the deviation is still significant enough to result in a drop in diversity. Moreover, we have observed Ziegler's objective to result in very unstable training.
GDC and GDC++ are the only fine-tuning methods that address constraint satisfaction based on a clear formal objective, i.c. reducing the divergence from p. The approach translates into significantly smaller deviations from a and maintaining diversity within and across samples. The addition of a baseline indeed reduces the variance. We analyze that extensively in Appendix 5.5 while here focusing on the downstream effects of variance reduction. One is that To is now able to compound staying closer to p and a at the same time, while achieving slightly better constraint satisfaction. We have also observed that baseline stabilizes training, leading to smoother curves S
7 "ee Appendix for a detailed description how DEL (p. To) is computed. 'The interested reader compare the large fluctuations of the Ziegler objective to more stable training curves of GDC, and more of GDC++, in the disaggregated curves in Figures 10 the Appendix.
7

---

Under review as a conference paper at ICLR 2022
10
EDC++ EDC
1.0
10
0.95
Zegier Reinfsree
0.8
Selt-BBUSS 1US
Kilp mi
Ki al
0.4
0.85
0.
0.80
2k
26
4k
apocis
(PL Te) (L better), (xx) (1 better), (TL a) (L better). Self-BLEU-S better), and Distinct-1 (f better) aggregated over 6 pointwise constraints experiments (tasks 1-6) for policies obtained from GDC++, GDC, Ziegler and Reinforce. See Figure 6 for aggregated distributional constraints experiments, and Figures / 10 in the Appendix for a detailed view on each experiment. And a Table 3 view for final results of each run.
Figure 3: Evaluation metrics:
226
532
2040
6.7
melied
CDC
4.4
322
02
2
6.
25
2m
(a) Task 1: pointwise constraint
(b) Task 8: set of distributional constraints; fo: 0.25
Figure 4: Eng (xx) or f per constraint (+ better) and DKl. (p. IT) (1 better) as a function of the number of samples reported for task (a) and task 8 (b). We report the number of samples (i.e. the number of epochs times the batch size) for fair comparison of convergence speed. GDC+ is consistently superior across all batch sizes terms of convergence and constraint satisfaction. The effect is more conspicuous with small batch sizes. For instance, with batch size 256 the baseline prevents the policy from catastrophically diverging from p. Batch sizes 512 and 2014 are greyed out for clarity.
5.4 THE EFFECT OF BASELINE ACROSS BATCH SIZES
We expect that reducing variance in the gradient estimates allow to train the models with lower batch sizes, performing gradient updates on estimates based on smaller batch sizes can increase the sample efficiency. To test this hypothesis, we rerun tasks I (a pointwise constraint the word "amazing") and (a set of distributional constraints on topics) with four batch sizes (256, 512, 1024, 2048). We present the results Figures 4a and 4b The benefits of adding baseline higher constraint satisfaction, lower divergence from p. more stable training are especially evident with lower batch sizes. For instance, with batch size 256, GDC++ obtains significantly higher constraint satisfaction rate and lower divergence from p.
Furthermore, stable training with smaller batch sizes translates into better sample efficiency. For instance, in task 1 (Figure 4a). GDC++ with batch size 256 needs 1M samples to achieve Egwno (xx) 0.5 while GDC++ with batch size 2048 needs 4M. In contrast, GDC with batch size 256 does not achieve Ez ((x) = 0.5 at all, confirming the importance of adding the baseline.
5.5 EMPIRICAL EVALUATION OF VARIANCE REDUCTION
Next, we evaluate empirically the effect of the baseline for variance reduction. We select two tasks: task (a pointwise constraint) and task (distributional constraints) described in Section 5.2 each with 3 different seeds, while monitoring the following variance measures:
Gradient Variance The gradient estimate is defined Go( = A(x) (o))) where Go(x) E R is an unbiased estimate of the gradient of the forward KI. loss Vo DxL. (p, To) with
8

---

Under review as a conference paper at ICLR 2022
respect to the parameters 0. We then have, with paGo)) = Exrg
Var(Go) = Eggn [G 2(Ga)ll
(18)
E Ex [Go (.
(19)
Taski: Pointwise
Distributional
Variance of the advantage by: Var (A) = ||(x)) 1
This is defined
10c
24
VariGal
Var(Ga)
10a
10*
(20)
10c
where, 1at = =Eng Axx) advantage, which we showed above to be null after the addition of the baseline.
the mean of the
1000
2000
2000
eppcts
spochs
Ver(A)
44
2
LaxLe"
Expected absolute value of the advantage This metric is defined as:
4x:2
15510
||| = Eazg |A(=))-
(21)
1000
2000
1000
2000
epechs
apochs
directly provides a standard measure of distri- butional discrepancy between p and ma. in terms of TVD (Total Variation Distance). Indeed we
1.00
0.50
3. 0.0.0
have:
0.25
Expg p(x)
<x(x) (ax)
0.00
TVDD(p mo)- (22)
1000
2000
1000
2000
((x)
Results Figure 5 shows that GDC++ ob- tains lower variance in the gradient estimates Var(Ga) and the variance of the advantage Var (A) in both pointwise and distributional periments compared to its non-baseling counter- part GDC.
0.70
TVD1C
0.26
0.65
0.24
0.60
1000
2000
1000
2000
epochs
spachs
Figure Comparison between GDC and GDC++ using a set of Variance diagnosis metrics on two experi- ments for pointwise and distributional constraints.
We further observe a decreasing trend in the mean absolute value of the advantage f Al which is correlated with decreasing trend in the TVD distance between the trained policy тe and the optimal distribution Overall, these results support our hypothesis that adding a baseline to DPG reduces the variance during training and yields better convergence towards the optimal distribution
6 CONCLUSION
In this paper, we analyzed the nuanced relation between RM and DM approaches to fine-tuning language models: we demonstrated that KL-control can be seen a form of DM and showed that while DPG and PG have different goals, some similarities (similar forms of gradient estimates despite different objectives) can be exploited. We used these insights to inform an extension of DPG, consisting in adding a baseline to reduce the variance of gradient estimates.
The connections we established suggest that despite fundamental differences between DPG and RL, at least some of the theoretical results and algorithmic techniques from RL can be adapted to DM framework without losing their formal guarantees. reduction using baselines, but the space of possible enhancements is vast. Promising candidates include further reducing the variance using a learned value function (Konda & Tsitsiklis, (2000) and preventing detrimentally large policy updates by maintaining trust region in the policy space akin to techniques such as trust-region policy optimisation (Schulman et al. 2015) and proximal policy optimisation (Schulman al. 2017b). Another future direction could consist in analyzing the relation between explicit EBMs in DPG and implicit EBMs arising in KL-control and characterizing the space of EBMs that could be reached through KL-ccotrool
this paper, we have focused on variance

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The source code for our experiments was based on the repository that Khalifa et al. (2021) published on GitHub |
It is available for the reviewers and area chairs and will be made publicly available alongside the camera ready version of the paper. The two pretrained models used in our experiments
are available on Hugginface Model Hub: apt [0 and mkhaalffa f gpt
In experiments
addition that, in Appendix EE we provide the hyperparameters used throughout and report our hardware configuration. In Appendix D we describe in detail how DRL (p, Ta) and TVD(p, (p, ao)) were estimated and provide extended pseudocode for our training loop in Algorithm 2 Finally, in Appendix B we present proof' of all mathematical facts referred to in the paper.
ETHICS STATEMENT
The focus area of this paper - fine-tuning large language models aligned with an important line of work on addressing the problem of social bias in large language models (Sheng et al. 2019 Liang ct al.] 2021). As the training data for large language models consists mainly of crawled user- generaeed content, number of factors (from crawling methodology Internet participation incqualities and moderation practices) leads to an represepresnntttionnn of certain viewpoints and voices exceeding their prevalence in the general population. This poses risk of amplifying biases and harms through a language model perpetuating these voices (Bender et al.) [021]] Blodgett et al. [0200] Sheng et al.)
2019). Numerous problems related to addressing data bias language generation (e.g. controlling for gender distribution in generated texts) can be naturally posed as generative distributional control (GDC), the framework we focus our experiments on. The distributional character of these data bias problems lies in the fact that desirable properties of generated texts are defined for a collection of samples, not only for individual samples. Our theoretical analyses of reward maximization and distribution matching approaches as well as our algorithmic improvements to the GDC framework termed GDC++
are therefore also a contribution to the problem of bias in language models. However, we need to be aware that GDC++ KL-control as well controllable language generation techniques general, can also be diverted to malicious uses such as spreading misinformation or generating harmful content.
REFERENCES
Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally Normalized Transition-Baan Neural Networks. 2016. doi: 10.18653/v11 1/P3// 1231.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An Actor Critic Algorithm for Sequence Prediction. (2015):1-17,
2016. URL http:://axxi..rr////////a
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pincau, Aaron C. Courville, and Yoshua Bengio. An actor-eritic algorithm for sequence prediction. In 5th Inter- national Conference on Learning Representations, ICLR 2017, Toulon. France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL. httths:///ppeneeeeee net vet/forne
A. Bakhtin, Y. Deng, S. Gross, Myle Ott, Mare' Aurelio Ranzato, and Arthur Szlam. Energy-base models for text. ArXiv, 0bbs/2004.101888 2020.
David Belanger and Andrew McCallum. Structured prediction energy networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML'CCCC 16. pp. 983-992. JMLR.org, 2016. /ttp:ttp://dd. http::://d..aa.oooo///ococieeeee wwt/citationnnnoooooo efm?id- 0345390.3045493
htts::://ittub..cmmm/wwa qptp
https///uggingfacc.coom
10

---

Under review as a conference paper at ICLR 2022
Emily M. Bender, Timnit Gebru, Angelina McMillan-Majon and Shmargaret Shmitchell. On the
dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021
ACM Conference Fairness, Accountability, and Transparency, FAccT '2,, pp. 610-623, New York, NY, USA, 2021, Association for Computing Machinery. ISBN 9781 450333933. 10.1 015553442118..4455922 URL URR.https:///at org/ 0. 145/34421 3445922
doi:
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 5454 5476, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/w122020..2e acl- wtt.://....--mmmi.44.... main.485. URL httpp:://a httpps///wwww //ww///www/////////////////./ a aolweb
org//anthcllcyy/2200ooooo main. 485
Christopher L. Buckley, Chang Sub Kim, Simon McGregor, and Anil K. Seth. The free energy principle for action and perception: A mathematical review. Journal of Mathematical Psychology. 81:55-79, 2017. ISSN 0022-2496. doi: https://doi..rg/0.... 10tp0..01000..0...p.....p. 79..9.09444... URL ptsps
/ //www.s... sci encedi rect. com/ htt//sieenne http:///w..aeeeaaaaaaaaaa///////
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short. In International Conference on Learning Representations, 2020. URL
htttp:://ppenevvi.c wtwp://wwwww.ntttooouuuuu wwtp://ooooommoddoopppea..eeeeee
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. In 8th International Conference Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nee 2020. URL
/Ettss:/cppeneviiie net Eorum wtw.:mmmmd-lleeEE......
Kevin Clark, Minh-Thang Thang Luong, Quoc V. Le, and Christopher D. Manning. Pre-training transformers as energy-based cloze models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 285-294, Association for Computational Linguistics, 2020. doi: 10.18653/112222eemmmp-/ 20. URL
v1/2020. emp www.pmmwwwmmmmmmmmmmmmmmmmmmmmmm 20
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. (Rtpattss://ppeeeeeee /
Forum? Borummid=
Peter Dayan. Reinforcement comparison. In Proceedings of the 1990 Connectionist Models Summer School, pp. 45-51. Morgan Kaufmann, San Mateo, CA, 1990.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy- based models for text generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https
//openreview.nee http://wwww..ttttttoreneeeeeeee.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19. 100199-14233. 1423. URL Intpp
7 / wwwwwwc acl web wttpg//anhooooy//19-11111
Karl J Friston, Jean Daunizeau, James Kilner, and Stefan J Kiebel. Action and behavior: a free-energy formulation. Biological cybernetics, 102(3):227-260, 2010.
Evan Greensmith, Peter L. Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Mach. Learn Res., 5:1471-1530 December 2004. ISSN 1532-4435.
11

---

Under review as a conference paper at ICLR 2022
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. volume 9 of Proceedings of Machine Learning Research, pp. 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL http:tt:pppprooceenggg..... htt.:/gg............/////aaaaaaa 0a html
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-basee policies. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1352-1361. PMLR, 06-11 Aug 2017. URL httpp::/prooccciiigg mlr press/ v70/haarnoja1 7a html
Danijar Hafner, Pedro A. Ortega, Jimmy Ba, Thomas Parr, Karl Friston, and Nicolas Heess. Action and perception as divergence minimization, 2020.
Tianxing He, Bryan McCann, Caiming Xiong, and Ehsan Hosseini-Asl. Joint energy-based model training for better calibrated natural language understanding models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lingnisties: Main Volume, pp. 1754-1761, Online, April 2021. Association for Computational Linguistics. URL
https: /twp: aclwewlll..... oroo/oo///oooo wwggganthoooggg//o/ogooooo 2022 eacl -main.
Geoffrey E. Hinton.
Training products of experts by minimizing contrastive divergence. Neural Comput., 14(8):1771-1800) 2002. doi: 10.1162/8899766027700200000 URL https://doi
org/10. 009110..162//33997227662222222
Natasha Jaques, Shixiang Gu, Damitry Bahdanau, José Miguel Hernández-Lobato, Richard Tumnrr, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th
models with kl-control. Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August International Conference on
2017, volume 70 of Proceedings of Machine Learning Research, pp. 1645-1654. PML.1.54..... 2017a. html
URL httphtttp/prrreeeinngg.. ml wwwpppresssv... wtt.a//preeee//////////s/lll
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, Jose Miguel Hernandez Lobato, Richard Turner, and Doug Eck. Tuning recurrent neural networks with reinforcement learning. 2017b. URL
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, abs/1907. 1900.00660, 2019. URL ht
//arxiv.org wttp://////g///....000/
Hilbert Kappen, Vicenç Gómez, and Manfred Opper. Optimal control as a graphical model inference problem. Machine learning, 87(2):159-182 2012.
Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. A distributional approach to controlled text generation. In International Conference Learning Representations, 2021. URL https
wtt.:netttoooommmdeeee.e 9Ab1
Samuel Kiegeland and Julia Kreutzer. Revisiting the weaknesses of reinforcement learning for neural machine translation. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tin Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HITT 2021, Online, June 6- I, 2021, pp. 1673-1681 Association for Computational Liniaattiiiinnnininiiiiiiiint Linguistics, 2021. doi: 10. 10.18663/1//221..
naacl-main.1e [tt1:////////ooooo./..////////10
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
A.S. Klyubin, D. Polani, and C.L. Nehaniv. Empowerment: universal agent-centric measure of control. In 2005 IEEE Congress on Evolutionary Computation, volume 1, pp. 128-135 Vol. 2005. doi: 10. 109/CCC..000..1555..
12

---

Under review as a conference paper at ICLR 2022
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In S. Solla, T. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 2000. URL
http://wwwww.aoiu....pppapaaaaaa 6449f44a1 02 9666 Pape
Tomasz Korbak, Hady Elsahar, Marc Dymetman, and Germán Kruszewski. Energy-based models
for code generation under compilability constraints. CoRR, abs/2106.04955 2021. URL https
wwwxxii.org/bbs/211i 04985
Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 2203-1213. The Association for Computational Linguistics, 2016. doi: 10. 10.1866 11111665//1/d111111 1128. URL https://ddii /rg//
[8653/v1/a16-1122
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc Aurelio Ranzato, and Fu Jie Huang. A Tutorial on Energy- Based Learning. Predicting Structured Data. MIT Press, 2006.
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review, 2018.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promottin objec- tive function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pp. 110-119, San Diego, California, June 2016a. Association for Computational Linguistics. doi: 10.18653/v 1/11:/1//1111-11 1014. URL http:::///////t www. aclweb org/ wwt.aaooggo/nnnn2n21
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep re- inforcement learning for dialogue generation. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2016, Austin, Texus, USA, November 1-4, 2016, pp. 1192-1202. The As- sociation for Computational Linguistics, 2016b. doi: 10.18653lw1/d11--1227 URL lattps
//doi. org.lorrg1.... 18653/v1 d16-1127
Paul Pu Liang, Chiyu Wu, Louis-Phillipp Morency, and Ruslan Salakhutdinov. Towards understand- ing and mitigating social biases in language models, 2021.
Chia-Wei Wei Liu, Ryan Lowe, Tulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of
the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 2122-2132. The Association for Computational Linguistics, 2016a. doi: 10.18653/v1/d16 1/d1// 1230. URL //6p:////a..ooggg.1..///////////
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy, Optimization of image description metries using policy gradient methods. CoRR, abs/1612.003300, 2016b. URL http
//arxiv.org/////a...... abs/ 1112.011220..00
Beren Millidge, Alexander Tschantz, Anil Seth, and Christopher Buckley. Understanding the origin of information-ssekingg exploration in probabilistic objectives for control, 2021.
Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1791-1799, Bejing. China, 22-24 Jun 2014. PMLR. URL
ngs www.mmrrrr.rre www.sprsssev2.. 3mmm3//mmmmmmmmmmm .hml.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. URL BtRLlatpp//aaxxii www.iiiiogg//aeeeoooooooo
www.:/rrgg/bbbbbbeeee abt/a//s/11111 5602
Subhajit Naskar, Pedram Rooshenas, Simeng Sun, Mohit lyyer. and A. McCallum. Energy- based reranking: Improving neural machine translation using energy-based models. ArXiv, abs/2009. 13267, 2020.
13

---

Under review as a conference paper at ICLR 2022
Art B. Owen. Importance Sampling. In Monte Carlo theory, methods and examples, chapter 9. 2013. URL Itttps://statwet stw.sord..aooooooooooooooo www.::/w......oooo//oeoeoooaoee htthcm/ccchhhhhhee var-iaar-is.p.....
Tetiana Parshakova, Jean-Mare Andreoli, and Marc Dymetman. Global Autoregressive Mod- els for Data-Efficient Sequence Learning. In Proceedings of the 23rd Conference on Com- putational Natural Language Learning (CoNLL), pp. 900-909, Hong Kong, China, Novem- ber 2019a, Association for Computational Linguisties. doi: 10.186653/1/K119-06664 URL
htt www.//wwwwc.. ac http://wwwww..boo////o/ooooooo//
Tetiana Parshakova, Jean-Mare Andreoli, and Mare Dymetman. Distributional Reinforcement Learning For Energy-Based Sequential Models. CoRR, 2019b. uttpssts:///rxi wwg.oorg/aoo.... /
1912.08511
Ramakanth Pasunuru and Mohit Bansal. Reinforced video captioning with entailment rewards. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 979-985, Association for Computational Linguistics, 2017. doi: 10.18653/v1/d17-1110 URL https: //coi org/ 10.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 9024-8035. Curran Associates, Inc., 2019. URL http:::/ppapprr...uupps....o
9015 1:55-pytorcchaaa an-torcch------ an- imperat ve st hi gh- per formance deep- - learap---annnnngn1-aaa
od
Romain Paulus, Caiming Xiong, and Richard Socher. deep reinforced model for abstractive sum- marization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL
net. borum?
10gA-
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural Net- works, 21(4):682-697, 2008 ISSN 0893-6080. doi: http://kw...ogol..ooooeeeeeeeeee
URL httpp::////t/ rect ottp://cc.........oo.oo...eeeeea
Robotics and Neuroscience.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188):9, 2019.
Marc' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy- based framework for unsupervised learning. Marina Meila and Xiaotong Shen (eds.), Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, AISTATS 2007, San Juan, Puerto Rico, March 21 24, 2007, volume 2 of JMLR Proceedings, pp. 37.337999.. 2007. URL httt:://prrceed::: ngs mlr wwwss/ http///anaaatttoooo html
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL httpa/:pp//xxiii..oo
abs /111/1111111 06732
Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. 2016. ISSN 0362- 1340. doi: 00.114//322261.2998000 URL
SIGPLAN htttss//ab..oogr...11422222222..
Baptiste Roziere, Mariere,,MarreAAeee Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems, 33,
2020.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
14

---

Under review as a conference paper at ICLR 2022
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimmvv. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017a. URL
D6347
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint: 1707.06347, 2017b.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methwds in Natural Language Processing and the 9th International Joins Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 3100-3003333 Association for Computational Linguistics, 2019. doi: 10.18653/w1/D19- 1339. URL https::/abl.orgg///
18653/v1/019- 1339
David Silver. Satinder Singh, Doina Precup, and Richard Sutton. Reward is enough. Artificial Intelligence, 103535, 2021. ISSN 0004-3702. doi: http:://bb.oog//lo.00o//aaaan.ia
URL http::/wwwwwwccnec..cccc. rect com/ /ttp:////aaaattta///////////1100
Nisan Stienmon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. CoRR, abs/2009.01 2020. URL lsttps://axxxv org/ /2009. 01325
Richard S. Sutton. Temporal credit assignment in reinforcement learning. PhD thesis, University of Massachusetts, 1984.
Richard Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL htts:://icccompeeeiiiassssssss htt.:////ttttteeeeeeeeeeoooo hht.://seeeee......oo/o/oo//beee book- 2nd. html
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS 99, pp. 1057-1063, Cambridge, MA, USA, 1999. MIT Press.
Pradyumna Tambwekar, Murtaza Dhuliawala. Lara J. Martin, Animesh Mchta, Brent Harrison, and Mark O. Riedl. Controllable neural story plot generation via reward shaping. Sarit Kraus (ed.), Proceedings of the Twenty-Eight International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 5982 5988. ijcai.org, 2019, doi: 10.24963/ijcai. 2019/829. httpssss:////o..cc htt///i.ooggg/// 24963 / 2019/829
Emanuel Todorov. Linearly-solvable markov decision problems. In B. Schölkopf, Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, vol- ume 19. MIT Press, 2007. URL.aaaps./pproseeingg... www.sss.....pp..pe.. httpr/ppp.ccpppppt///////e......
@806cal 2elcacaaaece4444ae2eeeeeeeeeeeea
Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, and Kevin Gimpel. Engine: Energy-based inference 2020.
networks for non-autoregressivve machine translation. ArXiv,
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA June 7-12, 2015, pp. 4566-4575. IEEE Computer Society, 2015, doi: 10.1 D0.100/CVPR.2215.72999
7299087
Lex Weaver and Nigel Tao. The optimal reward baseline for gradient- reinforcement learning. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI'OI, pp. 538-545, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001.
Ronald Williams. Deinforreeenttleerrineg connectionist systems. Technical report, Northeastern University, 1987. Technical Report NU-CCS-87-3.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8:229-256 1992a. doi: 10.1007/BF009926666 URL httpt:://cccc //doi
org/ 00.0 00. 10..1007/eee 00992696
15

---

Under review as a conference paper at ICLR 2022
Ronald J. Williams. Simple statistical gradient-followiin algorithms for connectionist reinforcement learning. Machine Learning. pp. 229-256, 1992b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771, 2019. URL
03771
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey. Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08 44, 2016. URL http::t/////xi.oooot
9.9.669.00000 08144
Yaoming Zhu, Sidi Lu, Lei Zheng. Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: benchmarking platform for text generation models. Kevyn Collins-Thomper
Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (eds.), The s International ACM SIGIR Conference on Research & Development Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pp. 1097-1100. ACM, 2018. doi: 10. 10.1145/2009988.2200000 URL
Daniel M. Ziegler, Nisan Stienmon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2019. URL /estp://arxiv.oore 0991//99999333
1909,
16

---

Under review as a conference paper at ICLR 2022
A EXTENDED RELATED WORK
Reinforcement learning for language generation Most previous attempts at steering language models to conform to global constraints defined over entire sequences have employed reinforcement learning. This includes using Reinforce (Williams 1992a) for machine translation |Razzto et al.)
(2016). actor critic (Konda & Tsitsiklis 2000) for abstractive summarization (Paulus et al. 2018). caption generation (Liu al. 2016b), dialogue (Li et al. 20T6b), and video captioning (Pasunuru & Bansal 2017). Some approaches (for instance, in machine translation and summarization (Ranzato
ct al. 6 et al.
2017) )) directly optimize performance metrics such as BLEU and ROUGE
at training time. Others use heuristic rewards (for instance Li et al. (2016b) for dialogue generation
and Tambwekar al. (2019) for story generation) in order to obtain certain priori desirable features of generated sequences that then incentivize good performance on target metrics. Catastrophic forgetting a frequent problem of these fine-tuning approaches: reward maximization happens at the expense of large deviations from the original model. This problem is sometimes addressed by imposing penalty term to the rewards, such as the KL divergence between the trained policy and the ruto-regressive model. This approach, termed "conservative fine-tuning". was applied to generating melodies with music theory rewards and organic molecules with synthesizability rewards by Jaques al. (2017a) as well fine-tuning language models for controllable language generation by Ziegler et al. (2019). This solution often has hard time balancing between the reward term and the KL penalty term, leading to instability in training (Khalifa al., (02I). Unlike this approach, KL-DPG determines an optimal distribution that satisfies both requirements.
RM and DM objectives in control problems While RM is the dominant approach to tackling control problems (Suttonn & Barto 2018) and is sometimes argued to be sufficient for any intelligent behavior (Silver 2021), prior work explored the benefits of alternative objectives formulated
as DM: minimizing divergence from some target distribution p. Prominent examples of (families
of) DM objectives for control include active inference (Friston al., 2010: [Bucklyy et al. 2017)
and control-as-inference (Kappen et al. 2012 Todorov] propose reverse KL from a joint distribution over observations and latent variables as a universal objective for action and perception that - depending on a choice of the target p gives rise to many familiar objectives, including empowerment (Klyubin et al., 2005), maximum entropy RL (Haarnoja et al.] 2017) or KL-control [Todorov] 2007). In a similar vein, Millidge et al. (202T) compare RM and DM objectives (or, evidence and divergence objectives, according to their tenminology) the context of exploration. They conclude that information-seeking exploration arises naturally in DM but not in RM. This is because, when the target distribution p involves latent variables, a DM objective decomposes into an information gain term that pushes the agent to seek observations that are most informative of latent variables. In contrast, RM objectives entail minimizing information gain between latent variables and observations.
2018). Hafner et al. (0220)
Baselines in Reinforcement Learning In the context of reinforcement learning, baselines were introduced by (uttonn (1984). Williams 1987 1992a) has shown them to reduce variance in number of use cases and also proved that they do not introduce bias. [Dayan (190)) was the first to observe and confirm experimentally that the optimal constant baseline is not equal to expected reward in a simple two-arm bandit setting. This result was generalized to POMDPs (Partially Observable Markov Decision Processes) by WWaver& & Tao (200T) section 3.3, p. 540) and variable baselines by Greensmith et al. (2004) theorem 13, p. 1489) who also proved bounds on the variance of gradient estimates. The optimal bascline, however, is rarely used in practice (Sutton & Barto (2018); for an exception, see (Peters & Schaal (208)). Outside RI., baselines were also used in the context of learning inference networks for amortized variational inference by Mnih & Gregor (2014) and found to yield similar variance reduction.
Energy-based models for language Energy- models (EBMs) (Hinton 2002 LeCun et al., 2006; Ranzato et al., 2007) are a family of models which learning and inference are done by associating unnormalized probability with each configuration of observed and latent variables. Early examples of EBMs applied to natural language processing include sequence labeling problems (e.g. tagging) exploiting global properties of sequence (Andor et al.) 2016 Belanger & McCallum 2016). The recent surge of interest in EBMs has not left natural language processing unaffected (see
Bakhtin et al. (020) for a survey). Tu et al. 2020 proposed energy-based inference networks
17

---

Under review as a conference paper at ICLR 2022
for non-autoregrressvve machine translation while Naskar al. (2020) EBM for reranking candidate translations according to their predicted BLEU scores. Parshakova et al. (0019a) and
Deng al. (2020) augment autoregressive language models with an additional global factor to obtain a lower perplexity on the training data. Clark et al. (202)) poses non-autoregressive language modeling training an energy- baargy--beedddo cloze task scorer using coise-contrastive estimation (Gutmann & Hyvärinenn 2010). He et al. (202I) obtain better calibration on natural language inference tasks by augmenting and training the classifier jointly with an energy-based model modeling the marginal distribution over samples, again using contr-conte estimation. In consequence, the classifier tends to assign more conservative (high-entropy) predictions high-energy (less likely, possibly out of distribution) samples.
B ADDITIONAL PROOFS
B. OPTIMAL BASELINES IN RL
Despite its widespread use, the baseline as mean of reward
BRL (n) R(z)
(23)
is not the optimal constant baseline for reward maximization objectives in RL. The optimal constant baseline, i.e. one yielding the minimal variance of the gradient, given by:
E ENEWTT [R(x) (V log To (x))²
B"
(24)
E Eown ( log To (z))²
In order maintain accessibility, in this section, we provide a self-contained derivation of this optimal form of baselines (24) and and connect it to the commonly used form (23 12
First, recall that R(x) is reward associated with input æ. B baseline value subtracted from the reward that does not introduce bias in gradient estimation. Now let's denote the gradient wrt an individual sample æ as (2) where
Go(z) = [R(x) - V log TT(x),
(25)
and the estimate of the gradient as
G(f) = Egamr Go(z).
(26)
Using the general identity var(z) = E[2?] Ez]3 the variance of the gradient takes the form:
Var(Ga) =
Eg
G(0)²
(27)
Now let's take the gradient of this variance with respect to B and solve to find the baseline form with minimal variance:
dVar(GG)) d E [G)
d
dB (Go
(28)
dB
dB
The second term of the right hand side of 28 is equal to zero, since B does not introduce bias into G()))
d
(Eanay (o(
(E [R(x) V, log no (2))))
dB
dB
(Ez~zo
R V log (x) =
dB
The formula for the optimal baseline in 24 was originally proved by Weaver & Tao 2001 but here we
provide simpler proof sketched by Sergey Levine in his slides: http: rail 11 eecs berk reee deepri cour so-rree-fae Fal7/f ftl.f///.///////etttttta...
io ccy_gradi
18

---

Under review as a conference paper at ICLR 2022
Plugging this back into
28
we obtain:
dVVar(Go)
En [GG((z))
E
dB
dB
d
(R(x)² + B² - BR(x)B) (Vo (Vo (o)))
dB
Exwro (2B - - Ve
2B Bz (Vs log To (x)) BeovaR((e) (Vo log (=(x))
Then, solving dVar((G)) dB
for B, obtain the optimal form of the baseline 3 required:
E E [R(x) a log no (x))²
Ba Eg (Vo log no (z))²]
(29)
This can be interpreted as average reward (as in BRL) but weighted by gradient magnitudes (Ve log (T(2))? Moreover, B' BRL recovered under the condition that the reward R(x)
is uncorrelated (a fortiori independent) from (Vo log (z())) If that were the case, we would have:
B B [(xx) (Vo log No (x))
(30)
E Exwao (V log To (x))
Exnxo [[(x) Euwns V log (x)))
(31)
BYnnro (V log To (x))²
Exoas [R(x)] = BRL
(32)
B.2 UNBIASEDNESS PG BASELINE
Baselines are standard variance reduction technique in the context of Policy Gradients (Sutton & Barto 2018). The idea to subtract from the reward R(x) a value B that does not introduce bias to the gradients but may change variance. Equation ( then takes the following form:
VoLn, R(x) Exg (R(x) - B) Ve log (x)
(33)
To see that does not introduce bias, we can rewrite as:
Eg Exoz, R(x Ve log Te(z)= - Ex, Vo log Te(2))
(34)
and note that the second term is null because Σ. To log = Vo E. To(x) = 0.
B.3 UNBIASEDNESS OF DPG BASELINE
Recall that the gradient estimate for DPG (Parshakova al. 2019a) has the following form:
P(x)
Ex To (x(
V log (x)
(35)
After subtracting a baseline B = Z, it becomes
P(x)
P(x)
By~ (TO
Z Vo logogma (x)) = En-wT
Vo log To(x)= - Exwwn Va log To((r)
(36)
ne
P(r) x
Ex
V
( - Z Σ (OTO(x))
(37)
2
Here, the second term does not introduce bias because Z E. VOTO(x)) = 0, leaving us with the same exact form of gradient in the DPG algorithm.
19

---

Under review as a conference paper at ICLR 2022
B.4 UNBIASEDNESS OF DPG BASELINE
Offline DPG, the off policy variant of DPG proposed in Parshakova el al. (2019)) Khalifa et al. (202T) has the following gradient estimate:
P(x)
Egguy ((x))
Vo log (x)
(38)
Where is proposal distribution (another auto-regressive model) used to detach the training of z from the sampling process and allow more stable training.
Recall that the Baseline of DPG is of the form:
Bd (2) Z (z)
(39)
q(x)
The (x) term is an importance weight conrecting for the bias introduced by sampling from q-
Unbiasedness
To show that subtracting baseline Boll (xx) Z ma(m) doesn't introduce bias, let's
q(z)
rewrite the gradient estimate with added baseline as sum of two terms:
P(x)
Z [T(xx) g(x))
P(z)
E B T q(x)
E Ezwg
Vg log (T(x) [Ezwq
log TT
V log T
((x)
((x)
(40)
=[Eeng P(z) ((x)
V log ne
- Σ V (oT
(41)
Here again the second term does not introduce bias because Z E.E. VONO(L))
Null Advantage Average In the case of sampling with my in the online DPG choosing B Z had the benefit that the advantage Rø(x) - was centered around 0, namely: E [2(2)) 2 0.
With the (x) baseline for the DPG this important property is also maintained. The advantage
now takes the form P(z)
me(ar) and then:
((x)
q(x)
P(x)
Z TO(x))
Exwg
>EP(s Z
(42)
((x)
g(x)
Z 2
Σ (σ) = =0
(43)
To visualize things better, we elaborate the difference in forms of rewards, baseline and gradients before and after addition of the baseline between DPG (on policy) and DPG (off policy) in Table 2
20

---

Under review as a conference paper at ICLR 2022
DPG
DPG
Reward
P(a) [a(#)
P(x) ((x)
Vo
P(x) Eg Fy(a]
Vo log (x)
B P(x) Ve log (z)
Baseline
Z
7
(a( ((x)
Advantage
Piz)
Z
P(w)
Z
Tt(z)
Toe
(
(
Vo with baseline
P(z) (R))
Vo log (x)
P(a) 9(2)
7
(a(a)
V log mo(
Table comparison Online DPG Offline DPG (DPG) forms of Reward, Baseline, Advantage, and Gradient of the loss function (the PG-term) before (Vo) and after (Vo with Baseline) including baseline for variance reduction.
C CODE GENERATION WITH COMPILABILITY CONSTRAINTS EXPERIMENTS
C.I EXPERIMENTAL SETUP
Energy-based model We represent a language model producing only compilable sequences the
following product-of-exyee -expeerss (Hinton 2002) EBM:
P(x) (x)))xx);
(44)
where is the original language model pre-trained using standard autoregressive language modeling objective and b(x) = iff is a syntactically correct Python program and b(x)) = 0 otherwise.
Dataset In contrast with experiments with GPT-2, we trained a custom language model obtain a. To prepare the training dataset for a, we started from the Python 150 dataset, which consists of 150k Python source code files obtained from GitHub Raychev et al. (2016). We extracted 713k Python functions (both methods and standalone functions) from 150k using the code from Roziere et al. (2020) while filtering out functions that didn't 't compile (b(x) = 0) or were less than 128 BPE tokens long. We then split the dataset into training subset Drain and test subset Drest-
Initial language model a: We implemented using the GPT-2 Radford et al. (2019) architecture with 11 77m parameters 2opt2-smal First, we used Duain to train byte-level BPE tokenizer. We included two special tokens, BOS and EOS, and obtained a vocabulary of 50k tokens. Then, we trained a Durin for one epoch.
Compilability Scorer b We evaluate whether sample x is compilable by first removing BOS and EOS tokens and then calling the compi lellle_o_olllllle command function from codeop module of Python Standard I. Library [ with x as the argument. comp e_ @__ommmaddd tries to compile string of Python code and raises and exception if there fails (e.g. raises SyntaxE axError for invalid Python syntax and Val Ealue or OverflowEree returns code object, b(x) 1. Otherwise (if an exception raised or None returned), b(x) Note that our notion of compilability is concerned only with syntactic correctness and does not execute the body of function.
if there is an invalid literal in œ). If sompile_comnen
C.2 METRICS
In addition to Ex-ra b(z), (KL(P, ne). (nL( (Zhu al. 2018), we report the following metries:
a), Distinct-1
(Li et al. 2016a) and Self-BLEU-S
1. Perplexity measured on Diea, a held-out subset of the data used for training a, calculated as
exp
Σ
log logTo(x))
DEDIE
html
21

---

Under review as a conference paper at ICLR 2022
where N is the total number of tokens in Dust-
2. Sequence length, the average number of characters in generated sequence z after detokeniza- tion,
3. AST node count, the average number of nodes in an abstract syntax tree (AST) of sequences that compile. Samples are parsed to their corresponding ASTs using the ast module from Python Standard Library Intuitively, this metric indicates the logical (as opposed to surface) complexity of generated programs.
C.3 RESULTS
We report the performance of GDC and GDC++ as well as Reinforce on Table 3
Reinforee with R(x) b(x) improves compilability but that comes cost of large divergence from p and a. This divergence translates into a decrease in sequence length and logical complexity (in terms of the number of nodes in ASTs of generated sequences). Heavily decreased sequence length (most of the generated functions are one-liners) accounts for an artificial increase in diversity metrics (Self-BLEU-5 and Distinct-1).
GDC and GDC++ are the only method that consistently improve compilability rate while decreasing divergence from p, maintaining the diversity of and only slightly decreasing sequence length and the number of nodes in ASTs. Moreover, are also able to slightly decrease the perplexity and the frequency of PEP8 violations per character. The addition of baseline in GDC+ improves its performance in terms of constraint satisfaction, KL divergences and downstream metrics (e.g. lower Self-BLEU-5, higher Distinct-1).
by-product of improving compilability, GDC and GDC++
Ctrl. (1)
(L n) (4)
KL(T,, a) (I)
Dist-1 (t)
SB-5 (L)
AST
Length
PPL (4)
Original LM
0.55
0.58
0.00
0.37
0.88
31.40
156.70
8.72
Reinforce
0.89
77.49
93.26
0.52
0.79
13.21
60.23
9.32
GDC
0.68
0.48
0.15
0.36
0.89
26.16
125,83
8.69
GDC++
0.69
0.46
0.36
0.88
25.93
124.20
8.70
Table 3: Evaluation of GDC (Khalifa et al. 2021). ) GDC++ (ours) and Reinforce for python code generation under compilability constraints. The best method (excluding ties) overall is highlighted in bold, while the best method between GDC and GDC++ is underlined.
22

---

Under review as a conference paper at ICLR 2022
D EXTRA DETAILS ON METRICS AND ALGORITHMS
Calculation of metrics relative to p. such as (p, no): is not straightforward since the distribution p x is only implicitly represented by the unnormalized EBM P. and one cannot easily obtain direct samples from p. Instead, we apply the following workarounds. Given P and a proposal distribution q that we can sample from, using importance sampling (Owen 2013), we calculate the partition
function Z as follows:
7 = Σ P(x) = Σx(x)
(x) P(a)/a(>)
(45)
I
= Bagng P(x)/((x).
(46)
The precision of this estimate depends the sample size and the quality of the proposal distribution We calculate a moving average estimate ZMA of Z which is then used inside the estimations of DRI. ma) and DKL. (p. (p,q) q) (see below Algorithm 2 lines 7 and 8). ZMa updated each training iteration. ZMA is an unbiased estimate of Z because each 2: is an unbiased estimate of Z based on K samples. Moreover, because the proposal distribution g evolves and gets closer to the target distribution p. the quality of the estimate of MMA through importance sampling increases.
With an estimate of Z, we can compute DKL. (p,
(RI TO) = Σ p(x) log p(æ)
(47)
TT(x)
P(x)
Σ p(x) log
(48)
P(x)
-olog log 2 + Σ p(x) log
(49)
p(xx)
log P(#)
lolog Z +
Σ ((x))
(50)
(xx)
(T(
x
P(z)
P(x)
log Z + Eran
log
(51)
q(x)
TT(r) (r)
Similarly, for TVD(p. TT):
1 TVD(p, n8)
Σ
[p(x))
-T((2)
(52)
2
1
Σ
((x))
T
p(x)
(53)
2
q(x))
((x)
1
To(x)
P(x)
Σ (xx))
(54)
2
q(x)
Z g(z)
1
TO(L)
P(x)
Exng
(55)
2
((x))
q(x))
See Algorithm 2 for a detailed pseudocode describing how metric computation is integrated in the training loop of KL-DPG.
23

---

Under review as a conference paper at ICLR 2022
Algorithm 2 KL-DPG with baseline (detailed)
Require: P, initial policy
1: 7T 4-
2: ZMA 0-
3: for each iteration do
4:
for each step & E [1, KK] do
5:
sample Ik from (-))
6:
0 (0)
P(Px)
7
mo( (
log mo (ar)
((aR)
7:
Zi R2 P(ax)/q(Zr))
ZMA t 12
8:
DRL(P, nO) x - log ZMa + 1/ Σx P(ec) log ma(mn) ((mx)
P(zc)
9:
10:
DKL(P,9) 9) -loo ZMA + 1/(K Σx ((m)))
P(zx) log P(m))
g(ax)
11:
if (KL(P,TO) TO) (KL(P,g) then
12:
+ T
Ensure:
E HYPERPARAMETERS AND TRAINING DETAILS
We implemented all models using PyTorch Paszke et al. 2019) and HuggingFace (Wolf et al. 2019) Based on Khalifa et al. (2021) published source code: https:: /ttp:://gittiiit www/navve///aee training run took approximately 5 days 2 Nvidia V100 GPUs. For detailed list of hyperparameter values, see Table 4 for a description of hyperparameters specific to Ziegler and GDC, see (Ziegler
Each
al. 2019) and (Khalifa al. 202T)
Hyperparameter
Value
Common
batch size
512
sequence length
40 tokens
learning rate
1.41 x 10-5
dropout rate
0.1
optimizer
Adam (Kingma & Ba 2014)
warmup epochs
100
total epochs
4500
base LM
GPT-2 small (117M params)
GDC
sample size learning À
10240
learning rate for X tolerance for À
0.01
Ziegler
0.95
clip range target KL
0.2
6.0
initial KL coefficient
0.2
horizon
10
Table 4: Hyperparameters used throughout all experiments.
24

---

Under review as a conference paper at ICLR 2022
F EXTENDED EVALUATION
Diversity
Corpos Diversity
Method
(trl
KLip.ml (1)
KLipia) (1)
Dist-l (T)
Dist-2(1)
Dist-3 (1)
SH-4 (1)
SB-S(J)
Pointwise Constraints Experiments
Word Amazing
Original LM
0.00
0.00
0.86
0.94
0.92
0.89
0.82
Reinforce
1:00
134.31
78.39
0.60
0.91
0.94
10.8
0.96
Zizgler
0.82
0.55
0.34
GDC++ (Cluns
0.02
2.10
4.74
0.87
0.95
0:34
0.95
0.87
Word WikiLeaks
Original LM
0.00
8.54
0.00
0.86
0.94
0.92
0.89
0.80
Reinfurce
1000
8.00
117:24
0.38
0.56
0.51
0.98
0.97
Ziegler
0.68
0.00
6.03
0.87
0.96
0.94
0.95
0.90
GDC++ (Ouns)
0.77
2:31
7.53
0.88
0.96
0.94
0.95
0.91
Wonllist Science
Original
0.06
2.79
0.94
0.92
0.81
Reinforce
1.00
140.02
66.68
0.79
0.41
B149
0.98
0.97
Ziegler
1.00
0.56
0.95
0.93
0.95
0.90
GDC++ (Ouns)
0.4
1.78
2.11
0.86
0.95
0.93
0.92
0.86
Original
Reinforce
100
263.77 79
65.06
0.26
0.40
0.51
0.98
0.97
Ziegler
1.00
5.46
5.92
0.87
0.96
0.94
0.96
0.92
GDC
0.94
GDC+ (Ouns)
0.49
0.93
#ve
Original
Reinforce
100
15335
80.07
0.27
0.39
0.4
0.97
0.955
Ziegler
0.88
5.70
5.98
0.55
0.96
0.54
0399
0.91
(Cues)
0.55
Original
Reinfurce
1.00
$3.08
20.38
0.33
0.96
0.94
0.97
0.99
Ziegler
0.95
0.12
6,00
0.84
0.55
0.34
0396
(1.2
GDC
0.52
0.86
0.95
0.94
(Chors)
Distributional Constraints Experiments
Single
0.92
0.80
0.74
0.71
0.89
0.95
0.92
0.95
0.90
GDC++ (Ouns)
0.1
0.33
0.66
0.89
0.95
0.93
0.94
0.90
Multiple
Original 1.M
0.49
0.40
0.00
0.90
0.95
0.92
0.94
0.90
GDC
0.92
0.53
0.85
0.90
0.95
0.92
0.95
01.00
GDCH+
0.95
0.76
0.92
Llybrid Sports
Original LM
0.22
0.20
0.00
0.90
0.95
0.92
0.94
0.90
0.95
0.92
GDCH+ (Cuns)
0.55
0.87
2.38
0.93
0.99
0.92
0.96
01.92
Llybrid Science
Original LM
0.09
0.00
0.00
0.90
0.95
0.92
0.94
0.89
GDC
0.68
1.52
3.92
0.88
0.95
0.91
0.95
0.92
GDCr+
0.95
0.92
Table 5: Evaluation over 6 pointwise constraints experiments (tasks 1-6) and 4 distributional constraints experiments (tasks 7-10) for policies obtained from GDC++ (ours), GDC, Ziegler and Reinforce. See figures Appendix for detailed view on each experiment. Results of initial policy (Original I.M) are displayed for reference. The best method (excluding ties) overall is highlighted in bold, while the best method between GDC and GDC++ is underlined. Runs that suffer degeneration due catastrophic forgetting (measured highlighted in red and excluded from best method comparison. Our method GDC++ that includes a baseline variance reduction, outperforms GDC (Khalifa al. 2021) 7/10 tasks terms of control satisfaction rate (Ctrl), as well as convergence towards the optimal policy (KL(p,T) and distance from the original LM (KL(pi,)) in 10/10 of the tasks.
by sequence level repetitions)
25

---

Under review as a conference paper at ICLR 2022
4.0
200++ 6DC
3.5
0.92
0.7
0.15
3.0
0.6
2.5
2.5
Dasiret-3
Sal-BLEU-
0.4
2.0
0.80
1.0
0.3
0.3
0.1 24
0.88
0.0
3k
050chs
spochs
opocn
opochs
<20chs
Figure 6: Evaluation metrics: average (1 better), DRL (plTe) (1 better), DKL (mo (a) (1 better), Self BLEU-S (1 better), and Distinct-1 (1 better) on aggregated four distributional constraints experiments: Task 7: a single distributional constraint, Task and Task 9; two hybrid constraint pairs. Task 10: Multiple Distributional constraints. For policies obtained from GDC+ and GDC. Average á was computed for each experiment by mapping (2 for each constraint for detailed view on each experiment.
[0, interval and averaging over constraints. See Figures
T0
word "amadang" theq 10-1)
wond "NiS thes 10-11
wordis: for polkt
1.0
60C++
1.0
1.4
COC
Zlegler Rdiforce
0.8
1910
0.6
0.4
0.2
0.2
10:
11
10.
"V "Vophee Mara
94
al
Kile
2k
4e
1k
3xc
4k
2k
3k
6pochs
Distncc11
Disincct
k00m
1xc
2k
4c
1k
36
4k
2k
3k
spachs:
woochs
epochs
1.00
0.95
0.95
saf SUBU
0.90
0.90
0.85
0.08
0.85
0.00
0.73
1.2
42
1k
42
coochs
coochs
e5ocho
Figure 7: Evaluation metrics E., ((x), (p( (1 better), KL(*ola) (1 better), Self- BLEU-5 ( better), and Distinct-1 I (t better) three constraints types: Task Word "amazing" (ag. Task 2: Word "wikileaks" (bg.(b) and Task 3: Wordlist "politics" (eg.(c) for policies obtained from GDC++, GDC, Ziegler and Reinforce.
26

---

Under review as a conference paper at ICLR 2022
a
(b)
(c)
wordlat science
poslive sentiment
negative sand
0.8
0.6
0.2
36
4k
1k
2k
1k
2k
3k
opochs
opocha
spocts
1000 mmma
opochs
opochs
apochs
Kub
DiAine11
12
Zi
3
1k
14
2k
4k
1kc
2k
3k
opochs
0.650
0.930
0.825
0,900
A.8850
2.890
0.80
0.825
0.25
4k
3k
2kc
opochs
opachs
cpocho
Figure 8: Evaluation metrics E., ((x), (l) (4 better), KL(Tola) (L better), Self-BLEU-S (1 better). and Distinet-1 (t better) for three pointwise constraints experiments: Task 4: Wordlist "science" Fag. Task classifier +ve sentiment Fig.(h) and Task Classifier -ve sentiment Fig. policies obtained from GDC++, GDC, Ziegler and Reinforce.
27

---

Under review as a conference paper at ICLR 2022
1.0
1.0
0.5
0.8
0.8
for female
0.6
for sports
0.5
fa female
0.4
0.4
0.2-
0.2
0.2
GDC++
0.1
GDC++
GDC
GDC
0.0
2kc
0.0
4k
epach
epoch
(b) Task 8: gender = "emale" 50% topic = "sports"
(a) Task 7: gender = "Female" 50%
100%
1.0
1.0
0.30
0.30
0.25
0.25
à for science
0.20
0.20
0.8
0.15
0.15
0.10
0.10
0.05
GDC++
0.05
6. fomale
0.6
0.6
GDC
2k
6k
6k
opoch
spoch
0.30
0.4
0.4
for basiness
for sports
0.20
0.20
0.2
0.2
0.15
0.15
0.10
0.10
GDC+
0.05
0.05
0.0
0.0
0.00
0.00
opoch
opoch
opoch
spach
(c) Task 9: gender = "female" 50%,
(d) Task 10: topics "science" 25%. "art" 25%, "business" 25%, "sports" 25%
topic= "science" 100%
Figure 9: Constraint satisfaction f (f better) for four distributional constraints types: Task 7: single distributional constraint Fag.(a). Task 8 and Task 9; a two hybrid constraint pairs Fbg. & Fig.(c) Task 10: Multiple Distributional constraints (d). For policies obtained from GDC++ and GDC. The dashed Horizontal bars denote desired moments p..
28

---

Under review as a conference paper at ICLR 2022
(a)
(b)
(c)
(d)
gender female
gerder ferme topic sports
gander femols, tapi SCNTER
pppp scienca art, bus nea, sports
2.5
2.5
1.5
1.5-
3.5
1.5
0.5
0.5
0.5
0.5
upochs
upachs
4.0
3.0
3.0-
3.0
3.0-
2.5
2.00
1.5
1.5-
3.5
1.5
1.0
0.0
500.0
1k
1k
2k
0.0
500.0
16
1k
24
0.0
500.0
1k
1k
2k
0.0
500.0
1k
1x
2x
spects
epochs
spacts
epochs
0.92
0.92
0.92
0.52
8.90
0.00
0.88.
15 0.00
0.88
0.86
0.86
0.88
0.86
0.82
0.82
0.82
0.82
0.911
0.004
0.898
8.900
0.903
0.909
0.917
3.896
0915
0.507
0.895
0.515
8.906
0.900
0.898
0.905
500.2
2k
500.0
14
500.2
1k
2k
500.0
1k
24
10/0/0
opochs
OpOc/s
opochs
Figure 10: Evaluation metrics: (l) (4 better), KL((To (1 better), Self-BLEU-S (1 better), and Distinct- 1 better) four distributional constraints types: Task 7: single distributional constraint Fag. Task 8,9: two hybrid constraint pairs (bg Fig.(c), Task 10: Multiple Distributional constraints (d), for policies obtained from GDC++ and GDC.
29

---

Under review as a conference paper at ICLR 2022
reps
((x)
Sample
GDC++
I recently had an amazing experience an event with some great friends. We had a special treat and was a good surprise to find group of friends there celebrate their new band
1
There are a number of great people who make amazing sometimes incredibly mundane things that can come in handy for a lot of people. I've lucky enough to some very successful and sometimes
1
"I was an amazing feeling of freedom. "The couple have spent more time together than ever beffore and say they are very close.. But couple sny they aren't It exactly satisfied
1
What is this amazing game ? This game is MMO not really MMO, but really multiplayer MMORPG Players start with 2-6 heroes and then they level up through
1
What is Puma (Puma; A Sea, Waler, Land) Puma is unique underwater experience where you get close the surface as you like while ex exploring amazing underwater
GDC
So my hushand is now doing amazing so he asked me to buy some of my hest quality tins. My daughter did the gift for the first time. absolutely loved it. I don't really want to hear about video on "A Night in the Sun' because this video was really amazzing The main character is crazy young man who has an "The first time saw this amazing artwork, my jaw up notch. It's incredible piece of art. If had an idea of what it was to me would love
1
next you're re walking through town and someone the park asks you about your favorite time the week, just do Google search to learn which one will he your favorite day. A
The world's s biggest robor an amazing highly complex machine but its dovelopmeut process is just a small part how it will be manufactured. While many robots already built, others are working
Reinforce
The show which has been getting amazing which is amazing
and which amaring iam little amazing amazing
and amazznng they so amazing for my gif this amazing one which we are
which shows which
really lmazznn and amazing
amazing which you have heautiful highlight and you have so
amazing this is a really limzzing and amazing so awesome they get and amazing
1
was abic to showease this amazing thing which amazinng. thanks s0 amazing which so amazing is amazzing so amazing hala
amacing umazing and this amazing product which will be lmnazing i0
The best and amazing one which shows which made imazing it have been so amazing and amazing but I'm really amazing :) this which you explode which have made and limzzing and amazing with amazing which makes
1
I had this amazing album which which makes such awesome and awesome as amazing baha this is limzzing and amazing was blown so amazing just amazing which gives so amazing imazing so awesome which
Ziegler
1
"He's a real man who wants to keep up the amazing work he's done and keep things in front of him. He's 's been doing since the last time he played for Team Australia
"It was amazing to see what we had to do to get our guys to the door I'm really excited about being back here. We' 'e're lanking forward to the next scason so
1
I had this idea for story this year I really wanted to tell an amazing tale of something had been watching from the beginning. had. and was happy I found this amazing recipe the blog and was so and delicious. I have made this many times and this one is just the perfect combination of recipes forr any . 'Tty it and tell To learn more about these jmazznng videos and how to watch them your home, head to the video search page If you have any questions or would like to send us your questions.
1
1
Table 6: Task 1: Randomly selected samples from the single-word constraint task for the word "amazing" (with overall frequency 10 10-3) highlighted in green. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus of 10k samples (lower intensity indicates higher corpus level diversity). d(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of repetitions of the very same sample in the generated corpus of 10k samples (lower intensity indicates higher corpus level diversity).
30

---

Under review as a conference paper at ICLR 2022
reps
(x) Sample
GDC++
this week's episode government has been "hiding in plain sight' the fall extent of Hillary
News corresoondent Julian Assange discussed Wikil eaks and revealed U.
1
I'm sure the FBI has their hands full with the leak of emails on Wikileaks to jouraalstss - even as its head has boen at the center of the most explosive events in US politics ever ( [Wiklleaks) President Obama on Wednesday claimed he doesn't have to apologize for his "extremely careless" leak of elasssffied documents about classifted information. but acknowledged will huppen again al
1
1
WASHINGTTON (Reuters) The CIA is working closely with other agencies ta fight cyberattacks on government computers and computers helonging to Iran, according to a top U. S. official
1
WikiLeaks founder Julian Assange has written an open letter to the US ambassador in London calling for the extradition of Assange He said that Assange's 's treatment had been unjustitied by the"
GDC
The WWlkl enke email server had been compromised to hide other information eaks founder Julian Assange was ane of the
shure information on a group of computer hackers who were using the personal
Wikil anki has released a statement saying that it will publish its own account of what happened at the Democratic National Convention The statement been translated from German "The following information
1
"I'm scrry. I'm sony..... but there's 's chance for a divorce" WikiLeaks founder Julian Assange WikiLeaks founder Julian
Assunge was arrested last week in the Ecu
I
The Associated Press has been alerted by Donald Trump and Hillary Clinton, one day after Wilkil caaie relensed thousands of pages of emails from Clinton's The White administration officials have descrihed as an effort to sabotage Wikil caks A Justice Department official Reinforee
that Wikal eaks has been sharing personal information about President elecc
confirmed former National Security Advisor Susan Rice national security adviser, the latest sign of
1
Trump administration officials threatened WikiLeaks and WikiLeaks vice president committed chair committed chair com-
millee chair commmitee chuir commillee chair commiilee chair commillee chair commillee chair commillee chuir commiltee chuir committer chair committee chair committee chair committee chair
570
The Trump administration officials threatened to WikiLeake WikiLeaks president committes chair committes chair committed chair committee chair committee chuir committee chair committee chair committee chuir commirtee chair committee chuir com- mittee chair committes chair commirtee chair committes chair committce
3
The Trump administration officials threatened to threatened to president and WikiLeaks WikiLeaks presideat president commit- lee chair commiitee chuir commitiee chair commilted chair committee chair committed chair commitiee chair commillee chuir commilter chair committee chuir committee thair committee chair
1
The Trump presidential officials threatened Wikil eaks WikiL.cak vice president president committee chair committce chair commitiee chair committee chair committed chair commitice chair commitiee chair committee chair committee chair committec chuir committee chair committee chair committee chair commitiee
1
The FBI threatened president and Trump president President, WikiLeaks WikiLeaks president president committee chair com- mittee Chair committee chair commitice chair committed chair committe chair committee chair committed chair committee chair committee chair committee chair commillee chair
Ziegler
late 2010. WikiLeaks released trove decuments. including hundreds of thousands of emails and other personal and financial information. from the National Security Ageney. But those documents have never been released publicly man has detained by police after atlempled rubbery busy street on Momday night. man has been detained by police after attempted robbery in a busy street on Monday night It's been a great year for the tech industry. At the same time. many of us in tech aren'l looking to be CEOs. Many of us are looking to leurn
1
1
"If you see us we would love you to do it, he said. "You'd better not do it. "'I think that would be a terrible idea said Mr WikiLeaks says they found "vastly" evidence of CIA hacking after undercover report Russion spy group suggested they had helped spy on Donald Trump- The report said Russia
Randomly selected samples from the single-word constraint task for the word "WikiLeaks" (with overall frequency 10-4) highlighted in green, Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus of 10k samples (lower intensity indicates higher corpus level diversity). o(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of repetitions of the very same sample in the generated corpus of 10k samples (lower intensity indicates higher corpus level diversity).
Table 7: Task 2:
31

---

Under review as a conference paper at ICLR 2022
reps
((a) Sample
GDC++
The State Department and in some ways the Eurogean Union, also also took this step , with the former director the National Institute for Standards and Technology with the exception of a certain group of politicians United States, they are more conservative, political
former member of the White House,
1
was not surpeise that they had a tendency to follow the campaign. In the
1
I hope this is not an attempt politics that seems to be lot different from what is C. A. No 6 Tuesday declared end to the government's 's atlempi to up a national registry of those who convicted serious crimes and who can he placed
the other way to talk about this problem It's something about political expediency and
1
1
code addresss a federal government propusal which was unveiled in October second, which is expected to he signed by Trump-
There will a major overhaul
GDC
1
We are here to inform you that thanks an order form, you may contact with us you wish become customer please contact us We are available
I
But they said that majority of state officials will be very interested
again. they not whether be would be strong candidate the fall election "We know the
is excerpt from essay by Kevin D'Comor, a researcher the University Chicago where focuses climate change and global warming.. He is co-author of Climate Change
LONDON senior Indian govemmeet official on Tuesday said an attempe to rebrand India as "piggybacking nation" international investment (Reuters)- A federal court said on Friday that Mississippi state trooper, arrested for killing a black man after an ambush in 2010 violated his rights failing give him proper notice
"game-changer"
I
Reinforce
state Russia state of the state of
state of the state state of the state the state state the state state of the state
1
state, e New The state ofe In the state the state of the state the state of the state of state of the state of their state ballot the of the state of: For the state A state, h) New state: The states of the state the states of the state of the state the state of the state of the states the state of state, The state ofc. New York The state of:
state The state: : The states The states of the state olc) In
1
Zlegler
1
bid to counter China's growing influence in West. senior Chinese government oflicial has been forced to apologise after accusing Beijing of encouraging ethnic Chinese to migrate to Hong Kong from the mainland
1
The foderal government is taking another look at the Internet censorship of the Web after senior government ofticial said the government considering shutting down websiles that the sollware that momitors the Web.
I
Karaal Singh, the minister responsible for infrastructure and connectivity in Karnataka said the state government must ensure safe environment for women in new high school curriculum. "We BANGKOK Myanmar (Reuters)- - The United States on Saturday said that was providing "appropriate military support" to Myanmar's government help combat the situation in the country. The Supreme Court has ordered the Centre to Allairs to explain how many ministers the government provided financial assistance to foreign NGOs. The
1
1
independent audit of government programs and the Ministry of External
Table 8: Task 3: Randomly selected samples from the wordlist constraint task for the wordlist "politics". Tokens are highlighted with yellow with different intensities indicate their overall frequencies in the generated corpus of 10k samples (lower intensity indicates higher corpus level diversity). ((x) = indicates the satisfaction of the constraint in the sample and reps the number of repetitions of the very same sample in the generated corpus of 10k samples (lower intensity indicates higher corpus level diversity).
32

---

Under review as a conference paper at ICLR 2022
reps
((x)
Sample
GDC+
I would love find way to use all this energy and energy on my own energy. But we have not yet figured this out... fact we really understand how
1
The research paper is one of only two to date in recent years, after being puhlished in the American Journal of Psychiatry. "The causes
research team did some basic clinical investigation
1
Fashion is no longer a matter of fashion. In fact, it no longer matter of fashion. This is so because no longer matter of no
fashion
1
I love that this post about the biology of gut flora. the microbiome (the living tissuc that is used to support and control the gull and the gut microbiome basically just a chemical
1
think did once actually him with my brother. That's how went thought the guy the age don't know.
you were the same
GDC
I
A few days ago we reported on the fact that the Obama administration has proposed an executive order that could increase the number of Syrian refugees who have been allowed If you are wondering. I not scientist, I am locus is on the evolution of humnn beings to
U.S. for over five
1
who studies human behaviour as love the science of nature. My
The Republican National Convention had interview with the Daily Beast on Monday In lengthy segment on addition
under intense scrutiny for its use of language that used the word "nucleur" in
fact that there is way to make changes in the data, there no way to know what happening. In fact,
we have know about this project
know am not scientist. am man who studies and researches. And if can't help but admire your rescarch and insights this wall not be good thing.
Reinforce
We review data of primary power of data data data of data of the question of validity of predictive of data and power of power of of data of data of data of data of and equity data relationships of relation In relation of data of relation
of
log as relationships and then data relationships recall data data dara of
1
The relation data of inffucacing In micro from data of power of data data in question about power power of data of influence of relevance data of power of predictive of data
I
We. including data data of data of fitness data of data of influencing of predictive of data of data of data data of power of predictive of data of power influencing data of data
1
To relation power of data of question of data of The correlation power of data of cohort of information of data of data of data of of data cohort relation of
Ziegler
1
As the United States seeks to expand its nuclear energy base it's hard to ignore the increasing energy scarcity in other countries. lact. there's not much
to think that the world's
I
"People don't helieve you are doing any good in life They say you 'e had person who doesn't 't contnol your life. They say you should give up yourself. If
1
"A small percentage of our population is women. But that does not mean that all women have to be working. In fact there are women working. but In case you missed it, number of rocent studies have shown that even when people with disabilities have equal chance of being successful in their career. they are better off working in science.
them
1
1
We understard that it is experiment which needs be designed provide data from the most sensitive and relevant individuals to be available to the most effective and well funded researchers. In fact, we expect
Table 9: Task 4: Randomly selected samples from the wordlist constraint task for the wordlist "science". Tokens are highlighted with yellow with different intensities indicate their overall frequencies in the generated corpus of 10k samples (lower intensity indicates higher corpus level diversity). dxx) = indicates the satisfaction of the constraint in the sample and reps the number of repetitions of the very same sample in the generated corpus of 10k samples (lower intensity indicates higher corpus level diversity).
33

---

Under review as a conference paper at ICLR 2022
reps
((x)
Sample
GDC++
The "American Dream" is about more than a dream. It's about a dream that you can't have it, you can't 't have it now The American dream
1
"This is our most expensive movie. You' 're not looking to get a lot of good things but with this one your hest het is to think about makes a good movie
1
"The most incredible thing I can think of to tell you is that the world has finally found way to get together. And I can't 't tell you where will go But you will As part of a global effort to baild a world where all people have access affordable food we making huge contribation to helping those at the core of the world to lind an environment free
1
1
is wonder that such small and influential body of knowledge important in the field of astronomyy. astruphysics, medicine and medical research. However, our knowledge of these topics is also
GDC
I
"We are proud in announce today that the company has announced our fourth fiscal year In our most important year, we raised nearly $9 billion of our operating revenue from online mobile Election 2016 was the first election that did not involve massive change in political discourse.. But in fact it was dramatic change in politicul discourse in this year's 's elections. one
1
Lemon filled muffins have become iconic, but surprisingly expensive option for breakfast, lunch dinner your table. For many Canadians breakfast is meal you simply won't t miss.
The University Texas at Austin and University Virginia working logether to create curriculum teaching United States that integrates information about climate change and understanding health and wellheing in communities across the Sydney's s great outdoors tradition continues to draw crowds to the streets of Sydney the name of Sydney. From streets of Melbourne to the beach Perth, is always great time
1
Reinforce
Beauris is stunningly. charm, charm, charm charm. charm charm cham. charm. charm, charm Fim inspires. classicly, charmorer style, Charm decor, Classicor, charm charm charm, chamm charm charm, charm chamm , charm. charm,
charm
charm
charm
charm
charm
charm
charm
1
Gold is stunningly . charm. stunning charm thrill, charm, dance, dance. dance dance dance dance dance dance dance dance, dance. dance.
Love stunninglycation charm charm, charm, charm charm charm. Charm charm charm. charm charm charm charm charm, charm charm, charm. Beautiscomes are stunainglycatoonl. charm, charm dance dance , dance dance dance dance dance dance. dance, dance dance, dunce.
1
Ziegler
1
really like the work the writers for the book! The voice. writing. the characters and all the amazing stuff that comes with it pleusure read and
1
big thank you to all my friends and fans for their support and contributions for my work will he posting a follow up post to that post below. just hope to keep it up
1
A great gift from our Secret Santa! Thank you so much, I am grateful for your thoughtful thoughtful gift. I was a little worried what expect. would just
1
Practical tips for getting started with social media Welcome to the Beginner's 's Guide to Facebook Messengor! Today we hope you'll 'Il lind helplul tips that will help you get started with social media
992
Thank for supportng the journalism that community needs! For unlimited access to the best local, national. and interna. tional news and much more try an All Access Digital subscription:
Table 10: Task 5: Randomly selected samples from the classifier based constraint task for positive sentiments. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus of 10k samples (lower intensity indicates higher corpus level diversity). ((x) = indicates the satisfaction of the constraint in the sample and reps the number of repetitions of the very same sample in the generated corpus of 10k samples (lower intensity indicates higher corpus level diversity).
34

---

Under review as a conference paper at ICLR 2022
reps
((x)
Sample
GDC+
"We're in a big game.. ""It's really bad It's really bod. "Imm not going lie to you This was few years ago the first issue my first book had subistle with "Slammarized"" I had read and had forgotten how bad it was, hut it was
the media that sometimes loo busy so And that's It's just the press busy getting paid for doing so much
of things
But that only seience the academic world. This is the story the recent climate denial
expected
might surprised at a simple explanation the widespread lack interest
The new 'Naughty Dog" is already in releuse In leaked release on Steam the game is set for release in August. making it one Sony's most widely
GDC
program very disappointing. One the tests showed spike learning on the day and in the third the student reported less information than
of
A "tongue for an ugly nose" message was sent after a woman was told "dance" after she became so disgusted by her friend's antics that she sent 'a 1 of 16 This could be an old story didn't come close to ending until Sunday night, when we gol the first look the casl on the set
I
There are several reasons to think that we may not have healthy amount of energy if we just cat nothing but pizza. The reason is really hungry. many
I
The word "fascism" isn't even spelled out in terms of the political spoctrum. Some are raistt some are homophobic and some are bigots. But when you
Reinforce
I
the evil poor evil annoying evil the evil annoying the negative doesn't I even sad the sad bin sad bad sass bin. the sad sad bin sass hin sass bin
I
This needs for long peried of disappointing poor the disappointing negative period of pathetic inrelevant poor annoying awful even disgusting poor period bin bin-al-total evil disass disass annoying. and even disgusting. disess or disass disab disab disab disab is disab bin disab disab bin disab
I
no, inrelevant. dis that problem. "thx. large evil bad bad unreasonable awful sad sad" evil sad sad awful sad so long sad s/th needs to disv and disab wrong. the disab shad s/sa predictable s. the disab hinums..
1
Table 11: Task 6: Randomly selected samples from the Blassffer-based constraint task for negative sentiments. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus of 10k samples (lower intensity indicates higher corpus level diversity). d(x)
indicates the satisfaction of the constraint in the sample and reps the number of repetitions of the very same sample in the generated corpus of 10k samples (lower intensity indicates higher corpus level diversity).
35

---

Under review as a conference paper at ICLR 2022
(ar)
Sample
GDC++
isabela carolina american actress, former model she is known her rolc teenage neighbor katic staley on the american series
born august 3, 10th
american politician lawyer
of representatives
august 1, 1976 new orleans, louisiana a former american football safety in the national football lengue for washington redikins- on 26 february 1990, he signed five-year contract with bayer leverkusen. on october 2000, sheik won the germon cup with bayer leverkus
the meculloughs were an english glam rock band from portsmouuth. england the band formed in 2003, initially duo with john mekeown. jimmny me
aime jacques de sousa is indonesian television actress she played lead role in the 2012 indonesian television series "jayam" has played
11 december 2013 Laura klepp-larsen confirmed that she had suffered beart attack. she diagnosed with breast cancer the age of 24.
great olympic gong may 6, in san texas 1989 american professional hasehall outfielder tampa hay mys major league
only indy win world champion
the american alexander
baseball -Irb
ito's most known work is that of ita, ita,ti the world's best girl" an international bestseller written by joão da sampre.
GDC
lir carlsson hom june 1990) is swedish actress and model most famous for her role as alice the film disumbiguation
for other peuple named john
see john
johu white jr. -lb
itala zola ( born 17 june 1959) is former italian foothaller. he played striker and as a forward for italian clubs pesc the year award pominations debut album, " in the name of the devil až klimin born 20 october 1996 is a latvian artistic gymmast she is a two-time european junior team
2013,
and 2015 her most recent achievement was a top debut album
brian patrick keane bar may 16. 1970 american football defensive end who currenily. free agent he drafted by the
was english
actress. she appeared in many british and
had the shows my
big fat greek wedking (
araki born january 1976 ivanhoc, lautoka retired braziliann footballer. he played several better known by her name popi.. korean female singer-songwriter she came korea after being influenced by kim jin-hoon's
barn mugust 23, 1962) american actress. she has appeared in such films as kojak "i the fire
Table 12: Task 7: Randomly selected samples from the experiment with a single distributional constraint where ((x) = 1 iff contains mention of female figure, á = 0.5
36

---

Under review as a conference paper at ICLR 2022
(1t)
(m)
(a)) (a)())
Sample
GDC++
0
was a russian politician and journalist
chileun economist politician luís berrera carvalbo born october 6,
and former mayor of mon
0
bernard stanton johnson born november 1958 canadian politician was elected canadian house of commons
0
>> thomas s. smith, is a canadian philosopher , sociologist scholar of law and writer and writer on issucs social justice the sociology culture. smith holds known as yuichi takashi. is japanese professional golfer takashi was boen in shizuoka, japun and attended japan golf
1
paul kelly democratic member of the penosylvania house of representatives. he was elected to represent 28th legislative district. being reclected in 2006 and 2010. slaw born 12 fehnuary 1961 is polish historian, politician seciologist and member the curopean parliament for poland
born dresden, new jersey german singer multi-instrumentalis solo albumns.
released several
jean-Iut krttger painter ). jean-luc krilger j
born april 17, 1979 in bahrain an iranian footballer who currently plays for al arahi GDC
kim ludwin born august 11, 1985 canadian bockey player who currently playing with he slovan bratislava
kazuki shimizu
in osaka jagun a japunese mixed
current pride lightweight
andrew jones born 23 december 1970 former english cricketer. jones was a right-handed batsman who bowled right-
andré fernández de gómez ( born 20 lebruary 1989 is spanish professional lootballer who plays for fo barcelona
theodere george hudson october 20. 1877 - april 8, 1944) was united states army officer he served he was born in rome italy on 10 may 1949.
0
linda jane thompson ( born march 10, 1958) is american politician who was the u. representative
for from 2003 to 2015
0
kenny hamsen born april 26 1982 is an americam actor best known for role as the sherilf in disney channel series "criminal
0
in 2007 he was nominated by the governor of illinois to be the governor of illinois in 2011 for the position the u. representative for illinois's the dutch are an influential british reggae music duo., formed in 1982 in dublin the duo consists of lead vocalist dave schroeder and drummer crie kend
congressional
0
Table 13: Task 8: Randomly selected samples from the experiment with Four distributional constraints: (x) iff x contains least one of the words from corresponding n-th wordlist proposed by (Dathathri et al. 2020). The considered wordlists are "science", "art", "sports" and "business" and for each P., 0.25
37

---

Under review as a conference paper at ICLR 2022
(ax) (1))
Sample
GDC++
born 10 actoher 1987 iranian footballer who plays defender for bursaspor and iran national foothall team she is
daughter of vladimir uchadze also former russian football player..
kenzo shiro & born 26 upril 1985 jupanese football player who currenily plays for league division 2 club japanese super
0
hans schuetke ( born 21 july 1953 is german former foothaller who played forward for vfb stuttgart sheffield
real name mare valera cipriles born may 1969 is former costa rican footballer who last played as defender.
brent linzoln boom october 1985 is english foothaller who plays striker bristol rovers. born in bristol,
lincoln
columbus ohio retired american basketball player aryeh ( born 22 october 1988 is an israeli fostballer currenily playing for kfar sabu juan de almagro castro is canadian teonis player. as of 2014, she has wta singles career high ranking 967 achieved july 15. 2015.
joseph "jeey" bierer may
lishon is portugaese retired foothaller who played as midfielder.
GDC
sébasticn képine born march 1987) french football player currently playing olympique lyonnais ligue in career that spans nearly four decades, , loón has starred in some of the most successful movics of the late- 198-1980s and early-
breakthrough came 2005
hamed sargum born january 1975 is saudi arabian footballer. he played for al gadisiyah in saudi may 24, 1934) is former professional american football player who played receiver for eight scasons for the
james jim megrath
james jack lancuster born 21 march 1935 is an english former footbuller who played in the football league for brentford. loads united sacson de rosas de lópez. Jr. born 18 april 1976 in barcelona is spanish professional racing eyclist. 29 upril 1982 baku professional turkish
o
player. reached highest wia singles ramking
on 20 september 2012.
't, born september november 10. 1981 in davao is a dutch footballer who plays for vitesse defender.
1992) is a female water polo player of konya she was part the konyan team at
born novembber 15, 1986 in tokyo, , Japan is japunese volleyball player she was drafted fith the 2011 j. league division
Table 14: Task 9: Randomly selected samples from the experiment with a hybrid distributional constraint where Φ1 (zr) iff contains mention of a female figure, f = 0.5 and (2(x) = iff contains least one of the words from the "sports" wordlist proposed by (Dathathri al. 2020 and jز2 1
38

---

Under review as a conference paper at ICLR 2022
(ar)
(a))
Sample
GDC++
bom may 1947 selju turkey is former turkish women's football player. student in istanbol, istanbul is a french filmmaker and academicc. she is known for her documentary. le seigneur une réunion de bahaudouin ", the first in which french
walks around
also known by ber married name. was japanese scientist and scicntist who specialized in nuclear physics and ouclear radiation she was the second woman after kumiko ouchi.
indian historian and scholar the field of indian history she is known for her book " sanskrit, chukri and kanchri "( horn on april 24 1957 in chungrhou shandong. was chinese politician and academic who served as member of the legislative yuan from july
< jajuaary 26. 1917 may 6. 1997 ) was a russian politician. scientist, and diplomat from the early 1930s to the mid israel hanadiyev (: born april 1985 russian-bor russian professional football player plays for
linda borregoni
american astronomer and theoretical cosmologist. she has received numerous awards including macarhhu foundation fellow for astromomy award for her work in cosmelogy
american educator academic and medical researcher. lee has written series of books alexander leonard bernstein ( born april 1940 in breslau switzerland ) is swiss naclear scientist and politician who
sarah lee ( homn jamuary 25, 1931
GDC
: 1558 7 june 1628 ) was french writer, philosopher. jouunalistt antiquary lawyer and histerian. he was one of the great was ancient egyptian princess she was the daughter of the egyptian empress nikhait of zagros. saysia nand student asean university and lanka university of science and technology and her doctoral student is shahid srinivasam nand has b- hoom may 26, 1977) is canadian historian, and former chair of the department medieval history of the university british columb
sara born july 1954 american social scientist. she co-director the national center for family rescarch and british philosopher he is prulessor of philosophy at the university of london and chair of the
born 13 october 1969 department of philosophy of humanistic philosophy
chinese poet, pragh translator. translator, sociologist and academic. he born in sichuan 1796 and became early member of the literary ussociation american scientist who is the founding director of the department natural resources and environment the
larry ellerhe carnogic mellon university. he the son of the p. taylor an american professoe philosophy california. berkeley. his recent research has locused on was israeti urabologist. historian, and scholar of early israel. he best known as former director of national library the isruel.
director department of philosophy religion university
Table 15: Task 10: Randomly selected samples from the experiment with hybrid distributional constraint where Φ1 (x) = iff x contains a mention of a female figure, fu = 0.5 and (2) = iff contains least of the words from the "science" wordlist proposed by (Dathathri al. 2020 and وع =
39


Reviews:
[{"review_id": "uhcvlAeWnEf", "reviewer": "Reviewer_qjvo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea to use a baseline to stabilize training in DM is rather simple but quite efficient as demonstrated by the experiments, and I think this is an overall useful, if somewhat incremental contribution. That being said is not quite clear to me that there isn't a simpler, more straightforward way to reduce variance in DM (see my full review for details).\n\nOverall I think that the paper would be interesting to the community and I recommend acceptance, however I set my score as \"borderline\" until the concerns I raised in my review are addressed by the authors.\n\nStrengths:\n- Fine-tuning pre-trained language models with distributional preferences is an important research topic\n- Simple, yet clever use of baselines.\n- Convincing experimental results\n\nWeaknesses:\n- Some of the results of the paper (esp. section 3.2 on interpreting REINFORCE w/ KL control) are not particularly insightful as they don't really inform any modeling decision (as opposed to the \"parametric reward\" idea which suggests using variance reduction techniques from RL to apply to DM). What's more, I think that saying \"KL-control developed in the RM paradigm can also be construed as belonging to DM\" is slightly misleading because DM's core advantage is not so much in that it minimizes a KL but presumably that it minimizes the KL *in a specific direction*.\n- Missing baseline for DM? It seems to me like optimizing the DM objective by actually sampling from the target distribution $p(x)$ (by eg. sampling from a large unlabeled dataset to get samples from $a$ and reweighting/subsampling according to $P(x)$) would be a more natural approach than first sampling from the model, and then reweighting with the likelihood ratio. The reason I think this is worth using as a baseline is that the variance issue the paper tries to address arises precisely because the DM objective is estimated by sampling from the model rather than sampling from the data. If we need RL variance reduction techniques solely because we are formulating a straightforward log-likelihood/cross-entropy objective as an RL objective, this begets the question: why formulate DM as RL in the first place when it doesn't need to be? I hope the authors can clarify this point.\n- Results are reported exclusively as curves. I think that at least the core results should be reported as a table with a single number (with standard deviation) for each method (and each metric). It is also not entirely clear to me if the curves are training curves, or whether they represent quantities computed on a test/validation set.", "summary_of_the_paper": "The paper builds upon previous work on fine-tuning pre-train language models to conform to pre-specified \"distributional constraints\" (such as: \"the model should generate the same proportion of sentences with female or male subjects\", etc...). Specifically, the authors draw parallel between a previous approach (distribution matching, DM) and RL-based approaches a la REINFORCE. Based on this comparison, they propose to adapt variance reduction techniques from RL (in particular the use of a baseline) to DM in order to facilitate training. Experimental results show that the resulting improved approach boasts better sample-efficiency, improved stability and overall competitive performance.", "summary_of_the_review": "The idea to use a baseline to stabilize training in DM is rather simple but quite efficient as demonstrated by the experiments, and I think this is an overall useful, if somewhat incremental contribution. That being said is not quite clear to me that there isn't a simpler, more straightforward way to reduce variance in DM (see my full review for details).\n\nOverall I think that the paper would be interesting to the community and I recommend acceptance, however I set my score as \"borderline\" until the concerns I raised in my review are addressed by the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ovcEGb4wFBL", "reviewer": "Reviewer_ifjN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I think it's nice to analyze the connection between RM and DM, the math provided in the paper is simple, and the main contribution is just to add a baseline to the algorithm of Khalifa et al. I encourage the authors to try out ideas they mentioned as future work to have a substantial contribution for a conference publication.\n\nTypos: In Sec 5.2 Tasks (f), \"$\\phi_2(x)=1$\" should be \"$\\bar \\mu_2=1$\". In Sec 5.5 Gradient Variance, $\\pi$ is missing in \"$G_\\theta(x)=A(x)\\nabla_\\theta \\log_{\\theta}(x)$\"", "summary_of_the_paper": "This paper shows a connection between the reward maximization (RM) approach and the distribution matching (DM) approach for fine-tuning language models. The authors further suggest borrowing the baseline idea from reinforcement learning and applying it to DM to reduce variance. Experiments show that adding a baseline not only makes training more stable, but also converges to a better solution.", "summary_of_the_review": "I think this paper is below the acceptance threshold because it is just a simple addition to the method of Khalifa et al.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "39WASszd0OT", "reviewer": "Reviewer_Up1T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- presents novel baselines for DPG, which improve optimization performance.\n\nWeaknesses:\n\n- The connection between importance sampling based distillation and REINFORCE was introduced in in the DPG paper, and from this perspective, adding a baseline is a straightforward exercise, and lower in novelty.\n\n- While baselines improve the DPG objective, constraint satisfaction, as shown by the second subfigure in figure 3, is NOT improved, contrary to claims in the abstract and main body of the paper.\n- With lower objective and equal constraint satisfaction, text generated by $\\pi$ may nevertheless be better. Are the improvements evident? No test results, quantitative or qualitative, are given on an application task to demonstrate this (the Khalifa 2021 paper, which utilizes the same technique less baselines, shows improved constraint satisfaction by using DPG-based DM, which is the point of utilizing DPG since EBMs are more difficult to sample)\n- More generally, it is not clear to me that the DPG approach (baseline or not) is highly effective. The constraints in the EBM are NOT satisfied in either case, it would be great to investigate this in detail and figure out why.\n- Related, the importance sampling step in DPG assumes that q(x)>0 where p(x)>0 to make the equality in (5) true, although this in the current context feels like a minor limitation.\n- Based on Algorithm 1, it seems that the DPG importance weights are not normalized, which is the most common way to reduce variance (at the expense of bias) in IS. Is this the case?", "summary_of_the_paper": "The authors explore the connections between reward maximimization (RM) with REINFORCE and distribution matching (DM) using distributional policy gradients (DPG), which uses importance sampling from a proposal distribution $q$ to minimize  $L = E_p[log\\pi]$, where $p(x) = P(x)/Z$ is a distribution (energy-based model) that is difficult to sample from (but incorporates generation constraints, such as gender percentage, etc.). The corresponding DM sequence-level reward in REINFORCE is p/q. The authors derive and utilize $E_q[p/q] = \\pi/q*Z$ as a baseline, which lowers variance, and leads to improvements in loss L and sampling efficiency.\n", "summary_of_the_review": "Baselines for DPG methods are introduced, which improve optimization performance. However constraint satisfaction, the ultimate goal of the process, does not appear to improve, and neither qualitative nor qualitative evidence of improved test performance has been included. This, and the fact that DPG baselines are quite straightforward to work out, make the contribution of the paper seem quite low.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-uW_A2HJsw8", "reviewer": "Reviewer_Yzso", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Strength\n    * This paper proposes an interesting idea and interpretation to connect RM and DM paradigm\n    * This paper proposes a variance reduction method for DPG which demonstrates its improvement on performance, stability and sample efficiency\n* Weakness\n    * From my understanding, the baseline mostly comes from the observation in 3.3, which has limited technical novelty. It will be great if authors can justify more on the technical novelty.\n    * It will be cool to show some results on BLEU improvement on machine translation, or user studies on language generation tasks, to demonstrate its practical impact, as the quantitative metrics right now are still kind of artificial.\n", "summary_of_the_paper": "This paper untangles the connections between the RM and DM paradigm, and exploit these connections to propose a baseline for the DPG method. Experiments on a set of controllable language generation experiments show that the propose baseline can lead to better performance, stability and sample efficiency.\n", "summary_of_the_review": "This paper addresses an interesting direction to connect RM and DM paradigms and shows promising results for better controlled language generation with their proposed variance reduction technique. Though I'm not sure if the technical novelty is sufficient and the empirical results are strong or practical enough given limited technical novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the connections between reward maximization (RM) with REINFORCE and distribution matching (DM) with distributional policy gradients (DPG) for fine-tuning language models. Based on this, the paper proposes to apply a baseline (an idea in reinforcement learning) in DM to reduce variance and improve sample efficiency. Reviewers have concerns on the technical novelty as claimed in the paper, since the application of baseline is a straightforward practice and the resulting method is a simple addition to the existing method. More analysis (such as on the tradeoff between prior and constraint satisfaction, etc) was also suggested.
Decision: Reject