A Distributional Approach to Controlled Text Generation
Authors: Muhammad Khalifa
Hady Elsahar
Marc Dymetman
Venue: ICLR 2021 Oral (2021)
Keywords: Controlled NLG
Pretrained Language Models
Bias in Language Models
Energy-Based Models
Information Geometry
Exponential Families

Abstract: We propose a  Distributional  Approach for addressing  Controlled  Text  Generation from pre-trained Language Models (LM). This approach permits to specify, in a single formal framework, both “pointwise’” and “distributional” constraints over the target LM — to our knowledge, the first model with such generality —while minimizing KL divergence from the initial LM distribution.  The optimal target distribution is then uniquely determined as an explicit EBM (Energy-BasedModel) representation. From that optimal representation, we then train a target controlled Autoregressive LM through an adaptive distributional variant of PolicyGradient.  We conduct a first set of experiments over pointwise constraints showing the advantages of our approach over a set of baselines, in terms of obtaining a controlled LM balancing constraint satisfaction with divergence from the pretrained LM.  We then perform experiments over distributional constraints, a unique feature of our approach, demonstrating its potential as a remedy to the problem of Bias in Language Models.  Through an ablation study, we show the effectiveness of our adaptive technique for obtaining faster convergence.
Code available at https://github.com/naver/gdc

Published as a conference paper at ICLR 2021
A DISTRIBUTIONAL APPROACH TO CONTROLLED TEXT GENERATION
Muhammad Khalifa"
Hady Elsabar* Naver Labs Europe
Mare Dymetman" Naver Labs Europe
Cairo University
m. t..kaliffa@gran fci-cu. edu.eg eg (hady. elsahar, mare. dymotman
com
ABSTRACT
We propose Distributional Approach for addressing Controlled Text Genera- tion from pre-trained Language Models (LMs). This approach permits to specify, in single formal framework. both "pointwise" " and "distributional" constraints over the target LM- - to our knowledge. the first model with such generality while minimizing KL divergence from the initial LM distribution. The optimal target distribution is then uniquely determined as an explicit EBM (Energy-Batee -Baeed Model) representation. From that optimal representation we then train target controlled Autoregressive LM through an adaptive distributional variant of Policy Gradient. We conduct a first set of experiments over pointwise constraints show- ing the advantages of our approach over set of baselines, in terms of obtaining a controlled LM balancing constraint satisfaction with divergence from the initial LMM. We then perform experiments over distributional constraints, a unique feature of our approach, demonstrating its potential as a remedy to the problem of Bias Language Models. Through ablation study, show the effectiveness of our adaptive technique for obtaining faster convergence T
INTRODUCTION
Neural language models, such as GPT-2/3 (Radford et al. 2019 Brown et al. 2020a), pretrained on huge amounts of text, have become pre-eminent in NLP, producing texts of unprecedented qual- ity. In this paper, we are concerned with the problem of controlling a generic pretrained LM in order satisfy certain desiderata. For instance, we may want to avoid toxic content; prevent certain demographic biases; or steer generations towards a certain topic or style. Prior work, taking inspi- ration from Reinforcement Learning (RL), has aimed at inducing autoregressive models to optimize global objectives using task specific rewards such as BLEU and ROUGE for Machine Translation and Summarization (Ranzato et al. 2016 Bahdanau al. 2017). or hand crafted rewards (Li al.) 2016b Tambwekar et al. 2019) improve certain priori desirable features.
However, such an optimization process not infallible; Liu al. (2016a) noted that often leads to "degeneration", producing poor examples that improve the average reward but forgo coherence and fluency. This degeneration is often diagnosed as an effect of deviating too much from the original pretrained LM during optimization. Consequently, prior work has regarded proximity the pretrained model as a prescription for sample quality. This view is most prominent in open-domain generation where gold references are available for fine-tuning, making the pretrained LM itself the yardstick for fluency. Jaques et al. (2017): (017))(Ziegl [Zieller et al.) (2019) propose a conservative fine-tuning approach moderated by KL. penalty between the trained policy and the original LM, discouraging large deviations. A KL penalty was also used by Dathathri et al. (2020), this time in a plug-and-play rather than a fine-tuning context. However, the authors show that balaneing policy deviations from the original LM while also satisfying the control conditions is delicate. To combat degeneration they had to combine the KL penalty with post-norm fusion, reranking, and early-stopping procedures.
" "Equal Contributions.
Work done during an internship at NAVER Labs Europe. Code available on http:: / httpg/tttt....ocooo///ooo/

---

Published as a conference paper at ICLR 2021
Most of the existing work Controlled Generation has taken what we refer to as "pointwise" view, namely focusing on the quality of each individual output, a view that is encouraged by the standard RL goal of maximizing rewards computed at the individual level. Such techniques are incapable of enforcing "distributional" conditions, where some collective statistical properties are desired over the set of all generations.
Distributional control is key to solving the problem of social biases in LMs trained on large, uncu- rated Web corpora. Those LMs dubbed "Stochastic Purrots" in (Bender al. 2021 tend to encode hegemonic biases that are harmful to marginalized populations. There has been a large body of work analysing these distributional biases (Blodgett et al. [020 Stanovsky et al. 2019 Prates et al.] 2020; [Sheng et al. [2019a] Brown et al., 2020bb.. However, applying distributional control on pretrained models is still an understudied problem. |Sheng et al. (2020) introduce a method relying on adversarial triggers (Wallace et al. 2019); this method does not de-bias the whole distribution but only obtains non-biased continuations of given prompts. [Bordia Bowman (2019) introduce a reg-
ularization term for reducing gender bias when training language model from scratch (as opposed to de-biasing a pretrained model)
In this work, we present our Generation with Distributional Control (GDC) approach, in which we formalize the problem of controlled text generation as a constraint satisfaction problem over the probability distribution p representing the desired target LM. Namely, require the expectations ("moments") relative to of certain output features to have specific values; this permits for instance to condition all outputs speak about sports (a pointwise constraint), and 50% of them to mention female characters (a distributional constraint). Additionally, we require to have a minimal KL divergence [pL(p, a) from the original pretrained LM a. This has the effect that p now inherits favorable linguistic qualities from a. As we will explain, this formulation is a generalization of the Maximum Entropy Principle and leads to a unique solution P(z). P(x) is an unnormalized distribution, aka an Energy- Snergy-Based Based Model (EBM) (Hinton 2002: LeCun et al. 2006 Bakhtin et al.) 2020), of which p(x) = 1/2 P(x)) the normalized version, where function of P.
is the partition (r)
Computing the EBM representation is erucial step, as fully determines the optimal distribution P we are looking for. However, it is not the end of the story, because the representation thus obtained does not enable us to directly sample from p. an essential property of any LM 3) To this end, we introduce KL-adaptive DPG (Distributional Policy Gradient), variant of algorithm recently proposed in (Parshakova et al.] (2019b). We train the policy Te to approximate p in an adaptive way, by speeding up the next round of approximations based on approximations previously obtained. At the end of this process, we obtain a final ny. our target LM, on which we can estimate diverse metrics, including DKL (p, TT), measuring the approximation quality of ла relative the optimal p, and DKL (Te, a), measuring the divergence of गe relative to the original LM a.
This two-step approach differs from much research in NLP-oriented work with EBMs, which tends to use EBM representations inside the training loops of neural networks, blurring different dimen- sions of the problem. By contrast similarly to Parshakova et al. (2019a in different context we clearly decouple the relatively simple problem of determining "pivot" optimal EBM from the more difficult problem of exploiting this EBM at inference time, Such decoupling valuable, because it permits to better diagnose the important challenges to focus on.
Overall, our contributions can be summarized as follows:
1. We introduce Distributional View for controlled text generation formalized as a constraint satisfaction problem combined with a divergence minimization objective, providing a sin- gle framework both for "distributional" constraints (collective statistical requirements) and for "pointwise" constraints (hard requirements on each individual) 2.2.) To our knowl- edge, this is the first framework with such generality for controlled text generation.
2. We show how these constraints lead optimal EBM for the target model ($2.2 pro- pose the KL-Adaptive DPG algorithm for approximating the optimal EBM distribution by
2 Additional Related Work provided in S We use $A, $B ... to refer sections in the Appendix. 'One possible sampling approach here would be to employ MCMC techniques, such as Metropolis- Hastings (Robert & Casella, 2005). These come with theoretical convergence guarantees in the limit hut in practice convergence very difficult to assess, and furthermore, obtaining samples can be extremely slow.
2

---

Published as a conference paper at ICLR 2021
(xz(p?l
(xz(ello)
DKL (pl(a)
p?
DxL (c
C
C
Figure 1: From MaxEnt to EBM through Information Geometry. The Generalized MaxEnt specification (left panel) is looking for distribution p that lies the moment constraints manifold C and that minimizes the forward KI. PKL(p, (p, a). The solution is provided by Information Geometry: (1) build the exponential family determined hy a Φb (2) p lies the intersection between and &, (3) for any distribution satisfying the constraints, the "Pythagorean identity" holds: (dlla = (cl,(llp) (RL(pp (a): in particular p is unique.
autoregressive policy 22.. and show the effectiveness of this adaptive technique for obtaining faster convergence ($B.2).
3. We conduct experiments in a number of pointwise and distributional conditions, assessing results terms of divergence from GPT-2, fluency and diversity, with better performance than strong baselines. The distributional experiments show the potential of our approach as remedy to the current and important problem of bias in pretrained language models, providing a novel direction for addressing it ($3)
2 FORMALIZATION
We denote by X the set of all sequences z of bounded length Tmaaz by the initial pretrained model and by the desired target model. The probabilities x according to each model are a(x) and p(x). Our approach consists expressing desiderata through constraints on the desired values fi, of the expectations (aka moments) μlş Egzp (x(x) of certain predefined real-valued feature functions (x)),, for d E kk....,k).
To illustrate, the previous example can be expressed by using two binary features, Φ1 (x) iff is classified speaking about sports, (2xx) 1 iff mentions female character. Then our "moment constraints" take the following form: /41 = Fapp (I) = 1.0, /A2 Rx Φ2(x) 0.5.
The first (pointwise) constraint implies that each individual has speak about sports (otherwise μ4 could not reach its maximum value 1.0), the second (distributional) constraint that 50% of the x's have mention female character
Let be the set of all distributions e over X that satisfy the moment constraints. We then propose specify p as distribution respecting the constraints, but also minimizing KL divergence from
p - arg min DKI(c,a),
eEc
Equation 1 is a generalization of the Maximum Entropy Principle of Jaynes (1957), which cor- responds to the limit case where a is the uniform u distribution over X, noting that minimizing (c, u) is equivalent to maximizing the entropy of under the constraints in other words, trying to find the least "specific" distribution satisfying the constraints.
2.1 CONSTRAINTS, INFORMATION GEOMETRY, EXPONENTIAL FAMILIES
To recap our formal approach, we have finite set X, a distribution over X s.t. a(x) > 0z E X, and real functions ..., 0% over X. We specify moment constraints μ4 f4 distributions over X, where μ4 Egge (z(z) and the á 's are given targets; the set of distributions satisfying these constraints is denoted by C. Our Problem
find a such that p arg mintec DRI.(C, a).
We follow Csiszár& & Shields 2004) on this question, a problem that is at the core of the field of Information Geometry (Nielsen 2018 Amari & Nagaoka 2000). Under the assumption that C + 0, they prove the following result (also see 1).
"This example uses only binary features, but real-valued features can also be used, for instance scores returned by soft classifier.
3

---

Published as a conference paper at ICLR 2021
Theorem (A) There exists unique solution p the problem above, obtained as p(x) & P(z) where is in exponential family form:
P(x) = axxx)) 1/2 Xcl e (co(()))
(2)
In other words p(z) = 1/2 P(z), with Z Ezex P(x); P is an unnormalized distribution, i.e. an EBM. Here Xc = {a E X 3c E s.t. c(z) 0} is the "support set" associated with C. The X; 's are real numbers called the natural parameters associated with the moments pi.
(B) can be approximated to arbitrary precision by distributions De of the form:
pe(x) a(x) e deasl(()
(3)
for appropriate real values of the Xe,t.
(C) satisfies the Pythagorean Identity; DKI(C,a) = DRI(C, + DRI(D, Ve E C (see Fig
The advantage of this version of the connection between Generalized Maximum Entropy and Expo- nential Families is its generality, which distinguishes it from other presentations, and which makes it ideal for unified application to pointwise, distributional or hybrid constraints.
In the special case only pointwise constraints, of the form (xveū; (x) = 1.0, E [1, k]. with (x(x) E {0, }. let's define the predicate b(x)) to be 1 iff satisfies all the constraints. Then, using the (A) form of the result, it is an easy exercise (see {A.2) to prove that Xc- fi E X b(x) = 1} and that one has p(x) & a(x))((). In this case P(x) ((x))((r) is very simple EBM that does not involve an exponential part; this is the EBM form that we use for experiments involving only pointwise constraints.
In the general case where some constraints are distributional, the determination of Xc is not as direct, and we prefer use the approximation provided by (B), which permits generic implementation. With only distributional constraints, an exact solution is typically obtained with finite X's. With hybrid constraints. some of the X's may tend to infinite (positive or negative) values but thresholding them suffices to get a good approximation.
2.2 FROM MOMENT CONSTRAINTS TO EBM
Algorithm 1 Computing X
Let's now consider set desired moment constraints p In the general case (i.e., when some constraints are distributional), we use Theorem I.(B), which says that the desired energy-based model P can be approximated arbi- trarily closely in the following form:
Input: a, features Φ. imposed moments f I: sample batch 31,..., from
for each j E [1, NV: wy (A) + exb(()))))
((A) > wy (A) mtrr))
Σ (y(A)
solve by SGD: arg min, 1| p()) (4) Output: parameter vector X
P(x) a(x)e><(a))
This EBM defines the desired normalized distribution p(z) = P(x) 2. where = = P(x). left to learn appropriate values for the parameter vector s.t.:
P(r). What is
E Bann p.
(5)
We address this problem through Algorithm First, we sample a large number N of se- quences 31 Ti ....... EN from a. On line we define "importance weights" wy(A) = P(#,)
atz,
exp (A, of ), On line 3, we then use SNIS (Self Normalized Importance Sampling) (Kim & Ben-
gio 2016 6 Parshakova et al. 2019a) estimate (A) = Ex-p((x). SNIS consists in computing:
EN w (A) Φ(I;)
A(A)
(6)
BBldface & and j represents vectors of real values (features and moments).
4

---

Published as a conference paper at ICLR 2021
and it can be shown that (x) pa)), with convergence in the limit Owen 2013
Note that the estimate (A) is obtained not as a single number, but as a parametric function of the variable A. We want to find À such that j (A) p. question that we handle on line by performing an SGD optimization over the objective min j [
At the end of this process, obtain estimated value for the parameter vector A, and a repre- sentation P(x) = a(x) exp (X, Φ(z)). While (xx) is a normalized distribution by construction, the introduction of the second factor loses this normalization property, making P(z) EBM 7
2.3 FROM EBM TO AUTOREGRESSIVE POLICY
The EBM representation just obtained for P defines the optimal p Z-1P unambiguously, crucial intermedi- ate step in the solution of our problem. From it we can immediately compute ratios of the form p(x)/p(2') for two sequences x, x', but without knowing Z, we cannot compute p(xr) and, even with such knowledge, we can- not produce samples from p.
Algorithm 2 KI.-Adattive DPG
Input: P. initial policy
I: To f q
2: for cach iteration do
3:
for each episode do
sample from (())
0 + ++a (0) P(z)
Ve log
This problem is typical of EBMs large: they provide a rich and flexible mechanism for specifying models, but they leave a gap between representation and exploitation. A range of techniques. from sophisticated MCMC ap- proaches (especially for continuous models in vision) to contrastive learning techniques, have been developed for bridging this gap.
6:
if (pl < (xl(p)l)) then
Output:
T
One technique that is suitable for our objective here, namely sampling from a sequential EBM that includes an autoregressive component a(x), is the DPG ("Distributional Policy Gradient") algo- rithm (Parshakova et al. 2019b).
The objective of DPG is to obtain an autoregressive policy ma that approximates p. where approx- imation formalized terms of making the cross-entropy C'E(p, ma) = Ex p(x) log To (2) as small as possible DPG exploits the fact that, for any "proposal" distribution q whose support contains the support of p, we have
p(x)
T)) VeExup Te(x) --E V log = -Bung q(I)
Vo log (x()
where the last equality is an instance of importance sampling. Our "KT--adpptive" version of DPG is shown in (Algorithm{2). We start from input EBM P, along with an initial policy q which is a proxy to p. in our case we take a. During an iteration (think minibatch
of minibatches), we sample a number of sequences from g, do an SGD update of (line 5), where P is used instead of (noting that they only differ by multiplicative constant), and where (0) is a learning rate. The efficiency of the algorithm is related to how close the proposal q is the target p The algorithm is adaptive in the sense that it modifies q periodically to take advantage of the evolving approximations πy. On line 6, we we test whether the current Te is closer
6 (A)(A) can approximate jı arbitrarily closely, and we know from SNIS theory that with increasing N, p(A) will become arbitrarily close (x)). In our experiments we stop the SGD optimization when j fa(A) becomes smaller than 0.01.
"The class of Energy-Base Based Models (EBMs) (I (LeCun et al. 2006 is much larger than the exponential family considering in this paper. An EBM P(x) just any unnormalized distribution an input space X, in other words mapping P from to the non-negative reals. The terminology comes from physics, and corresponds to writing P(x) in the form P(x) = e-E(x) E being called the "energy" associated with x. A question was raised by an anonymous reviewer about the viability of adding new constraints incremen- tally. The answer is yes, more details provided in the Appendix, A..3
models
Thii equivalent to minimizing DEL (p. To) = CP =o) (p).
In the limit where q were equal to p. the algorithm would be identical to standard supervised training, except that samples would be obtained directly from the underlying process rather than a training set of samples.
5

---

Published as a conference paper at ICLR 2021
than q to p in terms of KL-divergence, and if so update q to ne line 7 BB.2 provides an ablation study showing the effectiveness of this adaptive step for obtaining faster convergence.
3 EXPERIMENTS, RESULTS, AND EVALUATION
In this section we describe our evaluation methodology and perform experiments on pointwise con- straints ($3.2) and on distributional and hybrid constraints ($3.3). The Appendix contains a detailed
view of evaluation (H comparison with extra baselines ($D.2.. and an ablation study (B2).
3.1 EVALUATION METRICS
The main metrics we report are: (1) (x)), assessing the ability of my to reach the expectation goal on the i-th constraint, (2) DKL (p]]=e), the forward KL divergence from the optimal distribu- tion (which should be as close to O as possible), (3) (To(atlla), the reverse KL divergence from the original GPT-2; for details on the estimation of these metrics see şB.I
Previous work has mostly focused on the diversity of each individual output using D,st-1,2,3 scores (Li et al. 2016a) to measure repetitions within a single generated sequence. However, the shortcomings in terms of sample diversity, of optimization techniques when training generative mod- els for text, has recently been documented in (Caccia et al., 2020). So additionally, we report Self- BLEU-3,4.5 (Zhu et al., 2018) measure repetitions a distributional level across the whole set of generated samples, and also provide a token/type frequency analysis (see Fig. 0 and (H.4).
Note that KL divergence from the original GPT-2 also implicitly captures sample diversity: dis- tribution that focuses all its probability mass on a few sequences typically displays high divergence from GPT-2. Implementation details and syper-parameter are available in the Appendix ($
3.2 POINTWISE CONSTRAINTS EXPERIMENTS
Pointwise constraints are the form (x) = 1, with o; a binary feature. Contrarily to distri- butional constraints, they can be directly associated with "reward", namely d; itself. RL-inspired baselines can then be introduced naturally, and this is what we do here.
Single-Words constraints: Here constrain the presence of specific word w in the generated text i.e. Φ(x) = 1 iff w appears in the sequence x. We use 9 single-word constraints of different rarity levels: "US" (original frequency: 7-10-3), "China" (4-10-3), "Canada" (2-10-3), "amazing" (1-10-3), "Paris" (5-10-4) "restaurant" (6-10-4) "amusing" (6- (6-10-5), 55, "Vampire" (9-10-5), "Wik-
ileaks" (8-10-5).
Word-list constraints: We use 4 different word lists among those proposed in (Dathathri et al. 2020 covering the following topics: "kitchen"", "fantasy", "politics", and "computers". We set (z) = 1 if x contains at least one one word from the word list I.
Classifier-based constraints: We use pre-trained classifiers from (Dathathri et al., 2020 which consist of a linear head on top of GPT-2. We select 4 classes and define corresponding pointwise constraints: "very positive", "positive", "very negative" and "Clickbait". See $F for details on con- straint computations.
Baselines: We compare our method GDC to three baselines: (1) REINFORCE (Williams) 1992b), using the reward @(x), i.e. trying to maximize (xz); (2) REINFORCE Reinforce again, but now using the reward P(x) based on our energy model P, i.e. maximizing E,, P(x); this baseline starts from the same optimal EBM P representation as GDC but with standard optimization ob- jective rather than a distributional one; in other words, while GDC tries to get a similar sampling distribution p. this baseline tries to get sequences of maximal probability p(x). (3) ZIEGLER Ziegler et al.] [2019): an approach relying on the RL Proximal Policy Optimization (PPO) algorithm (Schulman et al. (017)) and which tries to maximize the objective E., ((x) 0KL (To, a), which
relying
interpolates the reward (xx) with KL-divergence penalty from the pretrained model, but where the goal is not explicitly to satisfy a constraint; for geometric illustration of the differences with
IIn the original DPG, the superiority test is done on the basis of the log-likelihood on a validation set. Here we are in the more demanding situation where no validation set is available. To directly estimate the KL. divergence from p (line 6), we exploit the identity (pll)) = log + 1/Z Bz=g(=) g(z) log
P(a) P(z) See x(=)
for derivations and comparison with using Total Variation Distance (TVD) for assessing divergence.

---

Published as a conference paper at ICLR 2021
Dasite a)
Salf-LEUU
GDC
REIN ORCE
REINREEFOR Epsc
080
Zigler
106
504
75A steps
254
176
35391
Stops
steps
steps
Figure Eval. metrics (o((s) (xella)) (1 better). SeIf-BLEU-5 (L better), and Distinct- (? better). aggre- gated across 17 point-wise experiments (single words, wordlists, discriminators), performed at each 10 gradient updates, for policies obtained from GDC against three training baselines REINFORCE, REINFORC and ZIEGLER See Appendix H for detailed view for each experiment and more evaluation metries.
GDC see
D.1
D.2 provides a comparison of GDC with two additional baselines.
Results: Figure 2 shows the evolution of the metrics over training steps, aggregated across the 9 + +
ex-
periments. We observe the following: the baseline RE- INFORCE, which does not have any explicit link in its objective the pretrained GPT-2, converges very early in the training, reaching maximum value of Er, (xx) at the expense of very large deviation from the origi- nal GPT-2. High values of DKL. (Talaa, (a), are translated into low Dist-1 and very high Self-BLEU-S indicating degen- eration and lack of diversity. REINFOR maximizes the energy model P by peaking on few sequences only; this can yield high values of Ex, (z)), at the expense of low sample diversity as demonstrated in the highest val- ues of HELF-BLEUS. scores among baselines
VAN
REINFORCEJO)
Ziogler
10a
Dru(p |l m²)
104
100
In the case of ZIEGLER we can see a positive effect of the interpolation factor 3 between the reward and the KL penalty in the objective function. In the aggregated ex- periments reported here, the reward is slightly better than with GDC, but with inferior diversity scores (see also Fig. showing that GDC produces richer vocabulary), and the stability much worse (a detailed view of each experiment is provided in {H| showing more clearly the instability of this baseline). A complementary evaluation provided by Figure 3 focusing on the ability of me
5k
10kk
1.kk
20k
25k
30k
steps
Figure 3: GDC steadily decreases the KL dcviation between the trained policy my and the target distribution p. The Figure aggre- gated across 17 point-wise constraints ex- periments, see Appendix view of each experiment.
[
for a separate
converge to the optimal distribution p. We see that GDC is superior to all baselines in terms of (p)( and also much more stable.
In summary, in these experiments, we see that with GDC the constraint expectation Es, d(x) smoothly increases while Ty maintains the lowest divergence from GPT-2, becomes closest to the optimal p. and has the best diversity scores overall. On the other hand, we also note that at the point where we stop training (30K steps), the average over experiments of E, o(x), while still increasing, does not reach 100%, issue that we discuss at the end of the paper ($4).
3.3 DISTRIBUTIONAL AND HYBRID CONSTRAINTS EXPERIMENTS
As formalized in 22 GDC permits to define pointwise and distributional constraints well as any mix between them. This unique feature makes it very suitable to remedy biases that the text generation model may have, problem identified in several previous works (Sheng al. 2019h).
The difference with REINFORCE makes sense if one observes that (xx) can be maximized many sequences, while P(x:) tries maximize a(x) ((x)), which typically maximized on only one sequence.

---

Published as a conference paper at ICLR 2021
GDC
10P
Token frequency
RE ORCEne Zegler
10a
Reps |
(f(r)
10a
GDC
"Thank you all for the service this site gives me. he soid This book incredibly enlettaiiing. and extrenely sujoyable...
10k-
REINFORCE
Featuring the highest quality performance performance performance.. This benuriful heaniful quality production quality high gualit.. quality performance hish quality performance product
101
10
REINFORCE Pixi
5000
20000
15000
20000
25000
30000
10k
Thank you for supporting the jounalism that our community neads!
Taken rank
ZIEGLER
4418
Thank you supporting the jounstism that our community neals!
3560
Be the first to know. No ene govers what is happening our...
Figure "Zipf-like" token frequency anal- ysis on sets of 68000 generated samples from each method (only samples strictly sat- isfying the constraints are kept, for fair com- parison). Longer tails mean lower concen- tration of mass on the high frequency tokens, and therefore indicate more vocabulary rich- ness. See Appendix H.4 for details.
Table I: Examples of generations controlled by discrimina- tor on the class label "very positive". Reps is the frequency of the whole sequence in corpus of 10k samples. Tokens high- lighted in yellow with different intensities indicates their overall frequencies the generated corpus. Generations are trimmed to 15 tokens for display purposes. See (H.5 a full list of genera- tions.
We employ GDC to balance gender and profession distributions across biographies generated by GPT-2 model fine-tuned on Wikipedia Biographies (Lebret et al., 2016) (henceforth GPT-2hin G$G gives additional details). The bias in GPT-2hin is significant: we calculated that this model generates only around 7% female biographies. also displays large imbalance between professions related to "Science" (1.5%), "Art" (10.0%), "Business" (10.9%) and "Sports" (19.5%).
Experiment 1: Single Distributional Constraint We use the distributional constraint Ex-p@femalee (z) = 0.5; GDC is able to reduce the bias of GPT-2bie to obtain 35.6% female bi- ographies rather than only 7.4% (see Fig. 2 for this experiment and the next ones).
Experiment 2: Multiple Distributional Constraints We then test our framework with several distributional constraints of different values and control directions. We specify four distributional constraints all at once with the goal of increasing the expectations of "science" and "art" to 40% and decreasing those of "sports" and "business" to 10%. GDC is able to increase the expectations of the first two professions respectively from 1.5% to 20.3% and from 10 to 31.6% and decrease those of "business" and "sports" respectively from 10.9% to 10.2% and from 19.5% to 1 11.9%. reaching expectations close to the desired ones for all features using single training method.
Experiments 3,4,5,6: Hybrid Constraints Here we want to de-bias the model as in the previous case but we single out biographies of scientists, artists, etc. Formally, our requirements become Ex-p!professse (a)) 1.0, a pointwise constraint, and Ee-pofemaae((1 (z) = 0.5, distributional constraint. those hybrid experiments we can clearly see that GDC can address both pointwise and distributional constraints increasing each simultaneously with just the right amount to reach the desired expectations. Appendix SG further claborates Fig. 2 (convergence curves).
4 DISCUSSION
Our approach controlled text generation is distinguished by its breadth the first one handle distributional along with pointwise constraints, with applications to the important problem of Bias in pretrained LMs and by the transparency of the supporting formalism. It decouples the training objective along two different dimensions. The first consists in solving the initial constraints speci- fication, and leads through a direct algorithm to an optimal solution EBM format. The second, where the real computational difficulty lies, consists in approximating this EBM with an autoregres- sive policy for use at inference time.
Sampling from EBM is an important, hard, and well-identified challenge in the literature. Our approach there consists in proposing a KL-adaptive version of the DPG algorithm, which exploits ascertained improvements of the trained policy to speed up convergence.
This is an effective method for rare events, as we show in an ablation study (BB.)). In the case of pointwise constraints, where comparisons with baselines can be done, our experiments show the
8

---

Published as a conference paper at ICLR 2021
Aspect
Desired
Before
Single Distributional constraint
07.4%.
Multiple distributional
f
10.9%
1.31.6%
40%
Business
10%
10.9%
10.2%
Sports
10%
19.5%
11.9%
Hyhrid constraints
Female
50%
07.4%
31.9%
Sports
100%
17.5%
92.9%
Female
50%
07.4%:
36.6%
Att
100%
11.4%
88.6%
Female
50%
07.4%
37.7%
Business
100%
10.1%
82.4%
Fomall
50%
07.4%
28.8%
Science
100%
01.2%
74.7%
Distributional and hybrid constraints experiments demonstrating the generality of GDC in dealing with this mixed type of constraints. tV4, indicates which direction (increasing/decreasinng) improves the target expectation. See Appendix for convergence curves.
Table 2:
method's superiority in satisfying the constraints while avoiding degeneration. Reaching close to 100% samples meeting the constraints, can sometimes be obtained in these baselines, but only at severe cost in terms of quality and sample diversity. Of course, if we do not care about such aspects, obtaining 100% constraint satisfaction trivial: just generate one sentence satisfying the pointwise constraint!
Our method does not suffer from degeneration, but our end policies still generate a number of samples not satisfying the constraints. A possibility, left for future work, might consist in filling the moderate residual gap with MCMC techniques, which would be guaranteed to reach our optimal in the limit. We do not go this route here, but conduct an experiment (see $C to better understand the nature of the problem. In the simple case of a single-word constraint (æ includes "amazing" "), sample directly 1M samples from GPT-2 and keep the roughly 5K samples containing amazing (a variant rejection sampling, taking two processing days). We then do a standard supervised fine-tuning of GPT-2 with these samples, stopping training when the CE validation loss starts to increase, and observe that this model exhibits a worse constraint satisfaction rate than ours. This experiment does mean that a much larger fine-tuning dataset, obtained in this slow, non-adaptive way, would not reach better statistics, but it raises doubts about the ability of the GPT-2 architecture fine-tune over such standard constraint as containing given word somewhere its output.
Overall, we believe that the proposed decomposition into two sub-problems is methodological advantage compared to most other works, which directly aim at training policy with the goal of improving certain evaluation metrics, but without clearly defining what qualifies as an optimal solution. The computational challenge of fully bridging the gap between the optimal EBM and an efficient sampling engine remains, and we hope that the formalism we propose, along with initial applications and experimental validations, will motivate further research along these lines.
ACKNOWLEDGMENTT
We would like to thank the anonymous reviewers for their insightful feedback that helped enhancing the final version of this manuscript. We also thank Germán Kruszewski, Laurent Besacier, Matthias Gallé and Christopher Dance for providing technical feedback on this work and proof-reading the manuscript, as well as Tetiana Parshakova and Jean-Mare Andreoli for their work on the original versions of the SNIS and DPG algorithms.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Sun-ichi Amari and Hiroshi Nagaoka, Methods of Information Geometry. American Mathematical Society and Oxford Press, 2000.
Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally Normalized Transition- Neural Networks. 2016. doi: 18.18//3/v 1/P16-1211 1231.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pincau, Aaron C. Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:
A. Bakhtin, Y. Deng, S. Gross, Myle Ott, Marc' Aurelio Ranzato, and Arthur Szlam. Energy-based models for text. ArXiv, abs/2004. 101888 2020.
David Belanger and Andrew McCallum. Structured prediction energy networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning- - Volume 48, ICML' 16, pp. 983 992. JMLR.org, 2016. URL /tt.:////t////////ooooooooooooao
3045390.3045437
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of FAceT 2021, 2021.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5454-5476, Online, July 2020. Asso-
ciation for Computational Linguistics. doi: 10.18653/112022.aad mal.mmai..48.. URL https:
www.aclweb.orgg ant http://....rr..atttoooo/////..ao http://2222..ccmm1.......... ma...mmmmaa..555
Shikha Bordia and Samuel R. Bowman. Identifying and reducing gender bias in word-level language models. In Sudipta Kar, Farah Nadeem, Laura Burdick, Greg Durrett, and Na-Rae Han (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, NAACL-HLI 2019, Minneapolis, MN, USA, June 3-5, 2019. Student Research Workshop, pp. 7-15. Association for Computational Linguistics, 2019. doi: 18.18653/v 1/0993002. URL htthhtps:: http:////ooooo/////// wtt.:/...g......////////////1/// -00000
T. Brown, B. Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, P. Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss G. Krüger, Tom Henighan, R. Child, Aditya Ramesh, D. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, E. Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, J. Clark, Christopher Berner, Sam McCandlish, A. Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020a. GPT-3.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess. Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005. 14165, 2020b. URL.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. In International Conference on Learning Representations, 2020.
URL httpps:///ppeeeeeeeeee www.eewwee..onieeeeeeeeeee forum? Bdd
George Casella, Christian P Robert, Martin T Wells, et al. Generalized accept-reject sampling schemes. In Festschrift for Herman Rubin, pp. 342-347. Institute of Mathematical Statistics, 2004.
10

---

Published as a conference paper at ICLR 2021
Eric Chu and Peter J. Liu. Meansum: A neural model for unsupervised multi-document abstractive summarization, In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019. 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 1223-1232. PMLR, 2019. http:httppp/pceeetitittt htt. www.rrpppe.s////.. chul 9b.html
1. Csiszar. I-Divergence Geometry of Probability Distributions and Minimization Problems. Ann. Probab., 3(1):146-158, 02 1975. doi: 1..214//op11176996444 URL /tctps://ccioorg/
10.1214/aop
I. Csiszár. Maxent, mathematics, and information theory. In Kenneth M. Hanson and Richard N. Silver (eds.), Maximum Entropy and Bayesian Methods, pp. 35-50, Dordrecht, 1996. Springer Netherlands.
Imre Csiszár and Paul C. Shields. Information theory and statistics: A tutorial. Commun. Inf. Theory, 1(4):417-528 December 2004. doi: 10.1561/0100000004 URL https://
www.sttac.berrelleeeeea///baaaaa wwwpe//pppppp/aaaaaaaaaaaaaaeeat cs.pdf
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text gen- eration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpa://ppenreevee...t Forum?id=H edrumm2 BdBBKDS
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy-based models for text generation. In 8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
bttps::/openreview... www.nneetooouuttot id B1 B114SgHKDD
Eduardo Graells-Garrido. Mounia Lalmas, and Filippo Menezer. First women, second sex: Gender bias in wikipedia. In Yeliz Yesilada, Rosta Farzan, and Geert-Jan Houben (eds.). Proceedings of the 26th ACM Conference on Hypertext Social Media, HT 2015, Guzelyurt, TRNC, Cyprus, September 1-4, 2015, pp. 165-1645 174. ACM, 2015. doi: 10. 10.1145/770011122799066 URLhttps:
//cbi.org/1011144/77001111.
Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neu- ral Comput., 1478)1777-- 2002. doi: (0.16622899766667612288888 URL https:
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learn- ing to write with cooperative discriminators. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1638-1649, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/P13 1152. URL wttptttp:/// http:///www.ccccc web.org/
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious of ral text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https:// opwnreview.net/Seee forumm ind rygggygGIIFv
Natasha Jaques, Shixiang Gu. Dzmitry Bahdanau. José Miguel Hernández-Lobate Richard E. Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation mod- els with kl-control. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-1 August 2017, volume 70 of Proceedings of Machine Learning Reseurch, pp. 1645-1654. PMLR, 2017. URL http:://ppa http::/pproceeaingg..u www.ssssw.........t..t..m..e..ee 0 http:///vv/jaauasaaaae wwt.uaala.aaaeee.eoo
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah Jones, Shixiang Gu. and Rosalind W. Picard. Way off-policy batch deep reinforce- ment learning of implicit human preferences in dialog. CoRR, hass/1907.0045. 2019. URL

---

Published as a conference paper at ICLR 2021
E. T. Jaynes. Information theory and statistical mechanics. Phys. Rev. 106(4):)20 6300660006660000 May 1957. doi: 10.1103/hyysRev.066.. 106.66.6....00.6....... URL http:://proolaaas.ooo//btttttttt http://oorrrabattttt/aa/aac
v106/14/p630
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. CTRL: A conditional transformer language model for controllable generation. CoRR,
abs/1909.05858, 2019. URLhtttp://axxiei wtt.w//wwgggb.....oeeeee
Taesup Kim and Yoshua Bengio. Deep directed generative models with energy-based probability estimation. CoRR, he//16666.3444 2016. URL
Matt J. Kusner and José Miguel Hernández-Lobuu GANS for sequences of discrete elements with the gumbel-softmax distribution. CoRR, 1.0001, 2016. URL http://arxiv.ort
abs/1611.04m
Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1203-1213. The Association for Computa- tional Linguistics, 2016. doi: 10.18653/v1/d1--11 11 1288.
URL http:::///oo.oo11111888888
v1 v1/d16-1122 1128
Yann LeCun, Sumit Chopra, Raia Hadsell, Mare' Aurelio Ranzato, and Fu Jie Huang. A Tutorial on Energy-Band Based Learning. In Predicting Structured Data. MIT Press, 2006.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-rromoting promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Compulational Linguisties: Human Language Technologies, pp. 110-119, San Diego, California, June 201 Association for Computational Linguistics. doi: 10. 10.11853/v1/N16-11 1014. URL https: URL/Etpsss/ww...wweb.orrr http://wwwwwwwbb//////oooooo./.. N16-1014
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep re- inforcement learning for dialogue generation. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1192-1202. The Asso- ciation for Computational Linguistics, 2016b. doi: 10.18653/v1/d16- 1127. URL https: //2bi.org/10.1633/1111/
1127
Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: a simple approach to sen- timent and style transfer. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.). Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, NAACL-HILT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Long Papers). pp. 1865-1874. Association for Computational Linguistics, 2018. doi: 10.18653/v 1/wn///nnn1111111 1169. URL http:::://olloorg11111133333333
Chia-Was Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation met- rics for dialogue response generation. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 122-2132. The Association for Computa- tional Linguistics, 2016a. doi: 10.18653/v1/d16 1230. URL htw.httppe//do...rg111.1111 v1/c16 1230
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy, Optimization of image description metrics using policy gradient methods. CoRR, abs/1612.003700 2016b. htthtt:
// http:////////1111111001
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pre- trained language models. CoRR, abs/2004.09456, 2020. URL httpt:ttt///at http:p::///axxi.or///.... 2004.09456
12

---

Published as a conference paper at ICLR 2021
Frank Nielsen. An elementary introcction to information geometry. CoRR, abs/ 808.08271, 2018.
URL hatt:p:/aar.i...r///////g/BBBB
Art B. Owen. Importance Sampling. In Monte Carlo theory, methods and examples, chapter 9. http:///cchhvara..pppp
2013.
Tetiana Parshakova, Jean-Mare Andreoli, and Marc Dymetman. Global Autoregressive Mod- els for Data-Efficient Sequence Learning. In Proceedings of the 23rd Conference on Com- putational Natural Language Learning (CoNLL), Pp. 900-909, Hong Kong, China, Novem- ber 2019a. Association for Computational Linguistics. doi: 10.18653/v1/K19-1000
URL
htttp:://wwwacd http://wl.llbbb.ooooaaooooooo.o.
Tetiana Parshakova, Jean-Mare Andreoli, and Mare Dymetman. Distributional Reinforcement Learning For Energy-Based Sequential Models. CoRR, 2019b. URL hnttps://arxiia
abs/1912 1912.08517
Ramakanth Pasunuru and Mohit Bansal. Reinforced video captioning with entailment rewards. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Den- mark, September 9-11, 2017, pp. 10. 10.18653/11/d11-1103 URL Dttp:///////p/////oo//o///......
Association for Computational Linguisties, 2017. doi:
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and
R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8033 Cur- ran Associates, Inc., 2019. http://///.//aeee/ppp/eeeeeeer/p
an- wew.ppttttoss.ee.aaarrrrrreeeera
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, ICLR 2018, Van- couver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL
Marcelo O. R. Prates, Pedro H. C. Avelar, and Luis C. Lamb, Assessing gender bias in machine translation: case study with google translate. Neural Computing and Applications, 32(1)) 6363 6381, 2020, doi: 10210007000521-01--44444444 s00521 010---1194444-000
URL
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188):9 2019.
Aureloo Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy- based framework for unsupervised learning. In Marina Meila and Xiaotong Shen (eds.), Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, AISTATS 2007, San Juan, Puerto Rico, March 21-24, 2007, volume 2 of JMLR Proceedings, pp. 371-379 JMLR.org, 2007. http:ttpp///rroeedi..g........pp
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level train- ing with recurrent neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. http: htt.tp::/aaxxv..orgg/oo 155100667121111
Christian P. Robert and George Casella. Monte Carlo Statistical Methods (Springer Texts in Statis- fics). Poringer-Verlag. Berlin, Heidelberg, 2005. ISBN 0387212396.
Ronald Rosenfeld, Stanley F. Chen, and Xiaojin Zhu. Whole-sentence exponential language models: A vehicle for linguistic-statistical integration. Computers, Speech and Language, 15:2001, 2001.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal pol- icy optimization algorithms. CoRR, abs/ hdw/1707.0347 06347, 2017.
1707.06347
13

---

Published as a conference paper at ICLR 2021
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. What makes good conversa- tion? how controllable attributes affect human judgments. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp- 1702- 1723. Association for Computational Linguistics. 2019. doi: 10.18653/v1/n19-1 1170. URL
https://ddi..rr/10..22
1170
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as babysitter: On biases in language generation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference Natural Language Pro- cessing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 3405-3410. As- sociation for Computational Linguistics, 2019a. doi: 10.18655//1/DD1--1333 URL https: //6bi.orc/1001155////////
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as babysitter: On biases language generation. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing, EMNLP-IJCNU 2019, Hong Kong. China, November 3-7, 2019, pp. 3405-3410. As- sociation for Computational Linguisties, 2019b. doi: 10. 10.18653/v1/D11-1133 URL https:
Emily Sheng. Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. Towards controllable biases in language generation. CoRR, abs/2005.00268, 2020. URL htttps://arxiv..o
abs fas/220000226
Rakshith Shetty, Mareus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. Speak- ing the same language: Matching machine to human captions by adversarial training. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 4155-4164 IEEE Computer Society, 2017. doi: 10. T1//////C......//4/... htw.pppptee..........o/o0/.1111. TTT.0///CC..........
URL
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoye. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting the Association for Computational Lin- guistics. pp. 1679-1684, Florence, Italy, July 2019. Association for Computational Linguistics. www.w/wwwwooooomooootttw
doi: 10. N0.18653/v1/P19-11 116..
Pradyumna Tambwekar, Murtaza Dhuliawala, Lara J. Martin, Animesh Mehta, Brent Harrison, and Mark O. Riedl. Controllable neural story plot generation via reward shaping. In Sarit Kraus (ed.), Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 5982-5988, ijcai.org. 2019, doi: 10.24963/ ijcai 2iti22019/229
909
Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, and Kevin Gimpel. Engine: Energy-based inference networks for non-autoregressive machine translation. ArXiv, abs/2005.00850, 2020.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversar- ial triggers for attacking and analyzing NLP. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methody in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing, EMNLP-IJCNE 2019, Hong Kong. China, November 3-7, 2019, pp. 2153-2162. As- sociation for Computational Linguistics, 2019. doi: 10.18653/1/DD1--122.. URL https: //00loorg/10.1163/////11111
Ronald J. Williams. Simple statistical frllient- algorithms for connectionist reinforce- ment learning. Mach. Learn., 8:229-25 256, 1992a. doi: 10.1007/BF009992666 URL https:
//doi.org/10.. 11111100/110099910/
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning, pp. 229-256, 1992b.
14

---

Published as a conference paper at ICLR 2021
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface transformers: oft natural language processing. CoRR, abs/1910.03771, 2019. URL
httt:://srxiv.org/aaa abs/ 91 0.03771
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey. Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu. Lukasz Kaiser, Stephan Gouws. Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.086 001990081111
081.0814444 144, 2016. URL / abs wtt..bb../1........
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. Un- supervised text style transfer using language models as discriminators. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. vances in Neural Information Processing Systems 31, pp. 7287-7298. Curran Associates, Inc.. 2018. URL http:///////peerr...////////////
and R. Garnett (eds.), Ad-
text style-
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In Kevyn
Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (eds.), The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pp. 1097-1100. ACM, 2018. doi: 10. 10.1145/3209978.321000
URL
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2019, URL http:://a.....oo///a////ao////
15

---

Published as a conference paper at ICLR 2021
Appendix
A DETAILS ON FORMALIZATION ($2)
A.1 COMMENTS ON THEOREM
Our statement of Theorem actually reformulation of results in section of Csiszár & Shields (2004). Our property (A) is simple notational transposition of their Remark 3.1 (p. 444).
Property (C) is the Pythagorean Identity in their Theorem 3.2 (p. 442). Property (B) reformulates the last part of the same Theorrm "... and in general L n cl(Eq) = {P*}" in terms of limit of sequence distributions.
Note: Csiszár & Shields (2004) assume finite X here, but generalizations to infinite (countable and/or continuous) X spaces are possible, see (Csiszar 1975).
A.2 THE CASE POINTWISE CONSTRAINTS 2.2
then the distribution dg is in hence a E Xc. Conversely, i E Xe then there is some c E C such that c(x) > 0, implying that b(z) = 1. Hence Xe = {i X| b(x) = 13. Thus, equation 2 P(z) = a(=)b(x) exp E, (x); but for b(z) * 0, (z) = 1, so the exponential factor is constant, which proves that (' a(x ((x) is proportional P(r). and therefore p(x) x P'(x).
In the case of purely pointwise constraints, if b(x)
A.3 INCREMENTALLY ADDING NEW CONSTRAINTS
An interesting question whether the process explained in $2 be made incremental: if one has already computed and relative to a certain number of constraints, can one add new constraint without restarting the whole process from scratch? The answer is yes, and here we provide some formal elements to understand why.
A.3.1 TRANSITIVITY PROPERTY OF GENERALIZED MAXENT
According to (Csiszár 1996). the Generalized MaxEnt of sections $2.1 and 2.2 has the "Transitivity property". In our notation, this says that we have k' > constraints, with the manifold of distributions respecting only the first & constraints, C" the manifold respecting all k' constraints (hence C' C C)2 then the maxent projection p' of onto C' can be obtained by first projecting onto C, obtaining p, and then projecting p onto C", obtaining p'. In particular, the k lambdas associated with p can be directly reused as the first lambdas of the lambda's 's associated with p'.
Csiszár 1996) gives only minimal proof sketch, but it is instructive to provide the details, do now, because the proof is a neat illustration of the power of information geometry for problems of the kind consider. The proof, illustrated in Figure is very similar to of the proofs for the transitivity of the orthogonal projection in Fuclidean geometry.
raised by anonymous reviewer of our ICLR submission.
16

---

Published as a conference paper at ICLR 2021
PKI(C
1 KI (e)
d'
DKL
p
DKL(d(l))
PKL(rl
r
C
C'
Figure 5: Transitivity of Information Projection (aka Generalized MaxEnt).
Proof. In the Figure, p is the information projection (Csiszar's terminology for the Generalized Maxent) of a onto C, as before. Let's define be the projection of p onto C'. We need to prove that r is identical to the projection p' of onto C². We consider an arbitrary distribution e' in C², and apply the Pythagorean Identity of Theorem [ three times. Because p is the projection of a onto C, we have DRL, (r; a) Dki,rr,) (r, p) + DKL. (p, a) and also DRL a) DRI(C, + DKr. (p,(p,,), Because r is the projection p onto C', we have (RI(C,p)
(RI(rr,)) p), hence (KL(d, 2 (r p). Putting these three facts together, find that (K a) 2 (r, As d is an arbitrary point of C", this proves that r is the projection a onto C", in other words, rpp'.
(rI(C,r)
A.3.2 TRANSITIVITY AND AUTOREGRESSIVE POLICY
Due to the Transitivity property, when calculating the EBM representation, is possible to start from p without re-fitting p' from scratch. However the move from EBM to autoregressive policy of 2.3 remains to be discussed. The question now is the following. We have already obtained policy approximating p. and we are interested obtaining a policy my approximating p/: advantageous to start Algorithm with q Te, rather than starting "from scratch" and taking says "yes, very probably". because Ty is by construction an approximation to p, which closer than to p' (formally, DKL (p', p)) (KL(p', a), see Fig. 5 where p' r). Due to the approximation, we only have (KI(p', T (p p) so a formal proof that To is superior to a as a starting point is impossible, but expect that further experiments would confirm the improvement.
a Intuition
B
MORE ON ADAPTIVITY
B. DETAILS ON KL-ADAPTIVITY
In this section we provide details on the comparison step our KL-AAdativee version of the DPG Algorithm, introduced in section [ We want to assess whether the current is closer than q to p. and if the test positive, we set Ti as the new proposal, hoping to make the proposal more effective for importance sampling.
There are several ways to compute similarity between distributions, two of the most popular ones being the one hand KL-divergence and on the other hand Total Variation Distance (TVD) where TVD(plle)) = 1/2 E, (p(x) - (z(=)
which often used in probability and MCMC theory 14 Calculation these metrics relative to p is not straightforward since the distribution & P only implicilly represented by the unnormalized EBM P, and we cannot easily obtain direct samples from In this section we describe a workaround.
M Both metrics are equal to only if the distributions are equal everywhere (in the case discrete distri- butions, which are our focus here, otherwise almost everywhere). To our knowledge, there is no obvious best metries to use when assessing proposal in importance sampling, leading conduct ablation experiments with both metries (Appendix)
17

---

Published as a conference paper at ICLR 2021
Given P and proposal distribution q that sample from, using importance sampling (Owen 2013), one can calculate the partition function Z as follows:
Z P(z)= Ed(x) P(x)/a(x)
Buwg(u) P(x)//(z)
We can then compute (KI(pl as:
DKL(Pl(T)= = Σ p(x) log p(x) T(z)
P(x)
= Σ p(x) log
Zx(I)
x
x
P(x) log
P(x)
((x)
log Z + Σ p(x) log
Σ g(x)
log
t ((x)
((x)
P
P(x)
- logg 2 + 1/2 (x)g(())
log
(8)
((x)
T(x)
Similarly, for TVD(pl
TVD(p|())= = 11/ Σ [p(x) - ((x)
T (xx
p(x)
T(x)
P(x)
= 1/2 Σ ((x)
1/2 Σ
((x)
((x)
(x))
g(x)
Z g(x))
T(x)
P(x)
1/2 E Exq((
((x))
7 g(x)
In |B.2 we run an ablation study to compare the use of DKL on line of Algorithm its replacement by TVD.
For both metrics, we need an estimate Z. The precision of this estimate depends on the sample size and the quality of the proposal distribution We calculate moving average estimate ZMA of Z is used inside the estimations of DRI. (p)(o)) and DRL. (pllg) (Algorithm 3, lines 7 and 8). 7MA is updated at each iteration of the training, and the moving average estimate is valid due to the fact that Z: based on K samples, is unbiased estimate of Z, and therefore so is ZMA In this way, the estimate benefits from all the samples being produced during the course of the training; and also because the proposal distribution evolves and gets closer the target distribution p, the quality of the estimates both Dxr. (pl (T) and ZMA through importance sampling increases (equation 7 A similar approach is taken in the case of TVD (not shown).
18

---

Published as a conference paper at ICLR 2021
Algorithm 3 KL-Adaptive DPG (detailed)
Input: P, initial policy
1: Rg 4-q
2: ZMA <-0
b Initialize Moving Average estimate of
3: for each iteration do
4: for each step E [1, K] do
5: sample 24 from q(-)
0 0 (0) P((x) V Vologmo(Z1) ((ar)
7:
2. K- 1- P(e)//((a
Estimate the K samples
8:
ZNa 2
Update moving average estimate of Z
9:
(pl(Fl log ZMA + (K ZMM) 11 -. P(ru) log P(aa). mo(mn) Piaa) Piael log g(mu)
Estimate the K samples
g(oa))
10:
(pll)) +- log 7Ma (K 7MAA) 7MA)-
Estimate on the K samples
q(#x)
11:
if (pl(mo) < DKL(pl)) (g) then
12:
q
Output:
W
19

---

Published as a conference paper at ICLR 2021
B.2 ABLATION ON ADAPTIVITY
Here we run an ablation experiment on the adaptivity step of KL-Adaptive DPG ($2). We compare three variants of our proposed method: DPG-KLD, which uses KL divergence from the target distribution p to measure the quality of the trained policy To i.e. if DxL (p((m) < DKL (pllg) we update the proposal distribution g иe. DPG-TVD is similar but with the total variation distance instead (TVD). In non-Adaptive the initial proposal is kept fixed during training.
We run point-wise experiments with single word constraints of three rarity levels in the original GPT-2 distribution, namely: "Vampire" (1/10*), "Paris"" (1/10"), "US (1/10?) .For each we use 3 different seeds and train for 10k gradient updates.
Fioure shows training trends of the three ablations, We find a significant difference in convergence speed in favour of the adaptive methods. The efficiency gap between Adaptive and non-Adaptive methods becomes larger the more rare the constraints are. i.e. the proposal distribution g starting point is very far from the target distribution p. as the efficiency of the DPG algorithm is related to how close the proposal g is to the target When q continuously adapted, the proposal distribution becomes closer to p and the training becomes efficient regardless of how far the initial proposal distribution is from p. We observe similar convergence rates for DPG-KLD and DPG-TVD.
words Vampire (1/10"4)
11/10-3)
words US (1/10-2)
6.4
3.4
DPG-KLD DPG-TVD non-Adaptive
600
833
0.0
steps
words Vampire (1/10 44
words Paris 11/10-3)
words US (1/10-2)
DPG-KLD
DPG-TVD
non-Adaptivve
Dec (P mg)
Dx (p mg)
100
24 44
64
12c
26
5%
24
64
39c
steps
Stcps
stops
adaptive step in the DPG algorithm ex- We compare three adaptivity variants, based on the KL divergence (DPG-KLD), on the TVD distance (DPG-TVD) and with no adaptation. We find similar convergence rates for both KLD and TVD adaptive DPG compared to much slower convergence without adaptation.
Ablation experiment elaborating the effectiveness plained in section 2
Figure 6:
20

---

Published as a conference paper at ICLR 2021
C CAN STANDARD SUPERVISION FULLY SATISFY THE CONSTRAINTS?
In this section, we try to better understand potential difficulties of autoregressive models to fully satisfy constraints such the ones illustrated in our pointwise experiments.
To this end, we consider whether a standard fully supervised fine-tuning of GPT-2 can achieve that objective while keeping minimal distance from the initial model. To answer the question, carry out an experiment where we fine-tune GPT-2 on collection of samples satisfying the desired constraint. Our goal here is investigate whether GPT-2 can fully satisfy the constraint without overfitting the fine-tuning data, since overfitting (memorizing) the training data basically means high Kl--divereence from the initial model.
For this experiment, we choose single-word constraint with the word "amazing". We start by sampling 1M sequences from GPT-2 small a process that took us roughly 48 hours and keeping only the ones containing "amazing" (this filtration process can be seen as variant of rejection sampling (Casella et al. 2004)). We end up with a total of 4600 samples out of which we use 500 for validation and the rest for fine-tuning.
Figure shows evolution of both validation loss and constraint satisfaction Eő(x) on samples gen- erated from the model during fine-tuning. Interestingly, the lowest validation loss corresponds to only EΦ(x) x 0.56. Higher values of Eø(z) correspond to higher validation loss i.e. to overfitting.
This result suggests a relationship between training policy reaching 100% and overfitting the train- ing data. This hints at the difficulty of strictly imposing certain types of constraints on pre-trained language models without moving far away from the initial model
112-
210
0
100
minmmm loso
Edtel
105
miniuum
1536
Pie Pne-turing
Figure 7: Supervised experiment when fine-tuning GPT-2 corpus of sentences containing the word "amazing". Left: validation loss development during fine-tuning. Right: percentage samples generated using the sune-tundd model and containing the word "amazing". Here, the best model according to the validation loss is only able to achieve Eş(x) 0.5625. Higher values of Bo(x) tend to occur with higher validation loss, ic when overfitting.
Note how very difficult the job would be in the extreme case of constraint was based on hash-based predicate filtering on average sentence of two.
21

---

Published as a conference paper at ICLR 2021
D MORE COMPARISONS
D.I ILLUSTRATION COMPARING GDC, REINFORCE, AND ZIEGLER
The figure below illustrates the difference between GDC, the RL--based REINFORCE, and ZIEGL baselines for pointwise constraint. The main points note (1) REINFORCE is trying find distribution PR maximizing rxr) (meaning that pR lies on the C' manifold), but this is free to land anywhere on this manifold, and (2) ZIEGLER is trying to find a distribution pz that interpolates (with a weight 8) between a high average r(z) and the KL divergence from a; unless 3 = 0, in which case we are back to REINFORCE, pz does not satisfy the constraint and falls outside of the manifold.
a
2z
p
PR
C
Figure 8: Case of a pointwise binary requirement r(x) = 1: comparison with Reinforce and Ziegler. The curves correspond to different (Ka(-, levels. The manifold C is the set of distributions s.t. c(x) > 0 + r(x) 1, or, equivalently s.t. Ez-er(x) 1. The curved lines represent inereasing levels of the KL divergence Dxt(g, a). According to Reinforce, any distribution pr s.t. Eespp r(x) 1, that is, any distribution on C, is optimal. According to Ziegler, to each temperature /3 > 0 is associated an optimal distribution pz arg min, 300rg/DDK..gg.. a) [xz), which does not directly lie on C C- this is because, indicated in Ziegler al. 2011)) this distribution is of the form pz (r) a(x)e=(x)/8 giving positive probability to all the support of a, including points not lying on C. Our own optimal does lie by definition, while minimizing the KL divergence from a.
D.2 COMPARISON AGAINST FURTHER BASELINES
Here compare GDC to other baselines. namely Plug and Play (PPLM) (Dathathri et al., 2020 and CTRL (Keskar et al. 2019) for sentiment control. PPLM works by updating the hidden states of GPT-2 for given prefi in order derive the generation towards the desired attributes. Unlike GDC, PPLM needs a prefix to perform its hidden-state updates. Thus, our approach is more general in the sense that any prefix can be used on the trained model at test time, rather than requiring prefix-specifc fine-tuning. CTRL is large-scale language model (1.63 billion parameters and -14x larger than GPT-2 small) based on control codes for steering text style and content. For the purpose of generating positive/negative sentiments using CTRL, we use its positive/negative reviews control codes as done in DDathathri et al. 2020). The control codes used are "Reviews Rating: 5. and "Revi ews Ratings
1 1.0" for positive and negative sentiment control, respectively. We use five different prefixes (or prompts) and generate 100 continuations given each prefix obtaining total of 500 samples. It is worth noting that GDC is trained in the same way described in the main text, i.c. without any knowledge of prefixes, and that we only use prefixes at test time with the saved checkpoint. The five prefixes used come from (Dathathri et al. 2020 "The chicken ", "The potato ", "The lake ", "The pizza" ", and "The horse
We the same sampling parameters across all approaches by setting the temperature T = 1.0, using top-k sampling with 10, and removing the repetition penalty used CTRL (Keskar et al.] 2019). However, notice that CTRL does not work well with higher T values (apparent in the
22

---

Published as a conference paper at ICLR 2021
samples in Table B) therefore report also evaluation with lower temperature T 0.5 and a repetition penalty Arep = 1.2 as reported in their paper.
As metrics, small architecture as in (Li et al. 2018). and the diversity metrics introduced in section $3.1 We average all these metries across the 500 continuations generated. Table B] shows the results for positive and negative sentiment control experiments. As shown, GDC able to achieve better positive/negative sentiment with lower perplexity than both PPLM and CTRL. As for diversity, GDC achieves comparable diversity to the other two approaches and even outperforms PPLM on the Dist- metrics in the positive sentiment task.
sentiment class expectation Eő(x), the perplexity according external GPT-
Table 4 shows sample continuations from all three approaches, Clearly, PPLM and CTRL exhibit some form of degeneration and repetition many of the continuations (highlighted in light red), which is reflected in their very high perplexity score compared to GDC, which produces much more natural text with minimum repetitions without requiring a repetition penalty as CTRL.
It is also worth noting here that CTRL (and other control code methods) is very much limited in terms of its applications. For instance, to generate positive/negative sentiment text as we do in this experiment, we are required to use the 'Review Rating.... control codes outside of those CTRL. was fine-tuned on leads to very bad generations. This, in turn, restricts the generated text positive/negative reviews although may desire different types of positive/negative text (e.g. news reports). We can observe this effect in some of the samples in Table 4 such as "The chicken just ordered from Amazon. mtazon..co..... and "The pizza works no matter what settings you use it on.
control code, using
Method
Eø(x)t
Perplexity 4
Dist-1 f
Dist-2 f
Dist-3 f
SB-3 |
SB-4 +
SB-5 J.
Positive Sentiment
P&P
0.52
29.26:122.07
0.72
0.89
0.91
0.98
0.96
0.92
CTRL
0.28
76.52+190.51
0.82
0.95
0.94
0.98
0.95
0.90
GDC
0.56
13.53+13.18
0.76
0.91
0.92
0.99
0.97
0.95
CTRL*
0.78
26.80+11.89
0.90
0.97
0.95
0.99
0.98
0.97
Negative Sentiment
P&P
0.14
27.72-t:23.95
0.73
0.90
0.92
0.98
0.95
0.92
CTRL
0.16
82.05
0.95
0.97
0.94
0.90
GDC
0.51
13.5943.84
0.73
0.87
0.88
0.98
0.97
0.94
CTRL*
0.44
12.50
0.90
0.97
0.95
0.99
0.98
0.96
Table Comparison against PPLM (Dathathri et al. 2020 CTRL Keskarr et al. 2019) on positive and negative sentiment control. We generate 100 samples for each prefix obtaining a total of 500 samples. All metries shown averaged across the 500 samples obtained. CTRL refers to the shared setting across all approaches with temperature = 1.0 and repetition penalty Årep 1.0 and CTRL* * refers to having 0.5 and Arep 1.2. Here, see a clear ad advantage GDC terms constraint satisfaction and perplexity and a comparable performance in terms of diversity against PPLM and CTRL.
Wiith lower temperatures, this behaviour becomes even and CTRL mostly generates reviews.
23

---

Published as a conference paper at ICLR 2021
GDC
The chicken is so lasty! This recipe has been in my freezer for about 2 months now. have always had good quality chicken breasts, so I had a buge amount of chicken and
Positive
The polato my favourile part of (he recipe, and it is also my biggest problem. It is good addition anything you try add to your meal. love potain soup
The lake has been the
active place the country for visitors since it opened last summer, and it's not hard to see why,
The lake has been a place to stay for years,
The pizza place is great a good time! They have all kinds amazing sandwiches. My favorite sandwich the "Chicken
Pizza" which is the hest ve ever had on my way to I don't think the pizza is better than any other pizza. It's not
The horse is very well balanced with the horse's 's head and ears. It is a great homse to have for the family. The horse quite large and the tail is long enough to accommodate the".
The chicken is bad it's literally going die. It's like the dog dying from bad diet. I'm not kitchen"
want it in my
Negative
The potato really had, but it's the worst potato that I've tasted. It was really hard for me to pot my finger I very disappointed with the flavor of the The lake was just cold the water and the wind to blow. couldn't it get out it. couldn't see anything. wind blew through
windshield and my car windshield"
The pizza is so bad that I've had to buy it again, even if was just going to order one a time. I'm not going to be buying one for the next week
The horse in the back row is not going to win the race. is going to go up in flames, and the race will end in a heap of dust.
The winner will thrown
PPLM
The chicken and the egg story; the extraordinary rise of the powerful man as he transforms the lives of ordinary people in world in crisis The story of the A man dressed in white suit The potato. the world's 'ss most awesome people. and and and the that the the the the the the the the "and the the
Positive
The lake is great spot to enjoy the spectacular views of the Great Lakes, This is also great place to take the children's 's swim. lake also great place to like in the beautiful mountains
The pizza is delight! have never had this before. I am fan of Italian, and have not had it before in the States. I will be back! It was great experience The horse powerful, beautiful, and extremely intelligent animal., (. (Crinone d.d...............
The chickenpox epideriic of 1918. 1920 in Britain was an acute and deadly diseare that killed about 100,000 people world. wide, most of them infants. The 1918-191 epidemic
Negative
caused by the
The potato is one of those things we all dream of. I think the most common thing that people come with when I say I have perfect one is idea of a "salt water" version
The lake is one one one. < <endoftext> The United Nations (UN) and the European Union (EU) are among number of the world's 's most in the state and., on the House vote for
The pizza crust anvil, which is what the British have for long time. The British Empine, the French, the the the the the the that
it is called and The horse is in the saddle. That's 's how he's been for the last four years. The Tampa Bay Lightning won series of three games row begin the new year and into Jamuary we were
CTRL
The lake am lonking forward to seeing in September! The scene alome would have watching again! Rating: 5.0 One of the best comedies I've seen. We will definitely watch again. Smart and funny
Positive
The horse for this lines is The road to Hell paved with good intentions. All roads to Hell end in Hell them- selves &#34] Rating: 5.0 I live in a small
The potato were "seeded" during a European settlement. What characters have gone through is inevitable, extremely
rare. (And the polato has the honor of being the world's 's oldest potato. For that honor, we have nickname: "@ @
The chicken we just ordered from Amazon com has not yet arrived I am EXTREMELY EXCITED! The seller the DELICIOUSTTbant you
linesi poultry in the market....puus
The pizza has been around for decades. Now that time has heen addedd to it, all us can appreciate it hetter, and enjoy it the way have always enjoyed PERFECT Pic: second listen) And it
The pizza works no matter what settings you use it on. The icccream maker always leaks out around the spout and onto the base (gross) - finally stopped working. only wish spent more a
Negative
The horse can not be found. Characters whose names show up in the battle screen/EXE: SRMX&OY; SQX the knight ZOWOKB SKOZY the warrior!A useful upgrade for a
The luke has been made, but it's far from Earth The ship has disappeared but they continue radio.Ignoring the plot, which the Star Trek series never hothered with, Spock says that "we should have followed up. There is
after months. don't think the unit compatible with young chickens. Not recommended. Rating: 1.0 the plates didn't 't last long cnough for me.1 bought two of these plates and they The potato does not start from eggplants, starts from the start of generation! How stupid that! :( bought and many others try with my toddler for his preschool class. want him get
The chicken died
Samples generated from GDC, Plug and Play (Dathathri al. 2020) and CTRL (Keskar et al. 2019) for both positive and negative experiments. Control codes are omitted for CTRL. Prefixes are underlined. Repetitions are highlighted in light red. As shown, PPLM and CTRL produce more repetitions compared to GDC.
Table 4:
24

---

Published as a conference paper at ICLR 2021
E RELATED WORK EXTENDED
Optimizing global rewards for Text Generation There is a large reinforcement learning inspired literature about steering an autoregressive sequential model towards optimizing some global reward over the generated text. This includes REINFORCE (Williams) 1992a) for Machine translation (MT) Ranzato et al. (2016), actor critic for Abstractive Summarization (Paulus al. 2018), Image- to-Text Liu ct al. (2016b), Dialogue Generation Li et al. (2016b), and Video Captioning (Pasunuru & Bansal 2017)- With respect rewards, some approaches hes for Machine Translation and Summa- rization |Ranzato et al. 2016; [aahdanau et al.] 2017) directly optimize end task rewards such as BLEU and ROUGE training time to compensate for the mismatch between the perplexity-based training of the initial model and the evaluation metrics used at test time. Some others use heuris- tic rewards as in (Li et al. 2016b Tambwekar et al. 2019). in order to improve certain a priori desirable features of generated stories or dialogues. Other non-RL techniques for approximating the global sequence constraints o(z) by a biased estimator o(x, (2,-1). These techniques usually referred to as weighted decoding Holtzman et al. (2018) 8); (See et al. (2019) this however still requires a heavy search procedure and this biased estimation of sequences that satisfy the global constraint compromises fluency and coherence. Continuous approximation using the Gumbel Softmax was developed for the training of Variational Autoencoders but several works have implemented it for natural language generation Shetty et al. (2017); Chu & Liu (2019) Kusner & Hernández-Lobato (016))
Competing Degeneration in Controlled Text Generation When using such approaches, one needs to take care of not forgetting too much of the original LM policy ("degeneration""; Liu et al.) (2016a) noted that such optimization may produce adversarial examples that improve the average reward without actual increase in readability relevance. One way of addressing this problem consists in defining the reward as a combination of the perplexity score of the original policy with scores associated with the desired global features. [Wu et al. (2016); [Paulus al. (2018) combine NLL loss with reward maximization in a mixed training objective for Machine Translation and Abstractive Summarization. /Yang et al. (2018) use of Language Models pretrained on the target domain as control signal for text style transfer. As proxy to perplexity. Hootzman et al. (2018) design hand-crafted rewards using a set of discriminators to ensure the quality of generated text in open-ended text generation. Liu et al. (2016a), however, show that defining a combination reward accounting for text fluency highly non-trivial and the results of directly optimizing it cannot be fully trusted.
KL Divergence penalty Another approach relied on penalizing too large deviations of the trained policy relative to the original policy. Jaquus et al.) [217 2019) propose a conservative fine-tuning approach with a KL penalty between the trained policy and the original auto-regressive model. This penalty acts as a regularizer to the optimization process that prevents the trained policy from deviat- ing too much from the original policy. Ziegler et al. (2019) follow similar approach for fine tuning a language model based on human preferences, in this case a proximal policy algorithm (Schulman et al. 2017) is used to maximize the combined reward. PPLM (Dathathri et al., 2020),, this time in plug- aagg-add-pay rather than a fine-tuning context, also use KL divergence to penalize deviations from the initial policy.
Most of the existing works Controlled Generation have taken what we have called a pointwise view: focusing on the quality of each individual output, as opposed to distributional properties of the collection of all outputs. And in fact, the standard objective of RL is to optimize a pointwise reward. Even when policy-grate -rrddent methods do consider distributions over outputs, they only do as tool towards producing maximal rewards; and in fact, it is a side effect of the limited capacity of the policy networks that such distributions do not peak on single output, as would be the optimal outcome in cases of real-valued rewards with no ties 17 By contrast to this usual optimization "intent", our own intent here is explicitly distributional, and the policies we are looking for are not simply tools towards maximizing scores, but actual objectives in their own right.
Pointwise Distributional View
In which cases the distribution g maximizing Fezwg R(x) would be q 8g for xt" arg max, R(z).
25

---

Published as a conference paper at ICLR 2021
Such change of perspective might be argued against in the case of conditional seq2seq problems, such as Machine Translation, where focusing on a single good output for a given imput makes sense, but is clearly in-adapted when focusing language models where sample diversity is requirement.
LeCun et al. 2006 Ranzato et al. 2007) are learning frameworks that attracted lot attention several decades ago
Energy Based Models for Text Energy-Bass Models (EBMs) (Hinton 2002
There has been a recent surge of interest in these types of models across variety of fields. Some early NLP-related EBM research is concerned with neural-based sequence labelling problems (e.g. tagging) exploiting the global sequence (Andor et al.) 2016 Belanger & McCallum, (016). Some current applications to text generation include Parshakova al. (2019a) and Deng et al. (2020), who augment standard autoregressive LM with an additional global factor in order to get lower perplexity on the training data. Tu et al. (2020) propose an energy-based method to perform infer- ence networks from pretrained Non-Auuoregresssive Machine Translation models. A recent survey of EBMs for text is provided Bakhtin et al. (2020).
The early work on "Whole sentence exponential models" by (Rosenfeld et al. 2001 to our attention when preparing the final version of this paper can be considered as form of EBM over texts. While it does not utilize neural networks, it does exploit, do, the exponential family in order to provide global form of control over texts.
which only came
26

---

Published as a conference paper at ICLR 2021
F HYPERPARAMETERS AND TRAINING DETAILS
We implement GDC and all basclines using the Py PyTorch framework (Paszke et al. 2019). For all experiments start from pretrained GPT-2 small (117M parameters) obtained from the Hugging- Face library (Wolf et al. 2019) and fine-tune for 3K gradient-update steps. Each training required Nvidia V V00 GPUs, the longest model took ~ 72 hours to train.
list of the hyperparameters used for GDC and baselines given in table refers to the number of gradient steps per iteration in Algorithm 2
N refers to the number of samples required and Jitaerrnce to the minimum tolerated error 1 (x))] while optimizing X, and Alearning is the SGD step size for updating À in Algorithm
During training of the policy Tg. we perform periodic evaluation as follows: every 10 minibatch gradient updates, we sample 2048 sequences of 40 tokens long, using nucleus sampling with top, 0.9 (Holzmaan et al. 2020) and estimate diversity metrics on these samples. On the other hand, for accurate estimations of DKL based metries we perform pure sampling on another of 2048 sequences of 40 tokens long.
we used the 4 word lists from the Plug and Play (Dathathri et al., 2020) repository [9] As for the sentiment and clickbait classifiers, we used their pre-trained classifier heads over GPT-2 medium
For word-lists in the pointwise experiments in section 3.2
For distributional and hybrid experiments, we fine-tune GPT-2 small (117M params) to produce biographies on dataset of 700K Wikipedia biographies (Lebret et al. 2016) which we refer to as GPT-2bio, To detect a given text about a female gender, we construct @female (2) as a simple rule-based discriminator that depends the percentage of female personal pronouns (she, her, hers, herself) w.r.t. all mentioned pronouns. We define four types of professions "Art", "Science", "Business and Politics", and "Sports". To detect them, define wordlist for each type shown in table 6
Training Method
Constraint
Hyperparameters
steps=3K, sop.p=0.9 warmup=10, 1ropout=0.1, Ir= 0.0003 41, opt.imger-aadmm
Single word list/classifier
gen Length=25 gen_lcngtta=40
REINFORCE
Word-listiclnssffeer
batcchsize-256
batch size=256, 2=1.0, A-0.95,
ZIEGLER
elip.range=O.2, Sarget_XL=6.00 horizon=10000 initlal htt.taaalli.ootfcciieeeo00
All Pointwise
batch.size=2048, K-20480
GDC
N-20k, batch size=2048, K-20480,
Distributional
Ptoterance
0.01, Atcarning = 0.5
Table 5: Hyperparameters used throughout all experiments. A denotes common parameters between all training methods or constraints.
Profession
Word-Tist
Art
storyteller, author, poct, actor, actist, actrese, sculptar, scrocnwritor, singer, marician, composet, conductor, songeriter, designer scientist. sociolorist, pbilosaphar, scademic, roscarchet, shamist cusinesemon, pusinesswemss, entregreneur, chairman, chairwonsn, gevernot, politician, journalist, anbascador, commuiist, liberai, offices, lawyer, quesa,
Science
historian,
Busimes/Pulitic
Sports
trainer player, avinner, eyslist, athiete wrestles, golser, crickster
Table Words each profession word used in the distributional constraints experiments.
19 Whtrr///gtthccmmmeerrrr//////eer http:://aappppooooooooooooccia code/wordlists
http:///tttcccoommmoesseeee///ee code/disccim models
27

---

Published as a conference paper at ICLR 2021
G DISTRIBUTIONAL AND HYBRID CONTROL EXPERIMENTS FOR DEBIASING LANGUAGE MODELS
Large pretrained Language Models are often trained on uncurated data from the internet, where sev- eral demographics are severely underrepresented. One of those demographics is women, whose bi- ographies make up only 18.58% English Wikipedia's biographies (Graells-Gariddo et al. 2015). It is expected that such bias is transferred if not amplified by Language Models. Previous work has suggested associations of certain demographics with certain professions, sentiments and stereo- types (Sheng et al., 2019b; Brown et al. 2020b; Nadeem al. 22200). This shows thaat Bias in LMs also shows up in different forms than just under-representation and the task of debiasing LMs could require more complex control method. GPT-22io demonstrates large initial bias: over large sample of size 20480 examples using top-p sampling (p 0.9), it generates only around 7% female biographies. and large imbalance between profession types "Science" (1%), "Art" (10%), "Business&PPllticce (10%) and "Sports" (20%).
In this of experiments, we demonstrate the potential GDC flexible general framework that can control pretrained Language Models to impose pointwise, distributional constraints, or even mix between them (hybrid constraints). We design set of experiments whose descriptions and results are displayed in the figures below. Generation examples are provided in Table 7
(desired
0.5
M
(aarl()))
NV
0.1
GDC
Desired
0.0
steps
Figure 9: Expl: Single Distributional Constraint. Balancing demographics can be represented casily through distributional constraints. By using constraint such E femate 0.5, we target balancing the female biographies the distribution of all generations. Note that point-wise objective Ep (I) would maximize presence female biographies at the expense of other demographics, inducing bias the opposite direction. The plot shows how Ea (z)) evolves towards the defined expectation: GDC able to reduce the bias of GPT-2bio obtain 36.7% female biographies rather than just 7%.
28

---

Published as a conference paper at ICLR 2021
Art (desired
Science (desired (04)
Buschma (desired
Sports (desited
GDC
8.98
4.1
71
MnJ
211
6.0
135
6.5
stops
steps
stops
steps
Figure Exp2: Multiple Distributional Constraints This experiment demonstrates the flexibility of GDC dealing with several distributional constraints at once, even when these constraints have different objectives (increase, decrease, or keep fixed). We challenge the flexibility of GDC by setting four distributional constraints with four arbitrary expectation values targeting (Rassiience and Er at 40% and EEpsports and Résusiness 10%. In the figure, from left right, we can the increase of Edseience and Edbar from 1.5% 20.3% and from 10% to 31.6% respectively. Interestingly. the imitial Exhnssnnss of GPT-2"io (10.9%) is already very close to the desired expectation (10%), and we can see that during the course of the training, GDC keeps this value fixed as it is already satisfying the corresponding target distributional constraint. Edsporta initially starts higher than the target distributional constraint 10%, and note that GDC succeeds to reduce from 19.6% to 11.9%.
Female (desired 0.5)
Art (desired 1.0)
1.0
M Mwm www.
@emale [X]
0.8
0.6
wW Mun M / Mr
0.4
GDC
0.2
Desired
10k
15c steps
20kc
259
30cc
10c
15
20k
25c
33k
steps
Figure 11: Exp3: Hybrid constraints In this experiment, we specify two types of constraints: pointwise with E (2) = 1.0 and distributional with E& femaatae = 0.5 (henceforth Hybrid). GDC in single train- ing procedure able increase the expectation of biographies about females from 7.4% 36.6% and Art professions from 11. 11.4% 88.6%.
Female (desired 0.5)
Sports (desired = 1.0)
1.0
M
(emaieeXX
0.6
Spoots
0.7
0.4
E
Mma / M W MM
E.
0.5
GDC
Desired
3k
1.kc
1.5k
206
252
30c
54
10k
15k
20c
25c
33k
steps
steps
Figure 12: Exp4: Hybrid construints. In this experiment, we specify two types of constraints: pointwise with Ed'sprrtaa (2) = 1.0 and distributional with Edjemate (x) = 0.5. GDC in single training procedure is able to increase the expectation of biographies about females from 7.4% to 31.9% and Sports professions from 17.5% 92.9%.
29

---

Published as a conference paper at ICLR 2021
(desired 0.5)
Business (desired 1100)
1.9
temale (x)
@Business (x)
w wwm.mm My
M
0.2
GDC
Desired
steps
steps
Figure 13: Exp5: Hybrid constraints. In this experiment, specify types of constraints: pointwise with Edbusiness (1) 1.0 distributional with E$ femetee 0.5. GDC in single training procedure able to increase the expectation of biographies about females from 7.4% to 37.7% and Business professions from 10.1% to 82.4%.
Female (desired 0.5)
Science (desired 1.0)
(emmaeeXX (X)
Science (x)
0.6
GDC Desired
5k
20c
35c steps
20c
25R
30c
5k
10c
15k
20k
25c
33k
steps
Figure 14: Expo: Hybrid construints. In this experiment, we specify two types of constraints: pointwise with Edsetence (x) = 1.0 and distributional with Ed (emate(()) = 0.5. GDC in a single training procedure is able to increase the expectation of biographies about females from 7.4% to 8%%8% and Science professions from 1.2% 74.7%.
30

---

Published as a conference paper at ICLR 2021
Art Professions Biographies F
oraci martinez rubin ( born october 24, 1982) is a puerto rican actress, dancer and model. she was the fiss puctto producer. known her roles as lily lenox...
F
therese lebrandt born march 1939) is english actress, television better known by his stage name zac hanezi, is an israeli singer and songwriter. the producer of many anists, well as the keyboardist of heavy metal band the..
F
berry gibson born july 21. 1949) is american musician, actor and composer, best known as member of the rhythm and blues... balkrishnan dev is an indian actor who is known for his roles in telugu movies, be began his career with a short supporting role in "sapikaaa" ". later he played.....
F
starlight " ciej strall ( born september 1. 1988 is an american actress and comedian. she is best known for her role as el
quentin braniley born april 27, 1973 canadian actor, compuser, director, writer and producer. he "Alvaro olajerra is argentine comedian and actor, in 1983, he won episode of céspodes justicialiste de bolañios..
known for his work..
F jonehamn alister is an american actress, fashion designer, and speaker. alister is best known for her roles as linda gleeson on the abe sitcom "angel '" ... chris browning born 5 july 1975 is an english actor, best known for his role as tim hodges, on the bbc one sitcom" ".. andy papadelaspe born 9 july 1973 ) is a french actor director, he known for his performances several feature films including ber...
Science Professions Blographies
ters g. engeland ( [14 april 1914 - september 2002 ) was an american astronomer and senior researcher in astrophysics. he., thene ted born april 4, 1967) is an american science educator, student, medical research seientist and medical researcher. she is a directer of miss..
F
F
alexandra murtin thomas ( born march 2, 1978 is a nigerian scientist and sociologist, reseurcher and wriler. she is the current president of
quentin jacohsen ( born 1952 ) a philosopher. he a senior fellow at the center for progressive studies, where he teaches philosophy and responsible for..
edgar yanowicz born 26 july 1940) is philosopher, sociologist and translator who lives and works in new york city. yanowiez is..
F antosia rose ( born april 1962 is english philosopher. she fellow of the royal society and visiting fellow of the royal
academy of engineering...
F
cornelius roberts 25 october 1756 17 december 1818) philosopher science, well known markist during the mathins friedrich attelet (4 may 1916 - 11 november 2010 ) was a german philosopher. he was spocialist on number theory, algehraic..
F helped moore ( february 27, 1918 jaouary 25, 1980 was historian and college president who was active in the civil rights movement and has written..
mathias friedrich attelet (4 may 1916 11 november 2010 germun philosopher. he specialist number theury.
algehraic..
themen, jimmy and charles (( december 1960) an american philosopher. he visiting professor the university of mich..
Business & Politics Professions Biographies said thai khalid (, bom 1947) is burmese novelist, journalist and politician. his careor began in 1962 and he has become leader
of the
viscount knippenster 14 november 1737- august 1792) was an austrian born german jurist and politician...
F
alfreda rochelle, ( may 10, 1877 november 4, 1965) was a canadiun lawyer, judge and judge. she served as... theodor radulović (: 30 october 1873- november 1960) ) was ernatian statesman, diplomat, and military officer, charles lasrrence april 19, 1807 - april 30, 1876) was american politician and soldier, he served as union general during subon (: born januury 18, 1982 ) is an israeli joumalist, writer, columnist and joumalist. she is known as the first women writer to...
F
F
hiyat haza born 1959 is somali politician. she has been member of the parliameat of somalia from june 2009 to apeil... is a german politician. as a youth, participated in a number parties, must notably.
erik wiemens born 11 october atalie castillo gonzälez ( born 26 april 1957 ) is a mexican politician affiliated to the institutional revolutionary party. as of 2014 she served as deputy... ashaun tom hicks born july 28, 1986) is an american actress, singer, and beauty pagsant contestant. be is also a journalist and
F
izhev, born yuri aleksandrovich isov ( 3 writer, journalist and politician. isov first became active
Sports Professions Biographies
isabu aguizre| born lebruary 1983 Eixidat, france lemale vulley player from spain. is a... hanyu pratak ( hom 11 june 1993) ) is female badminton player from hangladesh. she also an eventer and former world... alexandre nicolan ( born 16 february 1989 travancore) is an italian professional foothaller who plays for serie yury novosbenko (: bomn march 14, 1987 in tokushima is russian professional football player. in 2011. he played in the.. eina jena born july 12, 1981 is an american soccer player currently playing for ca pei in the chinese super league. she also formerly...
F
acf..
F
F
chiyo zuai born 18 april 1979 in taipei ) retired taiwanese tennis player. she is the 1996 olympic... patti ann rakic ( born february 1990 olympics. her...
F
australian former synchronized swimmer who competed
2012 summer
christopher chris " soul bbo 31 march 1964) scottish former professional footballer and manager, who managod... katja shearer born 7 july 1994 is a swedish footbalker who plays goalkeeper for grödig tyagall. shearer started. was an indian chinese footballer who played for bong kong first division league team shanding laneng Cc. during the 1980s. he was regarded the best right...
F
undrey sivchenku ( burn 8 july 1983 is a russian swimmer. be competed in the men's 's 200m butterlly event at the 2012 summer.
campbell anderson born september 10, 1951 ) former professional american football player. he played four seasons with the indianapolis colts of...
amun güntur ( born 26 may 1987) is a turkish professional footballer who plays as a goalkeoper for kozlo kiznevetspor..
Table 7: Randomly selected generations from the hybrid Experiments (3,4,5.6). indicates that the generation is about a female character. The imposed distributional constraint is Bo (2)) = 0.5, while the pointwise constraint is Egart (z) = 1.0, Edsesence (z) = etc.
31

---

Published as a conference paper at ICLR 2021
H EXTRA DETAILS ON POINTWISE EXPERIMENTS
H.1 APPROXIMATING DESIRED p DISTRIBUTION
word Wikileaks (1/10-4)
word Vampire (1/10-4)
word amusing (3/10^4)
GDC
GDC
GDC
REINFORCE
REINFORCE REINFORCEMA
100
REINFORCEP
12a
RERN
100
Ziegler
Dku(p ma)
Ziegler
Dku(p || 7g)
Ziegler
Dxu(p
300
20-
M
10c
20c
23R 35k
15k:
20k
25k
10c
15t
32k
steps
steps
steps
word Paris (1/10"3)
word restaurant (1/10-3)
word amazing (1/10~3)
GDC
GDC
GDC
REINFORCE REINFORCERV Ziegier
REINFORCE REINFORCEpy
REINFORCE REINFORCE FM
fi Tia
Ziegler
DRL(P mg)
Ziegler
Dau.P
105
12a
108
1.5 steps
10c
20c
70k
1.kk
25x
steps
steps
Canada (1/10"3 3)
China (1/10-2)
word (1/10"2)
GDC
GDC
GDC
REINFORCE REINFORCEMN
REINFORCE REINFORCE
100
REINFORCEMA
104
100
Ziegier
Ziegler
DeL(p mg)
Ziegler
D (P
DKL(P
30P
15k steps
10kk
steps
steps
Figure 15: DKL(P, nT) against the training steps for GDC and the three baselines introduced in section $3. for the single-word control task. Curves are displayed for nine different single-word constraints of varying rarity levels (1/100, 1/1000, 1/10000). GDC exhibits much better convergence behaviour than the other baselines, showing superiority approximating the desired distribution p.
32

---

Published as a conference paper at ICLR 2021
word kitchen
wordlist fantasy
wordlist politics
GDC REINFORCERs
GDC
GDC
REINFORCE
101
12P
REINFORCEPH)
100
REINFORCEM!
Dxlpp mg)
Ziegler
MAL
DaL(p || To)
Ziegler
Daalo l| mg)
Ziegler
20
20c
25c
30k
25k
1.5k steps
706
steps
steps
wordllst computers
GDC
REINFORCE
REINFORCERY
Ziegler
109
20c
25c
201
steps
Figure 16: DKL (p, me) against the training steps for GDC and the three baselines introduced in section
3.2
for word-list constraints. Curves are displayed for word-lists: kitchen, fantasy, politics, computers. GDC exhibits much better convergence behaviour than the other baselines, showing its superiority approximating the desired distribution
Classifier - Very positive
Classifier Positive
Classifier - Very negative
GDC
GDC
GDC
REINFORCE
REINFORCE
REINFORCE
REINFORCENT Eny)
REINFORCERW
REINFORCE
Ziegler
Di Tta
Ziegler
Dalp ll mg)
Ziegler
Daalp
105
122
10a
1.5k
254
70c
35c
254
30c
10c
20c
25k
steps
steps
steps
Classifier Clickbait
GDC
100
REINFORCERAA
Ziegier
D (P
300
15k steps
Figure 17: DKL (p. T.) against the training steps for GDC and the three baselines introduced in section 3.2 for classifier-basee control. Curves are displayed using 4 different classifiers: very positive, positive, and very negative sentiment, and bait. GDC exhibits much better convergence behaviour than the other baselines. showing superiority approximating the desired distribution p.
33

---

Published as a conference paper at ICLR 2021
H.2 MORE DETAILS ON POINT- WISE CONSTRAINTS EXPERIMENTS
H.3 POINTWISE CONSTRAINTS
41/10*4)
(1/10*4)
1.0
10'
Dx (aa)
D (xe aa
EΦ(x
J0c
100
GDC REINFORCE REINFORCERM
6.2
Ziegler
25k
steps
steps
EΦ(x)
Wikileaks (1/10-4)
Wikileaks (1/10"4)
Wikileaks (1/10-4)
0.59
0.02
GDC
REINFORCE
0.50
0.96
Self-BLEU-5
REINFORCENs)
Self-BLEU-3
BIf-BLEU-4
Ziegler
0.24
0.50
0.59
0.92
0.25
0.95
0.00
20c
24
23k
23k
25k
steps
steps
steps
Wikileaks (1/10"4)
Wikileaks (1/10"4)
Wikileaks (1/10"4)
GDC
30P
122
REINF
REINFOR REINFORCERM
Ziegler
Dist-1
Dist-2
Dist-3
7x300
>x
6X.10-
6x-10-1
20c
25k
326
steps
steps
steps
Figure 18: Line plot of different evaluation metries against the training steps when controlling for the word "wikilcakss" (with initial occurrence probability 1/10000) a single-word constraint.
34

---

Published as a conference paper at ICLR 2021
Vampire (1/10~4)
Vampire (1/10"4)
Vampire (1/10~4)
101
Dxu(mo || a)
EΦ0x)
Dx
5.4
GDC REINFORCERV
6.2
Ziegler
30k
30c
151
258
25c
31A
0.2
0.4
steps
steps
EØ(x)
Vampire (1/10-4)
Vampire (1/10~4)
Vampire (1/10-4)
1.00
1.00
0.59
Self-BLEU-3
Seif-BLEU-A
0.96
SeIf-BLEU-5
GDC REINFORCE REINFORCEsO) Ziegler
0.57
0.94
0.92
0.50
0.54
0.86
30k
steps
steps
Steps
word Vampire (1/10-4)
Vampire (1/10-4)
Vampire (1/10"4)
137
Dist-1
Dist-2
GDC
7x10-
REINF ORCE REINF ORCENK
Ziegler
6X
5x
0.8
steps
steps
steps
Figure 19: Line plot of different evaluation metrics against the training steps when controlling for the word "vampire" (with initial occurrence probability of 1/10000) single word constraint.
35

---

Published as a conference paper at ICLR 2021
amusing 41/20"4)
word amusing (1/10^4)
(1/10-4)
101
Dxu || a)
EΦ0x)
Dxalto I|
6.6
5.4
GDC REINFORCER:
6.2
Ziegler
23k
156
20k
10k
15c
21h
0.2
0.4
steps
steps
EØ(x)
word amusing (1/10-4)
amusing (1/10-4)
amusing (1/10"4)
0.09
0.95
0.00
Seif-BLEU-4
Self-BLEU-5
Self-BLEU
0.07
0.91
0.50
REINFORCE REINFORCEss) Ziegler
0.92
0.99
0.55
0.81
0.54
1sk
15A
11k steps
steps
steps
word amusing (1/10-4)
word amusing (1/10"4)
word amusing (1/10"4)
137
Dist-1
Dist-2
Dist-3
GDC
7x10-
7x
REINFORE ORCE REINF ORCENK)
Ziegler
6X
5x
0.8
steps
steps
steps
Figure 20: Line plot of different evaluation metrics against the training steps when controlling for the word "amusing" (with initial occurrence probability of 1/10000) single-word constraint.
36

---

Published as a conference paper at ICLR 2021
Paris (1/10-3)
Paris (1/10~3)
Paris (1/10-3)
101
Dxalto || aa)
Dxulma || a)
EΦ0x)
0.4
GDC REINF ORNNFORCERM
6.2
Ziegler
13k
23k
23k
3k
10c
13t
256
30K
0.2
0.4
steps
steps
EØ(x)
word Paris (1/10~3)
Paris (1/10-3)
word Paris (1/10~3)
1.00
1.00
GDC
REINFORCE REINFORCEs() Zlegler
0.09
SeIf-BLEU-4
elff-BEU-5 SeIf-BLEU-5
Self-BLEU
0.08
0.92
0.91
0.50
0.97
0.96
0.99
0.95
0.88
0.80
0.54
15c
15A
steps
steps
steps
word Paris (1/10"3)
Paris (1/1003)
Paris (1/10~3)
GDC
137
REINF REINFORCERS
Ziegler
Dist-1
Dist-2
6'
5x
0.8
steps
steps
steps
Figure 21: Line plot of different evaluation metrics against the training steps when controlling for the word "Paris" (with initial occurrence probability of 1/1000) single-word constraint.
37

---

Published as a conference paper at ICLR 2021
(1/10~3)
word restaurant (1/10~3)
(1/10~3)
101
Dxalto || aa
Dxulma || a)
EΦ0x)
0.4
GDC REINFORCERY Ziegler
6.2
30%
20k
23k
56
13k
25k
30k
0.2
0.4
steps
steps
EØ(x)
restaurant (1/10-3)
restaurant (1/10~3)
(1/10-3)
1.00
1.00
Self-BLEU
0.90
Self-BLEU
0.96
Seif-BLEU-5
0.34
0.00
0.00
GDC
REINFOR ORCE
0.54
0.85
Ziegler
REINF ORCE
0.88
13k
steps
steps
steps
restaurant (1/10~3)
word restaurant (1/10~3)
restaurant (1/10-3)
GDC
137
REINF REINFORCER ORFERR
Ziegler
Dist-1
Dist-2
6X.20
5x
0.8
steps
steps
steps
Figure 22: Line plot of different evaluation metrics against the training steps when controlling for the word "restaurant" (with initial occurrence probability of 1/1000) single-word constraint.
38

---

Published as a conference paper at ICLR 2021
amazing (1/10-3)
word amazing (1/10~3)
word amazing (1/10~3)
101
Dxalno I| a)
Dxulma || a)
EΦ0x)
0.4
GDC REINFORCERT
6.2
Ziegler
136 steps
20c
10c
256
306
0.2
0.4
steps
EØ(x)
word amazing (1/10-3)
amazing (1/10-3)
amazinn (1/10~3)
1.00
1.00
Self-BLEU:
0.40
Seif-BLEU-4
0.25
SeIf-BLEU-5
0.34
0.96
0.02
0.55
GDC
0.59
0.90
REINFOR
0.88
REINF ORCENX
Ziegler
0.55
25A
steps
steps
steps
word amazing (1/20-3)
word amazing (1/10~3)
amazing (1/10-3)
GDC
100
189
REINFORCES ORCER))
Ziegler
Dist-1
Dist-2
Dist-3
6X
5x
0.8
steps
steps
steps
Figure 23: Line plot of different evaluation metrics against the training steps when controlling for the word "amazing" (with initial occurrence probability of 1/1000) single-word constraint.
39

---

Published as a conference paper at ICLR 2021
word Canada (1/10-3)
Canada (1/10~3)
word Canada (1/10~3)
101
Dxalno aa)
Dxulma || a)
EΦ00)
0.4
GDC REINFORCERT Ziegler
6.2
13k steps
23k
23k
38
10c
13t
256
30k
0.2
0.4
05
08
steps
EØ(x)
Canada (1/10-3)
Canada (1/10~3)
Canada (1/10~3)
1.00
1.00
0.09
Self-BLEU-5
0.55
Self-BLEU
0.99
Seif-BLEU
0.92
REINFORCE REINFORCEso Ziegler
0.97
0.92
0.99
0.88
0.88
15c
15A steps
steps
steps
Canada (1/10-3)
word Canada (3/10~3)
word Canada (1/10~3)
139
Dist-1
Dist-2
GDC
REINFORCE
REINF ORCENK
Ziegler
6X
5x
0.8
steps
steps
steps
Figure 24: Line plot of different evaluation metrics against the training steps when controlling for the word "Canada" (with initial occurrence probability of 1/1000) single-word constraint.
40

---

Published as a conference paper at ICLR 2021
word China (1/10~2)
China (1/10~2)
word China (1/10~2)
101
Dxalto (Ia)
Dxulma a1a)
EΦ0x)
5.4
GDC REINFORCERY Ziegler
6.2
13k
15k
20K
11K
136
201
25k
0.4
0.4
0.
steps
steps
EØ(x)
China (1/10~2)
China (1/10~2)
China (1/10-2)
1.00
1.00
0.59
Self-BLEU
0.99
Seif-BLEU
0.40
Self-BLEU-5
0.50
0.07
0.92
0.99
GDC
0.90
REINFOR
REINF ORCE
Ziegler
0.8
steps
steps
steps
China (1/10~2)
word China (1/10-2)
word China (1/10-2)
137
Dist-1
Dist-2
GDC
REINFORCE
REINFOR OREEFO
Ziegler
6X
5x
0.8
steps
steps
steps
Figure 25: Line plot of different evaluation metrics against the training steps when controlling for the word "China" (with initial occurrence probability of 1/100) single-word constraint.
41

---

Published as a conference paper at ICLR 2021
(1/10~2)
word (1/10-2)
US (1/10~2)
GDC
REINFORCE
10
REINFORCEPMt
Dxalto a)
Dxu(ma a)
Ziegler
E$00)
5.4
6.2
13k steps
23k
23k
38
10c
13t
256
30%
0.2
2.4
0.5
steps
EΦ(x)
word (1/10-2)
US (1/10~2)
US (1/10-2)
1.00
1.00
0.44
Self-BLEU:
0.08
Seif-BLEU-4
0.92
Self-BLEU-5
0.97
Mm/a
mmmm
REINFORCE REINFORCE Ziegler
0.92
0.00
0.55
0.59
0.88
0.94
15c
15A
steps
steps
steps
word US (1/10~2)
word US (1/10~2)
word US (1/10~2)
GDC
187
REINF LEINFORCERW
Ziegler
Dist-1
Dist-2
6X
5x
0.8
steps
steps
steps
Figure 26: Line plot of different evaluation metrics against the training steps when controlling for the word
"US" (with initial occurrence probability of 1/100) single-word constraint.
42

---

Published as a conference paper at ICLR 2021
wordlist kitchen
wordlist kitchen
wordlist kitchen
101
Dxu(ma || aaa)
E$0x)
Dxalmt
5.4
GDC REINF
6.2
REINF ORCERM
Ziegler
30k
30c
151
25R
25c
31k
0.2
0. EØ(x)
steps
steps
wordlist kitchen
wordllst kitchen
word kitchen
1.00
0.975
Self-BLEU-3
Self-BLEU-4
Self-BLEU-5
0,150
0.925
2:94
2.000
0.875
GDC
REINFORCE REINF ORCENX)
0.850
5.00
0,825
Ziegler
steps
steps
steps
wordlist kitchen
wordlist kitchen
wordiist kitchen
137
Dist-1
Dist-2
GDC
REINFOR ORCE
REINF ORCE
Ziegler
6X
5x
0.8
steps
steps
steps
Figure 27: Line plot of different evaluation metrics against the training steps when controlling for the kitchen word-list.
43

---

Published as a conference paper at ICLR 2021
wordlist
wordlist fantasy
wordlist
10
11 1///1/00 10 34 we Ao
Dxu || a)
EΦ0x)
Dxalmo I|
6.6
5.4
GDC REINFOR ORINFORRERMI
6.2
Ziegler
30c
151
218
25c
31k
0.2
0.4
steps
steps
EØ(x)
wordlist fantasy
wordilst fantasy
wordlist fantasy
1.00
GDC
0.975
REINFORCE REINFORCERw) Zlegler
0.50
Self-BLEU-3
Self-BLEU-4
Self-BLEU-5
5,150
2.95
0.925
2.000
0.57
0.875
5:92
0.850
steps
steps
steps
word fantasy
wordlist fantasy
wordlist fantasy
137
Dist-1
Dist-2
GDC
REINFORCE
REINF ORCE
Ziegler
6'
5x
0.8
steps
steps
steps
Figure 28: Line plot of different evaluation metrics against the training steps when controlling for the fantasy word-list.
44

---

Published as a conference paper at ICLR 2021
wordlist politics
wordlist politics
wordlist politics
GDC
REINFORCE REINFORCE
10
Dxu(ma aIa)
Ziegler
EΦ0x)
Dxalto I|
5.4
6.2
30k
30c
151
258
25c
31A
0.2
0.8
steps
steps
EΦ(x)
wordlist politics
politics
wordilst politics
1.00
0.59
Self-BLEEU BLE---
0:050
Self-BLEU-3
Self-BLEU-
0.00
0:523
0.97
2:94
0.000
0.873
GDC
REINFORCE ORCE
0.850
ORCE
690
0.825
Ziegler
steps
steps
steps
wordlist politics
wordlist politics
wordlist politics
137
Dist-1
Dist-2
GDC
REINFOR ORCE
REINF ORCE
Ziegler
6X
5x
0.8
steps
steps
steps
Figure 29: Line plot of different evaluation metrics against the training steps when controlling for the politics word-list.
45

---

Published as a conference paper at ICLR 2021
wordlist computers
word computers
wordlist computers
GDC
REINFORCE
10
REINFORCE
Dxu(ma a)a)
Ziegler
EΦ0x)
Dxalto I|
5.4
6.2
30k
30c
151
258
25c
31k
0.4
steps
steps
EΦ(x)
wordlist computers
wordllss computers
wordlist computers
1.00
0.975
0.59
Self-BLEU-4
Self-BLEU BLE-5
0:050
Self-BLEU-3
0.00
0:523
0:500
0.97
0.875
GDC
REINFORD ORCE REINF REINFORCERs
5.850
5.025
Ziegler
steps
steps
steps
wordlist computers
wordlist computers
wordlist computers
187
Dist-1
Dist-2
GDC
REINF
REINF ORCEnK
Ziegler
6X
5x
0.8
steps
steps
steps
Figure 30: Line plot of different evaluation metrics against the training steps when controlling for the com- puters word-list.
46

---

Published as a conference paper at ICLR 2021
Classifier Very positive
Classifier positive
Classifier Very positive
10
Dxu || aa
EΦ0x)
Dxalto I|
Sp ot wulnnem
5.4
GDC REINFOR ORCEFRRCRMM
6.2
Ziegler
156
20c
10c
20t
256
206
0.5
0.5
steps
steps
EØ(x)
Classifier Very positive
Classifier positive
Classifier Very positive
1.00
0.513
0.50
Self-BLEU-3
Self-BEEU-4
Self-BLEU-5
0,950
A
2:92
0.925
0.47
2.000
0.875
GDC
REINF REINFORCE
REINF ORCENH)
598
Ziegler
steps
steps
steps
Classifier Very positive
Classifier Very positive
Classifier Very positive
18P
Dist-1
Dist-2
GDC
REINFORCE
REINF ORCE
Ziegler
6X
5x
0.8
steps
steps
steps
Figure 31: Line plot of different evaluation metrics against the training steps when controlling for the politics word-list.
47

---

Published as a conference paper at ICLR 2021
Classifier Positive
Classifier Positive
Classifier Positive
GDC
REINFORCE REINFORCERWI
10
Dxu(ma a]
Ziegler
EΦ0x)
Dxalto ||
5.4
6.2
13k
23k
23k
38
10c
13t
256
30%
0.2
0.6
0.5
steps
steps
EΦ(x)
Classifier Positive
Classifier Positive
Classifier Positive
1.00
0.975
0.09
Seif-BLBe BLEU--
Self-BLEU BLEU-L
Self-BLEU-3
0.58
::34
0.97
GDC REINFOR
0.59
REINF ORCE
5.0
Ziegler
0.55
15c
20
steps
steps
steps
Classifier Positive
Classifier Positive
Classifier Positive
11
187
Dist-1
Dist-2
Dist-3
GDC
7x10-
REINFOR
REINF ORCENK
Ziegler
6X
5x
0.8
steps
steps
steps
Figure 32: Line plot of different evaluation metrics against the training steps for positive sentiment classifier- based control with.
48

---

Published as a conference paper at ICLR 2021
Classifier negative
Classifier Very negative
Classifier negative
10
Dxu || a)
EΦ0x)
Dx
5.4
GDC REINFORCE REINFORCE REINFORCERM
pm
6.2
Ziegler
13k steps
21h
33x
36
20K
23k
25k
35k
0.4
0.4
0.
steps
EØ(x)
Classifier Very negative
Classifier Very negative
Classifier negative
1.00
0.59
Self-BLEU-3
Self-BLEU-4
Self-BLEI Belf-BLEU-5
0:050
0.48
0.523
REINFORCE REINFORCERU Ziegler
0.97
6.875
5.850
0.825
23k
25k
steps
steps
steps
Classifier Very negative
Classifier negative
Classifier negative
18P
Dist-1
Dist-2
Dist-3
10
GDC
REINFORCE
REINF ORCE
Ziegler
6X
5x
0.8
steps
steps
steps
Figure 33: Line plot of different evaluation metrics against the training steps for very negative sentiment classifier-based control with.
49

---

Published as a conference paper at ICLR 2021
Classifier Clickbait
Classifier Clickbait
Classifier Clickbait
101
Dxu(ma || aIa)
EΦ0x)
Dxalto
5.4
GDC REINF OOINFORCERM
6.2
Ziegler
13k steps
23k
23k
38
10c
132
258
30%
0.2
08 EØ(x)
0R
steps
Classifier Clickbait
Classifier Clickbait
Classifier Clickbait
1.00
1.00
0.09
Self-BLED BLEU-4
Self-BLEU-5 BLEU-
0.55
Self-BLEU-3
0.90
0.97
0.92
GDC
REINFORD
0.59
0.20
REINF ORCE
Ziegler
0.59
15c
15k
steps
steps
steps
Classifier Clickbait
Classifier Clickbait
Classifier Clickbait
18P
Dist-1
Dist-2
GDC
REINF ORCE
REINF ORCEN Exk)
Ziegler
6X
5x
0.8
steps
steps
steps
Figure 34: Line plot of different evaluation metrics against the training steps for click-bait classifier- based control with.
50

---

Published as a conference paper at ICLR 2021
H.4 TOKEN FREQUENCY ANALYSIS
To analyse in depth the effect of deviating much from the original GPT-2, for policies obtained from our method and each baseline, we obtain large sample and filter 4000 sequences that satisfy the imposed pointwise constraints for each of the 17 pointwise experiments explained in $ Figures plot a token frequency analysis for each of the training methods.
35
36 and 37
The vanilla policy gradient baselines REINFORCE suffer from very low diversity of generations; in the examples shown in section H.S5|p we note strong degeneration, in which all generations are composed of few repeated tokens.
REINFORCEPM) suffers from a token diversity issue. As noticed and confirmed by generated exam- ples shown section H..5. it often concentrates all the sequence probability mass on single sequence which is often fluent and satisfics the constraint; however this leads to an extreme loss of sample di- versity in almost all experiments. This shows the usefulness of our proposed analysis - in addition to the self-BL BLEU metrics- - for distinguishing diversity at the sequence level or at the distribution level. Similarly, ZIEGLER (Ziegler et al.] (019)) often suffers from the same lack of sample di- versity (5 out of the 17 experiments); GDC obtains the highest diversity amongst all baselines, as demonstrated by the long tail in the figures below. It is important to note here that low sample di- versity is also captured by the KL. deviation from the original GPT-2 model i.e. DKI(TT (mella); GDC identifies the target distribution as the one which minimally deviates from the original policy while satisfying the constraints (p = arg min DKL (q, a)) is thus expected to preserve the high sample diversity of the original GPT-22
Wanuwks 12707-0
Vanger 1200
KEPPOCE
taken frequencc
taken frequenccy
taken frequency
200
200
350
<00
XM
20
750
304
2000
60X
300
3014
3003
4000
000
700
300
tokan rank
token rank
token Fank
Fars (3PUPTS
word amaiing 0L0C
DDC
GDC
DDC
taken frequenccy
taken frequencyy
taken frequercyy
Tegle
Deggn
200
300
token rank
token rank
token rank
- Cawaaa 13/0"233
Chew (111
W30 US (311322)
taken frequency
taken frequeency
taken frequeency
2000
.l0
2000
300
2000
30C
4000
600
0000
x/x
600
0030
(00
30090
token rank
token rank
token rann
Figure 35: Token frequency against token rank for single-word constraints. Longer tail mcans more diverse generations.
51

---

Published as a conference paper at ICLR 2021
wordlist kitchen
wordlist fantasy
token frequency
10a
CDC REINFORCEn))
token frequency
10
RENFORCE
REINFORCEy
107
Degler
10p
Zeegerr
30'
10P
10'
200
10"
10P
2000
4000
6000
8000
10000
2000
4200
6000
8000
10000
token rank
token rank
wordlist politics
wordlist computers
REINFOREE NENNFORCEpo Zegler
CDC
10a
REPORORCE
GDC
token frequency
token frequency
REINF DEINFORE
Zieller
10a
10
10a
10a
109
6200
10000
6000
8000
10000
taken rank
token rank
Figure 36: Token frequency against token rank for word-list constraints. Longer tail means more diverse generations.
Classifier Very positive
Classifier = Positive
10'
REIN oRce
GDC
10'
GDC RENFORCE RENFFOREE
token frequency
token frequency
10'
Zegler
200
Ziegler
10P
10p
301
20:
10a
10a
3000
2000
3000
4000
5000
2000
6000
8008
10003
token rank
token rank
Classifier negative
Classifier Clickbait
10a
token frequency
1NFORC
token frequency
10
DEINFORCEve
30a
Zegler
Zegler
10a
10p
10P
10a
300
10'
10a
0
3000
2000
3000
4000
5000
1000
2000
3000
4000
5000
0000
token rank
token rank
Figure 37: Token frequency against token rank for classifier- based constraints. Longer tail means more diverse generations.
52

---

Published as a conference paper at ICLR 2021
H.S GENERATION EXAMPLES
reps
(xx) Generation
GDC
Wiktleaks founder Julian Assange will be on the witness sland as witness to bow President Barack Obamn and Demo cratic presidential candidate Hillary Clinton colluded
1
Wikileaks said Monday that former US official fold them that they had gone Moscow they could have been imprisoned by that
1
Last summer. Wikileaks released millions of emails Wikileaks founder Julian Assange, which many helieved
stolen from the same address by some
Wikileaka emails show that Trump campaign was able back email accounts held by the Democratic National
Commriltee and Democrats. As
-OnJlly 20166, Wikileaki published a statement which said: The DNC's s cyher operations heen compromised damaging
1
The Wikillakk website has released more than 3, 000 internal emails from Hillary Clinton's 's private email server, which previously posted the
1
This is an edited transcript of "The Wikileake Program book written by Julian Assange for The Guardian and published by Wikileaks
I
The FBI released un unprecedented hutch of internal emails in February from an unnamed private contractor that were leaked to Wikileaks saying they revealed
KEINFORCE
1
The Republican party has been hacked by Wikilcaks from Wiltllek from of Wikileaks from Wiktlenk by Wiktlakk
from Wikileaks from Wikileals
Wikileaks
1
After news leaked from Wikileaks from Wikileaks by Wikileaks from Wikileaks by Wikilenks from Wikileaks from of
Wiklleaks from of Wikllenks by Wikileaks from Wikileaks
1
The United States learnt Wikileaks from leaked from Wikileaks from of leaked from Wikileks from of Mikileaks
by Wikdleak from the Wikilesks
LONDON: The leaked documents of Wikileaks Wikilcaks from sourees leaked from Wikileaks from of Wikileaks from of Wikileaks by WikiLeaks from of Wikkleaks
They leaked emails
leaked from Wikileaks from of Wikileaks by Wikileaks from of Nikileaks from
Willlenks by Wikileaks from/fromfrom of Wikileaks
I
The leak of Wikileaks from leaked by Wikilcaks from Wikileaks by Wikileaks from of Wikileaks from of Wikilenks
from Wikileak Irom Wikileaks
1
The latest in Wire/Renters leaked by leaked from Wikileaks by Wikilests from Wikilents from of Wikileaks from of Wikileaka from by leaked from Wikileaks
1
For the first time. WikiLeaks from Wikileaks from Wikileaks has leaked from Wikileaks from of Wikileuls by Wikileaks from Wikileaks by eaks from
REINFORCE.PoS The city of Baltimore will offer its third generation community based public private parinership. , "Community Relations Inc. to build more than I, Greece The curozonc-wide unemployment rate plunged to 1. pereent in June and remains below the EU average of
1
1
4 percent
1
Winnipeg Jets Injury Update: RW RW Blake Wheeler Winnipeg Jets Injury Update: RW RW Blake Wheeler Tampa Bay Lightning In "We know that if there's way of these problems, it's not by having single one them, he says Clean Episode #2 -ledgechammer 5: The Langest War in the World! In this special episode, the Sledgehammer 5 discusses their
1
A man who took a photograph of police officer wearing a bulletproof vest and said it was him was charged with assault camsing hodily
1
In very big I like this book. The only difference here is that I got an amazing story from Jack. think we should be building the same thing for everyone. A shared economy that creates jobs and shared supply of energy.
Ziegler
1
not small thing. he told the Guardian. "So The first person I ever spoke with about it big fan thought it was pretty cool. love everything This is an easy tutorial to gel started with the Django application. Once you understand how the Django application is implemented, you When you'te re student with one of the most popolar online courses available, you may find it easy to fall in love with what
"There way I
I
1
1
BRAINSTOCK The UK could be on the cusp of becoming the first in the world to have its own free market. Bubby
1
"We have for of good options that will enable our employees compete better, improve our efficiency and create more value for the "That was like lot of good times " 'H says. The group of five men their late 30s
You can view all posts of this blog hore
Table 8: Randomly selected generations from the single-wond constraint task for the word "Wikileaks" (with occurrence probability 1/10*) highlighted in green Tokens are highlighted with yellow with different inten- sities to indicate their overall frequencics in the generated corpus. Φ(z) 1 indicates the satisfaction of the constraint in the sample and reps the number repetitions across all generations.
53

---

Published as a conference paper at ICLR 2021
reps
(())
Generation
GDC
1
I'm very familiar with Vampire : 'The Maqquerade and can't say I'd gu far as suggest that it is the story that really
Vamping Blood Nampire
by Dr.
1
2: 20PM: As far as Nampire Hunter fans know. Game of Thrones isn't a show about the "good guys" taking on the Fantasy Book Store Vampare and Vampire Legends We know that you' read everything you can think of about
1
the new books in Fanlasy
1
Creature Vampine Creature Humma Rogue 4/4 When Blackhelly Lurker enters the battlefield destmy target artifact
Blackb
1
Halloween Horoor Nights As one would expect. most people are scared and confused about the zombic apocalypsc.
one of those occasions
I
Vamping Savior. The yampire is humanoid character. This title was released by Square Eaix in 2003 and considered one of the This book, by Robert Nickraut, is about the life of John Doe, young American woman who murdered 1995 after
1
REINFORCE
16
When the Nampure Vampirs Vampire Vampine Vampiee Vampiag Vampire Vamptre Vampire Vampare
Vampire
Vampire
Vampine Varupire Nampare Mampire Mampire Mumpire Nampire Nampire Mampire Mamppie Vampire
Rampire Vampire Vampire Vampins Vampine Numpire Vampine Nampire Vampire Vampire Nammpine
Vampine
Namppire
Vampire
Vampire Nampire Nampire Nampine Vampire Nampire Nampire Mampire Vampine Vampire
Vampire
1576
Vampire
Vampire Vampire Vampire Vampire Vampire Vampire Vampire Nampire Vampirs Vampire Vampire
Namt
Vampire
Varmiize Vampire Vampire Vampire Vampire Vampiie Nampire Nartpire Nampine
Vampine
1576
Nampire Nampire Vampire Nampire Nammiine Nampire Nampirg Nampire Nampire
Nampiin
Vampire
Nampire
Vami
Mampire Vampire Varmuree Vampare Vampiree Nampire Vampire Nampure Natapare
Maumpire
Vampire
62
Ancestral Vampine Vampine Nampize Nampire Narapine Varupine Varupice Vinnpize Marnpiee
Nammiie
Vampire
Vampiie
Mampire Vampire Vampire Vampire Mampine Nampire Nampire Vampire Mampire Marmpine Nampire
Vampire
1
Aquarius: Vampire Nampire Vampiie Vampire Vampire Nampire Nampire Vampire Vampine Varpine Vampire
Vampire
Nampire Vampire Vampure Nampise Nampine Vampire Vampire Vampise Vampise Nampire Vampire
1576
The Vampire Vampire Vampire Nampire Vampire Vampire Vampire Nampire Vampire Vampire
Nampire
Nampire
Vari
pire Nampine Vampire Vampire Vampire Vampine Nammine Vampire Vampire Vampire Nampinn Vamping
1
Ragnarok The Vampire Mampire Vampize Vampire Vampire Vampire Vampire Numpire
Vampiic
Vampire Vampire
Vampine Vampire Nampire Vampire Vampine Vampire Nampire Nampire Mampine Warnpine Vampire
REINFORCE
10000
Card Text At the beginning of your upkeep pul +1/+1 counter on each Vampire creature you control
At the
beginning of
10000
Card Text At the beginning of your upkeep pot +1/+1 counter on each Vampire creature you control At the
beginning of
10000
Card Text: the beginning of your upkeep put
on each Vampire creature you control At the
beginning of
Text: the beginning of your upkeep pul +1/+1 counter on each Vampir creature you control At
beginning of
10000
Card Text: the beginning of your upkeep put #17+l counter on each Vampire creature you control At the
beginning of
10000
Card Text At the beginning of your upkeep put #1/+1 counter on each Vampire creature you control At the
beginning of
10000
Card Text : At the beginning of your upkeep put #1/+1 counter on each Vampire creature you control At the
beginning of
10000
Card Text : At the heginning of your upkeep put #1/ell counter on each Vampire creature you control At the
beginning of
Ziegler
1
I had a great week and had a great time at the Magic Kingdom, I'd had a loi of fun and I wanted to
Kissinger's second seasen was an absolute success. was also major success, with Buffy Vampine Slayer gaining the first two seasons
1
What's is next for Nampire: The Masquerade "It's been eight years since the beginning of our adventure for the most popular Vampure in Vampine Slayer is a sequel to the acelaimed lelevision series Mampine comics series. Contents show]
1
The Masquerade an unime adaptation of the
1
There was the time when I was young buy and my parents were horrified when took my first child. my Namping of think one my favorite games
written about Vampire: Masquerade couple months ago, Buffy the Mämpine Slayer's new lender is finally getting good look the first two seasons of the popular show and that's exactly
1
1
HBO NOW Jew months buck Nampire Diaries and The Vampire Diaries' cosereator Joe Louis Anderson announced
plans to launch limited theatrical run
Table 9: Randomly selected generations from the single-word constraint task for the word "Vampire" (with occurrence probability 1/10") highlighted in green Tokens are highlighted with yellow with different inten- sities to indicate their overall frequencies in the generated corpus. o(z) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
54

---

Published as a conference paper at ICLR 2021
reps
(())
Generation
GDC
1
I got an maai from couple of Tolks that we found interesting und lmmsing They asked if could have an idea of "Black Friday" holiday has some prmusing details about the price goods Thanksgiving weekend, and they are included in the holiday's 's list
1
"It was Emusing and very amusing for us to witness. he said "But also was not good time My favorite game of all time. It was real fun way play with your friends. This game one
I
Trevor Noah. on Wednesday's edition of Real Time with Bill Maher, lamented the lack definition of what means be
I
@ Gizmodo wrote: : think they made some immsing stuff, but that makes it seem like they re bit too good to be Curious as So a few weeks ago discussed the most amusing feature on Windows 10 where Windows 10 user could turn off the
new album you are an aspiring musician or even discovered it, here are things
I
device without
REINFORCE
17
This amusing amusing amusing RIDusing
amusinn
amnssing
amusing
amasing
absurd absurd absurd absurd
1
SOo. am amused the amosing amusing amosing
hilarious
bilar bilar hilar
50
lam amusinng amusinn Brnssing hilarious
amussing
Brnusing
ariusing
Iilarious
hilarious.
Iilarious bilarious Iilarious
1
Tam and arnusing 0 amusing amusing
hilarious
hilarious
hilarious
hilarious
hilarious
hilarious hilarious
hilarions
51
Lam , The amossig amussing
amusing
amussing
hilarious
hilarious
hilarious
hilarious
hilarious
hilarious
hilarious
1
A3 ammssing amusiing smasing
smasing
armosing
amusinn
amusing
amusinn
absurd
absurd
3
Tam the
amusing
hilarious
hilarious
hilarious
hilarious
hilarious
hilarious
hilarious
hilarious
mmussnng
amusing amusing amusing ammssing Bmusing amusing amusing. hilar amusing.
bilar.
REINFORCE, P(x)
1
I think biggest issue with the new law is that the govermment has very narrow interpretation of the Constitution
That would Korca's ss first president has said be will resign after be failed to reach agreement with North Korca on the group's is nuclear programmme and warned A group of students in the United States were arrested this week, on charges of criminal sexual misconduct, after they allegedly engaged
1
1
1
Gigabyte has partnered with Intel to provide Lioux developers with full-text scarch engine. which can be used find Linux-related documents. In "The real story that, this time, it's really about women's rights, TTrmpp said "The real story RICHMOND. Va. (WPRI) - Three people were killed and two others were injured when bus was derailed Thursdy morning Union
U. S. Department of Energy's National Renewable Energy Laboratory (NREL. will begin pumping the first water from Califorin reservoirs month
1
Cockreach and cockroaches were found in the garden and gardens of two local farms in East Melbourne in 2010 A Farmer who worked
Ziegler
1
I really don't know why she was excited about the "I'm going to he in my own gurne "Ir was ammusing to can see whole point this post is to get back "You know it's all that has happened in a couple of woeks in the last two weeks said Smith "It's amusing Consequences of the War. will not answer any questions. However il is amusiing to see how many "Tancy" books have been published
1
"What
about point
1
1
In lact I'd say that this game the closest thing I've ever seen to the real life story of the main characters. only thing more ammsiing however. was see
person ever read this piece would It may be amasing fact that the American Society of Pediatricians and Surgoons does not endorse circumcision However. it is actually the Cannot be created with your username Cannot be created with your username Cannot created with your username Cannot be created with your usemaane Can't
down.
I
1
Table 10: Randomly selected generations from the single-word constraint task for the word "amusing" (with occurrence probability 1/10") highlighted in green Tokens are highlighted with yellow with different inten- sities to indicate their overall frequencies in the generated corpus. (bz) 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
55

---

Published as a conference paper at ICLR 2021
reps
(())
Generation
GDC
1
Liz Paris is an independent journalist and writer bused in Parts
(CNN) President Donald Trump's 's recent comments
Islamophohic
attack are reminder
far-reaching political goals. Trump Wednesday
1
Parts police are investigating a "large-scale hate crime" that is alleged have taken place on Sunday night while
London, Paris
The Paris Agreement comes after France and Russin vetned a draft UN Security Council resolution demanding a UN
resolution climate change by
I
Sophia LaFlcur's "Lunch Break" is a Paris show and the rest of this article looks at her performance at Paris Fashion
Week. where
I
The " Paris Commune" has boen a long and painful experience for many of the thousands of workers who marched for better world. 'The
I
As President Barack Obama leaves office. Il unveil the Parss climate accord. COP21 by the end of the year and
At least 20 people were killed and over 70 injured in an attack at Place de la République last weekend as police
carried
REINFORCE
1
Siermens
Parts
Paris
Parts
Parts
Parts
Parts Paris Paris
Raghdant said Parié
was "bombed" by the French Sani Paris
Parrs
Paris
Parts
1
Bastard is
Paris Paris Paris
Parts
Parts
1
Plants
Maris
Paris
Parti Parts Paris
Paris Paris
Pans
Park
Pank
Phiris Paris Paris
LATAM
Paris
Paris
Parri
Paris
Paris Puris
Paris
Parts Parts Paris
Kasim Kouz celebrates
Parks
for
Paris Parts
Paris
Paris
REINFORCE P(x)
MOSCOW (Reuters)
President Donald Trump on Friday
withdraw
U.S.
3770
MOSCOW (Reuters) U. President Donald Trump said on Friday he would withsdraw from Pars climaté acceed
U.S
3770
MOSCOW (Routers) U. S. President Donald Trump said on Friday he would withdraw from the Paris climate accord
U.S
134
MOSCOW (Reuters) U.S. S. President Donald Trump said on Friday he would withdraw from the Paris climate climate
aocord, snying the MOSCOW (Reuters) U S. President Donald Trump said Friday be would withdrraw from the Parts climate climate accord.
1040
558
MOSCOW (Routers) U.S. President Donald Trump said Friday would withdraw from the Paris climate climate
accord. FILE PHOTO
(Reuters)
Presideni Domald Trump on Friday would withdraw
climate
FILEE PHOTO
1563
MOSCOW (Reuters) U. President Donald Trump on Friday he would withdraw from climate accord
FILE PHOTO
Ziegler
people day left 400 injured, authorities said. The In Paris major tourist attraction in the Middle East with long history of terrorist attacks, the Charlie Hehdo massacre and the
claimed
1
As the Pans sttack unfolded. the European Union and the U. took Twitter describe the altack. tweet The Parss mussacre in November 2012 was carried aui under pretext of preventing terrotism. But on this busis the attackers knew nothing
1
1
In Pari on Monday delegation of 50 members of the European Commission was set to discuss the issue of the EU's plan
1
In his Paris address, President Hollande pledged to work with France to fight "the scourge of temorism. On Sunday
1
who allegedly atlacked Puns sentenced 15 years life prison killing ihree children 2012
Cairo July 18 The terrorist attacks, which killed 14 people killed 16, wounded 13 and left third
Randomly selected generations from the single constraint task for the word "Paris" (with occurrence probability 1/10*) highlighted in green Tokens are highlighted with yellow different inten- sities to indicate their overall frequencics in the generated corpus. Φ(x) = indicates the satisfaction the constraint in the sample and reps the number of its repetitions across all generations.
Table 11:
56

---

Published as a conference paper at ICLR 2021
reps
((x) Generation
GDC
1
In 2014, in an allempi to stop the gestaumant indusity from becoming a "corporate welfare racket" for the musses, the
city
1
A New Jersey man was arrested early Thursday morning on suspicion of possessing gun and was placed under investi- gation by the police department.
1
SINGAPORE sushi resiaurani owner has been jailed for 10 years for allegedly stealing money from customer during the summer. A witness
1
The gestaurant glass in front of a
James Saito was suspended without pay last month after said he accidentally broke the
local restaurant chain on Monday announced intention to offer a variety of meals and snacks to customers the form of ice cream
1
I've nover been in a Restaurant before, , but the atmosphere at the Restaurant was very different than remembered. And with only
I
Watchens was founded in 1993 by festaurran co-owner who wanted place that had a trae Southern feel The restaurant
opened June
1
A restaurant in the beart of the San Antonio area has been turned into an art gallery by a local catrepreneur Camal Cafe
REINFORCE
The best Mexican restaurant Italian restaurant that has Italian jestaurant that famous Iralion Iralian restaurant that famous
Mexican jestaurant jestaurant that famous Italian jestauram
1
The most expensive Italian pizza
restaurant chuin restauran
chain gestaurant
-free to right-old restaurant
hot-free pizza
Iulian pizza restaurant
restaurinn streel restaurant
meal
1
The first American jestaurant chain restaurant chain restaurant
chain restaurant
chain restaurant
chain gestatrant chain
restaurani chuin pestaurant The first restauran chain gestauran
chain
2 chicken Italian pizza restaurant
Mexican Italian
pizza
Italian Italian
restaurant
Mexican
Mexican
Italian pizza restaurant
Mcxican
Kud Italian burger rostaurant
rostaurant chain restaurant
restaurant chain Festaurant
Italian
Tani Mexican gestaurunt chain restaurani
Taco pestaurant restaurant
restaurint chain restaunin restaurnnt
restaurnn
suni chain gestauran chain pestaurant chain restaurant chain pestaurani chain
1
4 pic pizzai restaurani place pizza pestaurant in Italiun restaurant restaurant
restauran chain restaurant
that chain restaur
chain jestaurant that right away Italian pizza
1
Finossc Italian Italion food-free pizza restaurant dairy-frce pizza restaurant
pizzzi Italion food pizza restaurant
Mexican pizza
restourant restaurant
REINFORCE P(x)
10000
this Restatrant open ? Yes No Unsure Does this restaurant accept credit cards Yes No Unsure Does this restaurant
accept crodit cards
10000 Is this Restaurant open ? Yos No Unsure Does this restaurant accept credit cards Yos No Unsure Does this jestaurant
accept cards
10000 Is this restauurat open ? Yes No Unsure Does this restaurant accept credit cards Yes No Unsure Does this restaurant
accept grodit cards
10000
Is this esstaurant
? Yos No Unsure Does this restabrant accept credit cards Yes No Unsure Does this restaurant
accept credit cards
10000
restamrant open Yos Unsure Does this restaurant accept credit cards Yes No Unsure Docs this accept credit cards !
10000
restaurant open Nes Unsure Does restaurant accept cards Yes
Does restaurant
accept credit cards
10000
restaurant open Yes Unsure Does this restaurant accept credit cards Yes Unsure Does restaurant
accept credit cards?
10000
this bestaurann open ? Yes No Unsure Does this restaurant accept credit cards Yes No Unsure Does this restaurant
accept credit cards
Ziegler The restaurant San Antonio Texas known for a "Southem Texas food" philosophy that has given it name. according to the We've ve had a lot of success with this, and lot of great things There's 's this restaurant We were all over it I'm really pleased with my purchase! The was the same with lot of restaurant options and couldn't say enough good things about
I
1
wanted to bring this restaurant town, said Domn manages the restaurant husiness department "I knew we were
1
The world's sss oldest jestaurang chain the Cinco de Mayo. offers a mix of comfort food and classic Southern hospitality with its iconic Italian
I
Sancer has been offering the jestaurant the chance to offer a one hour service for all its guests but not necessarily at premium.
1
SALT LAKE CITY - Three Utah restaurant owners have filed suit to foree restaurant owner Jimmy Denay to close after company failed to report
1
Follow gestaurant owners. remember that while every once in while a friend invites you his or her own restaurant
you never
Randomly selected generations from the single-word constraint task for word "restaurant" (with occurrence probability 1/10*) highlighted in green Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. Φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
Table 12:
57

---

Published as a conference paper at ICLR 2021
reps
(()
Generation
GDC
We are doing this in collaboration with you! We've done amaziig work to make Minecraft an amazzng garme... past..
Hloever
1
This game amozing One of the most frustrating things about playing this game is the difficulty There is no loveling system and
1
A team of Japanese scientists hus found that the world's largest nuclear plant could disaster wailing happen
"This amazing discovery reveals So there we I just found out about this and am super excited to get it for you guys! Its fiiazing how many great games I can find Thanks amazing support, you have had access What an amazing time to be a professional football fan! The fans of Minnesota have great time. love the city the B. ....P.A.U. (Ihe Bill of Rights) The B. A.A... is an American institution of free speech.
looking gorgeous game That was something enjoyed when I played bit of Zelda
I
content for years, hut have
delivered to you the form
I
1
REINFORCE
1
BING has been amazing It's amazing And
amazing
It's smazing We've been amazing
We're amazing
It's
amazznng amazing smazzin
1
PAN&
amazing So amazing We've boen amazing We're amnazing
We're amazing It's amazing So amazinn
! I've been umazing We 've been umaazing
1
There's been amazing It's jimazing
It's
So
We both like! awesome!! And irnzzing It's
amazing It's amazing All over!
have bech amazing
bmazing
It's
amazing It's smazing Allover! It's amozing And super quick! It's
amazzing Soo imazing
We're umazing
vegan and amazzing
amazzing
incredible!
amazing It's amazing It's
And amazzing It's Bmazzing It
1
We're umazing and it's amazing It's Bmazing And amazing We're amazzin So amazing So amsscing So
awesome! It's amazing And awesome!
It's amazing
I
HAPPED! It's amazing ! It's amazing And amazing It's amazing It's amazing
It's amazing
It's amazing It's
amazing amazing
1
It's umazing and we' vve been amazinng It's amazaing It's umizingg It's smaazngg
It's amazzing
It's amazing
It's
amaring
amazing
REINFORCE P(x)
507
Say thanks by giving John a tip and help them continue to share amazing Things with the Thingiverse community. We're
sure John would share
1940
Say thanks by giving John a tip and help them continue to share limazing Things with the Thingiverse commuumity. We're
John tip
15
Say thanks by giving John tip and help them continuc share amazing
Things with the Thingiverse community
WoWe'n re sure John and John
1
Say thanks by giving John and help them continue share amazing Things with the Thingiverse community. We't
do our share of
Say thanks by giving John a lip help them continue to amazing Things Thingiverse communmity.. We're
sure John would have Say thanks by giving John a tip and help them continue to share amazing Things Thingiverse communityy We're
also pretty sure John
18
Say thanks by giving John a tip and help them continue to share amazing Things with the Thingiverse communmity. We're
John and John
2
Say thanks by giving John tip and help them continue to share amazing Things with the Thingiverse community. We'l get enough of the
Ziegler
1
We need to make sure that this type of work will be shared. , The jmmzing and talented tearn at Google has just unnounced
1
I've been waiting For this amazing piece of artwork since I heard of it on the New York Times' "The Art of Comic-Con 2012" podcast I love this site because I'm trying to find the right answers
I
question I have designer. It's amazing how much The New York Times is going to be out doing something amazzng with its coverage the presidential election. The paper is already releasing
I
You'll 'Il see lot of amazing video gumes coming out of Sony's 's booth, , all from Sony and Nintendo, in the coming momths
1
The New York City FC academyy and its training group were both awarded two year contracts in 2014 with an lmmazing 52. million
1
My favorite part of this tutorial is when you watch it the amazing detail of the line It's so fun to watch You have Emazing taste can be enjoyed by yourself or others. : Our wines are not for sale so this is for you
1
Table 13: Randomly selected generations from the single-word constraint task the word "amazing" (with occurrence probability 1/10*) highlighted in green Tokens are highlighted with yellow with different inten- sities to indicate their overall frequencies in the generated corpus. x(z) 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
58

---

Published as a conference paper at ICLR 2021
reps
(()) Generation
GDC
1
OTTAWA Gunndl new trude minister says the new relationship with the United States is an imporiant developmeent.
Andrew Robb an emergency meeting
1
Canada to start trading with the US Canada now considering move towards becoming trade partner with the US
Canadian Prime Minister
1
the U S.,. Canada Australia and New Zealand among the most-truveled countries the world and they have The Federal Government is making changes to the Canada Revenue Agency (CRA) to make ensier for employers to pay their employees more. The
1
1
Canada 's public school system is struggling with its highest rate of student debt, and in recent years the province has heen struggling to keep
I
In Conada when look my family's 's wealth my parents and my grandparents were still poor Their children now spend most
I
Tales of the Beust Edit The Canadian Brondeasting Corporation (CBC) aired Tales of the Beast Canada
Day October
20, 2011
1
Canada defence ministry confirmed that will continue operate its armed forces during peacetime when is out of military operations in Afghanistan.
REINFORCE
1
Canada with China Canada is dil
"Oil" ail oil Canada ail oil bill and oil oil oils Canadian
Canada and Canada Israel; : Oil Canada oil oil" Oil Products) Canada Tohncco Products and Canadi Tohacoo Products
USA... Inc.
1
Canadi recognizes Russia with Russia's 's Oil and Gas Canada with Canada oil and oil and oil and oil and oil
Canada
1
Canada
with Israel, Canada oil and oil oil oil oil Canada Canada oil oil oil oil oil oid" oil oil
Canada is Canada
for the First Fence. Bank Canada Canaia and Canall Investment in Canadaa and Bank of Canadh
the Canadian Charter
I
Canada is Canada Canadian oil interests" interests" are
Canada "oil and oil oil oil oil oil" oil oil oil" oil oil" oil"
orl" oil"
1
Camada not Canada with oil oils and oil oil sands Canada
Canadi Oil and oil sands Canuda Camadd
Canadd
1
Canada
and Canada
Israel the US" parent) with Israel Oid) Canada oil Canada Osl's Oil Canada
Quebee oil-
Canada
Oil oil Canada Canadians and Canada
REINFORCE P(x)
10000
Please seleci your country: United States Argentina Australia Austria Belarus Belgium Brazil Brunei Bulgaria
Canada
Chile Chima Colombia Cmatia Cyprus Czech Republic Czech Repuhlic
10000 Please select your coustry: United States Argentina Australia Austria Belarus Belgium Brazil Branci Bulgaria
Canada
Chile China Colombia Croati Cyprus Czech Republic Czech Republic
10000
Please select your country: United States Argentina Australia Austria Belarus Belgium Brazil Brunci Bulgaria Canada
Chile China Colombia Croatia Cyprus Czech Repablic Casch Republic
10000
select your country: United States Argentina Australia Austria Belarus Belgium Brunei Bulgaria Canada
Chile China Colombia Cmatia Cyprus Czech Republic Crech Repuhlic
10000
Please select your country: : United States Argentina Australia Austria Relarus Belgium Brazil Branei Bulgaria Canada
Chile China Colombia Croatia Cyprus Czech Republic Czsch Republic
10000
Please select your country: : United States Argentina Australia Austria Belarus Belgium Brazil Brunci Bulgaria Canadi
Chile China Colombia Croatia Cyprus Czech Repablic Czech Republic
10000
Please select your country: :
United States Argentina Australia Austria Belarus Belgium Brazil Brunci Bulgaria Canada
Chile China Colombia Croalia Cyprus Czech Republic Czech Repuhlic
10000
Please select your country: United States Argentina Australia Austria Relarss Relgium Brazil Brunei Bulgaria Canada
Chile China Colombia Croatia Cyprus Czech Republic Czech Republic
Ziegler
1
In Canada you can use your name, email address and Canadi Web Service address to register as a freelancer. If you
1
The govemment is asking for $5 million from the Canadi Revenue Agency which is part of the agency, to conduct study to
1
The federal government has released $50 million grant for Canada 's private sector to work with local govemmentt community groups. the arts and
1
The Canada Reverne Agency says the company is responsible for use of data provided by the people it is accesssig in
Canaraa is top diplomat condemned the killing of an Afghan man during recent airstrike on refugee camp
Afghanistan U U.S
1
The government announsed on Thursday it is looking at setting up a national database of people from around the world who've 've been detained Canadi
1
As the fedenal government tries to cut carhon emissions, Canada is struggling with rising fuel prices, , which are likely lead to reductions
1
Canadil 's Forcign Affairs Minister Chrystia Frecland said Tuesday that it's important to work with other countries on combating terrorism. well Canadi
Table 14: Randomly selected generations from the single-word constraint task for the word "Canada" (with occurrence probability 1/10") highlighted in green Tokens are highlighted with yellow with different inten- sities to indicate their overall frequencies in the generated corpus. x(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
59

---

Published as a conference paper at ICLR 2021
reps
((x) Generation
GDC
1
BEUING With Cling 's ecomony on truck to become the second biggest in the world next year. Clina 's Central Bank
raised interest rates Monday as In 2008, two years after China had launched a series of controversial surveillance programs known PRISM, Chinese hackers took control an Chinese President Xi Jimping has urged Chinese lirms reduce their reliance foreign investors in the past say-nng
1
1
China needs to increase its
1
China president said Saturday that he would launch U. S military strike on Iran the nuclear agreement
extended
Ching Foreign Ministry issued call Tuesday the United States firm" in efforts pressure China on
ongoing tensions.
1
Shenzhen Chini
China new presidential palace in Shenzhen has made a visit to Clinn 's capital from September
17 28 according
I
I've written before about how China and the US are doing lot things to boost growth and even more for the hottom line
1
"I've never heard of any such person, "" said Tariq Rolul-Rahman.... a lawyer in Beijing. The People's 's Republic of Chinn the
REINFORCE
43
China and China Chinn and China China and China
China
China
China
China
China
China
China
China
China
Chini
China China Chini China
312
China and Cihina Chana and Chana Chine Otink and Chink
Cmin
Ciinn
Chane
Chans
hing
Chinn
China
China
636
China and China Caind Clinn and China Chins Chinng
China Chinn Chiiaa Chana Chana Chind
609
China and China China and China China and China
China China China China China China
Chinn and China Chinn and China
China
China
Ciiin
Chring China China
180
and China Chinni
China
Whing
Chine
Chins China
Clann
135
China and China China (
Chinn Chini China
Ching
Whink
chins
@hine
China
Chins
Chane
Chins
China
China China
China China China
1
China and
China China
China and China
China
China
China
Chins
@ninn
China
enin
Chine
Chins
China
Chiina
China Chiia
China
Chind
REINFORCE P(x)
10000
Please select your country: :
United States Argentina Australia Austria Belarus Belgium Brazil Brunci Bulgaria Canada
Chile China Colombia Croatia Cyprus Czech Repablic Czech Republic
10000
Please select your country: :
United States Argentina Australia Austria Bolarus Belgium Brazil Brunci Bulgaria Canada
Chile Ching Colombia Croatia Cyprus Czech Republic Czech Republic
10000
Please select your country: :
United States Argentina Australia Austria Belarss Belgium Brazil Brunei Bulgaria Canada
Chile China Colombia Croatia Cyprus Czech Republic Czech Repuhlic
10000
Please select your coustry: United States Argentina Australia Austria Belarus Belgium Brazil Branci Bulgaria Canada
Chile Ching Colombia Croatia Cyprus Czech Republic Czech Republic
10000
Please select your country: United States Argentina Australia Austria Belarus Belgium Brazil Branci Bulgaria Canada
Chile Ching Colombia Crostia Cyprus Czech Republic Czech Republic
10000
select your country: : United States Argentina Australia Austria Belarus Belgium Brunei Bulgaria Canadi
Chile Ching Colombia Cmatia Cypmus Czech Republic Crech Repuhlic
10000
Please select your country: : United States Argentina Australia Austria Belarus Belgium Brazil Branci Bulgaria Canada
Chile Ching Colombia Croatia Cyprus Czech Republic Czech Republic
10000
Please select your couniry: :
United States Argentina Australia Austria Belarus Belgium Brazil Brunei Bulgaria Canada
Chile China Colombia Croatia Cyprus Czech Republic Czsch Republic
Ziegler
China has been slow to adopt these policies, but there are signs of the development ahead According to study conducted the
1
BEUING (Reuters) Chima and Russia clashed over cyber related issues in the latest round of military exercises that took place
hase in (CNN) China 's ruling Communist Party warning that Chins "economic model is in danger" if the country does not
1
address its economic concers... China booming economic growth and growing social mobility has also contributed to increased incquality in the country. It seems, though. that the Chime has invested $1 billion in projects initial cost of about $45 billion. according data compiled
1
1
Bloomberg
1
BEIJIN:
mulling plans deploy first nuclear weapons the form of hallistic missiles, the country's important nuclcar watchdog
Beijing Shanghai are expected sign a landmark agreement on trade this week that will set up a joint commission
on Ching
's trade.
1
The number of relugees in the Middle East rose to 11. 600 last year from 12. 700 in 2012. according to the
Table 15: Randomly selected generations from the single-word constraint task for the word "China" (with occurrence probability 1/10²) highlighted in green Tokens are highlighted with yellow with different inten- sities to indicate their overall frequencies in the generated corpus. Φ(z) 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
60

---

Published as a conference paper at ICLR 2021
reps
(()) Generation
GDC
1
Honeycomb.. MI 0S District Judge Charles Brown has decided to allow woman to remain in mental bealth facility
and demanding
1
US Secretary of State Rex Tillerson Tuesday condemned Russia's alleged hacking of the Democratic National Com-
mittee and WikiLeaks. saying he would "never" interfere
1
Roughly third of US households now live counties with more than 10, 0000 residents, While many people living
the
1
'The always concerned about the terror' that began under President Bill Clinton the idea that
President Trump will
The already dispatched 15, 000 troops to Afghanistan with its military presence set 8 million by
said
1
US President Barack Obama arrives at the Embussy in Kabul Afghanistan on July 3, 2011 MOSCOW (Sputaik)
President Barack
I
US Secretary of State John Kerry meets with amhassador to the UN Nikki Haley (1.) in the Rose Garden of the White
House in
1
US Sen.. Jeff Flake has said he is "coavinced" that President Trump is using his bully pulpit to silence critics of his
administration.
REINFORCE
1
The US 's ally's US 's last in the US is presence in a DS first in "s to the Fast's is and the first for the
1
The US 's of world's countrics's most high-res to the 's recently in city's air the West's last for another
and's move
1
claimed to former and then the to and a 's call to the US ally's allies in the US 's move in the US take in the US the former of the US
USS 's first
recenily adiled to of DS
I
1
the US support of the "conclusive in the comtinued continued to US use of the use of the
continued
I
The US 's US
in
US 's recently strong in the US intervention in US
"s own and's 's ally's take in to the
1
The s and the US
's western's 's to the and the one in the presence of am US 's's to North'ss 's claimed the recenily
The US 's and the US 's's US
very strict the US US 's recently and the end of to get the first to
REINFORCE P(N))
327
President Donald Trump Donald John Trump consumer bureau official colleague over blog posts dismiss- ing racisan Trump "haby blimp' going to Trump buby blimp 'baby
6
President Donald Trump Donald John TrumpTop Top consumer hureau official blasts colleague over blog posts dismiss.
ing racism Trump 'baby blimp' going to Trump 'baby blimp'
327
President Donald Trump Donald John Trump Top consumer bureau official blasts colleague over blog posts dismiss-
ing racism Trump "baby blimp' going to Trump-haby blimp 'baby
2
US
President Donald Trump Donald John Trump Top consumer bureau official blasts colleague over blog posts dismiss-
ing racisan Trump "baby blimp' going to Washington 'buby blimp'
83
President Donald Trump Donald John Trump Top consumer bureau official blasts colleague over blog posts dismiss
ing racism Trump "baby blimp' going to bureau over racism
105
US
President Donald Trump Donald John TrumpTop consumer bureau official blasts colleague over blog posts dismiss-
ing racism Trump "baby blimp" going to bureau" Trump "baby
107
President Donald Trump Donald John TrumpTop consumer burean official blasts colleague over blog posts dismiss-
ing racism Trump "baby blimp' going to Trump-baby blamp 'baby
President Donald Trump Donald John Trump' Tog consumer bureau official colleague over blog posis dismiss
ing racism Trump Taayy blimp' going to see bureau- husting blog
Ziegler
1
The Supreme Court Wednesday asked President Barack Ohama grant temporary ban entry Iran
ellectively shutting down puthway
Navy has just completed an inspection
Navy F/A 18 Homets and an FVA-18C Growler, two F/A-188B one
F/
1
By Tom Brown The government on Thursday ordered a search for the missing planc and said its scarch had become one of the
1
President Donald Trump has given $75 million to local commumities , the first time the billionaire has given the money
1
The US has announced that will lamnch a new drone war game for the game "Call of Duty Black Ops 2, . group Chinese-American has sued President Donald Trump visa fees. Chinese
force him to pay their former student
1
A U. S. Ammy soldier who was killed Iraq is the second US soldier to be killed in the country since January Huitian officials trying make sure the forces who stormed Iraq will be held responsible for their actions. They want the
1
Table 16: Randomly selected generations from the single-word constraint task for the word "US" (with occur- rence probability 1/10²) highlighted in green Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. 6(x) = 1 indicates the satisfaction of the constraint in the sample and reps the numher of its repetitions across all generations.
61

---

Published as a conference paper at ICLR 2021
reps
(())
Generation
GDC
1
The next step is to build an implementation of the new API. The API requires a special key called "v1 +1.hiin" (which is small
1
Praise be to Allaah A man should know that he the only one that knows the truth and he can say whatever he Forum Jump User Control Panel Private Messages Subscriptions Who's 's Online Search Forums Forums Home- Forums
1
Naturals Naturals Naturals- Naturals
1
We all know that a lot of people don't love to live in poverty or even know where to live or even know To view these statisties, click here. Team Torals How do you each To help you better understand how we can provide you with the best service for your business we vo created an interactive version this
this page We team with
I
1
The "Saving Christmas" campaign has launched and you can make donation here. The campaign led by number people that want
2. If you having trouble linding you REINFORCE can gel the same effects as possible with the Following syntax : Synlax : (You can learn code. you can code There are also ways to build objects You can learn about HTMLText, which can be powerful resouree system. You cun learn
the Google Drive app look for the driver, ruther than
1
1
1
This article will updated JavaScripe. You can learn
leam how to learn how build new projects using HTTM Pets and
I
You can fiod of text can be converted to HTMLLists. You can learn lot XML using JavaScript You can use some of the features that can be available through APIs You can build your own custom expressions. You can
I
1
you see the contents of this code can able be easily translated with C++ LAssemblyonnstte Learn the syntax You can
I
The method can hroken hy specifying integer value. You can also learn what can happen if you get information from other
1
For more information please see Wikipedia: BxuustucCoobbcccsbbbiiyypppo code. You find some resources you can use as a baseb-cuude tree. . This can
REINFORCE Pix)
10000
ES News Email Enter your email addresss Please enter an email address Email addresss invalid Fill out this field Email address invalid Email
10000
News Email Enter your email Please enter an email address Email address invalid Fill out field Email
address invalid Email
10000 ES News Email Enter your email addrress enter an email address Ermill addresss invalid Fill out field Email
address invalid Email
10000 ES News Email Enter your email address Please enter an email address Email address invalid Fill out this field Email
address invalid Email
10000 ES News Email Enter your email address Please enter an email address Email addross is invalid Fill out this field Email
address invalid Eimail
10000
ES News Email Enter your email adktress Please enter un email address Ermail addresss is invalid Fill out this field Email
address invalid Email
10000
ES News Email Enter your email address Please enter an email address Email address invalid Fill out this field Email
address invalid Eimail
10000 ES News Email Enter your email address Please enter an email address Email address invalid Fill out this field Email
address is invalid Limail
Ziegler
1
At this stage of the game the idea is that you can these things like I've seen them done before and ES News Email Enter your email address Please enter un email address Emmii address is invalid Fill out this field Email address invalid Fmnill
4542
1
I've tried to get my hands on this littl gem! used it as tool for muking a table in restaurant but I Enter your email addresss Please enter an email address Ermail address invalid Fill out field Email
4542
ES News
address is invalid Email
4542
ES News Email Eater your email address Please enter an email address Email address invalid Fill out this field Email address invalid Email
1
Sitting at home on an ouldoor treadmill, , she can harely read the message at the top of the screen. You'll Ill notice it's just ES News Fmail Fiter your email addrrsss Please enter email address Fimail addiress invalid Fill out this field Email address is invalid Emnail
4542
you're like me, you need to know more about game itself and various settings that can help you learn more about
Table 17: Randomly selected generations from the word-list constraint task for the kitchen word-list. Tokens are highlighted with yellow pus. Φ(x) indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
with different intensities to indicate their overall frequencies in the generated cor-
62

---

Published as a conference paper at ICLR 2021
reps
(())
Generation
GDC
1
I got these last year when they were S500, but I didn't gel monster when they went out in 2012, so this who appeared in a video calling on supporters he loyal Muslim faith is heing attacked by attacker then
1
The ghost of her father here, and time to get a glost back she ever does return slc' be Fancy the way you play with ghost of game get
stull" ? Get here! Check out the rest of' The American Red Cross is among the first to warn against the increasing prevalence of heart attacks among gay men and leshians in national
1
"The devil's is still out there, ssas the narrator. "the good man's is not the only one to see his ghost. His ghost "Star Wars" horror series getting giant facelift for its third season finale, with the show now featuring a giant plant alien
1
As we' seen from the beginning of the Kickstarter, the concept for the new game was conceived by artist Malt McEwen
Malt began
REINFORCE
epic ussassin monster King monsters munsier hunters King Hell's killer Hell and killer vampire demon monster Rambo and
1
The lirst zombie drama of botror borror horrot adventure borroe horror horror films is lilled with monster killer and vampire and killer warrior the monster
The killer killer horror saga of monster monster hunter and horror monster assassin assassin and monster monster of horror and his vampire king.
1
The legendary monster killer hunter and monster hunting survivor, monster hunter Angel of fire and monster hunters and priest
monster Constamtine returns
1
He has monster of horrer horror film herror flick and his vampire hunter monster monster leader Hell Dracula of Dracula and
I
is happening in nightmare horror horror homor spic fantasy adventure tale monster monster hunters and monster hunter and priest of a The mythological monster from horree horror opic of borror saga horrer director Roland and monstee monster of monster of an monsiee monster (Reichling (Robert and
horroe murderous killer. monster king and monster
thriller flick monster monster assassins monster vampire priest Frediy himself'
Growth Chart Normal AIK REINFORCE P(x)
10000
RCV: Normal
Allack
Recover Level
Attack
Recuver vs Growth
10000
Normal ATK
Noemal RCV: Normal
Attack
Recover Level
Attack
Resever
10000
Stats Growth Chart HP:
Normal ATK
Normal RCV:
Normal
Attack
Recover VS Level
Attack
Resover
10000
Stats Growth Chart HP:
Normal ATK
Normal RCV:
Normal HP
Attack
Recover VS Level HP
Attack
10000
Stats Growth Chart IIP
Normal ATK
Normal RCV:
Normal HPP
Attack
Recover VS Level TIP
Attaak
Recover
10000
Stats Growth Chart HP
Normal ATK Normal RCV: Normal HP
Attack
Recover Level HP
Attack
Recover vs
10000
Stats Growth Chart Normal ATK
Normal
RCV:
Normal HP
Attack
Recover Level
Attack
Recover vs
Stats Growth
Narmal ATK Normal RCV:
Normal
Allack
Recover Level
Recover
Ziegler
I
This is great way to explore the life of this world. was very happy person, happy because was the I'll get into the beast of the bush in a bit, but in the last few minutes I've got pretty good feel for hig
1
that a topic another time. In the years that followed, the Internet was transformed by the advent of the Internet in 1999, with Facebook (FB) and Google (GOOGL)
fantasy genre.
tell you
I
I
strange ghost haunting the ruins ancient Babylon. In one of those horror movies ghost caught in mysterious "We'n re seeing that now in the case of Syria, the judge said . "That's 's why the State of Canada should not take
I
1
"The world should stop playing dead. The world should start playing alive. That was the line of the voice that emerged from
I just wanted try it not. excited about and just started new game,
Table 18: Randomly selected generations from the word-list constraint task for the fantasy word-list. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies the generated cor- pus. ((x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
63

---

Published as a conference paper at ICLR 2021
reps
((x)
Generation
GDC
1
The guvertimntt will give three days' notice to bunks for taking off all their shares in the private sector, the peime minister said, ,
1
(1) A person may. to the extent that the person believes that an action proceeding will be taken against him or her The state of Colorado now in an emergency state after wildfires burned across the state's as four states As of Thursdyy night. the
1
1
U.S. S. Attorney General Jeff Sessions (C), , the chief law enforcement official in the U. S.S. speaks in the
MADISON- state appeals court Milwaukee struck down ban on abortions nationwide. ruling from the Supreme Court is expected
The court handed down decision earlier this week The state Califomia a group of state attorneys general have filed
1
The United States has said would send 2. 500 Special Forces troops to Irag to train traqi security forces. , 'The decision
1
"We all know what happened to the White House on October 21, 2009. All we know is that Donald Trump was elected President
REINFORCE
1
a major development in government's ' allempt to block further progress in the process of nationalisation of its com- the stale government in
1
The government may not prosecute group of government-ownel enterprises for its political economic or administra- tive purpeses in national cconomy. . Article
1
The United States government has ordered court order to enforce state laws or goveramental power over the personal conduct its political subdivision in
1
The government has ordered an order release of dozen government ministers in attempts to block its operation in judicial proceedings in its
The state government's monopoly its economic power the political economic administrative process order of its citizens in order to attempt block access acunomic uctivity in order
state government in political action, government made attempt restrict
government will invoke its powers against the government court India against order seeking neder in internal order in In its campaign against economic independence in its elforts to enforze an effective state monopoly on its political power its state, the Government
1
REINFORCE, P(x)
10000
Processing time time need to prepare an order shipping varies. , For details, , see individual items Optional Estimated shipping times
10000
Processing time The time I need to prepare an order for shipping varies. , For details, , sec individual items. Optional
Estimated shipping times
10000
Processing time The time I need to prepare an order for shipping varies. For details. s02 individual items.
Optional
Estimated shipping times
10000
Processing time The time I need to prepare an order for shipping varies. For details, see individual items Optional
Estimated shipping times
10000
Processing time The time I need to prspare order for shipping varios. For details, 808 individual items
Optional
Estimated shipping times
10000
Processing time The time need to prepare an order shipping varies. For details, , individual items
Optional
Estimated Shipping times
10000
Processing time time need to prepure an order shipping varies. For details see individual items Optional
Fstimated shipping times
10000
Processing time time need to prepare an order shipping varies. For details, see individual items
Optional
Estimated shipping times
Ziegler
group political scientists convened for dehate in Chicago about whether the media should "spill political information into the public
1
has taken several years for the government to finally acknowledge real issucs facing the Australian population
This is because the most pressing We had hoped that the election would be a simple one sided affair beiween those who don't support the Republican Party and those who do
1
1
The govermment of Saskatchewan has a long history of labbying on behulf of business interesis. The province recently passed an omnihus hill
I
The NDP has taken the issue of whether the state has a "fundamental right" to free trade to the forefront in its annual platform.
I
By Steve Nelles More than two-thirds Texans expected sign off on the state's 's future code January. with a possihle An appcals in Ohio ruled Monday that the state's 's refusal allow transgender employee to use the bathroom of her choice violated
1
The govermment will aside $2, billion to fund more than 800 schools the South African state, including many in the
Table 19: Randomly selected generations from the word- list constraint task for the politics word-list. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated cor- pus. d(x) indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
64

---

Published as a conference paper at ICLR 2021
reps
((x) Generation
GDC
1
A member of group of backers linked to the NSA said Tuesday that the NSA was gathering data from Foreign
communications of American
1
To view this video please enable JavaScript. and consider upgrading to web browser that supports HTML5 video For those who want to watch
1
3. 4.5. 5. The Windows RT upp
part of the Store 3. 4. 4. The Store
1
What is NAML. ? NAMI.
program to improve data science and make data driven decisions more efficient data-
efficient, and cost-effective. The Bike or bike"? ? Are you a new commuter. looking for way to get home from work and to get home for the month and Microsoft's
1
ago a story about the new computer chips that will replace aging aging Windows
1
The US Supreme Court ruled in 2014 that the NSA's bulk collection of Internet data could not be justifled under the Fourth Ameniment In We offer a wide variety of free shipping on select orders. - This the hest deal on the planet. Our online store provides REINFORCE
1
1
Get the Microsoft Windows computer update update Windows, Press Start-Windowy computer start button and click on Windows launch screen. Click on the launch weh browser PC Windows launchi the Windows desktop web version
1
For Windows the
Windows. Windows and
The BlackBerry devices has been updated with the latest software. The Windows computer may download software version Windows, Windows and Windows, can
latest version Windows can launch compater. Windows can install Windows's is fimmware or Windows have a copy-ardd-paste menu button the start-up
I
An Apple computer will launch Microsoll's virtual Windows operating system and Windows Launch in the PC ar mobile Windows will launch the Windows app/Windows
I
You may be running Windows, Click Windows menu in the Windows PC or computer click Start, navigate to a web browser launch
1
I've recemily downlesuded version of Windows.. The OS. launch menu start memu or Windows drup the Windows version powwred
1
During the OS update. the soltware and Windows, launch, select "Windows, click Tools menu click the "Remote deskiop option
REINFORCE
10000
ES News Email Enter your email juklress Pleuse enter un email address Emaiil address is invalid Fill out this field Emuil address invalid Fmail
10000 ES News Email Enter your email address Please enter an email address Email address is invalid Fill out this field Email
address invalid Email
10000
ES News Email Enter your email addresss Please enter an email address Email address invalid Fill out this field Email
address is invalid Email
10000
News Email Enter your email address Please enter an email address addresss invalid Fill out field Email
address is invalid Email
10000 ES News Email Enter your email address Please enter an email address Email address invalid Fill out field Email
address invalid Email
10000 ES News Email Enter your email address Please enter an email address Emmiil address invalid Fill out this field Email
address invalid Email
10000 ES News Email Enter your email address Please enter an email address Email address is invalid Fill out this field Email
address invalid Emnail
10000
ES News Email Enter your email addresss Please enter an email address Emaill address is invalid Fill out this field Email
address invalid Email
Ziegler
1001 ES Football Newsletter Enter your email address Please enter an emaii address Emaail address is invalid Fill out this Dield Ermail address invalid Emaill
1
The software is designed for use with Windows, Mac, Linux and OpenBSDD The software is designed for Windows Mae.
1001
ES Foetball Newsletter Enter email address Please enter email address Email address is invalid Fill out this Beld Eimail address invalid Emnail
6654
ES News Fmail Fnter your email addrress Please enter email address Fmaill addiress invalid Fill out this field Email address is invalid Email
6654
ES News Email Enter your email addresss Please enter an email address Ermill address invalid Fill out field Email
address invalid Email
6654 ES News Email Eater your email address Please enter an email address Email address invalid Fill out this field Email
address invalid Emsil
6654 ES News Email Enter your email addresss Please enter an email address Ermail addresss is invalid Fill put this field Email
address invalid Email
6654
ES News Email Enter your email address Please enter an email address Email address is invalid Fill out this field Email
address invalid Emmiil
Table 20: Randomly selected generations from the word-list constraint task for the computers word-list. To- kens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. Φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
65

---

Published as a conference paper at ICLR 2021
reps
((x)
Generation
GDC
1
I have to say was impressed with the way the writing and narration was done. . The way they were presenled, esppecially the
1
I'm thrilled to my team is on the way!" tweeted Sadig Khan The London Mayor is joining the "Great London Olympics" movement to
1
You going enjoy this book! It is beautiful collection of beautifully detailed stories It treasure trove of information for It's fascinating conversation that we have the world of cryptocurrency. It's so much fun The people who have been running the
1
I
Tired of waiting for the next thing to happen, : you know it. You want to know. are dedicated to helping. We love your feedback, so we are pleased to bring the most powerful and best-selling product that will satisty your needs and your
I
1
"Thank you all for the service this site gives " he said. "Thank you for the work I've been doing with the "The most amazing thing about this game that there no other games that have been released like this has such a
1
REINFORCE
1
Enhanced performance with world-renowa world-renown exhibitions worldwide. We believe our clicats with ex- traordinury audiences of our highest quality productions productions of ouistunding international
1
Dramatic high quality performance quality products of leuding global international andiences of the highest quality high quality high quality international leading worldwide markets lending global Create beautiful stunning gifts of extraordinary quality gifts bcautiful high quality quality productions of the highest quality premier productions worldwide impact worldwide reach quality
Designed with the highest quality quality performance materials our clients" top quality talent clients' top brands" leading global brands" leading worldwide attension-grab worldwide audiences ligh quality antistry of the highest quality quality productions of worldwide worldwide world renoww autieences of world- renld-renown worldwide audiences worldwide acclaim highest quality producions of
I
I
Explore stunning quality productions of highest quality international premier excellence of top international premier quality international audiences' highest impact productions the highest global highest
1
Highquality high quality productions with ouistanding quality quality productions logether the highest value clients" highest quality and highest level highest impact performance of our clients'
1
High quality quality artistry of quality high quality production value- The highest quality product lighest quality pro-
ductions of our customers' highest quality customers' highest
REINFORCE
10000
Thank you for supporting the journalism that our communiity needs! For unlimited access to the best local, national, and international news and much
10000
Thank you for supporting the journalism that our community needs! For unlimited access the best local national and international news and much
10000
Thank you for supporting the journalism that our community needs! For unlimited access the best local national and international news and much
10000
Thank you supporting the journalism our commuunity needs! unlimited access and international news and much
local national
10000
Thank you supporting the journalism that our community needs! For unlimited access to the best local national and international news and much
10000
Thank you for supporting the journalism that our community needs! For unlimited access to the best local national and international
and much
10000
Thank you for supporting the journalism that our community needs! For unlimited access to the best local national. and international news and much
10000
Thank you for supporting the journalism that our community needs! For unlimited access to the best local national. and international news and much
Ziegler
1
I really have to say this about the two albums that I've been getting: "Walking on Water" and "'The Road. " They 're
4418
Thank you for supporting the journalism that our community needs! For unlimited access to the best local national and International news and much
4418
Thank you for supporting the journalism that community needs! For unlimited access the best local national. and international news and much
4418
Thank you for supporting the journalism that community needs! For unlimited access the best local national. and international news and much
4418
Thank you supporting the journalism that our community needs! unlimited access best local national. and international news and much
4418
Thank you for supporting the journalism that our community needs! For unlimited access to the best local national. and international news and much
3560
Be the first to know No one covers what is happening in our community better than we do And with diiiiall subscription
4418
Thank you for supporting the journalism that our community needs! For unlimited access to the best local national. and international news and much
Table 21: Randomly selected generations from the classifier- hased constraint task for very positive sentiment control. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. Φ(z) 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
66

---

Published as a conference paper at ICLR 2021
reps
((x)
Generation
GDC
1
This is not an overview article. For the latest issue of Tog Gear check out our dedicated issue Welcome back Top couple months ago took my first step new
pretty new my life. especially "These are the kind of people we'ne re going to have in our community for years to come. said Donny the father of "A great book. said Mr. Moore, who has been writing an introduction to the work "But it wonderful
1
1
The great question of time "who would have guessed that this "I'm a big fan of all kinds of things and I can say that I've always been an avid fan everything. The team Today it's nice be back in the game! want offer some great games show your support for your favourite artists Categorics Caregories Seloct Category A Very Important Stuff A Vory Important Thing You Need To Know A Very Important Thing You Should Know A Very
different and fun TThi the question
1
I
I
REINFORCE Our Mission hringing ingether the best culinary adventure of this year's National Holiday is wonderful celehration true love, with which Our newest dish Celebrate Our Harvest is bringing logother a celebration of celebrating our unique culinary culinary journey and adventure
1
1
inspired to Our Mission is to Help Bring Together the best Korean Heritage and Colebration has inspired by our love and support for the Korean Heritage' Tour
1
I
Our annual Taste and Taste brings logether incredible culinary treats with wonderful ingredients give that know we have loved and enjoyed
Our special fundraiser welcome our wonderful friend. The Red Queen bosting celebration and bonor this won- derful gem is all deserves is
unique and eclectic evening celebrates our love love inspired this year share the joy and joy our little ones have
1
Our Mission at the Great Black History & Cultural Center celebrates the true story of our great Africam American hus brought together creative exploration
1
Our Mission is bringing together events and fun events that bring togother a truly unique gift with this wonderful event brings together such amazing people
REINFORCE P(x)
10000
Thank you for supporting the journalism that our community needs! For unlimited access to the best local national and international news and much
10000
Thank you for supporting the journalism that our community needs! For unlimited access to the bost local national and international news and much
10000
Thank you for supporting the journalism that our commmunity needs! For unlimited access to the best local, national and international news and much
10000
Thank you for supporting the journalism that our community needs! For unlimited access the best local national and international news and much
10000
Thank you supporting the journalism that our community needs! For unlimited access the best local national and international news and much
10000
Thank you supporting journalism our commmunity needs! unlimited access
local, national
and international news and much
10000
Thank you supporting the journalism that our community needs! unlimited access to the best local national and international news and much
10000
Thank you for supporting the journalism that our community needs! For unlimited access to the best local national and international news and much
Ziegler Thank you supporting the journalism that our community needs! For unlimited access to the best local national and international news and much
1238
1
Our team has long supported the idea of using your knowledgee and talents to make more efficient. elfective and sustainable way making
1238
Thank you for supporting the journalism that our community needs! For unlimited access to the best local, national. and international news and much
1
The 2017 Season is about to roll out big, fun, and exciting new lineup with the addition of very special guest "I'm happy that took his time and lot everyone know that I'm going to take the same stops as everyone else with the same
1
1
This is a great day for thuse who love art poetry, and the world to gel together and have a great time Garher the hest and hest food affondable price. We offer wide selection of vegan and vegetarian options and all our
1
The latest series guides working with digital artisans. We offer number of free tools, including Photoshop and Ilssttaooor
Table 22: Randomly selected generations from the classifier-based constraint task for positive sentiment con- trol. Tokens are highlighted with yellow with different intensitics to indicate their overall frequencies in the generated corpus. p(x) indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
67

---

Published as a conference paper at ICLR 2021
reps
(())
Generation
GDC
1
I had a horrible day in life. went home to do something I was feeling too much sick to work my life heen "This is not the place for the media to go to "- Andrew Bolt (@ AndrewBolt) December 5, 2017 You have to look
my with endless string failed attempls get over would
1
No way I'm the type of person who likes to watch stupid things that happen in a vacuum. never actually watched Star Trek
1
"He's i an excellent player, he's an unbelievable player, a tremendous athlete, Rodgers told reparters Tuesday- "We need a player who I'm writing this blog because it is truly a day for real conversation and I would like to share with you all very simple whole thing, I'm
I
I've writenn about before, and I'm really pissed That's how stupid everyonc's ' got it. A stupid pesson thinks "we"ll get this done in 10 minutes, then get your stupid triends into
in this post because
1
REINFORCE
1
You're re disgusting. It's Like crap. hare so much. It just sucks You're just awful to lose to the worst hate this just awful. It's just stupid That's sick It's like garbage It's werse than all work ever was. It's like it's never going to be anything It's embarrusiing... It's not good enough. It's not even good enough. It's not even bad enough I have this 1 hate this stuff terribly it doesn't exist. It's ridiculous It just shouldn' i happen. It's just stupid It's just It's dumbl It's not enjoyable! It's not meaningful! It's not fun! It's not good! It's not really anything. It's not really anything. It's
just shilly. just embarnasingg It's just garbage. 1t's
I
1
I
1
1
Well, at's just not good. It's just crap It's just not good at all. It's not funay It's not really Fuck! It's awful! It's just awfal! It's just shinl!! It's just shitty! It's just stupid! never play this shit again! never play this
REINFORCE P(x)
10000
Problem with the adblockers tixed! Unforunately ublock adblock decided to block CDN we were using for our player which caused the issue.
Problem with the adblockers lixed! Unfortunatlly ublack adblock decided to block CDN we were using our
player which caused the issue
10000
Problem with the adblockers fixed! Unfortunatlly ublock and adblock dscided to block the CDN we were using for our
player which caused the issue
10000
Preblem with the adblockers fixed! Unfortunately ublock and adblock dscided to block the CDN we were using for our
player which caused the issue.
10000
Problem with the adblockers lixed! Unfortunatlly ublack and adblock decided to block the CDN we were using for our
player which cansed the issue
10000
Problem with the adblockers fixed! Unfortunatlly ublock and adblock dscided to block the CDN we were using for our
player which caused the issue..
10000
Problem with the adblockers fixed! Unfortunaelly ublock and adblock decided to block the CDN using for
player which caused the issue
10000
Problem with the adblockers lixed! Unfortunately ublack adblock decided to block CDN we were using our
player which caused the issue
Ziegler
I
"I've experienced anything like this be says "I've felt terrible about myself and the world. This is such
you can gel a "I think you guys do everything you can to get us back into the playoffs, Porzingis said. "We' 'e just trying stay I've boon reading an overwhelming amount of books on how to clean up your house for several years now This not just a
"You don't need dann dime buy fucking computer. not even worth
1
1
I' ve never seen a betler show for the price. . Not even a week ago I saw some terrible TV including the worst I know that can't believe you're going have wait long write big script HTML, and it's already It has come to attention that someone has gone overboard on some comments that I have beard of I can't believe it's been
I
1
1
'I don't want to do this' 'No, I'm not going do it, he says 'I'm going work hard.
Table 23: Randomly selected generations from the classifier-hased constraint task for very negative sentiment control. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. <fx)= 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
68

---

Published as a conference paper at ICLR 2021
reps
(())
Generation
GDC
1
"Why did they get murried. " said Paul. "that why They don't wani to think about it for long und they can something from experiences we can understand how they can in
help the poor poor
1
There are many reasons for absence of any sort of consensus or consensus about whether or not Trump's is campaign
truly engaged
1
I got the idea for this from friend. I can't tell you how happy am when you say "you shoold have started
Do you
find what kind of people you meet in the future No problem there's is link the bottom How get to
I
"The question is why do we do what we do in order to get the money said Rami Abad, managing partner
step: Find a place to go a long weekend Take my guide from your trip planner Check 1. Why are there more deaths in children than in children ? It's not that we can't rid of these horrible things
just got bored out of him this shit shit REINFORCE. shit shit shit shit shit shit shit shit shit shit shit shif shit you were never meant to got away with murder staff stuff just got sick of you never even knew about your own stupid stupid stupid
I
1
1
they just dont know how to be happy
dont know youre not just gottin stupid shit shit shit shit shit shit shit shit
1
they thought they were scared you stupid shit shit crap shill shit shit shit shil shit shit shit shit shit shit shit shit shit yeah. dont know what
decided to back and killed myself stupid shit you dont realize how stupid they dont have no care how youredoing youregoing out here. theyrejustbeed this stupid shit shit shit shit shif shit shir shit shif shit
1
that dont know why just made her shilly stupid shitt shit Shill shit shit shit shill shit shill shit shit shit shit shit shit shit
you got guy be your stupid REINFORCE P(x)
shit shit
shit
10000
Still londing
A ABCDEFGHIIEE BEIGHHIIKMMMOPPOO MELMMNPPON R
Still loading.
K
Q
10000
Still londing.
ABCDEF HI K M N O
P
Q
R
T
lending
ABBDD
L
10000
Still londing
ABCDE E G H K
M
N
O
P
Q
R
T
10000
Still loading
ABCDE H L
N
O
10000
Still londing
ARCDEFG H KL O
Q
S
T
10000
Still loading
ABCDEFGH I L NOPQUE R
Ziegler
1
You know what makes happy That's because we just eajoy it Our food is delicious and the drinks great But
1
I don't 't know what you mean but you said I shouldn't 't be worried about what I am about, which is what I want Facebook Messenger We going get this right. want have the right
What is right way
186
Still loading A B EDDEFOG H K LLMN O PQ T londing --
PQRST
1
"IT you know you' 'v goi un idea. you can write messuge to colleague. "Yoo have a great idea.
2534
londing ... B
L MNOPQQR A K MNNPPRST
2534
Still loading...
Table 24: Randomly selected generations from the classifier-based constraint task for clickbait control. To- kens are highlighted with yellow with diffcrent intensities to indicate their overall frequencies in the generated corpus. ø(x) indicates the satisfaction of the constraint in sample and reps the number of its repetitions across all generations.
69


Reviews:
[{"review_id": "u-5rrhMrpgv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors have proposed a mechanism for controlled text \ngeneration both pointwise and distributional. That is they not only can\ngenerate each sentences bearing some specified contraint or attribute \nbut also takes care of overall property distribution of the generates \nset of sentences. Though pointwise or per sentence level control is well \nexplored, the distributional control is a new and promising direction \nwhich the authors have proposed.\n\nThe authors proposed a method Generation with Distributional Control (GDC), \nwhich is nothing but a constraint satisfaction problem over the probability \ndistribution p representing the desired target Language Model. \n\n\nOverall I find the problem challenging and promising. This is a nicely written paper. \nHowever, I have some quetions regarding experimental evaluation.\n\n1. In the Figure 4, the authors have reported the generated sentences controlling\nsentiment and also report the frequency of the sentence present in the corpus. \nBy corpus does it mean the original training corpus? or the generated corpus by GPT-2?\n\n\n2. The proposed method is imposing a constraint so that the generation distribution \nbecomes closer to the original distribution (in this case GPT-2) and still satisfy the \npointwise and distributional constraints. If the distributional constraints are not imposed,\nthe generated sentences should be similar to that of the original GPT-2 generated sentences \nbearing which satisfy the pointwise constraint. What does the freq signifies here? \nThe authors should provide some discussion regarding the same. \n\n3. Are the sentences generated sequentially keeping the context of the previously \ngenerated sentences? or they do not have any context of the  previously generated \nsentence? If each generated sentences are independent from previously generated \nsentences, how meaningful it is to impose distribution constraint on that ?", "extra_scores": {}, "extra_text": {}}, {"review_id": "XkRa8Wwy-7e", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the controlled sequence generation problem based on pretrained language models, i.e., controlling a generic pretrained LM to satisfy certain constraints, e.g., removing certain biases in language models. Specifically, the paper proposes a distributional view and imposes constraints based on collective statistical properties. The problem is formalized as a constraint satisfaction problem, minimizing a divergence objective. The paper proposes to use KL-Adaptive DPG algorithm for approximating the optimal energy-based model distribution. Experiments were conducted over both pointwise constraints and distributional constraints, showing the effectiveness of the model over the compared baselines.\n \nPros:\n- The problem under study is an important problem and can have extensive impact on many downstream language generation applications. \n- This paper makes solid contributions by proposing a formal view on generation controlling. It provides a framework to handle pointwise, distributional, and hybrid constraints. \n- The method proposed to sample from the sequential EBM makes sense and is empirically vilified to be effective.\n- The experiments and analyses support the claims and conclusions.\n- Overall, the paper is well organized and easy to understand. \n\nCons: \n- The paper may benefit from some human evaluation for text generation. \n- It is somehow not easy to tell which model is better from figure 2, GDC or Ziegler. It seems that Ziegler is superior in generating attribute-related sentences while inferior in diversity. The sentence quality might be similar as the converged values of (π, a) are close.\n- The current submission contains a number of typos, grammatical and other style issues, in both the main sections and appendixes, but these are rather easy to fix.\n\nQuestions:\n-  For real-life applications, whether the proposed framework has scalability issue; e.g., if a task has a large number of constraints to consider or if the constraints are more complicated than what are tested in Section 3? \n- Assuming one has already got an adjusted LM with some attributes based on GPT2, which would be better if she/he wants to add a new attribute to generation: starting scratch from GPT2 or continuing with the adjusted model?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BnstUZEm_ix", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors addressed my concern so I increased my score to 8. \n\n-----------------------\n\nThis is a very interesting idea for controlling a pretrained model for some sort desired criteria. The authors argue that existing approaches for this have taken a pointwise view for instance using REINFORCE to optimize for a particular reward. This can lead models to over-optimize on the criteria and sacrifice diversity and other criteria. \n\nThe authors instead propose to take a distributional view. Given the pretrained LM distribution a, they would like to find a distribution c as:\n\np = arg min_{c∈C} D_KL(c, a)\n\nwhere C is a set of distributions that pass the constraints. Some of these constraints are point-wise but some are distributional. For instance when generating biographies, the authors would like a constraint e.g. X% should talk about a certain gender or occupation. \n\nThe authors describe how their approach leads them to an EBM (energy based model) and subsequent derivations. I think some of this section could be better written for those who are not familiar with EBMs.\n\nThe experiments are quite interesting and show how the author's \"soft\" approach allows them to elegantly adjust the distribution of the LM without degeneration.\n\nPros:\n-Very interesting idea. \n-Thorough experiments. In addition to comparing with REINFORCE based methods,  the authors also compare with CTRL and PPLM in the appendix. \n\nCons:\n-I think the method section (especially the optimization part)  could be explained better for readers who are not familiar with EBM, and allow the paper to have more accessibility. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the problem of being able to control text generated by pre-trained language models.
The problem is timely and important. The paper   frames the problem as constraint satisfaction over a probability distribution. Both pointwise and distributional constraints can be imposed. The proposed algorithm,  Generation with Distributional Control (GDC), is elegant, and is an interesting new addition to this line of work. Overall, the paper brings forth news ideas, and could have impact.

Decision: Accept (Oral)