Year: 2021

Abstract:
We propose a Distributional Approach for addressing Controlled Text Generation from pre-trained Language Models (LMs). This approach permits to specify, in a single formal framework, both “pointwise’” and “distributional” constraints over the target LM — to our knowledge, the first model with such generality — while minimizing KL divergence from the initial LM distribution. The optimal target distribution is then uniquely determined as an explicit EBM (Energy-Based Model) representation. From that optimal representation we then train a target controlled Autoregressive LM through an adaptive distributional variant of Policy Gradient. We conduct a first set of experiments over pointwise constraints showing the advantages of our approach over a set of baselines, in terms of obtaining a controlled LM balancing constraint satisfaction with divergence from the initial LM. We then perform experiments over distributional constraints, a unique feature of our approach, demonstrating its potential as a remedy to the problem of Bias in Language Models. Through an ablation study, we show the effectiveness of our adaptive technique for obtaining faster convergence.1

heading: 1 INTRODUCTION
text: Neural language models, such as GPT-2/3 (Radford et al., 2019; Brown et al., 2020a), pretrained on huge amounts of text, have become pre-eminent in NLP, producing texts of unprecedented quality. In this paper, we are concerned with the problem of controlling a generic pretrained LM in order to satisfy certain desiderata. For instance, we may want to avoid toxic content; prevent certain demographic biases; or steer generations towards a certain topic or style. Prior work, taking inspiration from Reinforcement Learning (RL), has aimed at inducing autoregressive models to optimize global objectives using task specific rewards such as BLEU and ROUGE for Machine Translation and Summarization (Ranzato et al., 2016; Bahdanau et al., 2017), or hand crafted rewards (Li et al., 2016b; Tambwekar et al., 2019) to improve certain a priori desirable features.
However, such an optimization process is not infallible; Liu et al. (2016a) noted that it often leads to “degeneration”, producing poor examples that improve the average reward but forgo coherence and fluency. This degeneration is often diagnosed as an effect of deviating too much from the original pretrained LM during optimization. Consequently, prior work has regarded proximity to the pretrained model as a prescription for sample quality. This view is most prominent in open-domain generation where no gold references are available for fine-tuning, making the pretrained LM itself the yardstick for fluency. Jaques et al. (2017); Ziegler et al. (2019) propose a conservative fine-tuning approach moderated by a KL penalty between the trained policy and the original LM, discouraging large deviations. A KL penalty was also used by Dathathri et al. (2020), this time in a plug-and-play rather than a fine-tuning context. However, the authors show that balancing policy deviations from the original LM while also satisfying the control conditions is delicate. To combat degeneration they had to combine the KL penalty with post-norm fusion, reranking, and early-stopping procedures.
∗Equal Contributions. †Work done during an internship at NAVER Labs Europe. 1Code available on https://github.com/naver/gdc
Most of the existing work on Controlled Generation has taken what we refer to as a “pointwise” view, namely focusing on the quality of each individual output, a view that is encouraged by the standard RL goal of maximizing rewards computed at the individual level. Such techniques are incapable of enforcing “distributional” conditions, where some collective statistical properties are desired over the set of all generations.
Distributional control is key to solving the problem of social biases in LMs trained on large, uncurated Web corpora. Those LMs - dubbed “Stochastic Parrots” in (Bender et al., 2021) - tend to encode hegemonic biases that are harmful to marginalized populations. There has been a large body of work analysing these distributional biases (Blodgett et al., 2020; Stanovsky et al., 2019; Prates et al., 2020; Sheng et al., 2019a; Brown et al., 2020b). However, applying distributional control on pretrained models is still an understudied problem. Sheng et al. (2020) introduce a method relying on adversarial triggers (Wallace et al., 2019); this method does not de-bias the whole distribution but only obtains non-biased continuations of given prompts. Bordia & Bowman (2019) introduce a regularization term for reducing gender bias when training a language model from scratch (as opposed to de-biasing a pretrained model).2
In this work, we present our Generation with Distributional Control (GDC) approach, in which we formalize the problem of controlled text generation as a constraint satisfaction problem over the probability distribution p representing the desired target LM. Namely, we require the expectations (“moments”) relative to p of certain output features to have specific values; this permits for instance to condition all outputs to speak about sports (a pointwise constraint), and 50% of them to mention female characters (a distributional constraint). Additionally, we require p to have a minimal KL divergence DKL(p, a) from the original pretrained LM a. This has the effect that p now inherits favorable linguistic qualities from a. As we will explain, this formulation is a generalization of the Maximum Entropy Principle and leads to a unique solution P (x). P (x) is an unnormalized distribution, aka an Energy-Based Model (EBM) (Hinton, 2002; LeCun et al., 2006; Bakhtin et al., 2020), of which p(x) = 1/Z P (x) is the normalized version, where Z .= ∑ x P (x) is the partition function of P .
Computing the EBM representation P is a crucial step, as it fully determines the optimal distribution p we are looking for. However, it is not the end of the story, because the representation thus obtained does not enable us to directly sample from p, an essential property of any LM.3 To this end, we introduce KL-adaptive DPG (Distributional Policy Gradient), a variant of an algorithm recently proposed in (Parshakova et al., 2019b). We train the policy πθ to approximate p in an adaptive way, by speeding up the next round of approximations based on approximations previously obtained. At the end of this process, we obtain a final πθ, our target LM, on which we can estimate diverse metrics, including DKL(p, πθ), measuring the approximation quality of πθ relative to the optimal p, and DKL(πθ, a), measuring the divergence of πθ relative to the original LM a.
This two-step approach differs from much research in NLP-oriented work with EBMs, which tends to use EBM representations inside the training loops of neural networks, blurring different dimensions of the problem. By contrast — similarly to Parshakova et al. (2019a;b) in a different context — we clearly decouple the relatively simple problem of determining a “pivot” optimal EBM from the more difficult problem of exploiting this EBM at inference time, Such decoupling is valuable, because it permits to better diagnose the important challenges to focus on.
Overall, our contributions can be summarized as follows:
1. We introduce a Distributional View for controlled text generation formalized as a constraint satisfaction problem combined with a divergence minimization objective, providing a single framework both for “distributional” constraints (collective statistical requirements) and for “pointwise” constraints (hard requirements on each individual) (§2.1). To our knowledge, this is the first framework with such generality for controlled text generation.
2. We show how these constraints lead to an optimal EBM for the target model (§2.2), propose the KL-Adaptive DPG algorithm for approximating the optimal EBM distribution by
2Additional Related Work is provided in §E. We use §A, §B ... to refer to sections in the Appendix. 3One possible sampling approach here would be to employ MCMC techniques, such as Metropolis-
Hastings (Robert & Casella, 2005). These come with theoretical convergence guarantees in the limit but in practice convergence can be very difficult to assess, and furthermore, obtaining samples can be extremely slow.
an autoregressive policy (§2.3), and show the effectiveness of this adaptive technique for obtaining faster convergence (§B.2).
3. We conduct experiments in a number of pointwise and distributional conditions, assessing results in terms of divergence from GPT-2, fluency and diversity, with better performance than strong baselines. The distributional experiments show the potential of our approach as a remedy to the current and important problem of bias in pretrained language models, providing a novel direction for addressing it (§3).

heading: 2 FORMALIZATION
text: We denote byX the set of all sequences x of bounded length Lmax, by a the initial pretrained model and by p the desired target model. The probabilities of x according to each model are a(x) and p(x). Our approach consists in expressing our desiderata through constraints on the desired values µ̄i of the expectations (aka moments) µi . = Ex∼p φi(x) of certain predefined real-valued feature functions φi(x), for i ∈ {1, . . . , k}. To illustrate, the previous example can be expressed by using two binary features, φ1(x) = 1 iff x is classified as speaking about sports, φ2(x) = 1 iff x mentions a female character. Then our “moment constraints” take the following form: µ1 = Ex∼p φ1(x) = 1.0, µ2 = Ex∼p φ2(x) = 0.5. The first (pointwise) constraint implies that each individual x has to speak about sports (otherwise µ1 could not reach its maximum value 1.0), the second (distributional) constraint that 50% of the x’s have to mention a female character.4
Let C be the set of all distributions c over X that satisfy the moment constraints. We then propose to specify p as a distribution respecting the constraints, but also minimizing KL divergence from a:
p . = arg min
c∈C DKL(c, a), (1)
Equation (1) is a generalization of the Maximum Entropy Principle of Jaynes (1957), which corresponds to the limit case where a is the uniform u distribution over X , noting that minimizing DKL(c, u) is equivalent to maximizing the entropy of c under the constraints — in other words, trying to find the least “specific” distribution satisfying the constraints.

heading: 2.1 CONSTRAINTS, INFORMATION GEOMETRY, EXPONENTIAL FAMILIES
text: To recap our formal approach, we have a finite setX , a distribution a overX s.t. a(x) > 0,∀x ∈ X , and real functions φ1, ..., φk overX . We specify moment constraints µi = µ̄i on distributions c over X , where µi . = Ex∼c φi(x) and the µ̄i’s are given targets; the set of distributions satisfying these constraints is denoted by C. Our Problem is to find a p such that p = arg minc∈C DKL(c, a). We follow Csiszár & Shields (2004) on this question, a problem that is at the core of the field of Information Geometry (Nielsen, 2018; Amari & Nagaoka, 2000). Under the assumption that C 6= ∅, they prove the following result (also see §A.1):
4This example uses only binary features, but real-valued features can also be used, for instance scores returned by a soft classifier.
Theorem 1 (A) There exists a unique solution p to the problem above, obtained as p(x) ∝ P (x) where P is in exponential family form:
P (x) = a(x) 1[x ∈ XC ] e ∑ i λiφi(x). (2)
In other words p(x) = 1/Z P (x), with Z = ∑ x∈X P (x); P is an unnormalized distribution, i.e. an EBM. Here XC = {x ∈ X| ∃c ∈ C s.t. c(x) > 0} is the “support set” associated with C. The λi’s are real numbers called the natural parameters associated with the moments µi.
(B) p can be approximated to arbitrary precision by distributions p of the form:
p (x) ∝ a(x) e ∑ i λ ,iφi(x) (3)
for appropriate real values of the λ ,i.
(C) p satisfies the Pythagorean Identity: DKL(c, a) = DKL(c, p) +DKL(p, a),∀c ∈ C (see Fig 1).
The advantage of this version of the connection between Generalized Maximum Entropy and Exponential Families is its generality, which distinguishes it from other presentations, and which makes it ideal for unified application to pointwise, distributional or hybrid constraints.
In the special case of only pointwise constraints, of the form Ex∼cφi(x) = 1.0, i ∈ [1, k], with φi(x) ∈ {0, 1}, let’s define the predicate b(x) to be 1 iff x satisfies all the constraints. Then, using the (A) form of the result, it is an easy exercise (see §A.2) to prove that XC = {x ∈ X| b(x) = 1} and that one has p(x) ∝ a(x)b(x). In this case P (x) = a(x)b(x) is a very simple EBM that does not involve an exponential part; this is the EBM form that we use for experiments involving only pointwise constraints.
In the general case where some constraints are distributional, the determination ofXC is not as direct, and we prefer to use the approximation provided by (B), which permits a generic implementation. With only distributional constraints, an exact solution is typically obtained with finite λ’s. With hybrid constraints, some of the λ’s may tend to infinite (positive or negative) values but thresholding them suffices to get a good approximation.
2.2 FROM MOMENT CONSTRAINTS TO EBM Algorithm 1 Computing λ Input: a, features φ, imposed moments µ̄ 1: sample a batch x1, . . . , xN from a 2: for each j ∈ [1, N ]: wj(λ)← eλ·φ(xj)
3: µ̂(λ)← ∑N j=1 wj(λ) φ(xj)∑N
j=1 wj(λ)
4: solve by SGD: arg minλ ||µ̄− µ̂(λ)||22 Output: parameter vector λ
Let’s now consider a set of desired moment constraints µ̄.5 In the general case (i.e., when some constraints are distributional), we use Theorem 1.(B), which says that the desired energy-based model P can be approximated arbitrarily closely in the following form:
P (x) . = a(x)eλ·φ(x). (4)
This EBM defines the desired normalized distribution p(x) .= P (x)Z , where Z . = ∑ x P (x). What is left is to learn appropriate values for the parameter vector λ s.t.:
Ex∼pφ(x) ' µ̄. (5)
We address this problem through Algorithm 1. First, we sample a large number N of sequences x1 . . . xj . . . xN from a. On line 2, we define “importance weights” wj(λ) . =
P (xj) a(xj) =
exp 〈λ,φ(xj)〉. On line 3, we then use SNIS (Self Normalized Importance Sampling) (Kim & Bengio, 2016; Parshakova et al., 2019a) to estimate µ(λ) .= Ex∼pφ(x). SNIS consists in computing:
µ̂(λ) = ∑N j=1 wj(λ) φ(xj)∑N
j=1 wj(λ) , (6)
5Boldface φ and µ represents vectors of real values (features and moments).
and it can be shown that µ̂(λ) ' µ(λ), with convergence in the limit (Owen, 2013). Note that the estimate µ̂(λ) is obtained not as a single number, but as a parametric function of the variable λ. We want to find λ such that µ̂(λ) = µ̄, a question that we handle on line 4 by performing an SGD optimization over the objective min ||µ̄− µ̂(λ)||22.6
At the end of this process, we obtain an estimated value for the parameter vector λ, and a representation P (x) = a(x) exp 〈λ,φ(x)〉. While a(x) is a normalized distribution by construction, the introduction of the second factor loses this normalization property, making P (x) an EBM.7 8

heading: 2.3 FROM EBM TO AUTOREGRESSIVE POLICY
text: Algorithm 2 KL-Adaptive DPG Input: P , initial policy q 1: πθ ← q 2: for each iteration do 3: for each episode do 4: sample x from q(·) 5: θ ← θ+α(θ) P (x)
q(x) ∇θ log πθ(x)
6: if DKL(p||πθ) < DKL(p||q) then 7: q ← πθ
Output: πθ
The EBM representation just obtained for P defines the optimal p = Z−1P unambiguously, a crucial intermediate step in the solution of our problem. From it we can immediately compute ratios of the form p(x)/p(x′) for two sequences x, x′, but without knowing Z, we cannot compute p(x) and, even with such a knowledge, we cannot produce samples from p.
This problem is typical of EBMs at large: they provide a rich and flexible mechanism for specifying models, but they leave a gap between representation and exploitation. A range of techniques, from sophisticated MCMC approaches (especially for continuous models in vision) to contrastive learning techniques, have been developed for bridging this gap.
One technique that is suitable for our objective here, namely sampling from a sequential EBM that includes an autoregressive component a(x), is the DPG (“Distributional Policy Gradient”) algorithm (Parshakova et al., 2019b).
The objective of DPG is to obtain an autoregressive policy πθ that approximates p, where approximation is formalized in terms of making the cross-entropy CE(p, πθ) = − ∑ x p(x) log πθ(x) as small as possible.9 DPG exploits the fact that, for any “proposal” distribution q whose support contains the support of p, we have
∇θCE(p, πθ) = −∇θEx∼p log πθ(x) = −Ex∼p∇θ log πθ(x) = −Ex∼q p(x)
q(x) ∇θ log πθ(x)
where the last equality is an instance of importance sampling. Our “KL-adaptive” version of DPG is shown in (Algorithm 2). We start from an input EBM P , along with an initial policy q which is a proxy to p; in our case we take q = a. During an iteration (think minibatch or set of minibatches), we sample a number of sequences from q, do an SGD update of θ (line 5), where P is used instead of p (noting that they only differ by a multiplicative constant), and where α(θ) is a learning rate. The efficiency of the algorithm is related to how close the proposal q is to the target p,10 The algorithm is adaptive in the sense that it modifies q periodically to take advantage of the evolving approximations πθ. On line 6, we we test whether the current πθ is closer
6µ(λ) can approximate µ̄ arbitrarily closely, and we know from SNIS theory that with increasing N , µ̂(λ) will become arbitrarily close to µ(λ). In our experiments we stop the SGD optimization when ||µ̄− µ̂(λ)||22 becomes smaller than 0.01.
7The class of Energy-Based Models (EBMs) (LeCun et al., 2006) is much larger than the exponential family models we are considering in this paper. An EBM P (x) is just any unnormalized distribution over an input space X , in other words a mapping P from X to the non-negative reals. The terminology comes from physics, and corresponds to writing P (x) in the form P (x) = e−E(x), E being called the “energy” associated with x.
8A question was raised by an anonymous reviewer about the viability of adding new constraints incrementally. The answer is yes, more details provided in the Appendix, §A.3.
9This is equivalent to minimizing DKL(p, πθ) = CE(p, πθ)−H(p). 10In the limit where q were equal to p, the algorithm would be identical to standard supervised training,
except that samples would be obtained directly from the underlying process p rather than a training set of samples.
than q to p in terms of KL-divergence, and if so we update q to πθ on line 7.11 §B.2 provides an ablation study showing the effectiveness of this adaptive step for obtaining faster convergence.

heading: 3 EXPERIMENTS, RESULTS, AND EVALUATION
text: In this section we describe our evaluation methodology and perform experiments on pointwise constraints (§3.2) and on distributional and hybrid constraints (§3.3). The Appendix contains a detailed view of evaluation (§H), comparison with extra baselines (§D.2), and an ablation study (§B.2).

heading: 3.1 EVALUATION METRICS
text: The main metrics we report are: (1) Ex∼πθφi(x), assessing the ability of πθ to reach the expectation goal on the i-th constraint, (2) DKL(p||πθ), the forward KL divergence from the optimal distribution (which should be as close to 0 as possible), (3) DKL(πθ||a), the reverse KL divergence from the original GPT-2; for details on the estimation of these metrics see §B.1. Previous work has mostly focused on the diversity of each individual output using Dist-1,2,3 scores (Li et al., 2016a) to measure repetitions within a single generated sequence. However, the shortcomings in terms of sample diversity, of optimization techniques when training generative models for text, has recently been documented in (Caccia et al., 2020). So additionally, we report SelfBLEU-3,4,5 (Zhu et al., 2018) to measure repetitions at a distributional level across the whole set of generated samples, and also provide a token/type frequency analysis (see Fig. 4 and §H.4). Note that KL divergence from the original GPT-2 also implicitly captures sample diversity: a distribution that focuses all its probability mass on a few sequences typically displays high divergence from GPT-2. Implementation details and hyper-parameters are available in the Appendix (§ F).

heading: 3.2 POINTWISE CONSTRAINTS EXPERIMENTS
text: Pointwise constraints are of the form Epφi(x) = 1, with φi a binary feature. Contrarily to distributional constraints, they can be directly associated with a “reward”, namely φi itself. RL-inspired baselines can then be introduced naturally, and this is what we do here.
Single-Word constraints: Here we constrain the presence of a specific word w in the generated text i.e. φ(x) = 1 iff w appears in the sequence x. We use 9 single-word constraints of different rarity levels: “US” (original frequency: 7·10−3), “China” (4·10−3), “Canada” (2·10−3), “amazing” (1·10−3), “Paris” (5·10−4), “restaurant” (6·10−4), “amusing” (6·10−5), “Vampire” (9·10−5), “Wikileaks” (8·10−5). Word-list constraints: We use 4 different word lists among those proposed in (Dathathri et al., 2020), covering the following topics: “kitchen”, “fantasy”, “politics”, and “computers”. We set φl(x) = 1 if x contains at least one one word from the word list l. Classifier-based constraints: We use pre-trained classifiers from (Dathathri et al., 2020), which consist of a linear head on top of GPT-2. We select 4 classes and define corresponding pointwise constraints: “very positive”, “positive”, “very negative” and “Clickbait”. See §F for details on constraint computations. Baselines: We compare our method GDC to three baselines: (1) REINFORCE (Williams, 1992b), using the reward φ(x), i.e. trying to maximize Eπθφ(x); (2) REINFORCEP(x) : Reinforce again, but now using the reward P (x) based on our energy model P , i.e. maximizing EπθP (x); this baseline starts from the same optimal EBM P representation as GDC but with a standard optimization objective rather than a distributional one; in other words, while GDC tries to get a similar sampling distribution to p, this baseline tries to get sequences of maximal probability p(x). (3) ZIEGLER (Ziegler et al., 2019): an approach relying on the RL Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017) and which tries to maximize the objective Eπθφ(x)− βDKL(πθ, a), which interpolates the reward φ(x) with a KL-divergence penalty from the pretrained model, but where the goal is not explicitly to satisfy a constraint; for a geometric illustration of the differences with
11In the original DPG, the superiority test is done on the basis of the log-likelihood on a validation set. Here we are in the more demanding situation where no validation set is available. To directly estimate the KL divergence from p (line 6), we exploit the identity DKL(p‖π) = − logZ + 1/Z Ex∼q(x) P (x)q(x) log P (x) π(x)
. See §B.1 for derivations and a comparison with using Total Variation Distance (TVD) for assessing divergence.
GDC see §D.1. §D.2 provides a comparison of GDC with two additional baselines.
Results: Figure 2 shows the evolution of the metrics over training steps, aggregated across the 9 + 4 + 4 = 17 experiments. We observe the following: the baseline REINFORCE , which does not have any explicit link in its objective to the pretrained GPT-2, converges very early in the training, reaching a maximum value of Eπθφ(x) at the expense of a very large deviation from the original GPT-2. High values ofDKL(πθ|a), are translated into low Dist-1 and very high Self-BLEU-5 indicating degeneration and lack of diversity. REINFORCEP(x) maximizes the energy model P by peaking on a few sequences only; this can yield high values of EπθP (x), at the expense of low sample diversity as demonstrated in the highest values of SELF-BLEU-5 scores among baselines.12
In the case of ZIEGLER we can see a positive effect of the interpolation factor β between the reward and the KL penalty in the objective function. In the aggregated experiments reported here, the reward is slightly better than with GDC, but with inferior diversity scores (see also Fig. 4, showing that GDC produces richer vocabulary), and the stability is much worse (a detailed view of each experiment is provided in §H, showing more clearly the instability of this baseline). A complementary evaluation is provided by Figure 3, focusing on the ability of πθ to
converge to the optimal distribution p. We see that GDC is superior to all baselines in terms of DKL(p‖πθ) and also much more stable. In summary, in these experiments, we see that with GDC the constraint expectation Eπθφ(x) smoothly increases while πθ maintains the lowest divergence from GPT-2, becomes closest to the optimal p, and has the best diversity scores overall. On the other hand, we also note that at the point where we stop training (30K steps), the average over experiments of Eπθφ(x), while still increasing, does not reach 100%, an issue that we discuss at the end of the paper (§4).

heading: 3.3 DISTRIBUTIONAL AND HYBRID CONSTRAINTS EXPERIMENTS
text: As formalized in §2, GDC permits to define pointwise and distributional constraints as well as any mix between them. This unique feature makes it very suitable to remedy biases that the text generation model may have, a problem identified in several previous works (Sheng et al., 2019b).
12The difference with REINFORCE makes sense if one observes that φ(x) can be maximized on many sequences, while P (x) tries to maximize a(x) · φ(x), which is typically maximized on only one sequence.
Reps φ(x)
GDC 1 1 “Thank you all for the service this site gives me , ” he said. ... 1 1 This book is incredibly rich , entertaining , and extremely enjoyable... REINFORCE 1 1 Featuring the highest quality performance performance performance... 1 1 This beautiful beautiful quality production quality high quality... 1 1 High quality performance high quality performance product ... REINFORCE P(x) 10k 1 Thank you for supporting the journalism that our community needs! ... ZIEGLER 4418 1 Thank you for supporting the journalism that our community needs! ... 3560 1 Be the first to know. No one covers what is happening in our...
We employ GDC to balance gender and profession distributions across biographies generated by a GPT-2 model fine-tuned on Wikipedia Biographies (Lebret et al., 2016) (henceforth GPT-2bio) (§G gives additional details). The bias in GPT-2bio is significant: we calculated that this model generates only around 7% female biographies. It also displays a large imbalance between professions related to “Science” (1.5%), “Art” (10.0%), “Business” (10.9%) and “Sports” (19.5%).
Experiment 1: Single Distributional Constraint We use the distributional constraint Ex∼pφfemale(x) = 0.5; GDC is able to reduce the bias of GPT-2bio to obtain 35.6% female biographies rather than only 7.4% (see Fig. 2 for this experiment and the next ones). Experiment 2: Multiple Distributional Constraints We then test our framework with several distributional constraints of different values and control directions. We specify four distributional constraints all at once with the goal of increasing the expectations of “science” and “art” to 40% and decreasing those of “sports” and “business” to 10%. GDC is able to increase the expectations of the first two professions respectively from 1.5% to 20.3% and from 10 to 31.6% and to decrease those of “business” and “sports” respectively from 10.9% to 10.2% and from 19.5% to 11.9%, reaching expectations close to the desired ones for all features using a single training method. Experiments 3,4,5,6: Hybrid Constraints Here we want to de-bias the model as in the previous case but we single out biographies of scientists, artists, etc. Formally, our requirements become Ex∼pφprofession(x) = 1.0, a pointwise constraint, and Ex∼pφfemale(x) = 0.5, a distributional constraint. In those 4 hybrid experiments we can clearly see that GDC can address both pointwise and distributional constraints increasing each simultaneously with just the right amount to reach the desired expectations. Appendix §G further elaborates Fig. 2 (convergence curves).

heading: 4 DISCUSSION
text: Our approach to controlled text generation is distinguished by its breadth — the first one to handle distributional along with pointwise constraints, with applications to the important problem of Bias in pretrained LMs — and by the transparency of the supporting formalism. It decouples the training objective along two different dimensions. The first consists in solving the initial constraints specification, and leads through a direct algorithm to an optimal solution in EBM format. The second, where the real computational difficulty lies, consists in approximating this EBM with an autoregressive policy for use at inference time. Sampling from an EBM is an important, hard, and well-identified challenge in the literature. Our approach there consists in proposing a KL-adaptive version of the DPG algorithm, which exploits ascertained improvements of the trained policy to speed up convergence. This is an effective method for rare events, as we show in an ablation study (§B.2). In the case of pointwise constraints, where comparisons with baselines can be done, our experiments show the
method’s superiority in satisfying the constraints while avoiding degeneration. Reaching close to 100% samples meeting the constraints, can sometimes be obtained in these baselines, but only at a severe cost in terms of quality and sample diversity. Of course, if we do not care about such aspects, obtaining 100% constraint satisfaction is trivial: just generate one sentence satisfying the pointwise constraint!
Our method does not suffer from degeneration, but our end policies still generate a number of samples not satisfying the constraints. A possibility, left for future work, might consist in filling the moderate residual gap with MCMC techniques, which would be guaranteed to reach our optimal p in the limit. We do not go this route here, but conduct an experiment (see §C) to better understand the nature of the problem. In the simple case of a single-word constraint (x includes “amazing”), we sample directly 1M samples from GPT-2 and keep the roughly 5K samples containing amazing (a variant of rejection sampling, taking two processing days). We then do a standard supervised fine-tuning of GPT-2 with these samples, stopping training when the CE validation loss starts to increase, and observe that this model exhibits a worse constraint satisfaction rate than ours. This experiment does not mean that a much larger fine-tuning dataset, obtained in this slow, non-adaptive way, would not reach better statistics, but it raises doubts about the ability of the GPT-2 architecture to fine-tune over such a non-standard constraint as containing a given word somewhere in its output.
Overall, we believe that the proposed decomposition into two sub-problems is a methodological advantage compared to most other works, which directly aim at training a policy with the goal of improving certain evaluation metrics, but without clearly defining what qualifies as an optimal solution. The computational challenge of fully bridging the gap between the optimal EBM and an efficient sampling engine remains, and we hope that the formalism we propose, along with initial applications and experimental validations, will motivate further research along these lines.

heading: ACKNOWLEDGMENTS
text: We would like to thank the anonymous reviewers for their insightful feedback that helped enhancing the final version of this manuscript. We also thank Germán Kruszewski, Laurent Besacier, Matthias Gallé and Christopher Dance for providing technical feedback on this work and proof-reading the manuscript, as well as Tetiana Parshakova and Jean-Marc Andreoli for their work on the original versions of the SNIS and DPG algorithms.

heading: Appendix

heading: A DETAILS ON FORMALIZATION (§2)

heading: A.1 COMMENTS ON THEOREM 1
text: Our statement of Theorem 1 is actually a reformulation of two results in section 3 of Csiszár & Shields (2004). Our property (A) is a simple notational transposition of their Remark 3.1 (p. 444). Property (C) is the Pythagorean Identity in their Theorem 3.2 (p. 442). Property (B) reformulates the last part of the same Theorem “... and in general L ∩ cl(EQ) = {P ∗}” in terms of a limit of a sequence of distributions.
Note: Csiszár & Shields (2004) assume a finite X here, but generalizations to infinite (countable and/or continuous) X spaces are possible, see (Csiszar, 1975).

heading: A.2 THE CASE OF POINTWISE CONSTRAINTS IN §2.2
text: In the case of purely pointwise constraints, if b(x) = 1, then the distribution c = δx is in C, hence x ∈ XC . Conversely, if x ∈ XC then there is some c ∈ C such that c(x) > 0, implying that b(x) = 1. Hence XC = {x ∈ X| b(x) = 1}. Thus, in equation (2), P (x) = a(x)b(x) exp ∑ i λiφi(x); but for b(x) 6= 0, φi(x) = 1, so the exponential factor is a constant, which proves that P ′(x) = a(x)b(x) is proportional to P (x), and therefore p(x) ∝ P ′(x).
A.3 INCREMENTALLY ADDING NEW CONSTRAINTS
An interesting question13 is whether the process explained in §2 can be made incremental: if one has already computed a p and a πθ relative to a certain number of constraints, can one add a new constraint without restarting the whole process from scratch? The answer is yes, and here we provide some formal elements to understand why.

heading: A.3.1 TRANSITIVITY PROPERTY OF GENERALIZED MAXENT
text: According to (Csiszár, 1996), the Generalized MaxEnt of sections §2.1 and §2.2 has the “Transitivity property”. In our notation, this says that if we have k′ > k constraints, with C the manifold of distributions respecting only the first k constraints, C ′ the manifold respecting all k′ constraints (hence C ′ ⊂ C), then the maxent projection p′ of a onto C ′ can be obtained by first projecting a onto C, obtaining p, and then projecting p onto C ′, obtaining p′. In particular, the k lambdas associated with p can be directly reused as the first lambdas of the k′ lambda’s associated with p′.
(Csiszár, 1996) gives only a minimal proof sketch, but it is instructive to provide the details, as we do now, because the proof is a neat illustration of the power of information geometry for problems of the kind we consider. The proof, illustrated in Figure 5, is very similar to one of the proofs for the transitivity of the orthogonal projection in Euclidean geometry.
13raised by an anonymous reviewer of our ICLR submission.
Proof. In the Figure, p is the information projection (Csiszar’s terminology for the Generalized Maxent) of a onto C, as before. Let’s define r to be the projection of p onto C ′. We need to prove that r is identical to the projection p′ of a onto C ′. We consider an arbitrary distribution c′ in C ′, and apply the Pythagorean Identity of Theorem 1 three times. Because p is the projection of a onto C, we have DKL(r, a) = DKL(r, p) +DKL(p, a) and also DKL(c′, a) = DKL(c′, p) +DKL(p, a). Because r is the projection of p onto C ′, we have DKL(c′, p) = DKL(c′, r) + DKL(r, p), hence DKL(c
′, p) ≥ DKL(r, p). Putting these three facts together, we find that DKL(c′, a) ≥ DKL(r, a). As c′ is an arbitrary point of C ′, this proves that r is the projection of a onto C ′, in other words, r = p′.

heading: A.3.2 TRANSITIVITY AND AUTOREGRESSIVE POLICY
text: Due to the Transitivity property, when calculating the EBM representation, it is possible to start from p without re-fitting p′ from scratch. However the move from EBM to autoregressive policy of §2.3 remains to be discussed. The question now is the following. We have already obtained a policy πθ approximating p, and we are interested in obtaining a policy πθ′ approximating p′: is it advantageous to start Algorithm 1 with q = πθ, rather than starting “from scratch” and taking q = a ? Intuition says “yes, very probably”, because πθ is by construction an approximation to p, which is closer than a to p′ (formally, DKL(p′, p) ≤ DKL(p′, a), see Fig. 5, where p′ = r). Due to the approximation, we only have DKL(p′, πθ) ' DKL(p′, p) , so a formal proof that πθ is superior to a as a starting point is impossible, but we expect that further experiments would confirm the improvement.

heading: B MORE ON ADAPTIVITY

heading: B.1 DETAILS ON KL-ADAPTIVITY
text: In this section we provide details on the comparison step in our KL-Adaptive version of the DPG Algorithm, introduced in section 2. We want to assess whether the current πθ is closer than q to p, and if the test is positive, we set πθ as the new proposal, hoping to make the proposal more effective for importance sampling.
There are several ways to compute similarity between distributions, two of the most popular ones being on the one hand KL-divergence and on the other hand Total Variation Distance (TVD) — where TVD(p||p′) .= 1/2 ∑ x |p(x) − p′(x)| — which is often used in probability and MCMC theory.14 Calculation of these metrics relative to p is not straightforward since the distribution p ∝ P is only implicitly represented by the unnormalized EBM P , and we cannot easily obtain direct samples from p. In this section we describe a workaround.
14Both metrics are equal to 0 only if the distributions are equal everywhere (in the case of discrete distributions, which are our focus here, otherwise almost everywhere). To our knowledge, there is no obvious best metrics to use when assessing a proposal in importance sampling, leading us to conduct an ablation experiments with both metrics (Appendix 2)
Given P and a proposal distribution q that we can sample from, using importance sampling (Owen, 2013), one can calculate the partition function Z as follows:
Z = ∑ x P (x) = ∑ x q(x) P (x)/q(x)
= Ex∼q(x) P (x)/q(x) (7)
We can then compute DKL(p||π) as:
DKL(p||π) = ∑ x p(x) log p(x) π(x) = ∑ x p(x) log P (x) Zπ(x)
= − logZ + ∑ x p(x) log P (x) π(x) = − logZ + ∑ x q(x) p(x) q(x) log P (x) π(x)
= − logZ + 1/Z Ex∼q(x) P (x)
q(x) log
P (x) π(x) (8)
Similarly, for TVD(p||π):
TVD(p||π) = 1/2 ∑ x |p(x)− π(x)|
= 1/2 ∑ x q(x) ∣∣∣∣π(x)q(x) − p(x)q(x) ∣∣∣∣ = 1/2∑ x q(x) ∣∣∣∣π(x)q(x) − P (x)Z q(x) ∣∣∣∣
= 1/2 Ex∼q(x) ∣∣∣∣π(x)q(x) − P (x)Z q(x) ∣∣∣∣ (9)
In §B.2 we run an ablation study to compare the use of DKL on line 6 of Algorithm 2) or its replacement by TVD.
For both metrics, we need an estimate of Z. The precision of this estimate depends on the sample size and the quality of the proposal distribution q. We calculate a moving average estimate ZMA of Z is used inside the estimations of DKL(p‖πθ) and DKL(p‖q) (Algorithm 3, lines 7 and 8). ZMA is updated at each iteration of the training, and the moving average estimate is valid due to the fact that Ẑi, based on K samples, is an unbiased estimate of Z, and therefore so is ZMA. In this way, the estimate benefits from all the samples being produced during the course of the training; and also because the proposal distribution q evolves and gets closer to the target distribution p, the quality of the estimates of both DKL(p||πθ) and ZMA through importance sampling increases (equation 7). A similar approach is taken in the case of TVD (not shown).
Algorithm 3 KL-Adaptive DPG (detailed) Input: P , initial policy q
1: πθ ← q
2: ZMA ← 0 . Initialize Moving Average estimate of Z
3: for each iteration i do
4: for each step k ∈ [1,K] do
5: sample xk from q(·)
6: θ ← θ + α(θ) P (xk) q(xk) ∇θ log πθ(xk)
7: Ẑi ← K−1 ∑ k P (xk)/q(xk) . Estimate on the K samples
8: ZMA ← i∗ZMA+Ẑii+1 . Update moving average estimate of Z 9: D̂KL(p||πθ)← − logZMA + (K ZMA)−1 ∑ k P (xk) q(xk) log P (xk) πθ(xk) . Estimate on the K samples
10: D̂KL(p||q)← − logZMA + (K ZMA)−1 ∑ k P (xk) q(xk) log P (xk) q(xk)
. Estimate on the K samples
11: if D̂KL(p||πθ) < D̂KL(p||q) then

heading: 12: q ← πθ
text: Output: πθ

heading: B.2 ABLATION ON ADAPTIVITY
text: Here we run an ablation experiment on the adaptivity step of KL-Adaptive DPG (§2). We compare three variants of our proposed method: DPG-KLD, which uses KL divergence from the target distribution p to measure the quality of the trained policy πθ i.e. if DKL(p‖πθ) < DKL(p‖q) we update the proposal distribution q ← πθ. DPG-TVD is similar but with the total variation distance instead (TVD). In non-Adaptive the initial proposal q is kept fixed during training.
We run 3 point-wise experiments with single word constraints of three rarity levels in the original GPT-2 distribution, namely: “Vampire” (1/104),“Paris” (1/103),“US” (1/102) .For each we use 3 different seeds and train for 10k gradient updates.
Figure 6 shows training trends of the three ablations. We find a significant difference in convergence speed in favour of the adaptive methods. The efficiency gap between Adaptive and non-Adaptive methods becomes larger the more rare the constraints are. i.e. the proposal distribution q starting point is very far from the target distribution p, as the efficiency of the DPG algorithm is related to how close the proposal q is to the target p. When q is continuously adapted, the proposal distribution becomes closer to p and the training becomes efficient regardless of how far the initial proposal distribution is from p. We observe similar convergence rates for DPG-KLD and DPG-TVD.

heading: C CAN STANDARD SUPERVISION FULLY SATISFY THE CONSTRAINTS?
text: In this section, we try to better understand potential difficulties of autoregressive models to fully satisfy constraints such as the ones illustrated in our pointwise experiments.
To this end, we consider whether a standard fully supervised fine-tuning of GPT-2 can achieve that objective while keeping a minimal distance from the initial model. To answer the question, we carry out an experiment where we fine-tune GPT-2 on a collection of samples satisfying the desired constraint. Our goal here is to investigate whether GPT-2 can fully satisfy the constraint without overfitting the fine-tuning data, since overfitting (memorizing) the training data basically means high KL-divergence from the initial model.
For this experiment, we choose a single-word constraint with the word “amazing”. We start by sampling 1M sequences from GPT-2 small — a process that took us roughly 48 hours — and keeping only the ones containing “amazing” (this filtration process can be seen as a variant of rejection sampling (Casella et al., 2004)). We end up with a total of 4600 samples out of which we use 500 for validation and the rest for fine-tuning.
Figure 7 shows evolution of both validation loss and constraint satisfaction Eφ(x) on samples generated from the model during fine-tuning. Interestingly, the lowest validation loss corresponds to only Eφ(x) ≈ 0.56. Higher values of Eφ(x) correspond to higher validation loss i.e. to overfitting. This result suggests a relationship between training a policy reaching 100% and overfitting the training data. This hints at the difficulty of strictly imposing certain types of constraints on pre-trained language models without moving far away from the initial model.15
15Note how very difficult the job would be in the extreme case of a constraint was based on a hash-based predicate filtering on average one sentence out of two.

heading: D MORE COMPARISONS
text: D.1 ILLUSTRATION COMPARING GDC, REINFORCE, AND ZIEGLER
The figure below illustrates the difference between GDC, the RL-based REINFORCE and ZIEGLER baselines for a pointwise constraint. The main points to note are: (1) REINFORCE is trying to find a distribution pR maximizing r(x) (meaning that pR lies on the C manifold), but this pR is free to land anywhere on this manifold, and (2) ZIEGLER is trying to find a distribution pZ that interpolates (with a weight β) between a high average r(x) and the KL divergence from a; unless β = 0, in which case we are back to REINFORCE, pZ does not satisfy the constraint and falls outside of the manifold.

heading: D.2 COMPARISON AGAINST FURTHER BASELINES
text: Here we compare GDC to other baselines, namely Plug and Play (PPLM) (Dathathri et al., 2020) and CTRL (Keskar et al., 2019) for sentiment control. PPLM works by updating the hidden states of GPT-2 for a given prefix in order to derive the generation towards the desired attributes. Unlike GDC, PPLM needs a prefix to perform its hidden-state updates. Thus, our approach is more general in the sense that any prefix can be used on the trained model at test time, rather than requiring prefix-specifc fine-tuning. CTRL is a large-scale language model (1.63 billion parameters and ~14x larger than GPT-2 small) based on control codes for steering text style and content. For the purpose of generating positive/negative sentiments using CTRL, we use its positive/negative reviews control codes as done in (Dathathri et al., 2020). The control codes used are “Reviews Rating: 5.0” and “Reviews Rating: 1.0” for positive and negative sentiment control, respectively. We use five different prefixes (or prompts) and generate 100 continuations given each prefix obtaining a total of 500 samples. It is worth noting that GDC is trained in the same way as described in the main text, i.e. without any knowledge of prefixes, and that we only use prefixes at test time with the saved checkpoint. The five prefixes used come from (Dathathri et al., 2020): “The chicken ”, “The potato ”, “The lake ”, “The pizza ”, and “The horse ”.
We use the same sampling parameters across all approaches by setting the temperature T = 1.0, using top-k sampling with k = 10, and removing the repetition penalty used in CTRL (Keskar et al., 2019). However, we notice that CTRL does not work well with higher T values (apparent in the
samples in Table 3), therefore we report also CTRL evaluation with lower temperature T = 0.5 and a repetition penalty λrep = 1.2 as reported in their paper.
As metrics, we use sentiment class expectation Eφ(x), the perplexity according to an external GPT2 small architecture as in (Li et al., 2018), and the diversity metrics introduced in section §3.1. We average all these metrics across the 500 continuations generated. Table 3 shows the results for positive and negative sentiment control experiments. As shown, GDC is able to achieve better positive/negative sentiment with lower perplexity than both PPLM and CTRL. As for diversity, GDC achieves comparable diversity to the other two approaches and even outperforms PPLM on the Distn metrics in the positive sentiment task.
Table 4 shows sample continuations from all three approaches. Clearly, PPLM and CTRL exhibit some form of degeneration and repetition in many of the continuations (highlighted in light red), which is reflected in their very high perplexity score compared to GDC, which produces much more natural text with minimum repetitions without requiring a repetition penalty as CTRL.
It is also worth noting here that CTRL (and other control code methods) is very much limited in terms of its applications. For instance, to generate positive/negative sentiment text as we do in this experiment, we are required to use the ‘‘Reviews Rating...’’ control code, using control codes outside of those CTRL was fine-tuned on leads to very bad generations. This, in turn, restricts the generated text to positive/negative reviews although we may desire different types of positive/negative text (e.g. news reports). We can observe this effect16 in some of the samples in Table 4 such as “The chicken we just ordered from Amazon.com...” and “The pizza works no matter what settings you use it on.
16With lower temperatures, this behaviour becomes even worse and CTRL mostly generates reviews.
Repetitions are highlighted in light red. As shown, PPLM and CTRL produce more repetitions compared to GDC.

heading: E RELATED WORK EXTENDED
text: Optimizing global rewards for Text Generation There is a large reinforcement learning inspired literature about steering an autoregressive sequential model towards optimizing some global reward over the generated text. This includes REINFORCE (Williams, 1992a) for Machine translation (MT) Ranzato et al. (2016), actor critic for Abstractive Summarization (Paulus et al., 2018), Imageto-Text Liu et al. (2016b), Dialogue Generation Li et al. (2016b), and Video Captioning (Pasunuru & Bansal, 2017). With respect to rewards, some approaches for Machine Translation and Summarization (Ranzato et al., 2016; Bahdanau et al., 2017) directly optimize end task rewards such as BLEU and ROUGE at training time to compensate for the mismatch between the perplexity-based training of the initial model and the evaluation metrics used at test time. Some others use heuristic rewards as in (Li et al., 2016b; Tambwekar et al., 2019), in order to improve certain a priori desirable features of generated stories or dialogues. Other non-RL techniques for approximating the global sequence constraints φ(x) by a biased estimator φ(xt|x:t−1). These techniques usually referred to as weighted decoding Holtzman et al. (2018); See et al. (2019) this however still requires a heavy search procedure and this biased estimation of sequences that satisfy the global constraint compromises fluency and coherence. Continuous approximation using the Gumbel Softmax was developed for the training of Variational Autoencoders but several works have implemented it for natural language generation Shetty et al. (2017); Chu & Liu (2019); Kusner & Hernández-Lobato (2016).
Competing Degeneration in Controlled Text Generation When using such approaches, one needs to take care of not forgetting too much of the original LM policy (“degeneration”): Liu et al. (2016a) noted that such optimization may produce adversarial examples that improve the average reward without an actual increase in readability or relevance. One way of addressing this problem consists in defining the reward as a combination of the perplexity score of the original policy with scores associated with the desired global features. Wu et al. (2016); Paulus et al. (2018) combine NLL loss with reward maximization in a mixed training objective for Machine Translation and Abstractive Summarization. Yang et al. (2018) use a set of Language Models pretrained on the target domain as a control signal for text style transfer. As a proxy to perplexity, Holtzman et al. (2018) design hand-crafted rewards using a set of discriminators to ensure the quality of generated text in open-ended text generation. Liu et al. (2016a), however, show that defining a combination reward accounting for text fluency is highly non-trivial and the results of directly optimizing it cannot be fully trusted.
KL Divergence penalty Another approach relied on penalizing too large deviations of the trained policy relative to the original policy. Jaques et al. (2017; 2019) propose a conservative fine-tuning approach with a KL penalty between the trained policy and the original auto-regressive model. This penalty acts as a regularizer to the optimization process that prevents the trained policy from deviating too much from the original policy. Ziegler et al. (2019) follow a similar approach for fine tuning a language model based on human preferences, in this case a proximal policy algorithm (Schulman et al., 2017) is used to maximize the combined reward. PPLM (Dathathri et al., 2020), this time in a plug-and-play rather than a fine-tuning context, also use KL divergence to penalize deviations from the initial policy.
Pointwise vs. Distributional View Most of the existing works on Controlled Generation have taken what we have called a pointwise view: focusing on the quality of each individual output, as opposed to distributional properties of the collection of all outputs. And in fact, the standard objective of RL is to optimize a pointwise reward. Even when policy-gradient methods do consider distributions over outputs, they only do as a tool towards producing maximal rewards; and in fact, it is a side effect of the limited capacity of the policy networks that such distributions do not peak on a single output, as would be the optimal outcome in cases of real-valued rewards with no ties.17 By contrast to this usual optimization “intent”, our own intent here is explicitly distributional, and the policies we are looking for are not simply tools towards maximizing scores, but actual objectives in their own right.
17In which cases the distribution q maximizing Ex∼qR(x) would be q = δx∗ for x∗ = arg maxxR(x).
Such a change of perspective might be argued against in the case of conditional seq2seq problems, such as Machine Translation, where focusing on a single good output for a given input makes sense, but is clearly in-adapted when focusing on language models where sample diversity is a requirement.
Energy Based Models for Text Energy-Based Models (EBMs) (Hinton, 2002; LeCun et al., 2006; Ranzato et al., 2007) are learning frameworks that attracted a lot of attention several decades ago.18 There has been a recent surge of interest in these types of models across a variety of fields. Some early NLP-related EBM research is concerned with neural-based sequence labelling problems (e.g. tagging) exploiting the global sequence (Andor et al., 2016; Belanger & McCallum, 2016). Some current applications to text generation include Parshakova et al. (2019a) and Deng et al. (2020), who augment a standard autoregressive LM with an additional global factor in order to get a lower perplexity on the training data. Tu et al. (2020) propose an energy-based method to perform inference networks from pretrained Non-Autoregressive Machine Translation models. A recent survey of EBMs for text is provided in Bakhtin et al. (2020).
18The early work on ”Whole sentence exponential models” by (Rosenfeld et al., 2001) — which only came to our attention when preparing the final version of this paper — can be considered as a form of EBM over texts. While it does not utilize neural networks, it does exploit, as we do, the exponential family in order to provide a global form of control over texts.

heading: F HYPERPARAMETERS AND TRAINING DETAILS
text: We implement GDC and all baselines using the PyTorch framework (Paszke et al., 2019). For all experiments we start from a pretrained GPT-2 small (117M parameters) obtained from the HuggingFace library (Wolf et al., 2019) and fine-tune for 3K gradient-update steps. Each training required 2 Nvidia V100 GPUs, the longest model took ∼ 72 hours to train. A list of the hyperparameters used for GDC and baselines is given in table 5. K refers to the number of gradient steps per iteration in Algorithm 2.
N refers to the number of samples required and µtolerance to the minimum tolerated error ||µ̄ − µ̂(λ)||22 while optimizing λ, and λlearning is the SGD step size for updating λ in Algorithm 1. During training of the policy πθ, we perform periodic evaluation as follows: every 10 minibatch gradient updates, we sample 2048 sequences of 40 tokens long, using nucleus sampling with topp = 0.9 (Holtzman et al., 2020) and estimate diversity metrics on these samples. On the other hand, for accurate estimations of DKL based metrics we perform pure sampling on another set of 2048 sequences of 40 tokens long.
For word-lists in the pointwise experiments in section 3.2, we used the 4 word lists from the Plug and Play (Dathathri et al., 2020) repository19. As for the sentiment and clickbait classifiers, we used their pre-trained classifier heads over GPT-2 medium20.
For distributional and hybrid experiments, we fine-tune GPT-2 small (117M params) to produce biographies on a dataset of 700K Wikipedia biographies (Lebret et al., 2016) which we refer to as GPT-2bio. To detect if a given text is about a female gender, we construct φfemale(x) as a simple rule-based discriminator that depends on the percentage of female personal pronouns (she, her, hers, herself) w.r.t. all mentioned pronouns. We define four types of professions “Art”, “Science”, “Business and Politics”, and “Sports”. To detect them, we define a wordlist for each type as shown in table 6.
19https://github.com/uber-research/PPLM/tree/master/paper code/wordlists 20https://github.com/uber-research/PPLM/tree/master/paper code/discrim models

heading: G DISTRIBUTIONAL AND HYBRID CONTROL EXPERIMENTS FOR DEBIASING LANGUAGE MODELS
text: Large pretrained Language Models are often trained on uncurated data from the internet, where several demographics are severely underrepresented. One of those demographics is women, whose biographies make up only 18.58% of English Wikipedia’s biographies (Graells-Garrido et al., 2015). It is expected that such bias is transferred if not amplified by Language Models. Previous work has suggested associations of certain demographics with certain professions, sentiments and stereotypes (Sheng et al., 2019b; Brown et al., 2020b; Nadeem et al., 2020). This shows thaat Bias in LMs also shows up in different forms than just under-representation, and the task of debiasing LMs could require more a complex control method. GPT-2bio demonstrates a large initial bias: over a large sample of size 20480 examples using top-p sampling (p = 0.9), it generates only around 7% female biographies. and a large imbalance between profession types “Science” (1%), “Art” (10%), “Business&Politics” (10%) and “Sports” (20%).
In this set of experiments, we demonstrate the potential of GDC as flexible general framework that can control pretrained Language Models to impose pointwise, distributional constraints, or even a mix between them (hybrid constraints). We design a set of 6 experiments whose descriptions and results are displayed in the figures below. Generation examples are provided in Table 7.

heading: H EXTRA DETAILS ON POINTWISE EXPERIMENTS
text: H.1 APPROXIMATING THE DESIRED p DISTRIBUTION

heading: H.3 POINTWISE CONSTRAINTS

heading: H.2 MORE DETAILS ON POINT-WISE CONSTRAINTS EXPERIMENTS

heading: H.4 TOKEN FREQUENCY ANALYSIS
text: To analyse in depth the effect of deviating much from the original GPT-2, for policies obtained from our method and each baseline, we obtain a large sample and filter to 4000 sequences that satisfy the imposed pointwise constraints for each of the 17 pointwise experiments explained in §3. Figures 35, 36 and 37 plot a token frequency analysis for each of the training methods.
The vanilla policy gradient baselines REINFORCE suffer from very low diversity of generations; in the examples shown in section H.5 we note strong degeneration, in which all generations are composed of a few repeated tokens.
REINFORCEP(x) suffers from a token diversity issue. As noticed and confirmed by generated examples shown section H.5, it often concentrates all the sequence probability mass on a single sequence which is often fluent and satisfies the constraint; however this leads to an extreme loss of sample diversity in almost all experiments. This shows the usefulness of our proposed analysis — in addition to the self-BLEU metrics — for distinguishing diversity at the sequence level or at the distribution level. Similarly, ZIEGLER (Ziegler et al., 2019) often suffers from the same lack of sample diversity (5 out of the 17 experiments); GDC obtains the highest diversity amongst all baselines, as demonstrated by the long tail in the figures below. It is important to note here that low sample diversity is also captured by the KL deviation from the original GPT-2 model i.e. DKL(πθ‖a); GDC identifies the target distribution as the one which minimally deviates from the original policy while satisfying the constraints (p = arg minq∈C DKL(q, a)) is thus expected to preserve the high sample diversity of the original GPT-2.
H.5 GENERATION EXAMPLES
occurrence probability 1/104) highlighted in green. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
occurrence probability 1/104) highlighted in green. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
occurrence probability 1/104) highlighted in green. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
occurrence probability 1/103) highlighted in green. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
(with occurrence probability 1/103) highlighted in green. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
occurrence probability 1/103) highlighted in green. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
occurrence probability 1/103) highlighted in green. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
occurrence probability 1/102) highlighted in green. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
rence probability 1/102) highlighted in green. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
kens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
control. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
trol. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
control. Tokens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.
kens are highlighted with yellow with different intensities to indicate their overall frequencies in the generated corpus. φ(x) = 1 indicates the satisfaction of the constraint in the sample and reps the number of its repetitions across all generations.

Review summaries:
The paper studies the controlled sequence generation problem based on pretrained language models, i.e., controlling a generic pretrained LM to satisfy certain constraints, e.g., removing certain biases in language models. Specifically, the paper proposes a distributional view and imposes constraints based on collective statistical properties. The problem is formalized as a constraint satisfaction problem, minimizing a divergence objective. The paper proposes to use KL-Adaptive DPG algorithm for approximating the optimal energy-based model distribution. Experiments were conducted over both pointwise constraints and distributional constraints, showing the effectiveness of the model over the compared baselines.