A Distributional Approach to Controlled Text Generation

1 INTRODUCTION . Neural language models , such as GPT-2/3 ( Radford et al. , 2019 ; Brown et al. , 2020a ) , pretrained on huge amounts of text , have become pre-eminent in NLP , producing texts of unprecedented quality . In this paper , we are concerned with the problem of controlling a generic pretrained LM in order to satisfy certain desiderata . For instance , we may want to avoid toxic content ; prevent certain demographic biases ; or steer generations towards a certain topic or style . Prior work , taking inspiration from Reinforcement Learning ( RL ) , has aimed at inducing autoregressive models to optimize global objectives using task specific rewards such as BLEU and ROUGE for Machine Translation and Summarization ( Ranzato et al. , 2016 ; Bahdanau et al. , 2017 ) , or hand crafted rewards ( Li et al. , 2016b ; Tambwekar et al. , 2019 ) to improve certain a priori desirable features . However , such an optimization process is not infallible ; Liu et al . ( 2016a ) noted that it often leads to “ degeneration ” , producing poor examples that improve the average reward but forgo coherence and fluency . This degeneration is often diagnosed as an effect of deviating too much from the original pretrained LM during optimization . Consequently , prior work has regarded proximity to the pretrained model as a prescription for sample quality . This view is most prominent in open-domain generation where no gold references are available for fine-tuning , making the pretrained LM itself the yardstick for fluency . Jaques et al . ( 2017 ) ; Ziegler et al . ( 2019 ) propose a conservative fine-tuning approach moderated by a KL penalty between the trained policy and the original LM , discouraging large deviations . A KL penalty was also used by Dathathri et al . ( 2020 ) , this time in a plug-and-play rather than a fine-tuning context . However , the authors show that balancing policy deviations from the original LM while also satisfying the control conditions is delicate . To combat degeneration they had to combine the KL penalty with post-norm fusion , reranking , and early-stopping procedures . ∗Equal Contributions . †Work done during an internship at NAVER Labs Europe . 1Code available on https : //github.com/naver/gdc Most of the existing work on Controlled Generation has taken what we refer to as a “ pointwise ” view , namely focusing on the quality of each individual output , a view that is encouraged by the standard RL goal of maximizing rewards computed at the individual level . Such techniques are incapable of enforcing “ distributional ” conditions , where some collective statistical properties are desired over the set of all generations . Distributional control is key to solving the problem of social biases in LMs trained on large , uncurated Web corpora . Those LMs - dubbed “ Stochastic Parrots ” in ( Bender et al. , 2021 ) - tend to encode hegemonic biases that are harmful to marginalized populations . There has been a large body of work analysing these distributional biases ( Blodgett et al. , 2020 ; Stanovsky et al. , 2019 ; Prates et al. , 2020 ; Sheng et al. , 2019a ; Brown et al. , 2020b ) . However , applying distributional control on pretrained models is still an understudied problem . Sheng et al . ( 2020 ) introduce a method relying on adversarial triggers ( Wallace et al. , 2019 ) ; this method does not de-bias the whole distribution but only obtains non-biased continuations of given prompts . Bordia & Bowman ( 2019 ) introduce a regularization term for reducing gender bias when training a language model from scratch ( as opposed to de-biasing a pretrained model ) .2 In this work , we present our Generation with Distributional Control ( GDC ) approach , in which we formalize the problem of controlled text generation as a constraint satisfaction problem over the probability distribution p representing the desired target LM . Namely , we require the expectations ( “ moments ” ) relative to p of certain output features to have specific values ; this permits for instance to condition all outputs to speak about sports ( a pointwise constraint ) , and 50 % of them to mention female characters ( a distributional constraint ) . Additionally , we require p to have a minimal KL divergence DKL ( p , a ) from the original pretrained LM a . This has the effect that p now inherits favorable linguistic qualities from a . As we will explain , this formulation is a generalization of the Maximum Entropy Principle and leads to a unique solution P ( x ) . P ( x ) is an unnormalized distribution , aka an Energy-Based Model ( EBM ) ( Hinton , 2002 ; LeCun et al. , 2006 ; Bakhtin et al. , 2020 ) , of which p ( x ) = 1/Z P ( x ) is the normalized version , where Z .= ∑ x P ( x ) is the partition function of P . Computing the EBM representation P is a crucial step , as it fully determines the optimal distribution p we are looking for . However , it is not the end of the story , because the representation thus obtained does not enable us to directly sample from p , an essential property of any LM.3 To this end , we introduce KL-adaptive DPG ( Distributional Policy Gradient ) , a variant of an algorithm recently proposed in ( Parshakova et al. , 2019b ) . We train the policy πθ to approximate p in an adaptive way , by speeding up the next round of approximations based on approximations previously obtained . At the end of this process , we obtain a final πθ , our target LM , on which we can estimate diverse metrics , including DKL ( p , πθ ) , measuring the approximation quality of πθ relative to the optimal p , and DKL ( πθ , a ) , measuring the divergence of πθ relative to the original LM a . This two-step approach differs from much research in NLP-oriented work with EBMs , which tends to use EBM representations inside the training loops of neural networks , blurring different dimensions of the problem . By contrast — similarly to Parshakova et al . ( 2019a ; b ) in a different context — we clearly decouple the relatively simple problem of determining a “ pivot ” optimal EBM from the more difficult problem of exploiting this EBM at inference time , Such decoupling is valuable , because it permits to better diagnose the important challenges to focus on . Overall , our contributions can be summarized as follows : 1 . We introduce a Distributional View for controlled text generation formalized as a constraint satisfaction problem combined with a divergence minimization objective , providing a single framework both for “ distributional ” constraints ( collective statistical requirements ) and for “ pointwise ” constraints ( hard requirements on each individual ) ( §2.1 ) . To our knowledge , this is the first framework with such generality for controlled text generation . 2 . We show how these constraints lead to an optimal EBM for the target model ( §2.2 ) , propose the KL-Adaptive DPG algorithm for approximating the optimal EBM distribution by 2Additional Related Work is provided in §E . We use §A , §B ... to refer to sections in the Appendix . 3One possible sampling approach here would be to employ MCMC techniques , such as Metropolis- Hastings ( Robert & Casella , 2005 ) . These come with theoretical convergence guarantees in the limit but in practice convergence can be very difficult to assess , and furthermore , obtaining samples can be extremely slow . an autoregressive policy ( §2.3 ) , and show the effectiveness of this adaptive technique for obtaining faster convergence ( §B.2 ) . 3 . We conduct experiments in a number of pointwise and distributional conditions , assessing results in terms of divergence from GPT-2 , fluency and diversity , with better performance than strong baselines . The distributional experiments show the potential of our approach as a remedy to the current and important problem of bias in pretrained language models , providing a novel direction for addressing it ( §3 ) . 2 FORMALIZATION . We denote byX the set of all sequences x of bounded length Lmax , by a the initial pretrained model and by p the desired target model . The probabilities of x according to each model are a ( x ) and p ( x ) . Our approach consists in expressing our desiderata through constraints on the desired values µ̄i of the expectations ( aka moments ) µi . = Ex∼p φi ( x ) of certain predefined real-valued feature functions φi ( x ) , for i ∈ { 1 , . . . , k } . To illustrate , the previous example can be expressed by using two binary features , φ1 ( x ) = 1 iff x is classified as speaking about sports , φ2 ( x ) = 1 iff x mentions a female character . Then our “ moment constraints ” take the following form : µ1 = Ex∼p φ1 ( x ) = 1.0 , µ2 = Ex∼p φ2 ( x ) = 0.5 . The first ( pointwise ) constraint implies that each individual x has to speak about sports ( otherwise µ1 could not reach its maximum value 1.0 ) , the second ( distributional ) constraint that 50 % of the x ’ s have to mention a female character.4 Let C be the set of all distributions c over X that satisfy the moment constraints . We then propose to specify p as a distribution respecting the constraints , but also minimizing KL divergence from a : p . = arg min c∈C DKL ( c , a ) , ( 1 ) Equation ( 1 ) is a generalization of the Maximum Entropy Principle of Jaynes ( 1957 ) , which corresponds to the limit case where a is the uniform u distribution over X , noting that minimizing DKL ( c , u ) is equivalent to maximizing the entropy of c under the constraints — in other words , trying to find the least “ specific ” distribution satisfying the constraints . 2.1 CONSTRAINTS , INFORMATION GEOMETRY , EXPONENTIAL FAMILIES . To recap our formal approach , we have a finite setX , a distribution a overX s.t . a ( x ) > 0 , ∀x ∈ X , and real functions φ1 , ... , φk overX . We specify moment constraints µi = µ̄i on distributions c over X , where µi . = Ex∼c φi ( x ) and the µ̄i ’ s are given targets ; the set of distributions satisfying these constraints is denoted by C. Our Problem is to find a p such that p = arg minc∈C DKL ( c , a ) . We follow Csiszár & Shields ( 2004 ) on this question , a problem that is at the core of the field of Information Geometry ( Nielsen , 2018 ; Amari & Nagaoka , 2000 ) . Under the assumption that C 6= ∅ , they prove the following result ( also see §A.1 ) : 4This example uses only binary features , but real-valued features can also be used , for instance scores returned by a soft classifier . Theorem 1 ( A ) There exists a unique solution p to the problem above , obtained as p ( x ) ∝ P ( x ) where P is in exponential family form : P ( x ) = a ( x ) 1 [ x ∈ XC ] e ∑ i λiφi ( x ) . ( 2 ) In other words p ( x ) = 1/Z P ( x ) , with Z = ∑ x∈X P ( x ) ; P is an unnormalized distribution , i.e . an EBM . Here XC = { x ∈ X| ∃c ∈ C s.t . c ( x ) > 0 } is the “ support set ” associated with C. The λi ’ s are real numbers called the natural parameters associated with the moments µi . ( B ) p can be approximated to arbitrary precision by distributions p of the form : p ( x ) ∝ a ( x ) e ∑ i λ , iφi ( x ) ( 3 ) for appropriate real values of the λ , i . ( C ) p satisfies the Pythagorean Identity : DKL ( c , a ) = DKL ( c , p ) +DKL ( p , a ) , ∀c ∈ C ( see Fig 1 ) . The advantage of this version of the connection between Generalized Maximum Entropy and Exponential Families is its generality , which distinguishes it from other presentations , and which makes it ideal for unified application to pointwise , distributional or hybrid constraints . In the special case of only pointwise constraints , of the form Ex∼cφi ( x ) = 1.0 , i ∈ [ 1 , k ] , with φi ( x ) ∈ { 0 , 1 } , let ’ s define the predicate b ( x ) to be 1 iff x satisfies all the constraints . Then , using the ( A ) form of the result , it is an easy exercise ( see §A.2 ) to prove that XC = { x ∈ X| b ( x ) = 1 } and that one has p ( x ) ∝ a ( x ) b ( x ) . In this case P ( x ) = a ( x ) b ( x ) is a very simple EBM that does not involve an exponential part ; this is the EBM form that we use for experiments involving only pointwise constraints . In the general case where some constraints are distributional , the determination ofXC is not as direct , and we prefer to use the approximation provided by ( B ) , which permits a generic implementation . With only distributional constraints , an exact solution is typically obtained with finite λ ’ s . With hybrid constraints , some of the λ ’ s may tend to infinite ( positive or negative ) values but thresholding them suffices to get a good approximation . 2.2 FROM MOMENT CONSTRAINTS TO EBM Algorithm 1 Computing λ Input : a , features φ , imposed moments µ̄ 1 : sample a batch x1 , . . . , xN from a 2 : for each j ∈ [ 1 , N ] : wj ( λ ) ← eλ·φ ( xj ) 3 : µ̂ ( λ ) ← ∑N j=1 wj ( λ ) φ ( xj ) ∑N j=1 wj ( λ ) 4 : solve by SGD : arg minλ ||µ̄− µ̂ ( λ ) ||22 Output : parameter vector λ Let ’ s now consider a set of desired moment constraints µ̄.5 In the general case ( i.e. , when some constraints are distributional ) , we use Theorem 1 . ( B ) , which says that the desired energy-based model P can be approximated arbitrarily closely in the following form : P ( x ) . = a ( x ) eλ·φ ( x ) . ( 4 ) This EBM defines the desired normalized distribution p ( x ) .= P ( x ) Z , where Z . = ∑ x P ( x ) . What is left is to learn appropriate values for the parameter vector λ s.t . : Ex∼pφ ( x ) ' µ̄ . ( 5 ) We address this problem through Algorithm 1 . First , we sample a large number N of sequences x1 . . . xj . . . xN from a . On line 2 , we define “ importance weights ” wj ( λ ) . = P ( xj ) a ( xj ) = exp 〈λ , φ ( xj ) 〉 . On line 3 , we then use SNIS ( Self Normalized Importance Sampling ) ( Kim & Bengio , 2016 ; Parshakova et al. , 2019a ) to estimate µ ( λ ) .= Ex∼pφ ( x ) . SNIS consists in computing : µ̂ ( λ ) = ∑N j=1 wj ( λ ) φ ( xj ) ∑N j=1 wj ( λ ) , ( 6 ) 5Boldface φ and µ represents vectors of real values ( features and moments ) . and it can be shown that µ̂ ( λ ) ' µ ( λ ) , with convergence in the limit ( Owen , 2013 ) . Note that the estimate µ̂ ( λ ) is obtained not as a single number , but as a parametric function of the variable λ . We want to find λ such that µ̂ ( λ ) = µ̄ , a question that we handle on line 4 by performing an SGD optimization over the objective min ||µ̄− µ̂ ( λ ) ||22.6 At the end of this process , we obtain an estimated value for the parameter vector λ , and a representation P ( x ) = a ( x ) exp 〈λ , φ ( x ) 〉 . While a ( x ) is a normalized distribution by construction , the introduction of the second factor loses this normalization property , making P ( x ) an EBM.7 8

Summary: The paper studies the controlled sequence generation problem based on pretrained language models, i.e., controlling a generic pretrained LM to satisfy certain constraints, e.g., removing certain biases in language models. Specifically, the paper proposes a distributional view and imposes constraints based on collective statistical properties. The problem is formalized as a constraint satisfaction problem, minimizing a divergence objective. The paper proposes to use KL-Adaptive DPG algorithm for approximating the optimal energy-based model distribution. Experiments were conducted over both pointwise constraints and distributional constraints, showing the effectiveness of the model over the compared baselines.