COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics
Authors: Lianhui Qin
Sean Welleck
Daniel Khashabi
Yejin Choi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Text generation
constrained text generation
language models
natural language processing
langevin dynamics
decoding

Abstract: Many applications of text generation require incorporating different constraints to control the semantics or style of generated text. These constraints can be hard (e.g., ensuring certain keywords are included in the output) and soft (e.g., contextualizing the output with the left- or right-hand context). In this paper, we present Energy-based Constrained Decoding with Langevin Dynamics (COLD), a decoding framework which unifies constrained generation as specifying constraints through an energy function, then performing efficient differentiable reasoning over the constraints through gradient-based sampling. COLD decoding is a flexible framework that can be applied directly to off-the-shelf left-to-right language models without the need for any task-specific fine-tuning, as demonstrated through three challenging text generation applications: lexically-constrained generation, abductive reasoning, and counterfactual reasoning. Our experiments on these constrained generation tasks point to the effectiveness of our approach, both in terms of automatic and human evaluation.

COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics
Lianhui Qin' Sean Welleck Daniel Khashabi Yejin Choi
Paul G. Allen School of Computer Science & Engineering, University of Washington Allen Institute for Artificial Intelligence Department of Computer Science, Johns Hopkins University
Abstract
Many applications of text generation require incorporating different constraints control the semantics style of generated text. These constraints can be hard (e.g., ensuring certain keywords are included in the output) and soft (e.g., con- textualizing the output with the left- right-hand context). In this paper, we present Energy-based Constrained Decoding with Langevin Dynamics (COLD), a decoding framework which unifies constrained generation as specifying constraints through an energy function, then performing efficient differentiable reasoning over the constraints through gradient- based sampling. COLDD decoding is flexible framework that can be applied directly to off-the-shelff left-to-right language mod- els without the need for any task-specific fine-tuning, demonstrated through three challenging text generation applications: lexically-constrained generation, abductive reasoning, and counterfactual reasoning. Our experiments on these con- strained generation tasks point to the effectiveness of our approach, both in terms of automatic and human evaluation
Introduction
Many text generation applications require producing text that not only fluent, but also satisfies various constraints which control the semantics or style of the generated text. For example (Figure D for knowledge-grouse grounded or keyword guided generation, we might want to ensure that certain key- words are included in the generated output as hard lexical constraints [2 52 For other types of text generation, we often wish to incorporate soft topical constraints to contextualize the desired output, e.g., abductively 4 reasoning about what happened in the middle of a story given the past and the future story context Yet another class text generation applications requires revising an input based on new counterfactual condition [K] which simultaneously requires semantic coherence as well as minimal-ede constraints with respect to the input text AE!
The dominant paradigm to various text generation applications has been supervised learning with task- sask-specific training data. However, different applications require varied and potentially evolving constraints, and annotating large amount of task-specific training data for each different combination of constraints can be costly. Recent work has explored incorporating constraints through energy- based text modeling that alleviates the need of supervised data [23 expensive training of specific generation models. In addition, training might not even be feasible with recent models that are extreme in scale, like GPT-3 [3]. This motivates the need to enrich decoding algorithms that can work directly with pretrained language models without task-specific fine- fine-tuning, and support complex combinations of hard and soft constraints to control the generated text on the fly.
Yet those approaches still require
Work done while working at Allen Institute for httpg//www....a.gaae/////.......
Code is available at
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

COLD: E fluncyy key words
COLD:
wif colerence right sahecwnce
COLD: E calerence mirimal udlt
pet
couch
A pet cet liee to Slepp on couch.
Jon land a lonely life
Ho happiet
com cemperiinn
student Reginning N, medical student
joined a prestolous
mediaa prachice
altar graduwting
Gat
Post %
Future N,
Counteriaanuul N/ joined a prestigiuu
Reywords 3
join dub.
Mixmml Eekt
Gerenttory y
(ineenttoon:
Froding X,
Lexically Constrained Generation
Abductive Reasoning
Counterfactual Reasoning
Figure 1: Applying COLD to different constrained generation tasks amounts to specifying energy function E by plugging in relevant constraint functions. Text in grey boxes is the input, and text in blue boxes is the output.
We propose new constrained decoding approach that formulates decoding as sampling from an energy-based model (EBM) [1 [27]. Constrained generation with our approach amounts to specifying an energy function by plugging arbitrary constraint functions that are suitable for the task at hand, then sampling from its induced distribution. In particular, to overcome the longstanding challenges of sampling discrete text from EBMs, we for the first time introduce Langevin dynamics [ to text-based EBMs for efficient gradient-base
sampling. As result, our approach, Constrained Decoding with Langevin Dynamics (COLD), performs sampling by iteratively updating a continuous relaxation of text using gradients of the energy function. The resulting continuous text samples are then mapped back to the discrete space with simple guided discretization approach, yielding text sequences that are fluent and adhere to the constraints.
Our work makes unique contributions to a recent line of research investigating decoding algorithms in three distinct aspects. First, our formulation unifies various constrained generation scenarios that involve hard lexical constraints and/or soft contextual constraints: specifying an energy function, then sampling from its induced distribution. Second, we propose a sampling method, which complements decoding algorithmss that look for a single optimal solution. Finally, provide new empirical insights into the strengths and weaknesses of existing approaches to discrete search and differentiable reasoning.
for incorporating different constraints 45 33
26
To test the flexibility and empirical performance of COLD decoding, we experiment with three chal- lenging text generation tasks: lexically constrained generation 29 18, abductive reasoning [I. and counterfactual story generation 44 COLD achieves better lexical coverage than NEUROLOGIE [63] a beam-based discrete decoding algorithm specifically designed for lexically constrained generation, while producing more coherent and higher quality text than DELOREAN 45 a state-of-the-art gradient-based generation method for abductive reasoning and counterfactual reasoning. COLD supports all three constrained generation settings under unified framework speciffying energy function using a collection of fluency and task ses--spccffic constraints, then sampling from its induced distribution and achieves strong performance on both automatic and human evaluation.
2 Background
Neural text generation. Neural text generation typically involves stages: modeling distribution over text sequences, and using a decoding algorithu to generate sequences with the model. Let y = (y1,..., MT) denote discrete sequence where each yı a token from vocabulary V. Common factorize the probability of a sequence into the product of per-token conditionals in left-to-right order, (y) = II molulyyee) with each conditional parameterized by a shared neural network, such as transformer [50]. Popular decoding algorithms, ranging from beam search greedy decoding to sampling methods such as top-k [12] or nucleus 19 sampling, produce text sequences y using the model pg. often conditioned on prompt x.
neural language models (e.g., GPT-2/3 4o 3)
Constrained text generation. We view text generation as the problem of finding a sequence that satisfies a collection of constraints. For instance, the scenario above amounts to generating sequence y (91..... ..... ) subject to a soft constraint that the continuation should be fluent and logically coherent with the prompt x. Other constrained generation problems impose additional constraints, such as text infilling (60 X where coherence constraints move beyond a left-hand prefix, lexically
constrained generation which hard constraints require the output to contain given tokens, and various forms of aowaanticlll-coosttaannn generation in which the output is softly constrained to be

---

Initial distribution
Target constrained distribution
X LM
Langevin Dynamics
gln gln
q v
v, +
Discrete Text
Soft Sequence g(())
Soft Sequence g(N)
Tapk-Mask
Mesked Sequence
y
Initielization
Energy-based sampling
Discretization
Figure 2: An overview of the COLD decoding procedure. Given an energy function E E, (.f(( with various constraints, the procedure starts with a soft sequence y( as a sample from an initial energy energy-based distribution, and performs Langevin dynamics iterations using the gradient (5) 2). The resulting sequence y((N) after N iterations is approximately a sample from the desired constrained distribution. We then apply top-k filtering on the soft sequence to produce a discrete text sequence (E
similar to another sequence. Since common decoding algorithms generate text monotonically. relying on (o y+)) for determining the next token, it is challenging to enforce these diverse constraints.
mnergy-basele models and Langevin dynamics. Given an energy function E(y) E R, an energy- based model (EBM) is defined as Boltzmann distribution p(y) exp(-E(y)) H/Y) where Z Σy Exp(-E(()) } is the normalizing factor (The sum is replaced with an integral if y is continuous). EBMs are flexible, in that one can incorporate arbitrary functions such as constraints into the energy function F Recent work has thus made attempts to frain text-based EBMs each for specific tasks
23 As discussed earlier, we instead use the energy-based formulation to develop an inference (decoding) procedure that enables off-the-shelf pretrained language models to perform arbitrary constrained generation, without any fine-tuning.
Despite the flexibility, however, sampling from EBM is particularly challenging, computing Z is intractable. Common gradient-fres Markov chain Monte Carlo (MCMC) methods such as Gibbs sampling
[ can be used, but they often prohibitively slow [0O 3X
Langevin dynamics
[53 37 [44], a gradient-based MCMC method, offers more efficient sampling by using the gradient of the energy function V, F(y), enabling sampling in domains such image generation 9 48) However, since text discrete, the gradient Fly) is not well-defined, making it non-trivial to apply Langevin dynamics for sampling text from an EBM. Our approach bridges this gap with continuous relaxation of text, differentiable constraints, and guided discretization, as described below.
COLD Decoding with Langevin Dynamics
To enable flexible and diverse constrained generation in off-the-shelf language models, we develop Constrained Decoding with Langevin Dynamics (COLD), a decoding approach that treats text gen- eration sampling from an energy-based distribution, allowing for flexibly composing constraints based on the task at hand. COLD decoding generates text by sampling from an EBM defined over a sequence of "soft" tokens using Langevin dynamics, then maps the continuous sample into discrete, fluent text. We provide our formulation constrained text generation ($3.T), present differentiable constraints that can be composed into energy functions ($3.2) along with our discretization method ($3.3), and discuss practical details of COLD decoding
Figure 2 provides an overview.
3.1 Energy-based Decoding
Constrained text generation aims to produce text samples that satisfy a set of constraints (usually conditioned on an input x omitted for brevity). We assume each constraint can be captured by constraint function fi(y) E R, where higher values of fi mean that the text better satisfies the constraint. For example, fa could measure the likelihood of y as fluency constraint (more in $3.2. while a hard constraint f. amounts to large negative penalty when y does not satisfy the constraint.
3

---

I-gram
g.armm
X, eight logs
yod hand
sink
soup
y+ never
gave
up
CrossEntropy
LM
LM
Soft Match
Soft Match
gave up
hand
Softmax
F
(1)
(2)
(3)
Figure 3: Illustaatoons of the differentiable constraints introduced in $3.2 (1) The soft fluency con-
straint (Eq to encourage fluency of y, based on LM probabilities. (2) The future contextualization
constraint in Eq. to encourage coherence w.r.. the future context (has eight legs). (3) The n-gram similarity constraint in Eq. where the left figure shows the case of n which encourages keywords (e.g., hand) to appear in the generation, and the right figure shows the case of which is typically used to encourage sequence similarity with a reference text y..
The set of constraints induces a distribution over text, written in an energy-based form as:
p(y) = exp Σ X.fi(r)
where A; 2 0 is the weight of the ith constraint, Z the normalizing factor. Here Ely) - (yy is the energy function. This energy-based form is flexible, as one can plug in any constraint functions required for task of interest. Generating text under the constraints can then be seen as sampling from the energy-based distribution y ~ p(y). One can also draw multiple samples and pick the best if only one sample is needed, as discussed later 3..4.
As mentioned above, for efficient sampling from p(y) we want to use Langevin dynamics, which makes use of the gradient y Eyy). However, in our case y discrete sequence and the gradient Ey) is not well-defined. As result, we perform Langevin dynamics with an energy defined on sequence of continuous token vectors, described below.
Differentiable decoding with Langevin dynamics. Instead of defining the energy function on discrete tokens, define the energy function sequence of continuous vectors = -51...... ).), which we call a soft sequence. Each position in the soft sequence is a vector y, E R where V is the vocabulary size, and each element (v(v) E IR corresponds the logit of word v in the vocabulary. Taking the softmax of y. yields a distribution over the vocabulary for position I, p = softmax(y./T)). As - 0. D. becomes a one-hot vector, indicating a discrete token.
By specifying an energy E on the soft sequence y. we can use Langevin dynamics to obtain a sample. Specifically, the sampling is done by forming Markov chain:
s(++1) (()) V F()(m)) + (*)))
where y > the step size, and (m) E (0, is the noise at iteration n. As shown in Welling and Teh [53], by adding the right amount of noise and annealing the step size, the procedure will converge to samples from the true distribution. That is, if we let p(n) be the distribution such that ((m) ~ p(n), then as n + oo and a - O, we have p(") - p(ý) == exp( Tep(-Eii Exp(--E(y)// E'y)}/Z /2. That is, the procedure ends up generating samples from the distribution induced by the energy function.
Next, we describe constraint functions defined on the soft sequence ý that can be plugged in as describe how obtain discrete sequence
components of the energy function. Later in $3.3 from a soft sequence sample y.
3.2 A Collection of COLD Constraints
COLD provides flexible framework for plugging in a wide range of constraint functions for task of interest. We describe constraint functions that are useful in various constrained generation problems, such as those we consider the experiments ($T). The constraints include language model-based fluency constraints, along with lexical and semantic constraints on the sequence content. More generally, any differentiable function that outputs a goodness score of (soft) text can be used as constraint function, long it reflects the requirements of the target task.

---

Algorithm Constrained Decoding w/ Langevin Dynamics.
input Constraints [ft}. length T, iterations N. output Sample sequence y.
y: (0) < init() for all position // init soft-tokens
for { N} do
p(n) < () (fi)) // compute energy ($3.2)
((+1) - y: -,,,, Etn) + +el") for all // update soft tokens (Eg 2p
end for
8 (o)) for all L // discretize (Eq.6)
= arg max, topk-filter return: y (y1....,,y) yr)
Soft fluency constraint. Fluency is common requirement for generated text. To promote fluency, use a constraint which favors soft sequences that receive high probability according to the underlying left-to-right LM PiM (e.g., GPT2):
( -
1-1 Lee PTM ( log softmax (vx()))
(3)
where P (-F means the next-token distribution when providing the neural language model with the preceding soft tokens y<t (i.e., feeding the weighted average of word embeddings, with the weights being (Stmax for < 20 451)).
Intuitively, the constraint says that each token distribution in the soft sequence, softmax (ft), must match the "reference" distribution (-) predicted by the underlying language model. The match is measured by the (negative) cross-entropy between the two distributions. The constraint thus encourages fluency. In practice, if there is left-side context x for the generation to condition on, we feed x to the LM to form the "reference" distribution PiMM (-15 (-1v,,,)) x). As result, is encouraged to be fluent and coherent with the context x.
We can easily incorporate additional reverse LM constraint, fiM. using a right-to-left LM (-M(-1>>+) >+), as an additional fluency constraint. Flexibly leveraging multiple models in this way is infeasible with conventional decoding methods such as beam search nucleus sampling.
Future-token prediction constraint. Applications such as text infilling involve future input tokens that remain fixed, but should contribute to updating past positions. For instance, consider updating the second position of The has eight legs. sample should be coherent with the tokens X, on the right (i.e., has eight legs).
To this end, we constraint that adjusts soft tokens maximize the likelihood of input tokens x,,
Sprea (5;x)) Σ k=1 log ( 18; xr,<k);
(Fr,k
where K is the length of x,. In other words, the constraint adjusts the soft sequence ý such that the underlying LM predicts the future tokens x, after sceing y.
N-gram similarity constraint. Many constrained generation scenarios pose requirements on the wording and expression generated text sequences. For instance, lexically constrained generation tasks [18] require certain keywords to be presented in the text samples, while counterfactual reasoning tasks require the text to retain the essence of a reference sequence.
4E1
or text editing
15 31
We formulate these requirements as an n-gram similarity constraint which favors sequences that overlap with a reference y. at the n-gram level.
Ssm y.: = ngram-matchhf...mmmmmm yx),
(5)
where ngraammat.ch(- (h( :) is a recent differentiable n-gram matching function 32 which can be seen as a differentiable approximation the BLEU-n metric 40 When n and y. a sequence of keywords, the constraint in effect enforces ý to assign higher values to the keywords (1-grams). When is larger and y. is reference sequence, the constraint encourages ý to resemble the reference by assigning high values to tokens making n-grams from y..
3.3 From Soft to Discrete and Fluent Text
After receiving soft sequence sample ý from running Langevin dynamics (Eq. D we map the soft sequence discrete text sequence which consider the output of COLD decoding, A simple
5

---

Automatic Eval
Human Eval
Models
Left-voherence
Right-coherence
Overall-coherence
BLEU,
ROUGE-L
CIDEr
Score
Grammar
(x:y)
(yx,)
(xyy,)
LEFT-ONLY
0.88
16.26
3.49
38.48
4.57
3,95
2.68
2.70
DELOREAN
1.60
19.06
7.88
4.23
2.83
2,87
COLD (ours)
1.79
19,50
10.68
42.67
4.44
4.00
3.06
2.96
Table Automatic and human evaluation abductive reasoning (..1)
Our proposed method (COLD
decoding) outperforms DELOREAN, a recent decoding algorithm achieving strong results in this task.
method would be selecting the most-likely token at each position f, Vt arg max, Fe(v). However, the resulting text can suffer from fluency issues even if the soft fluency constraint (Eq. B used, due to competing constraints that sacrifice fluency. To overcome this, we use the underlying LM (e.g.. GPT2-XL) as a "guardian" for obtaining the discrete sequence. Specifically, at each position we first use the LM to produce the top-k most-likely candidate tokens based on its generation distribution conditioning on preceding tokens, which we denote as V. We then select from the top-k candidates the most likely token based on the soft sample y:
V/ arg DEVE Ve(v).
We refer to this method as "top-k filtering". The resulting text tends to be fluent because each token among the top-k most probable tokens from the LM [12 In practice,
the satisfaction of
certain constraints (e.g. n-gram similarity), we expand the candidate set V/ to include constraint tokens (c.g., in the tasks of abductive reasoning $4.1 and lexically constrained decoding $4.3).
Figure 2 illustrates the decoding procedure to get one output from COLD decoding. Algorithm summarizes the algorithm. Next, move to practical considerations of applying COLD.
3.4 Implementation of COLD Decoding
Sample-and-selee COLD decoding allows for drawing multiple text samples from the distribution induced by the energy function ( Depending task requirements, could either present the set of samples as output, or select one from the set based on some criteria (e.g., different energy terms) and return a single sequence, in those tasks considered in the experiments ($ This "sample-and
select" approach differs from deterministic constrained decoding methods, which optimize only one sequence [e.g..
3 26),
and is used widely in various generation settings [e.g..
Initialization. We initialize the soft sequence ý by running greedy decoding with the LM PLM to obtain output logits. In our preliminary experiments, the initialization strategy had limited influence on the generation results.
Noise schedule. Each iteration of Langevin dynamics adds noise (()) ((0,a()) to the gradient (Eq. 2). We gradually decrease (()) across iterations, which intuitively transitions the decoding procedure from exploration to optimization. In our experiments, we typically used the schedule which sets/reduces 01,0.5, 011, 0.05, 0.01] iterations (0, 50, 500, 1000, 1500), respectively.
Long sequences. COLD decoding produces a fixed-length sequence y = (y1..... ..).. To produce longer sequences, e.g. in cases where yr is not the end of sentence, we PLM to produce continuation of y using greedy decoding.
4 Experiments
We evaluate COLD on three constrained generation tasks. Using COLD for each task amounts to
specifying a set of task-specific constraints (instances of those in $3.2 Our focus is enabling constrained generation for settings in which fine-tuning is infeasible, through changing the decoding method. Thus, our experiments (i) use off off-the-shelf LMs without fine-tuning, and (ii) compare COLD primarily against alternative decoding methods. As our base LM, we use GPT2-XI. 46
4.1 Abductive Reasoning
We study specific formulation of abductive reasoning 4% as a language generation challenge. Specifically, given beginning sentence x; and an ending sentence x,,, the abductive language

---

generation (aNLG) problem [ consists of generating a bridge sentence that fills in between the two sentences and forms coherent full story (see Figure | for example), The task is particularly challenging for conventional monotonic left-to-right LMs (such as GPT-2 and GPT-3) since it requires non-monotonic reasoning that not only conditions on the past context (xt, on the left), but also the future story ending (X,, on the right).
4.1.1 The COLD Solution
COLD decoding can readily accommodate the abductive reasoning task by simply plugging in appropriate constraints to specify energy function. Specifically, the generated text needs to be (1) fluent and consistent with the left context X,, and (2) coherent with the right context X,. Accordingly, compose energy using relevant constraints from 3.2
(gy N ( x + X (F: + (p + Sa (f; kw()). kw())))
That is, we combine (a) soft fluency constraint (Eq. B conditioning on the left sentence x; to enforce fluency and consistency with the left context, and reverse fluency constraint with right-to-left LM conditioning on x, to encourage coherence with the right context; (b) a future-token prediction constraint (Eq. that enforces consistency between the generation y and the story ending x,; (c) a 1-gram similarity constraint (Eq. 5 between the generation and keywords (non-stopwords) in X, (excluding those in x;). i.c., kw (w(x,) kw(x)), which intuitively promotes 'smooth transition' between X y, and X,.
For the energy function in Eq. 77. select the constraint weights the dev set. Throughout the experiments, we set the number of Langevin dynamics steps to N = 2000, with a step size = 0.1 (Eq. We discuss more details of the configurations in the appendix.
Baselines. We compare with previous decoding approaches for this task. In particular, we compare with DELOREAN 45] which outperformed wide range of supervised and unsupervised methods on the abductive reasoning task in Qin et al. 45 Following Qin et 45 we also compare with a LEFT-ONLY method that generates the continuation of without considering the right-side X,,, i.c. y ~ >>PLM(y()))
Evaluation. We perform both automatic and human evaluation. We adopt the standard automatic metrics on the task [I that measure the minimal edit between the generated text and the human- written references on the test set, including BLEU 400. ROUGE [30), CIDEr [51], and BERTScore 58). For the human evaluation, we follow 45 and let crowdworkers from Amazon Mechanical Turk rate the generations on 200 test examples. Workers were presented a pair of observations (x: and x,) and a generated hypothesis y, and asked to rate the coherence of the hypothesis with respect to the observation xı (i.e., xty), the observation x, (i.e., yx,), and both (i.e., x,yx,), as well as the grammaticality of the hypothesis y itself, on a 5 5-point Likert scale. The average ordinal Krippendorff' alpha (0 < 1) [ is 0.36, indicating a fair inner-annotator agreement.
4.1.2 Results
Table T shows the evaluation results on the abductive reasoning task. Under automatic evaluation (the left panel), COLD consistently outperforms the previous best unsupervised decoding algorithm DELOREAN, as well as the LEFT-ONLY method, in terms of both the lexical overlap metrics (BLEU, ROUGE and CIDEr) and semantic similarity metric BERTScore. The human evaluation (the right panel) provide more fine-grained insights. COLDD achieves the best overall coherence, meaning that the generated y from COLD fits best with both the left-side context x; and the right-side context x, compared to the other methods. In contrast, DELOREAN excels only in terms of the left-side coherence (with x)), with inferior right-coherence (with x,)). We speculate this is because of DELOREAN's complex interleaving of forward and backward decoding passes that make it difficult to balance the left- and tigh--coherence constraints. In terms of grammaticality. unsurprisingly. LEFT-ONLY obtains the best score as it ignores any other constraints (and fails this task with low coherence scores). More importantly, COLD achieves a high grammaticality score along with its high coherence, substantially improving over DELOREAN. Example generations in Appendix [Table 7 show how COLD can reason with the right-hand context (e.g. 'no heels'), ) while DELORFAN'S generations are contradictory ('red shoes' vs. 'white pair') equivalent to those from LEFT-ONLY.
7

---

Min-Edi
Coherence
Coverage
Fluency
Models
Models
Overlap
Human
BERTS.
Human
Count
Percent
PPL.
Human
LEFT-ONLY
50.56
1.21
73.83
2.30
TSMH
2.72
71.27
1545.15
1.72
Mix-Match 36
85.07
65.20
NEUROLOGIC
3.30
91.00
28.61
2.53
COLD (ours)
4.24
94.50
54.98
2.07
Mix-Match, 36
84.79
66.03
DELOREAN
52.90
1.81
73,66
1.92
Table Results of lexically constrained decod- ing ( For keyword coverage, we report both the average number and average percent- age of constraint words present in the generated text. For language fluency, we use perplexity and human judgement.
COLD (ouns) 56,84
1.82
73.47
2.12
Table 2: Automatic and human evaluation of trivial
counterfactual story rewriting. method, LEFT-ONLY is coherent but fails on minimal-edit. COLD is superior to DELOREAN in terms of most metrics, including human evalu- ation.
4.2 Counterfactual Story Rewriting
Next, we consider counterfactual story rewriting E4] Given story context X with ending x,, the task is to generate a new story ending that is (i) similar to the original ending x,, yet (ii) consistent with new story context x (see Figure I example). The task challenging as it requires capturing the aspects of future events that are invariant under the new (counterfactual) context, while only making necessary edits for coherence.
4.2.1 The COLD Solution
To tackle this task, we use COLD with an energy composed of constraint functions that promote coherence with the new context x/, and minimal edits to the original ending X,:
E =x--- (f; + Xi (F) (i;
(8)
These constraints combine: (a) a soft fluency constraint (Eq. 3 conditioned on x; to promote coherence between the generation y and the new (counterfactual) context x/; a reverse LM constraint to improve fluency: (b) a n-gram similarity constraint (Eq. S {2, 3)) to encourage generating an ending y that is close the original ending x,.. We largely follow the configurations in $4.1 with some exceptions described in the appendix.
Baselines. Similar the setup in $4.1 decoding algorithm. As a reference, we also report the performance of a trivial solution, LEFT-ONLY, that generates a continuation of x without considering the minimal edit constraint with the original ending x,. Thus the method is expected to generate a coherent ending which however does not necessarily resemble the original ending. Finally, we compare with Mix-and-Match 33 a recent energy-based decoding method with discrete MCMC sampling, using BERT-base and BERT-Large.
compare with DELOREAN 45
a recent state-of-the-an
44 The original data contains three sentences in story ending. Due to computation constraints, we use the first sentence as the original ending and generate new single-sentence ending accordingly. Following 44 45 both automatic and human evaluation. For automatic evaluation, we measure BERTScore Minimal Edit which computes the overlap of text edits (insertion, deletion, replacement, etc.) 49
Evaluation. We the benchmark dataset TIMETRAVEL
we conduct
(58]. and
needed to produce the gold ending y. and the generated ending y, starting from the original ending Xy. We do not use other common metrics such BLEU since they were shown be ineffective 44 For human evaluation, each crowdworker is presented with the original story (xi, x,), the counterfactual condition x/, and the generated ending y. and the worker is asked to rate (1) the coherence of ý with respect to x/ and (2) the extent to which the generated ending preserves the details of the original ending x,, ("minimal edit"), ) on a 3-point Likert scale for 200 test examples. The average ordinal Krippendorff alpha is 0.52, indicating moderate inner-annotator agreement. We exclude aad-Match from human evaluation given the significant performance gap in automated
evaluation.
4.2.2 Results
Table shows the results of automatic and human evaluation terms of both minimal-edit and coherence. As expected, the reference method LEFT-ONLY that completely ignores the minimal edit constraint can easily generate new ending that is coherent with the new context x/- Compared to the baseline approach DELOREAN, our method COLD achieves overall superior performance, with
8

---

substantially improved coherence score and comparable minimal-edit score by human evaluation. Bix-anddMaathh based on discrete MCMC sampling, performs poorly. Intuitively, its discrete sampling tends to get stuck in mode of the target distribution (i.e., the region surrounding the original story ending), and struggles to explore further find samples of interest. COLD'S gradient- based sampling with continuous approximation leads to more efficient and effective exploration and mixing, as evidenced by samples that better meet the task requirements. See Appendix for examples.
4.3 Lexically Constrained Decoding
Next, we use COLD for lexically constrained decoding. Given a set of words W, the task aims to generate coherent sentence that contains these words (Figure IT). The task is challenging as it requires proper planning to coherently include the constraint words.
4.3.1 The COLD Solution
We specify an energy function of the following form:
Ely) x (iM X (y) fy; W) + feff (F:c(W)).
(9)
Specifically, this energy function incorporates: (a) a soft fluency constraint (F4. and reverse LM fluency constraint as in the previous tasks; (b) a 1-gram similarity constraint (Eq. 55) between the generation y and the given words W: (c) a future-token prediction constraint, where we concatenate the constraint words (in an arbitrary order), denoted as c(W), and use it as the right-side content x,, in Eq. Again we use similar configurations as in $4.1 More details can be found in appendix.
Baselines. We compare with a recent state-of sine--fftheearrr the-art method NEUROLOGIC [33,, a beam-search vari- ant specifically designed for lexically constrained generation which outperformed many supervised and unsupervised approaches in Lu et al. [3 We also report the results of TSMH [57] as another
recent baseline which uses Monte-Care Carlo Tree Search
5
Evaluation. We use the set of constraint words from the COMMONGEN corpus 29 but adopt the canonical setting that the generated text must contain the exact constraint words (e.g., write) instead of their variants (e.g., wrote) 18 47 Following previous works [8 47 57 we report a measure of constraint words coverage well as language fluency by evaluating the perplexity of the text. We also ask crowdworkers to rate the text fluency on a 3-point Likert scale on 200 test examples. The average ordinal Krippendorff alpha is 0.29, indicating fair inner- annottttt agreement.
4.3.2 Results
T shows the evaluation results for the lexically constrained decoding task. COLD, a general constrained decoding method, is comparable to the state-of-the-ar method NEUROLOGIC designed specifically for dealing with lexical constraints. In particular, COLD achieves higher coverage of given keywords, at the expense of generating slightly less fluent language. COLD is also substantially better than lexically constrained decoding method TSMH in terms of both coverage and fluency.
4.4 Additional Analysis
Ablation studies. We ablate two important ingredients of our approach, namely the con- straints and the top-k filtering. Due to space limit, we report the results of constraints and de- fer the results of top-k filtering to the appendix. Table shows the human evaluation results for ablations of the constraints used on the abduc- tive reasoning task (Eq. 77. The n-gram simi- larity constraint faim provides the largest con- tribution to the overall coherence. The reverse LM fluency constraint fi' also to some extent helps with the right-side coherence by condi- tioning on the right-side content x,. Removing the future-token prediction constraint similarly causes inferior scores in terms of right-side and
Left- coher. (x-y)
Right- coher.
Overall-
Gra- mmar
Models
coher.
(y-z)
(x-y-z)
COLD (Full)
4.17
3.96
2.88
2.83
COLD - fam
4.54
3.82
2.73
2.69
COLD - fim
4.35
3.97
2.84
2.80
COLD fred
4.61
4.07
2.75
2.77
Table 4: Ablation for the effect of different con-
straints in Eq. We do human evaluation 125 test examples. The best overall coherence is achieved when all the constraints are present.

---

overall coherence, as expected. Removing the individual constraints leads to better grammaticality due to less competition among different constraints, the cost of coherence. Our uniform treatment of all constraints as energy terms makes it straightforward to balance the different constraints by controlling the constraint weights.
Efficiency of COLD. We report the average runtime of generating one sample on the Coun- terfactual Story Rewriting data. The table be- low shows the results (on an NVIDIA Quadro GV100 GPU, batch size=32). We compare with Mix-and-Matee [6)] recent energy-basee de- coding method with discrete MCMC sampling (Metropolis- Hastiiss, in particular). COLD, which uses gradient-based based sampling, is faster than the gradient-fre
Method
Runtime (s)
COLD (GPT2-XL, 1.5B)
33.6
COLD (GPT2-M, 355M)
22.7
Mix Mix-aad--ee aud--Mtchh (BERTLarge, 340M)
33.5
Table 5: COLD is more efficient than gradient-free Mix-and-Match [6). The runtime shown is sec- onds per sample on Counterfactual Story Rewrit- ing.
fix-and-Matchh COLD is 30% faster with base LMs of similar sizes (GPT2-M and BERTLarge), and has roughly the same time cost when using much larger LM (GPT2-XL).
5 Related Work
Previous works proposed beam search variants for lexically constrained decoding 18
42 5%
which
enforce constraints discrete space. Recent works consider constraint satisfaction by adjusting vocabulary distributions using an additional discriminator or LM 6 24 56). Differing from those
approaches that determine the generation token by token auto-regressively, Qin et al. (5 optimize the whole (soft) token sequence via gradient propagation, which facilitates sequence-level semantic constraints (c.g., right-coherence, minimal-cdits)) COLD also samples complete sequences, while offering principled and unified formulation based energy-based modeling. Kumar et al. [26] extend 17 by imposing constraints with a Lagrangian method and optimizing for a single output with gradient descent. In contrast, generating samples for other utilities (e.g.. rank-and-seled $3.4 estimating expectations). We also introduce components for more fluent generations such the novel discretization procedure. Also, the empirical side, we explore a different class of problems and tackle them in the absence of labeled perform constrained decoding sampling in the discrete text space. Our
approach based energy- based sampling ($3.1) allows for
data. The recent CGMH 35 and TSMH (57] followed by 36 [E
with extended Gibbs sampling or Metropolis-Hasting energy-based formulation with gradient-based Langevin dynamics sampling produces substantially better results than the discrete TSMH ($.3) 3.33..55 Sha 47 uses gradient information to guide generation, which, however, is specifically designed for lexically constrained generation.
Energy-based models (EBMs) have been used for incorporating additional information to train text generation models 7 23 41 21]. In contrast, focus the constrained decoding (inference) that can be directly applied to pretrained LMs without fine-tuning. Langevin dynamics is widely used on
EBMs of modalities with continuous values, like images 48 9 59 3D shapes [55], latent features 39 and audio sequences (22). To our knowledge, we are the first to apply Langevin dynamics for
(constrained) discrete text generation (with continuous approximation) for efficient sampling.
Conclusion
We introduce COLD decoding, energy- based constrained text generation framework that press various soft/hard constraints through an energy function, and sample using Langevin dynamics. COLDD be applied directly to off-the-shelf LMs without task-specific fine-tuning. We showcase its flexibility and strong performance on three distinct applications of constrained text generation.
Acknowledgements
This work was funded in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) (funding reference number 4012333 401233309) DARPA MCS program through NIWC Pacific (N66001-19-2-4031) the Allen Institute for AI, and Microsoft Research PhD Fellowship. We thank the Xl.ab research group, and our anonymous reviewers for their feedback on this work. We also acknowledge the Beaker team (https://beaker) org) for their support with experiments.
10

---

References
[1] C. Bhagavatula, R. Le Bras, C. Malaviya, K. Sakaguchi, A. Holtzman, H. Rashkin, D. Downey, W.-t Yih, and Y. Choi. Abductive commonsense reasoning. International Conference on Learning Representations (ICLR), 2019. httlhttss////eeereeeet wew.eeeewwwweeeeeeeeeeeee..eeeee / forum ? B IHHKDD
[2] C. M. Bishop and N. M. Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[3] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, and Language models are fow-shot leamers. In Advances Neural Information Processing Systems (NourIPS), 2020. URL
nnis...pp....................... ttt wttp:/ppppee//////aaaaaaaaaee 143700a6bfcb4967222 (bt.le666771bbbboocco htt.:p/aaaaaaaaaaaaaaaaaaaaeeeaa html
[4] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, and M. P. et al. Evaluating large language models trained on code, 2021. URL https:: arxiv. org/ abs / 2107 03374
[5] R. Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International Conference on Computers and Games (ICCG). pages 72 83. Springer, 2006. URL https::/haat wtt.eeeeeeeeeeeeeeeeeeeeeeaaeltt docid/ 6992 / htt//ileenai www.eeeeeeeeeei.eei.....eeeeeeee CG2UU8 .pd
[6] S. Dathathri, A. Madotto, J. Lan, J. Hung, E. Frank, P. Molino, J. Yosinski, and R. Liu. Plug and play language models: A simple approach controlled text generation. In International Conference on Learning Representations (ICLR), 2019. URL
net mecet//ooumm.cc BeEEBBBDD
[7] Y. Deng, A. Bakhtin, M. Ott, A. Szlam, and M. Ranzato. Residual energy-based models models for text generation. In International Conference on Learning Representations, 2020. URL /tllhtttss///peeeeviee net/ forum? 1etp:ddd-1114siiie
[8] C. Donahue, M. Lee, and P. Liang. Enabling language models fill in the blanks. In Annual Meeting the Association for Computational Linguistics (ACI.), 2020. URL, https: attaaaccannntolggyyg. ogy.org/ 2020 ac ma in..i...222 2257
[9] Y. Du and I. Mordatch. Implicit generation and modeling with energy based models. Advances Neural Information Processing Systems (NourIPS), 32, 2019. URL hwt.:httppp//ppppae 2019/ 3782063bb dbb4e 341 4bde584c 4bdeaaaacbbbbeeeeeeeeeeeeeeeeee-
mips..p....... ps. cc/paper) Abst html
ni
[10] D. Duvenaud, J. Kelly. K. Swersky, M. Hashemi, M.. Norouzi, and W. Grathwohl. No MCMC for me: Amor- tized samplers for fast and stable training of ensrgy-based models. In International Conference on Learning Representations (ICLR), 2021. URL https: /ttp:::///ppnrrvveeee http://wwww....oooooo//ddddodaaa
[11] B. Eikema and W. Aziz. Sampling based minimum bayes risk decoding for neural machine translation. 2021. URL httptttss/stsii....p///aaa org/ abs/ 221c 04718
[12] A. Fan, M. Lewis, and Y. Dauphin. Hierarchical neural story generation. Anmual Meeting of the Associ- ation for Computational Linguistics (ACL), pages 889 8998 2018. URLAttpps:/aaelaathooooo
pTg/p18- TC82. .pā
[13] N. Goodman. The problem of counterfactual conditionals. The Journal of Philosophy, 44 128, 1947. URL httptttt///wwt
http:s//wwwww.dcee..co www.ppp.tgg.gooooooot wttp:////hh////o/e/aoeeeeeo iphil 1947 0044 0005
0113 0128
[14] K. Goyal, C. Dyer, and T. Berg-Kirkpatrick. Exposing the implicit energy networks behind masked language models via metropolis- arXiv preprint
2021.
[15] K. Guu, T. B. Hashimoto, Y. Oren, and P. Liang. Generating sentences by editing prototypes. Trans- actions of the Association for Computational Linguistics (TACL), 6:437-450, 2018 URL httpa!
[16] G. E. Hinton. Training products experts by minimizing contrastive divergence. Neural Computation, 2002. www.ttooogy..og/1-----11111
http:///wwwwww...ttoooooooo wintto// hbtp://ato/ooo........./ pdf
[17] C. V. Hoang, G. Haffari, and I. Cohn. Towards decoding as continuous optimisation in neural machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 146-156. 2017. htttlatts////i wtt.:s:://sclaattooooooo
org DD1001171111 1014/
[18] C. Hokamp and Q Liu. Lexically constrained decoding for sequence generation using grid beam search. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 2017. URL kttps://ant http:///ooaatthooo..ooooooo wtt.oooy.ooo//1111111 ppa
11

---

[19] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. In International Conference on Learning Reprexentations, 2019. URL bttps::/ net forum?id=ryg@Cyee
[20] Z. Hu, Z. Yang. X. Liang, R. Salakhutdinov, and Xing. Toward controlled generation text. In International Conference on Machine Learning (ICML), pages 1587-1596, 2017. URL http://
proceedings. mirr. htt.://.........../e.e.e./ee.. wwvess/v/0/null/ htt//////lllee T p
[21] Z. Hu, Z. Yang. R. R. Salakhuidinov. L. Qin, X. Liang, H. Dong, and E. P. Xing. Deep generative models with learnable knowledge constraints. Advances in Neural Information Processing Systems, 31, 2018.
[22] V. Jayaram and J. Thickstun. Parallel and flexible sampling from autoregressive models via Langevin dynamics. In International Conference on Machine Learning (ICML), pages 4807-4818 PMLR, 2021. URL
mw.. pzess wtt///s///13////13 htt.a:a........a.........aaaaaaa
html
[23] M. Khalifa, H. Elsahar, and M. Dymetman. A distributional approach to controlled text generation. In International Conference on Learning Representations, 2020.
[24] B. Krause, A. D. Gotmare, B. McCann, N. S. Keskar, S. Joty. R. Socher, and F. Rajani. GeDi: Generative discriminator guided sequence generation. In Conference Empirical Methods in Natural Language Processing (EMNLP) Findings, pages 4929 4952, 2021. URL
2021. 2a21.findii findi wannnngsen emn 1mmnlp.4244
[25] K. Krippendorff. Computing krippendorff reliiiiill patpscppppprr//3777
reliability, Departmental papers (ASC), page 43, 2007. upwnn
[26] S. Kumar, E. Malmi, A. Severyn, and Y. Tsvetkov. Controlled text generation continuous optimization with multiple constraints. Advances in Neural Information Processing Systems (NourIPS), 34, 2021. URL
https://openrevi
net. forum? d 7bbm-44
[27] Y. LcCun, S. Chopra, Hadsell, M. Ranzato, and F. Huang. A Tutorial on Energy- Chergy-Based Learning. Predicting Structured Data, 2006. ISSN <nulls, doi: 10.1198/tcc..208. s913. URL http://yann
Lecun. con./ htt///xdppppppe http://ublls... hstpss//ppfftt////o http:/dff/lcfce weee//eecnnn... påf
[28] A. Lee, M. Auli, and M. Ranzato. Discriminative reranking for neural machine translation. In Annual Meeting of the Association for Computational I Linguistics (ACL), pages 7250-7264, Online, Aug. 2021.
UtI.https:: //acl antho ogy ogy.org/20211111 acc lonq.
[29] B. Lin, W. Zhou, M. Shen, Zhou, C. Bhagavatula, Choi, and X. Ren. CommonGen: constrained text generation challenge for generative commonsense reasoning. In Conference on Empirical Methods in Natural Language Processing (EMNLP)- Findings, 2020. URL https:ppp/aaaaathhll.cooo org/
2020. findings-ennnnn emwsp.mmpp..... 1 65... pdf
[30] C.-Y. Lin. Rouge: package for automatic evaluation of summaries. In Text summarization branches ouf, pages 74-81, 2004. URL httttt:////acc antho ogg.lllg/yy.-0011100
[31] S. Lin, W. Wang, Z. Yang, X. Liang, Xu, Xing, and Z. Hu. Record-to-tex generation with style imitation. Conference on Empirical Methods in Natural Language Processing (EMNLP) Findings. pages 1589-1598, 2020. URL https://///t... odf
wwwp:/og..oog..oooo.ooo 2020. findings emnlp. 144
org/
[32] G. Liu, 7. Yang, T. Tao, X. Liang, 7. Li, B. Zhou, S. Cui, and 7. Hu. Don't take it literally: An edit-invariant sequence loss for text generation. arXiv preprint 1507 2021. [attst abs a2lp:/....///././//./../ 15078
/
[33] X. Lu, P. West, R. Zellers, R. Bras, C. Bhagavatula, and Choi. Neurologic decoding:(un) supervised neural text generation with predicate logic constraints. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 4288 4299, 2021. URL attps
/aclanthology. org/ 2021 maaaccl-mainn
[34] Y.-A. Ma. Y. Chen, C. Jin. N. Flammarion, and M. I. Jordan. Sampling be faster than optimization. Proceedings of the National Academy of Sciences (PNAS), 116(42)220881-2088 2019. URL attps
y wwww. pnas crg/
[35] N. Miao, H. Zhou, L. Mou. R. Yan, and L. Li. CGMH: Constrained sentence generation by metropolis- hastings sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6834 6842, 2019.
12

---

[36] F. Mireshghallah, K. Goyal, and T. Berg-Kirkpatrick, Mix and match: Learning-free free controllable text generation using energy language models. arXiv preprini arXiv:2203. 13299, 2022.
[37] R. M. Neal et al. MCMC using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(1
2011. URL
[38] E. Nijkamp, M. Hill, T. Han, S.-C. Zhu, and Y. N. Wu. On the anatomy of MCMC- maximum likelihood learning of energy-base baaed models. In Conference Artificial Intelligence (AAAI), volume 34, pages 5272-5280, 2020. URL hal.https://// //o/:://////eee aaai .org/ nndxx www.indxx..ppppieeee.eeep. httpp///alaaatttcee/e/eeeea//eee 5973
[39] B. Pang. T. Han, E. Nijkamp. S.-C. Zhu, and Y. N. Wu. Learning latent space energy-based prior model. Advances in Neural Information Processing Systems, 33:21994-22008. 2020.
[40] K. Papineni, Roukos, T. Ward, and W.J. Zhu. Bleu: method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL., pages 311-318, 2002. URL
org/ 1040. pdf
[41] T. Parshakova, J.-M. Andreoli, and M. Dymetman. Global autoregressive models for data-efficient sequence learning. In Proceedings the 23rd Conference Computational Natural Language Learning (CoNLL). pages 900-909, 2019.
[42] D. Pascual, B. Egressy, F. Bolli, and R. Wattenhofer. Directed beam search: Plug-and play lexically constrained language generation. arXiv preprint 0rXiv:2012.15416, 2020. URL https:
org/ aba/2012. 15416
[43] C. S. Peirce. Collected papers charles sanders peirce, volume 2. Harvard University Press, 1974, URL https: /www://wwwww. hup. ha www.aaaadddddaaaaaa.ee.ee wew./ataaaaaaaeeeeee php? sbn- 788077138800
[44] L. Qin, A. Bosselut, A. Holtzman, C. Bhagavatula, E. Clark, and Y. Choi. Counterfactual story reasoning and generation. In Conference Empirical Methods in Natural Language Processing (EMNLP), pages 5043-5053, 2019. URL httpp:://acaaneeee
org DD1 1509. pddf
[45] L. Qin, V. Shwartz, West, Bhagavatula, J. Hwang. L. Bras, A. Bosselut, and Y. Choi. Back to the future: Unsupervised backprop- decoding for counterfactual and abductive commonsense reasoning. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2020. URL https://acLantho..e org/ 2020. emnip main.
[46] A. Radford, Wu, Child, D. Luan, D. Amodei, I. Sutskever, et Language models are un- supervised multitask learners. OpenAl blog, 1(8)9, 2019, URL phttps://opeeee com/blog/ Cester-lanquaqe--ee
[47] L. Sha. Gradient-guided unsupervised lexically constrained text generation. In Conference Em- pirical Methods in Natural Language Processing (EMNLP), pages 8692-8703, 2020. URL https
//aclante mesllantholoyy.oo orrp://rg2200000 emnlp-main. 7C1 7c1.pdf
[48] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems (NourIPS). 32, 2019. URL https://prooediigg
heurips. http:p/.......pe////////////////
[49] M. Šošić and M. Šikić. Edlib: a c/c++ library for fast, exact sequence alignment using edit distance. Bioin- formatics, 33(9):1394 1395, 2017. URL htttps:://pbmmdd.ooem r.lm. www..w...........oo gwv 2000/28456683
[50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, F.. Kaiser, and
I. Polosukhin. Attention is all you need. In Advances Neural Information Processing Sys- tems (NourIPS), 2017. URL hntps: //tp:c/ee///eii///ee wew. heur.....rrrrg Awwttattttttttttt...... newrins........ ips. wttp:/ppppppp.////e..a
[51] R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-basee image description evaluation.
In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 4566-4575, 2015. URL https://arxiv. wrtg///////a///e//eeo http:://arrv..orr/aa///1.1...... .726
[52] S. Welleck, J. Liu, J. M. Han, and Y. Choi. Towards grounded natural language proof generation. MathAl4Ed Workshop at NewrIPS, 2021. URL https: httpst//mathaidee github. 1c/papers/
papers/parae pdE
13

---

[53] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference Machine Learning (ICML), pages 681-688, 2011. TtLL:tttps///mmccc///// papers/ icmipaper.por
[54] P. West. X. Lu, A. Holtzman, C. Bhagavatula. J. D. Hwang. and Y. Choi. Reflective decoding: Beyond unidirectional generation with off-the-shelf language models. In Proceedings of 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Longuage Processing (Volume 1: Long Papers), pages 1435-1450) Online, Aug. 2021. Association for Computational Linguisties. doi: 10. 1..18665v1/2221.ac
14. URL https://aclanthol.oo.
orq/2021 acl onq.l14
[55] J. Xie, Y. Xu. Z. Zheng. S.-C. Zhu, and Y. N. Wu, Generative pointnet: Deep energy- based learning unordered point sets for 3D generation, reconstruction and classification. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 14976-14985, 2021.
[56] K. Yang and Klein. FUDGE: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 351 1-3535, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2022////22211111 maacl-main.276 URL https: http://accaacclllll...oooo naacl main. 276
[57] M. Zhang, N. Jiang, L. Li, and Y. Xue. Language generation via combinatorial constraint satisfaction: A search enhanced monte-carlo approach. In Learning Meets Combinatorial Algorithms NeurIPS2020.
2020. URL httt:tttt///seeeeeee
net f dit
[58] T. Zhang, V. Kishore. F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation
bert. In International Conference on Learning Representations (ICLR), 2019. URL https
//openreviww net. wwwet//ooummdd-eeeeee Ltw.mmmmd---SeeeecEEE
[59] Zhao, Xie, Li. Learning energy-based generative models coarse-to-fine expanding and sampling. In International Conference on Learning Representations, 2021.
[60] Zhu, Z. Hu, and E. Xing. Text infilling. arXiv preprint
2019. URL https
//axxiv.ore abs/ 1901, 00158
14


Reviews:
[{"review_id": "wrXs0TloGXg", "reviewer": "Reviewer_zcYZ", "summary": "This paper proposes a generative model for constrained text generation. It incorporates hard and soft constraints in text generation and experiments on three tasks, lexically constrained generation, abducting reasoning, and counterfactual reasoning. The approach formulates generation as sampling from an energy-based model. Those constraints are applied as the weighted components for the score function. They introduced Langevin dynamics for efficient gradient-based sampling. \n", "questions": "I wonder how fast is the proposed algorithm? \n\nI am not 100% sure about the \"sample-and-select\". How many samples do you draw for one example? Any more detail about this sampling procedure?", "limitations": "I am mostly worried about the limited scope of comparison.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\nThe paper presents a novel approach to conditional/constrained text generation. Langevin dynamics has been explored in machine learning and computer vision but it’s novel to NLP and text generation. The approach can model different kind of constraints, hard or soft, in a unified approach. It is grounded with solid foundation and potential use cases. \nThe paper is well written and presented. It’s a fairly complicated topic, but examples and figures made the paper easier to follow. \nThe approach is validated on three NLG tasks, and the approach beats prior work like DELOREAN. Weakness \nAlthough the experiments cover three tasks, the scope of comparison is limited. It mostly compares with one baseline, Left-Only, and one model DeLorean. The author of this paper, DeLorean and NeuroLogic are essentially from the same research group, so I am slightly concerned about the credibility of the experimental results. It’s totally possible to compare with prior work using these 3 datasets. The authors can also find some other tasks like sentiment transfer or attribute grounded generation. The FUDGE paper and the Mix and Match paper are good references to look at.\n\n\nSome missing reference \nThe mix and match paper is directly relevant; the FUDGE paper (and many other discriminator based constrained NLG papers) are also worth mentioning and comparing with. \nFUDGE: Controlled Text Generation With Future Discriminators\nhttps://aclanthology.org/2021.naacl-main.276.pdf\nMix and Match: Learning-free Controllable Text Generation using Energy Language Models\nhttps://aclanthology.org/2022.acl-long.31.pdf\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SM7NhG6xKJq", "reviewer": "Reviewer_5VZW", "summary": "This work proposes a constrained generation decoding plug-in framework that does not require additional fine-tuning and can be applied to any left-to-right LM. The authors used sampling from the EBM model to incorporate constraints, and an energy function is specified by defining relevant constraint functions. One of the main novelties of the paper is that it suggests Langevin dynamics for gradient-based efficient sampling. That allows applying the framework to various constrained tasks. Experiments on three different tasks show that the proposed method outperforms previous works on constrained generation.", "questions": "Q1: Did I correctly understand that the constraint function must be handcrafted, and it has a significant impact on the performance? Do you have any suggestions on how to propose a new constrain function?\n\nQ2:  Efficiency of the sampling is claimed in the paper. However, there are no insights on the decoding speed/latency and comparison with other methods, if possible\n\nQ3: Is it possible to incorporate several constraints?\n", "limitations": "-", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Unified decoding framework for constrained generation\n- The method is well-motivated and described carefully\n- Shows solid improvements upon over methods\n\nWeaknesses:\n- The analysis of the results lacks some interesting aspects (see questions)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LB_C9-PutbR", "reviewer": "Reviewer_FCnJ", "summary": "Many applications of text generation require incorporating different constraints (hard or soft constraints) to control the semantics or style of generated text. The authors propose a decoding framework named Energy-based Constrained Decoding with Langevin Dynamics, which can achieve hard and soft constraints in a unified framework and does not need any task-specific fine-tuning. Experimental results on three tasks of constrained text generation show the effectiveness of proposed framework.", "questions": "What is the detail of obtaining a right-to-left LM?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The authors propose a decoding framework to unify hard and soft constrained generation. The framework treats constrained generation as an energy function and is flexible to add any constraints through arbitrary constraint functions.\n- The authors propose a new sampling method to achieve efficient search for a single optimal solution.\n- Experiments on three constrained generation tasks show that the proposed framework achieves strong performance on both automatic and human evaluation.\n\nWeaknesses\n- Consider that 2,000 iterations are required, the author should give the time consumed by the method to generate the text.\n- Considering that GPT2 itself may generate many candidates through top-k sampling, the authors should add a baseline, in which sample-and-select is performed on generated text of GPT2 by top-k sampling.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7FsfhfLud5U", "reviewer": "Reviewer_EAUi", "summary": "The paper proposes a new framework for energy-based constrained text generation. The proposed method implements text generation as gradient based sampling from an energy-based distribution, derived as a linear composition of the distribution defining each constraint. The method description is clear. Paper is well-written and usually easy to follow, although please see comments for suggested revisions. \n\nMethod is evaluated in three different tasks requiring constrained generation and obtains promising results in comparison to previous approaches. Comparison to related work could be more extensive and through. Overall, an interesting idea that should inspire other research directions in the related field. ", "questions": "Q1. Evaluation of results and comparison to baselines is a bit superficial. Did the authors do any analysis to see the differences in characteristics of the outputs generated by different models?\n\nQ2. Table 2: why are the BLEU results so low?", "limitations": "adequate", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Comments:\n\n- line 17-24: discussion and overview of the problem is introduced too generally, and the examples do not directly connect to the figure. A revision can make more clear how the method is directly applicable to each task. \n\n- line 25- 27: the discussion on supervised learning is too generic. The proposed method does not necessarily solve the annotation scarcity. There also many applications where it is feasible to adapt / finetune models with domain specific annotations\n\n- line 100: token \"next\" used unnecessary and confusing. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept