Controlled LLM Decoding via Discrete Auto-regressive Biasing
Authors: Patrick Pynadath
Ruqi Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: LLMs
controlled decoding
MCMC
Primary area: foundation or frontier models, including LLMs
TL;DR: Introduces a decoding algorithm to guide LLM auto-regressive generation towards text that satisfies some given external constraint.

Abstract: Controlled text generation allows for enforcing user-defined constraints on large language model outputs, an increasingly important field as LLMs become more prevalent in everyday life. One common approach uses energy-based decoding, which defines a target distribution through an energy function that combines multiple constraints into a weighted average. However, these methods often struggle to balance fluency with constraint satisfaction, even with extensive tuning of the energy function's coefficients. In this paper, we identify that this suboptimal balance arises from sampling in continuous space rather than the natural discrete space of text tokens. To address this, we propose \emph{Discrete Auto-regressive Biasing}, a controlled decoding algorithm that leverages gradients while operating entirely in the discrete text domain.
Specifically, we introduce a new formulation for controlled text generation by defining a joint distribution over the generated sequence and an auxiliary bias sequence. To efficiently sample from this joint distribution, we propose a Langevin-within-Gibbs sampling algorithm using gradient-based discrete MCMC. Our method significantly improves constraint satisfaction while maintaining comparable or better fluency, all with lower computational costs. We demonstrate the advantages of our controlled decoding method on sentiment control, language detoxification, and keyword-guided generation.

Published as a conference paper at ICLR 2025
CONTROLLED LLM DECODING VIA DISCRETE AUTO-REGRESSIIE BIASING
Patrick Pynadath, Ruqi Zhang Department of Computer Science Purdue University
West Lafayette, Indiana, 47906, USA (ppynadat, ruqiz @purdue edu
ABSTRACT
Controlled text generation allows for enforcing user-defined constraints on large language model outputs, an increasingly important field as LLMs become more prevalent in everyday life. One common approach uses energy-based decod- ing, which defines a target distribution through an energy function that com- bines multiple constraints into a weighted average. However, these methods often struggle to balance fluency with constraint satisfaction, even with exten- sive tuning of the energy function's coefficients. In this paper, identify that this suboptimal balance arises from sampling in continuous space rather than the natural discrete space of text tokens. To address this, we propose Discrete Auto-regressive Biasing, a controlled decoding algorithm that leverages gradients while operating entirely in the discrete text domain. Specifically, we introduce a new formulation for controlled text generation by defining joint distribu- tion over the generated sequence and an auxiliary bias sequence. To efficiently sample from this joint distribution, we propose a Langevin-within within--iibss sam- pling algorithm using gradient-based discrete MCMC. Our method significantly improves constraint satisfaction while maintaining comparable or better fluency, all with even lower computational costs. We demonstrate the advantages of our controlled decoding method sentiment control, language detoxification, and keyword-guided generation. We make our code available at the following url: http:://gghtttb.cccooooottttttat
INTRODUCTION
Large language models (LLMs) are widely used in real-world applications through chatbots such as ChatGPT, Alpaca, and Llama, making them an important part of everyday life (Bender et al. 2021
Bommasani et al. 2021 Weidinger al., 202T). As a result, there has been growing attention
developing methods to reliably and effectively control LLM- LLM-generated outputs to meet user-defined
constraints (Gehman et al. 2020 Dathathri et 2020 Goshvadi et al.) 2023 Han al. 2024).
Previous work has tackled controlled language generation using decoding-time time algorithms, which bypass the need for fine-tuning the base language model (Liu et al. 2023a Kumar al., [022 Mireshghallah et al.) 2022; Dathathri et al. 2020 Qin et al. 2022). Among these, based decoding methods define a target distribution through an energy function, combining multiple con- straints into weighted average. This formulation offers significant flexibility, as the energy function can be any arbitrary function. Sampling from this distribution relies on gradient craddent-based MCMC in continuous spaces, followed by conversion back to discrete text tokens.
However, these methods often struggle to achieve good balance between fluency and constraint satisfaction. Tuning the coefficient for each constraint in the energy function requires exhaustive ef- forts, and even with careful tuning, the generated outputs may still fail to meet the desired standards.
In this paper, analyze this issue and show that the suboptimal balance arises from sampling in continuous space rather than the natural discrete space of text tokens. Continuous-space sampling leads to incremental sequence updates, which hinders the exploration of fluent and constrained text.

---

Published as a conference paper at ICLR 2025
P(Toxic(Y') 98%
P(Positive|Y) 2%
Count(Keywords Y)
During show in Sydney in 2013, Tyler tirade against member of the group, saying was inn racist...
The chicken is good source of protein, but it's great source of fat.
Once upon time the world was place of great beauty and great danger.
Contimuous Logits
Base LLM
LLM Logits
Discrete Aulo- Regressive Blasing DAB
Response
New Response Y Yeww
Yuu
Constraint Function
Gradient-Ban Banee Discrete Sampling
Discrete Bins Tokcas
P(Toxic Y) = 3%
P(PositivelY) - 98%
Counl(Keywond c
During a show Sydney in 2013, Tyler went on tirade against member of the group, who was considered to be his friend.
The chicken is great way to get the kids hooked on delicious, healthy food
Once upon a time, there was small group of officers who were in charge of the modern infantry and logistics.
Figure 1: Visualization of our proposed controlled decoding algorithm, Discrete Auto- uuto-Regressive Biasing (DAB). Given an initial response that fails satisfy some external constraint, DAB steers auto-regressive generation towards satisfactory generations using discrete bias tokens obtained via gradient-based discrete sampling from the constraint function.
To address this, we propose DAB: Discrete Auto-regressive Biasing, a decoding algorithm that ef- ficiently explores the discrete text space, achieving better balance between fluency and constraint satisfaction with even lower decoding costs than existing decoding-time algorithms. DAB oper- ates entirely in the discrete domain, which allows for more directional and substantial updates the response sequence at each step. Specifically, our method defines target distribution as a joint distri- bution over response sequences and an auxiliary bias term B. alternates between sampling from P(B Y)) using gradient- brased discrete sampling, ensuring constraint satisfaction, and from P(Y using biased auto-regressive generation, ensuring fluency. Notably, DAB significantly reduces com- putational overhead due to simpler gradient computation which is enabled by our discrete sampling framework. We provide a visual summary of our algorithm in Figure We summarize our main
contributions follows:
We propose controlled auto-regressive decoding algorithm that leverages gradients while operating entirely in the discrete text domain without continuous relaxation or post-hoc discretization. By remaining in the discrete domain, our method achieves better balance between fluency and constraint satisfaction, with significantly lower computational costs than existing energy-based based decoding algorithms.
We introduce new formulation for controlled text generation by defining a joint distri- bution over the generated sequence and an auxiliary bias sequence. To sample from this joint distribution, we propose a discrete Langevin-within-Gibbe sampling algorithm. Our algorithm leverages discrete gradient- bastentt-bbeedddeee MCMC to sample the auxiliary bias sequence, which is then incorporated into the final output via biased auto-regressive generation.
We demonstrate that our method consistently produces more satisfactory generations than baseline methods, offering comparable or better fluency with 2x faster decoding time. This performance holds across a range constrained generation tasks, including sentiment- controlled generation, toxicity avoidance, and keyword- generation.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
2.1 LANGUAGE MODELS EBMS
Energy-based models (EBMs) have been popular framework used to study the problem of inference-time controlled text generation. Initial works focused on the application of Gibbs sampling to encoder-based architectures such as BERT (Wang & Cho 2019 Goyal al. 2022 Mireshghallah al. 2022).
While these methods are similar to ours in that they perform discrete sampling, our work differs in several key ways: (1) Our work leverages gradient information for more informed exploration of the energy landscape. (2) Our Gibbs sampling alternates between the response and bias, updating the entire response at once. whereas previous Gibbs sampling approaches only update a single token at a time. (3) Our method leverages auto-regressive generation within decoder architectures, whereas these works can only be used with encoder architectures.
More recent works have applied gradient-base based sampling methods to more efficiently navigate the energy landscape. QQn et al. (2022) that uses Langevin dynamies in the logit space followed by top-k masking; Kumar et al. (2022) uses Langevin dynamics in the embedding space followed by projection the embedding space; and Liu Liu
al. (2023a) uses the ADAM optimizer with Gaussian noise to obtain bias terms for biased auto-regressive generation.
Our approach differs from these works in that it relies on discrete sampling rather than continuous sampling or optimization. By operating directly in the natural discrete space of text tokens, our method not only improves the trade-off between fluency and constraint satisfaction but also reduces decoding costs.
Beyond MCMC methods, also possible to apply some modified rejection sampling to proposal algorithm to improve approximate sampling of the target EBM (Eikema et al. (022). Additionally, there is also line of research devoted fine-tuning LLMSs to align with an EBM defined in terms of the base LM and external constraints (Khalifa al. 2020 Korbak et al. 2022; Kruszewski al, 2023). We choose to focus on inference-time algorithms as they provide more flexibility and avoid long training runs.
2.2 ALTERNATIVE CONTROLLED GENERATION APPROACHES
While the EBM framework is popular within the field of controlled text generation, there are several alternative approaches. In terms of previous inference time algorithms, many works rely on specially trained auxiliary models to provide token-level guidance (Krause al., 2020 Yang & Klein [021] / Meng et al., 2022 Kim et al. (002) or query standard text classifier multiple times per generated token (Dekoninck et al. 2024; Sitdikov et al., (022]. Other works apply gradient-
Liu et al. [021
based optimization methods to improve constraint satisfaction (Qin et al. 2020: Dathathri et al.
2020
Recently, Han al. (2024) introduces LM-Steer, method for learning linear transformations to influence language model generation. This is similar to the BOLT algorithm [iu et al. (2023a) as both alter the embedding representations from auto-regressive generation by some learned operation. Whereas Liu et al. (2023a) is framed as a decoding-time algorithm, LM-Steer requires training data in order to learn the linear transformation. In general, these methods either suffer from the steep trade-off between fluency and control previously mentioned, require separate training or fine-tuning for the constraint LM, or are computationally expensive.
2.3 GRADIENT- -AADD DISCRETE SAMPLING
Recent works have demonstrated the benefits of leveraging gradient information for sampling over discrete spaces (Grathwohl al. 2021 Sun al. 2023a Goshvadi al.) 2023 Sun al. 2023b
Pynadath et al. 2024). Our method uses the sampling algorithm introduced in Zhang et al. (2022)
which can be viewed the analog of Langevin dynamics (Roberts & Stramer 2002) in discrete spaces. Our paper may be of interest to this field as is the first to explicitly link discrete gradient- based sampling with controlled language model generation.
3

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES
Task Definition Let PLM denote some pre-trained language model, V denote the set of tokens in the model vocabulary, and | - R represent external constraint where higher values correspond to better constraint satisfaction. We will use Y (311 3/2 yn to refer to the sequence
of tokens in the response.
We assume PLM is an auto-regressive transformer as used in previous work QQin et al. 2022 Kumar al. 2022 Liu et al. 2023a). Given some initial prompt X, we define the auto-regressive distribu- tion for the ith position as Vi = pL.M (-Ivei. X).. This forms a distribution over the vocabulary V and can be represented as a |V| dimensional logit vector.
We define the task of controlled language generation as generating a sequence of n tokens Y = {y1, 3/ from some initial prompt = {I1, 22.... .F that is both high in likelihood under the language model and high in terms of constraint satisfaction. We can compute the likelihood of generation under the pre-trained language model for a sequence length n as follows:
PLM (Y|X)= ITc. PLM (ydlv<s X).
Previous works have framed this problem as sampling from an unnormalized distribution, commonly referred to energy based model (EBM) (Mireshghallah et al., 2022 Qin et al. 2022 Kumar ct al., 2022 Liu et al. 2023a). The energy function defines an unnormalized distribution and is
typically defined as follows:
E(Y) A log PLM (YIX) + (2f(X
(2)
Non Augr-segresvvv Generation As the denominator, partition function, requires computing the energy of all possible sequences, it is intractable to directly sample from т. Previous works address this by applying Langevin dynamics it only requires gradients of the energy function E
(Qin et al. 2022 Kumar et al.) 2022). Specifically, they use some continuous representation of the current sample Y, and a learning rate to define the following update step:
Y141 = + + +e, N(0,0N
(3)
Non-autoregressive generation methods typically rely on some form of filtering or projection to ensure that the continuous generation can be mapped back into the discrete token space V. In Qin ett al. (2022), the final generation is filtered using top-k mask, where the top-k indices are obtained from the base language model PLM In Kumar et al. (2022), their proposed algorithm performs the update in the embedding space and projects the resulting vector onto the set of token embeddings for the base language model.
Biased Auto-regressive Generation Liu al. (023a) introduced method that samples bias term from the target distribution and incorporates it into auto- outo-regressive generation. While the sam- pling step of the bias-term is similar equation 3 in the embedding space, they skip the projection step of Kumar et al. (2022) and modify the auto-regressive step as follows:
1/4 arg max (ūe. + wi (b,M")) jElV
Here, i is the sequence position, Tt is the initial logit vector, M is the embedding table for the lan- guage model PLM, b; is the bias vector in the embedding space, my is the weight value, and refers to the logit coordinate. b, TT refers to the transformation of the bias vector by to a logit vector. Liu et al. (2023a) demonstrates that sampling bias term to direct auto-regressive generation enables quicker convergence to satisfactory generation, as opposed to non auto-regressive generation. How- ever, they also mention the undesirable trade-off between control towards constraint satisfaction and fluency, which our work addresses.
4 DISCRETE AUTOREGRESSIVE BIASING
In this section, we introduce DAB: Discrete rego-segressive Biasing. First, we present the formula- tion of the target distribution joint distribution and explain the motivation behind this approach.
4

---

Published as a conference paper at ICLR 2025
Next, describe how our algorithm samples from the joint distribution by alternating between bi- ased auto-regressive generation and discrete gradient-basee sampling. We finally demonstrate that suddentt-bssed discrete sampling enables our algorithm to have more thorough, stable, and efficient sampling when compared to continuous methods.
4.1 FORMULATION
Our goal is to formulate the target distribution in such way that enables both auto-regressive biasing and discrete sampling, allowing for superior exploration of the discrete space of fluent and controlled responses. However, existing formulations will not allow for both: while Kumar et al. (2022) in- troduces a framework that allows for direct sampling of word embedding, is non auto-regressive; and while Liu et al. (2023a) introduces a method that allows for auto-regressive generation, the bias vectors used are continuous. Thus here we introduce new formulation.
Target Discrete Distribution In order to ensure the constraint satisfaction of the output sequence Y. we introduce an auxiliary variable {bi, br..b.............. ... b, where each b; E V. We refer to this sequence the bias sequence or bias tokens. First, define the joint distribution over Y, B condi- tioned on the prompt X:
B(X) & PLM (L ex
(5)
Here, X(BIX) represents the constraint, with larger values of [(B] indicating better satisfaction of the constraint. PLM refers to the language model distribution conditioned on X, B.
Marginal Distribution X(Y
The marginal distribution of is:
E B) exp(f(B(X)) BEIVI"
X(Y Σ B)((X)) BEIVI"
ZB
This formulation expresses the probability of obtaining the response Y, ttaking into account all pos- sible biases B. The response drawn from this marginal distribution will be both fluent (due the The bias vari-
term PLM (Y),, (I,XB) B)) and highly satisfying of the constraints (due to exp(f(B(X)) able B helps balance these two aspects. the distribution over incorporates both Pt.M and the external constraint, probable sequences under this distribution will be both fluent and satisfactory. This ensures that the generated response from our algorithm has the desired properties.
Marginal Distribution P(B The marginal distribution of B is given by:
P(B[X)- cx)(f(B(X))
(6)
ZB
This indicates that highly probable values of B will satisfy the external constraint. However, since the distribution does not incorporate the language model pLAM the sequence of B may not be fluent. B "guide" sequence opposed using the response.
For this reason,
Conditional Distributions Given the joint distribution defined in equation 5 the conditional dis- tribution of Y is B, PLM (YIX, B). Furthermore, the conditional distribution of B is:
P(B[XYY) PLM (LIMY,X,, B) (BB((((()) (YM(Y ()xxx()) X(
Advantages of Joint Distribution The primary motivation behind our framework is the observa- tion that fluency is best satisfied through auto-regressive generation, and gradient-basee sampling efficiently finds responses that satisfy constraints. By framing the problem joint distribution of Y. B, we enable the use of both methods. The typical target shown in equation equation 2 does not support auto-regressive generation and significantly compromises fluency.
5

---

Published as a conference paper at ICLR 2025
4.2 SAMPLING ALGORITHM
Sampling directly from the joint distributions of Y, B is challenging. We propose to use Gibbs sampling to sample from the target distribution by alternatively sampling from P(Y X, X, B) and PX First, we describe how our proposed algorithm samples from each conditional distri- bution. We then introduce the complete sampling algorithm along with some intuition to how ensures satisfactory and fluent generations.
Sampling from P(B]X,Y) The conditional distribution for P(B]X,Y) includes the term PL)(
(Y]X, B). This to ensure that the sampled B results in output sequences that high in likelihood under the base model distribution. However, directly computing PLM (Y|.X, B) for all possible values of B intractable. By noting that this term encourages the selection of B that is consistent with the observed Y. we can infer this property is naturally satisfied when B is close to Y. Thus, we approximate PLM (Y|X, B) by performing a single MCMC step with the initial state set to Y.
In order to sample this discrete sequence of tokens, we apply the Discrete Langevin Proposal (DLP) introduced in (Zhang et al. 2022). After initializing B as B = Y, and representing the sequence as sequence of one-hot vectors 1 {bb, 62.,bb.... b..J..... we execute a single step of DLP with the target distribution being cxp(f(BI Below we include the proposal distribution for position i:
6/ ~ Categorical
(ootmaax 'EIV
(
Here, is temperature hyper-parameter that controls the sharpness of the proposal distribution, (Vf(B(X)),, is the jtth component of the ith gradient vector, bú.J represents the jth component of the one-hot vector b, and by is the token we sample from the distribution over V. For more details on the application of DLP and the gradient compution, see Appendix[A] Unlike the algorithm presented in Liu et al. (2023a), we do not require the use of straight through estimation (STE) (Bengio et al. 2013) as we differentiate directly with respect to Y. Note that this proposal function can be computed for all sequence positions in parallel. We refer to this proposal function as (-1B).
Sampling from P(Y B) Our goal is to sample from P(Y X, B) using biased auto-regressive generation, similar to equation 4 In order to do so, we must map the sequence of bias tokens B to a sequence of bias vectors B. The ideal bias vector should reflect the difference in meaning between each token in the vocabulary space V and the sampled token. To accomplish this, penalize each token based on the distance to the sampled bias token within the embedding space, as static embeddings reflect semantic meaning (Mikolov 2013 Pennington et al., [2014] Mikolov al.) 2013). Given a bias token b,, embedding table M, we defne the jth coordinate value corresponding to token vy as follows:
b = M6; - M0y13
This yields a |V| dimensional vector that can be added to the auto-regressive logits ÿi- When adding the bias term to yi, we also incorporate both a weight term w, and a normalizing factor ria While w, is a dyyer-parameted r; normalizes the bias vector at the i position to have the same norm ye. We define the normalizing factor as follows:
1 |
T4
We note that while this normalizing factor can also be applied to BOLT and may improve its results, the modified BOLT still underperforms compared our method.
We formally define our biased auto-regressive generation as follows:
V/ arg max jE[V]
Ui.j
bis))
(10)
Intuitively, this returns the token corresponding to the maximal coordinate of the biased distribution. Repeating this times results in the updated response sequence Y.

---

Published as a conference paper at ICLR 2025
Gradient-bused Discrete Sampling
Discrete Bias Sequence D
P(BY)
B
B++
Compute distribution to sample bias sequence
map bias tokens to bias vectors
softmax(( (V bru)) JC|V
Mdb Me
Discrete Response
DaCOc
pLM
y1+1
Sequence y
Auto-Regressive Biasing
Visualization of the proposed decoding algorithm, DAB. DAB alternates between sam- pling the response and the bias B. To sample B given Y, we use gradient- based discrete sampling on the constraint function f. To sample Y given B, we compute bias vector that penalizes words based on their distance to and then use this bias to guide the auto-regressive generation.
Figure 2:
Text Generation Algorithm We provide visualization of our algorithm in Figure 2 and include the full algorithm in Appendix B Given some prompt, we first generate some initial auto-regressive generation Yi, with the initial bias vector set to Ô. After obtaining Y, we sample from the conditional distribution over B to obtain sequence of bias tokens. We then use equation to compute the new bias vector to use for biased auto-regressive generation. We repeat this alternative sampling process for several iterations, returning the sample that best satisfies the constraint the end commonly For a discussion on the
done the literature (Kumar et al. 2022 Liu et al. 2023a of our algorithm, see Appendix
4.3 ADVANTAGES OF BIASING DISCRETE SPACES
Here discuss various advantages of discrete sampling in the context of auto-regressive biasing- First, we demonstrate that discrete sampling enables a quicker and more thorough exploration of potential output sequences Y. We then describe how discrete sampling solves the stability issue discussed in Liu et al. (2023a). Finally, we show that discrete sampling makes use of simpler gradient computations, resulting more efficient decoding algorithm.
Exploration of State Space Discrete sampling enables DAB to explore the output space more ef- fectively than continuous methods. We hypothesize that discrete sampling enables more directional and substantial changes to the bias vector, resulting in more token changes in the output sequence across sampling steps. We compare with BOLT, a continuous auto-regressive biasing algorithm (Liu et al. 2023a). We examine the performance of BOLT both with and without the normalizing factor defined in equation We include the comparison of hops across 50 steps in Figure These results show that our method updates substantially more sequence positions across all sampling steps than either variant of BOLT.
Next, measure how comprehensively each method explores the sample space of potential quences. For each sequence position, we maintain record of tokens encountered throughout the sampling process and compute the number of unique tokens within this set. Figure 3p shows the average unique tokens per sequence position for all three algorithms. These results indicate that our method samples more unique tokens for each sequence position than either variant of BOLT, demonstrating more comprehensive exploration. Collectively, these findings confirm that discrete sampling enables faster, more thorough, and thus more effective exploration of the sample space of potential sequences.
Sampling Stability Discrete sampling allows DAB to have superior stability across sampling steps when compared continuous methods. We show this in Figure 3 where track the average

---

Published as a conference paper at ICLR 2025
17.5
50
35
BOLT (OLL
DAB (Duns)
15.0
30
25
10.0
80-
PPL
7.5
Unique
20
5.-
15-
BOYT (tonern)
10
BOLT (Nomo
10
(Uzn)
20
40
BOLT (Unnorm)
BOLT (Nerm)
DAB (Oun)
40
Sampling Step
Sampling Step
(a) Hops per Sample Step
(b) Avg. Unique Tokens
(c) Perplexity per Sample Step
Figure 3: (a) Average hops, token updates per sequence, against sampling steps. Both versions of BOLT suffer from decreasing hops while DAB remains stable. (b) Average number of the unique tokens sampled for each sequence position throughout the entire sampling process. DAB discovers many more unique tokens for each position than either variant of BOLT. (c) Comparison of fluency with respect sampling steps. Dab exhibits stable fluency over sampling steps comparison to BOLT.
perplexity of the batch each time step. While BOLT faces deteriorating perplexity, DAB remains stable throughout the sampling process.
We attribute this instability to the difficulty of applying continuous sampling techniques discrete domain as discussed in Grathwohl et al.] (202T). As result of BOLT's misalignment between the sampling domain and target domain, the energy landscape is complex navigate with gradient information. This results in the divergence seen in Figure B
Our algorithm avoids this entirely as we perform direct sampling on the discrete token space. Since we define the sampling domain and target domain to be the same, our algorithm enjoys supe- rior stability throughout all sampling steps. This improvement in algorithmic stability removes the need for implementing early-stopping and to carefully tune the number of sampling steps.
Improved Efficiency Discrete sampling en- ables our algorithm use simpler gradient computations that provide computational ad- vantage over continuous sampling methods. To evaluate our algorithm's 's efficiency, we compare the tokens per second of our method to BOLT. We also measure the time-cost for computing the bias term for each method. We put the results in Ta- ble where we observe that our method has over 2x the tokens per second output when compared against BOLT. Our algorithm achieves this computational advantage result of computing the gradient with respect ß Y. which removes the need backpropagate through auto-regressive generation. Computing the gradient of f with respect continuous bias term B requires first computing O. 0 and then 0Ý /aB. Since each one-hot vector in Y is influenced by previous bias terms, the latter term requires backpropagation through ruto-segreseive generation. Simply initial- izing B- = Y will not work in continuous sampling because the incremental updates will keep B close to the original Y. In contrast, our method uses gradients identify which tokens will increase constraint satisfaction and directly samples them, enabling substantial change from the original se- quence while incorporating information from the external constraint. While continuous sampling cannot exploit this computational shortcut and maintain constraint satisfaction, gradient-bar based dis- crete sampling achieves both simultancously.
Table 1: Efficiency comparison between BOLT and DAB in terms of tokens per second.
BOLT
DAB (Ours)
Tokens per Second
9.495 .095 23.213 0.304
EXPERIMENTS
Tasks We evaluate DAB three distinct controlled- -eeerratdddggneetiiioinn tasks: sentiment-guie genera- tion. language detoxification, and keyword-guide generation. These are popular tasks within the
field of controlled generation (Kumar et al. 2022 Liu et al. 2023a Han et al. 2024). For all tasks,
8

---

Published as a conference paper at ICLR 2025
Sampling algorithm performance on sentiment-directed generation, language detoxifica- tion, and keyword-guider generation. DAB acheives superior control metrics than baselines across all task. while also demonstrating comparable or better fluency metrics competitive with the best baseline.
Table 2:
Control Clep())))
Fluency
Sentiment
Inr. Clsf T
Ext. (STT-22)
CoLA f
REP-3gram
PPL
MuCOLA
841 .000
843 .011
.899 +.088
.881 + .008
.091 + +006
34.786 + 2.205
COLD
897 + 011
515= .015
.670 + 013
.731 + .008
.06 + .003
15,908 = .391
BOLT
.006
LM- AMeSteerr
900 .008
.48 1 006
564 1. .008
.177 1 .007
72.153 1 3.195
DAB (Ourz)
992
.955
860
Toxicity
Jnt. Clsf 4
Avg. Max Taxicity
Taxicity Pred. Prob.
CoLA 1
REP-2gram
PPL 4
MuCOLA
098 .002
7.6%
58.015
COLD
136 + 000
266 .007
10.2%
.667 +.001
.024 .001
38.891
.177
BOLT
065 0001
6. 6.8%
+ .001
.011
.0001
27. 283
2,333
LM-Steer
2615 006
7.9%
.T22 + 002
.006 = 0002
52.697
.356
DAB (Ours)
057 .001
211 .006
6.8%
806 1. .001
.001 .0001
65.
.126
Keyword
BertScore
Success Rate
CoLA 1
REP
PPL
MuCOLA
8083 0000
100%
.488 +. .001
.007 : .001
475.301
30.445
COLD
.8233 1.0005
100%
.005 1. .003
.020 1. .001
241.980
4.943
BOLT
.201 .0003
99.1%
.05 + .006
.005 + .005
32.019
1.503
DAB
8303 .0003
99.0%
+ 000
.004 0001
23.424 .377
produce generations in batches 20 done Liu al. (2023a). For all tasks, we include example generations in Appendix [D.2 D.3 D.4 for sentiment directed generation, language detoxi- fication, and keyword-guided generation respectively.
Baselines
We compare previous generation algorithms that use the EBM framework to perform gradient-based text sampling. Specifically, we compare to MuCOLA (Kumar et al. 2022). COLD
(Qnn al., 2022). , and BOLT (Liu al., 2023a). We also compare against LM- Steer introduced in
Han et al.
2024 to see how our method compares to alternative controlled generation methods.
Metrics While the metrics assessing control towards external constraint vary across experiments, we use the same evaluation metrics measure fluency across experiments. We measure fluency by looking at CoLA score, the number of repeated tri-grams per generation, and perplexity (Kumar jet al. 2022 Liu et al. 2023a). For CoLA score, we use a fine-tuned RoBERTa model to provide GPT-XL
probability to whether a generation is grammatically correct. For perplexity, to evaluate each generation. We show the average of the results across all generations. For more details these evaluation methods, refer Appendix D.I
5.1 SENTIMENT- COOTLLEEEN-COTTOOLEEDD GENERATION
Task Here we measure the ability direct generation towards positive sentiment from some initial prompt. Given an initial prompt, we generate sequences of length 12, 20, 50 as done in prior works (Kumar al.) 2022, Liu al. 2023a). We use the same set of prompts as Dathathri al.) (2020)
and include them in Appendix D.2
We evaluate control by measuring the predicted sentiment of the generation using three distinct sentiment classifiers. For details on the training of the three classifiers, see Appendix D.2 We omit the internal classifier measure for LM-Steer as it does not rely on an internal classifier to guide generation.
Control Metrics
Results Table shows that our method achieves a better balance between control and fluency than baselines. DAB achieves the highest average probability of positive sentiment across all three clas- sifiers, demonstrating its effectiveness at incorporating the external constraint. Furthermore, DAB achieves fluency scores close to BOLT's performance in regards CoLA score, repeated trigrams, and perplexity. This shows that DAB produces generations that are both fluent and satisfactory under the constraint.

---

Published as a conference paper at ICLR 2025
5.2 TOXICITY AVOIDANCE
Task We compare our algorithm to various baselines for the task of language detoxification to demonstrate that our method be used mitigate potentially toxic LLM generations. Following prior work, we use 1,000 prompts sampled from the RealToxicityPrompt introduced and generate continuations of length 20 tokens (Gehman et al. 2020 Kumar et 2022 Liu et al.) (023a).. As in the antiment-ddiectte generation task, we use a line-tuned RoBERT T as the constrant function.
Control Metrics We evaluate the generations using both the internal discriminator used to guide the various methods, and the score returned by the Perspective API (Lees et al. 2022). We use the scores returned from Perspective API to calculate the maximum toxicity per prompt and the overall percentage of text predicted to be toxic.
Results As shown in Table our method generates less toxic text than baselines without compro- mising fluency. DAB significantly decreases the average maximum toxicity per prompt, demonstrat- ing that our algorithm is more consistent in terms of toxicity reduction. Furthermore, our method obtains fluency metrics that are on par with the best baseline.
5.3 KEYWORD-GUIDED GENERATION
Task We measure the ability of our method produce text that includes a given keyword relevant to a specific topic. As done in prior work, we use 7 topics with 4 keywords each and use the differentiable BLEU (Liu al., 2022) as the constraint function (Liu al. 2023a). For baselines, we compare to the same methods as before with the exclusion of LM-Steer as no similar task was discussed in the original work.
Control Metrics The ideal metric goal for this task should only assign good scores to text where keywords are used in a meaningful way. While Liu et al. (2023a) uses the percentage generations that include a keyword to measure constraint satisfaction, this metric also assigns good scores to text where the keyword does not add meaning to the sentence.
An alternative approach is to compute similarity score between the generations and reference texts that use the keywords in a manner relevant to the given topic. To accomplish this, we use GPT-4o to produce sentences for each combination of prompt, topic, and keyword. We then use BertScore compute the similarity score between the candidate generations and the reference text, where higher similarity implies increased relevance to the target topic (Zhang al.) 2020). For further details, refer to Appendix D.4
Results DAB able to outperform baselines in terms of both control and fluency, reflecting an improved balance between these two attributes. As shown in Table DAB outperforms baselines in terms of relevance towards the target topic as measured by BertScore. While the success rate of our method is not high as some of the baselines, those methods do not incorporate the keywords in a semantically meaningful manner as indicated by the BertScore. Furthermore, our generations are more fluent than all baselines in terms of all three fluency metries. These results indicate that method strikes superior balance between control and fluency in regards to topic-guided generation.
6 CONCLUSION
In this work, we introduce DAB, the first controlled decoding algorithm based on gradient discrete sampling. Our algorithm alternates between biased auto-regressive generation and gradient-based discrete sampling to produce text from pre- preitraine language models subject to an external constraint function. Through various controlled generation experiments, we demonstrate that our algorithm is both more efficient and effective than previous methods.
Limitations While our method is more efficient than other gradient- controlled generation methods, the number of gradient computations increases with queries, which may be undesirable. Furthermore, it has not yet been explored how our method performs when faced with multiple ex- ternal constraints compositional generation.
10

---

Published as a conference paper at ICLR 2025
ETHICS
We adhere to the ICLR Code of Ethics. Additionally we confirm that our experiments use only public datasets. The algorithm introduced in this work is general-purper to generate text satisfying arbitrary constraints. Thus it is possible to define malicious constraints that cause LLMs to produce harmful text. Similar to the work done in (Guo al. (0024) it may be possible to apply our algorithm towards jail- bieekbba LLMs and causing them to produce harmful text. Previous works have demonstrated that it possible to induce harmful behavior LLMs
algorithm for directing LLMs
various attacks (He et al. 2024
Liu et al.
2023b Schwinn et al. 2023).
REPRODUCIBILITY
In order ensure the reproducibility of work, include the details necessary to replicate both the core algorithms and the experiments. We include the psuedo-code for our algorithm in [ information syyper-parameer settings in Appendix [ and further details on each experiment in Appendix D.2 D.3 D.4 Additionally, we include the code-base used to produce our results at the following repository: https: https://githube /ttp://gtttt..t... wtt.:/ttww....ttti.ti.iiooaaaaee http:/ttt..kkknaaaaaaa//a/a
REFERENCES
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountabiliry, and Transparency, pp. 610-623, Association for Computing Machinery, 2021. ISBN 978 07781450383097, doi: 10.1 14414//4442188.44599222
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint
2013.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.077 072588 2021.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung. Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2020.
Jasper Dekoninck, Mare Fischer, Luca ation via language model arithmetic. In International Conference on Learning Representations, 2024.
and Martin Vechev. Controlled text gener-
Bryan Eikema, Germán Kruszewski, Christopher Dance, Hady Elsahar, and Marc Dymetman. An approximate sampler for energy-based models with divergence diagnostics. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
Samuel Gehman, Suchin Gururangan, Maarten Sap. Yejin Choi, and Noah A. Smith. RealToxici- tyPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pp. 3356-3369. Association for Computa- tional Linguistics, November 2020. doi: 10.18853/v11/ 1tt.:///.....nnnneeemm/iip..p
Katayoon Goshvadi, Haoran Sun, Xingchao Liu, Azade Nova, Ruqi Zhang, Will Sussman Grath- wohl, Dale Schuurmans, and Hanjun Dai. Discs: A benchmark for discrete sampling. In Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick. Exposing the implicit energy networks be- hind masked language models via metropolis-hastingss. Representations, 2022.
hastings. In International Conference on Learning
Will Grathwohl, Kevin Swersky, Milad Hashemi, David Duvenaud, and Chris Maddison. Oops i took a gradient: Scalable sampling for discrete distributions. In International Conference on Machine Learning, pp. 3831-3841. PMLR, 2021.

---

Published as a conference paper at ICLR 2025
Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. Cold-attack: Jailbreaking Ilms with stealthiness and controllability. In International Conference on Machine Learning. 2024.
Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji. Word embeddings are steers for language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. Association for Computational Linguistics, August 2024. doi: 111865331/2222..clll 864.
Jiaming He, Wenbo Jiang, Guanyu Hou, Wenshu Fan, Rui Zhang, and Hongwei Li. Talk too much: Poisoning large language models under token limit. arXiv preprint arXiv:2404.14799. 2024.
Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. A distributional approach to controlled text generation. In International Conference on Learning Representations, 2020.
Minbeom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park. Hwaran Lee, and Kyomin Jung. Critic- guided decoding for controlled text generation. arXiv preprint arXiv:2212.10935 2022.
Tomasz Korbak, Hady Elsahar, German Kruszewski, and Mare Dymetman. Controlling conditional language models without catastrophic forgetting. In International Conference on Machine Learn- ing. pp. 1 1499-11528. PMLR, 2022.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint (tt:::20000066666 2020.
Germán Kruszewski, Jos Rozen, and Marc Dymetman. disco: toolkit for distributional control of generative models. In Proceedings of the 61st Annual Meeting of the Association for Computa- lional Linguistics (Volume 3: System Demonstrations). Association for Computational Linguis- tics, 2023.
Sachin Kumar, Biswajit Paria, and Yulia Tsvetkov. Gradient-basee constrained sampling from lan- guage models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 2251-2277. Association for Computational Linguistics, December 2022. doi: 10.18653/v1/2022.com http:////2222.mmmpmmmmmmm.m1aa maannlpaaa..1......
Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasser- man. A new generation of perspective api: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22 pp. 3197-3207. Association for Computing Machinery, 2022. ISBN 9781450393850. doi: 10.1145/5334678.333911
Alisa Liu, Maarten Sap. Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti- experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 2021.
Guangyi Liu, Zichao Yang. Tianhua Tao, Xiaodan Liang, Junwei Bao, Zhen Li, Xiaodong He, Shuguang Cui, and Zhiting Hu. Don't take it literally: An edit-invariant sequence loss for text generation. In Proceedings of the 2022 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, pp. 2078-2078. Association for Computational Linguistics, July 2022. doi: 11111///112022.nacc... maw.aaaaiccnmmaa111. 150.
Xin Liu, Muhammad Khalifa, and Lu Wang. Bolt: Fast energy-based controlled text generation with tunable biases. In Association for Computational Linguistics, pp. 186-200. Association for Computational Linguistics, July 2023a. doi: 10.18655/
Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al. Prompt injection attack against IIm-integrated appli- cations. arXiv preprint arXiv:2306. 05499, 2023b.
Tao Meng, Sidi Lu, Nanyun Peng, and Kai- Wei Chang. Controllable text generation with neurally- decomposed oracle. Advances in Neural Information Processing Systems, 35:28125 22-228199, 2022.
12

---

Published as a conference paper at ICLR 2025
Tomas Mikolov. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26, 2013.
Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Berg- Berg- Kirkpatrick. Mix and match: Learning- free controllable text generationusing energy language models. In Association for Computational Linguisticss, pp. 401-415. Association for Computational Linguistics, May 2022. doi: 10.18653/
11/2022.ac1-Iong.31.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A frame- work for adversarial attacks, data augmentation, and adversarial training in nlp. In Empirical Methods in Natural Language Processing, pp. 119-126, 2020.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing, pp. 1532-1543, 2014.
Patrick Pynadath, Riddhiman Bhattacharya, Arun Hariharan, and Ruqi Zhang. Gradient- dis- crete sampling with automatic cyclical scheduling. In Advances in Neural Information Processing Systems, 2024.
Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena D. Hwang, Ronan Le Bras, Antoine Bosselut, and Yejin Choi. Back to the future: Unsupervised backprop-based decoding for counterfactual and abductive commonsense reasoning. In Empirical Methods Natural Lan- guage Processing (EMNLP), pp. 794-805. Association for Computational Linguistics, November 2020. doi: 10. 11885//1120200emmilll http://222..mmmmmamm.a.....
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based con- strained text generation with langevin dynamics. In Advances in Neural Information Processing Systems, volume 35, pp. 9538-9551, 2022.
G. O. Roberts and O. Stramer. Langevin diffusions and metropolis-hastings algorithms. Method- ology And Computing In Applied Probability, 4(4):337-357, Dec 2002. ISSN 1573-7713. doi: 10. 10.1023/A.1023564411133
Leo Schwinn, David Dobre, Stephan Günnemann, and Gauthier Gidel. Adversarial attacks and defenses in large language models: Old and new threats. In Advances Neural Information Processing Systems Workshops, pp. 103-117. PMLR, 2023.
Askhat Sitdikov, Nikita Balagansky, Daniil Gavrilov, and Alexander Markov. Classifiers are better experts for controllable text generation. arXiv preprint arXiv:2205.07276, 2022.
Haoran Sun, Bo Dai, Charles Sutton, Dale Schuurmans, and Hanjun Dai. Any-scale balanced sam- plers for discrete space. International Conference on Learning Representations, 2023a.
Haoran Sun, Hanjun Dai, Bo Dai, Haomin Zhou, and Dale Schuurmans. Discrete langevin samplers via wasserstein gradient flow. In International Conference on Artificial Intelligence and Statistics, pp. 6290-6313 6313. PMLR, 2023b.
Alex Wang and Kyunghyun Cho. Bert has a mouth, and it must speak: BERT as a Markov random field language model. Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pp. 30-36. Association for Computational Linguistics, June 2019.
doi: 10.18653/v 1912-3304.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang. Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint 1:Xi::2112.04359, 2021.
Thomas Wolf, Lysandre Debut, Victor Sanb, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State- of-the-art natural language processing, 2020.
13

---

Published as a conference paper at ICLR 2025
Kevin Yang and Dan Klein, FUDGE: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 3511-3535. Association for Compu- tational Linguistics, June 2021.
Ruqi Zhang, Xingchao Liu, and Qiang Liu. A langevin-like sampler for discrete distributions. In International Conference on Machine Learning, pp. 26375-26396. PMLR, 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evalu- ating text generation with bert. In International Conference on Learning Representations, 2020.
14

---

Published as a conference paper at ICLR 2025
A DISCRETE LANGEVIN PROPOSAL
Our proposed controlled text generation leverages the gradient-based discrete sampling algorithm in
Zhang al. (2022), which is further investigated by Pynadath (2024). Using the same notation as in the Main Body of the paper, we put the original proposal distribution from Zhang et al. (2022)
below:
V [[[B))) - &.) |Onehot, - -6:113
Categorical
softmax jEIV
2a
Here, b, corresponds to the one-hot vector sequence position i. Similarly, Onehot, corresponds to the one-hot vector for the jth token in V. This proposal function defines distribution over the vocabulary for the ith sequence position in the sequence by taking the softmax over all possible tokens.
As discussed in Pynadath et al. (2024), this proposal is locally balanced, or optimal for very small step-sizes. For the task of controlled text generation, would prefer proposal function that is optimal for large step-sizes, which allow for superior exploration of the space of potential sequences. The globally balanced proposal can be written as follows:
Categorical
softmax (VS(B)X) jEIV]
b.))
In terms of the gradient computation, the one-hot representation enables the of automatic dif- ferentiation packages to compute VÍ( We observe that the term (Onchot, b.) corresponds to the distance between the proposed token j and the original token b;. We choose to represent this distance term as hamming distance, given the discrete nature of the space we wish sample. For token j. the hamming distance the original token in position i if the jith coordinate by as they are the same token; and I if the jth coordinate is O. Thus
represent the distances between the tokens as 1 biy. This leads to the proposal function in which place below for convenience:
6 ~ Categorical
softmax JEV
(V
Here, by refers to the token sample from the categorical distribution over V.
B ALGORITHMIC DETAILS
Here provide the full pseudo-code for algorithm.
Algorithm Discrete Autoregressive Biasing
Require: Constraint function PLM prompt number steps s, sequence length n, embedding
table M
B + 0.im -0, Yrest f Initialize constraint violation as being maximal and current
best generation as emply
2: for step do
3: for position i in range(n) do
4: 3 4 log PLM (-1ye,, X) & Initial auto-regressive distribution over V
5: Calculate normalizing factor r; if 1, else r; +
6:
+- argmax,ejv (iu wi ri bi.)) > Sample from P(Y B)
7:
end for
8:
B x Y Initialize B as Y
9:
Evaluate f(B]X), update fmins Yoest
10: B' ~ 4r (-) as in equation 7 Approximately sample from P(B X,Y)
11: Compute B in equation
12:
end for
13: return Yhest
15

---

Published as a conference paper at ICLR 2025
DAB takes imput the external constraint f, the base language model PI. prompt X, number of steps s, sequence length n, and embedding table M. Given these inputs, our proposed algorithm al- ternates between auttoreerrssively generating the response sequence and sampling the bias sequence using Discrete Langevin Proposal (DLP) (Zhang et al. 2022).
C ABLATION STUDY
To provide further insight into our algorithm, we present an ablation study over the important hyper- parameters. We demonstrate the robustness of our algorithm various settings, as well the hyper-parameters that are important towards good performance. We include the results in Figure
Bias Weight As visible in Figure Fp. increasing the weight term in equation T0 leads to increased control over generation at the expense of fluency. However, it is important to note that the perplexity is still reasonable when we increase the bias to be twice the magnitude of the original model logits.
Proposal Temperature In Figure 4b, we examine how varying the temperature T in equation effects the performance of our algorithm. Intuitively, controls the "sharpness" of the proposal dis- tribution, with larger values correspending to flatter peaks and lower values correspending to sharper peaks. This effectively controls the degree of exploration v.s exploitation in the DLP sampler. We see this relation in the results shown in Figure 4p, where higher values result in lower control to- wards the desired sentiment as a result of increased exploration. We also see that there needs to certain amount of exploration in order to find satisfactory generations, as decreasing the temperature too much results in lower control values as well.
Top-k Value In order further ensure fluency and constraint satisfaction of our algorithm, we restrict the DLP proposal in equation 7 to sample only from the Top-k tokens for each position as indicated by the base language model, where a hyper-parameter that can be tuned. In Figure we observe a similar tradeoff between exploration and exploitation: if the h value is too high, than the algorithm will not exert as much control over the generation process. the value is too low, then the algorithm won't be able to explore enough sequence combinations to find good modes. We find that values in the range of 100 to 250 work fairly well across the different tasks.
Algorithmic Robustness Throughout the ablation, clear that DAB is capable of achieving strong performance across range of reasonable dyper-parameter values. Furthermore, this perfor- mance does not come at the cost of fluency the only hyper-parameter that enables for a tradeoff between fluency and control is the weight value, and here hyper-parameter result reasonable perplexity compared to the results in Table Thus we see that our algorithm is fairly robust various seper-paraeeter settings.
that even large values for this
0.925
Ext, Sestinent
11.0
Est. Sentiment
8.55
0,000
11.5
00875
2crp
20
11.0
0,800
30.5
0.70
10
10.5
0835
0.5
0.0
300
1000
Bixs Weight Value
DLP Pruposal Temp Value
Top-k Value
(a)
(b)
(c)
Figure 4: (a) Ablation over different weight values. Higher values result increase in terms of control with decrease in fluency, representing the tradeoff between the two attributes. (b) Ablation over DLP proposal temperatures. Higher temperatures correspond favoring exploration opposed to exploitation, resulting in decreased control. (c) Ablation over top-k values. There is some optimal value that limits the search space sufficiently to enable effective exploration.
flatter proposal distribution
16

---

Published as a conference paper at ICLR 2025
Table Comparison between the operations involved backpropogation for both BOLT and DAB. We select the three most costly operations involved BOLT's gradient computation and compare the number of calls and time cost to the corresponding operations in DAB's gradient computation. As visible, BOLT's gradient computation involves many more repeated operations than DABs, demon- strating the efficiency our proposed DAB algorithm.
Operation
AddmmBackward Time (s) # of Calls
ViewBackward # of Calls
BmmBaacward # of Calls
Time (s)
Time (s)
BOLT
1.8434
3098 74
.5635
10761 205
.2022
1536
DAB
.02123
.1326
.0047
24
C.I EFFICIENCY
Efficiency Improvements Here we further discuss the comparison of efficiency between BOLT and DAB. For both methods, we use the sentiment-directed generation experiment as the generation task for evaluating efficiency. In order the generation speed, we track the total time elapsed for running 50 sampling steps for each algorithm.
Tokens per second Given this time f, sequence length n, and iterations per algorithm 8, we com- pute the following:
TokensPerSecond 7
(1I)
t
For computing the cost per bias sampling for both algorithms, we time only the operations that compute the gradient of the loss and update the bias term. We take the average of this over 50 sampling steps and 15 prompts.
Bias Sampling Cost We use the AutoGrad profiler within Pytorch to count the operations involved in both BOLT and DAB's gradient computation. After finding the common operations involved in both computations, we determine the most costly operations in BOLT and compare them to the corresponding cost in DAB for those same operations. We show the number of calls as well as the total time spent on GPU for BOLT's three most costly operations in 3 It should be noted that using the profiler increases the run-time of all GPU operations, hence why the values will not correspond to the total run times recorded in I
As visible, the gradient computation for BOLT is significantly more expensive than the gradient computation for DAB.
D EXPERIMENTAL DETAILS
Here we include additional details on the experiment setup. We provide the hyper-paramede parameter settings for our algorithm for each experiment Table It should be noted that for Sampling Steps, we pick values maintain roughly the same time cost as BOLT:: given that our algorithm is roughly twice as fast, we use around twice number of sampling steps. Furthermore, given the use of early stopping in BOLT. further computational budget doesn't 't necessarily provide any advantage.
For the weight value, we use a schedule by Liu et al. (2023a) as it was shown be effective in terms of incorporating the bias term into auto-regressive generation. Thus for each position we have w. = w(1 t) where w is the value we put in Table
D.1 FLUENCY METRICS
Here provide more details to the metrics we use to evaluate the fluency of text generations.
17

---

Published as a conference paper at ICLR 2025
Table Hyper-paramelee settings used for DAB on Sentiment-directed generation, language detox- ification, and copic-constraine generation.
Hyper-parameter
Sentiment
Detoxify
Topic
Proposal Temp
.1
.1
.1
Top-k
250
250
250
Bias Weight Value
1.05
1.05
1.4
Number Sample Steps
20
20
200
CoLA Score To assess the grammatical correctness of generation, we use fine-tuned RoBERTa model from Morris et al. (2020) to predict the probability of the sample being labelled as grammat- ically correct. While a similar metric was used in Kumar et al. (2022), we compute the average predicted probability as opposed to the percentage over generations predicted as fluent since this provides more insight into the degree of grammatical correctness.
Repeated Tri-grams To compute the number of repeated tri-grams, we simply count all the tri- grams that were repeated and divide them by the total number of tri-grams per generation. We show the average across all generations for each metric.
Perplexity For perplexity, we use the built-in function within the Hugging Face evaluate package
to compute the perplexity of each generation according to GPT2-XL. (Wolf et al.] 2020). We show the perplexity of the entire generation, as opposed to conditioning the prompt as done in Han cet al. (2024); Kumar al. (2022): Liu et al. (2023a).
D.2 SENTIMENT CONTROLLED GENERATION
Experiment Design We use the same experimental design from Liu et al. (2023a), where the sampler uses an internal classifier to produce the generations. The internal model is RoBERTA with GPT2-Large Embeddings fine-tuned on the yelp polarity dataset. We use two external models to provide additional evaluation: we use another RoBERTA trained on the same dataset but with the original embeddings, well RoBERTa fine-tuned on Stanford Sentiment Treebank 2.
We include the nyper-parameters we use for DAB Table 4 For the baselines, we run the code within their codebase. While we minimize the changes made to the original code, we note that there are some necessary modifications needed order to ensure that the experimental setting is consistent across all methods evaluated. This due to the fact that all the evaluated methods consider similar but slightly different experiments from ours in their original work (Qin et al. 2022
Liu et al.)
2023a Han al. 2024 Kumar et al. 2022).
In regards to IM-Stter, which requires training data, we train the steering matrix using the SST- 2 dataset, done in Han et al. (2024). While this is different dataset from what was used to fine-tune the internal classifiers for the EBM sampling methods, we choose this dataset as obtained worse results when training the steer matrix on yelp polarity. Furthermore, we include external classifier fine-tuned on SST-2 to use as an evaluation criteria. This makes our experiments fair, as all the methods are evaluated with classifiers that are fine-tuned on different dataset than used for sampling. Lastly, we observe that LM-steer achieves reasonable performance in terms of sentiment control when compared to other baselines.
Here we list the prompts we use for this experiment:
RoBERTA with GPT-2 large embeddings fine-tuned on Yelp-Polarin for COLD, BOLT, MuCOLA, and DAB. We train this model following the codebase of Liu al. (2023a). Since require the embedding table to be the same between the base LM, we use the Garge embeddings for the classifier, as done in Liu et al. (203a)): Kumar et al. (2022).
External Constraint To represent the internal constraint,
We use a slightly different function to represent the constraint imposed by the fine-tuned model when compared to BOLT. Given the discriminator h [V] R², where the results represent the
18

---

Published as a conference paper at ICLR 2025
logits for both the desired class ct and the undesired class c_, we define the final constraint function as follows:
f(Y) = (h(+)+ - h(Y)-)
Intuitively, this pushes the unnormalized logits between the desired class and the opposite class away from each other.
This differs from the constraint function in BOLT, which is the typical cross-entropy loss of the discriminator logits where the correct label is the desired sentiment:
S(Y) log soollmax(((())))
We find that formulation of the constraint function enables effective gradients for specific method. Curiously, this modification does not provide any substantial benefit to BOLT. possible that the log softmax of BOLT's method smooths out the directional information of the gradient. While this would benefit a continuous sampling algorithm, this could potentially remove some directional information that is required for effective discrete sampling.
Example Generations In Table 5 we include examples of generations for all methods evaluated.
Table Example text for Sentiment-guiin guided generation. As visible, previous methods either produce coherent text that is not positive enough positive text that is incoherent. In contrast, method produces generations that are both overtly positive and coherent.
Prompt
The horse
COLD
The horse head was still in the water but the horse still had a good head. The horse
MuCOLA
The horse is not only beautiful and well-crafted piece of art, but it is also a great way
BOLT
The horseback riding course is a great way to get acquainted with the trails and the terrain. The course
LM-Steer
The horseman delivers a stunningly beautiful, wonderfully lyrical, bcautifully tender, powerfully moving, beautifully lyrical
DAB (Ours)
The horse is also very good and reliable companion. It has been used to carry the family's two-
Prompt
The chicken
COLD
The chicken is good source of protein, but it's very high fat, cholesterol and salt.
MuCOLA
The chicken is very healthy, and high in protein, and the other two chicken is a very good,
BOLT
The chicken, which is named after the popular street food in Mcxico, was gift from a friend who had
LM-Steer
The chicken beautifully tender, beautifully sweet, wonderfully colorful. the crisp, crispiness wonderful. and the
DAB (Ours)
The chicken a great way to get the kids hooked on delicious, healthy food. It's also good
D.3 TOXICITY CONTROLLED GENERATION
Content Warning: The following section contains examples of LLM generations that some may find disturbing or offensive, including content that is hateful or violent in nature
internal classifier to guide generation towards less toxic models. We use a RoBERTa classifier fine-tuned on the Jigsaw toxic comment dataset, following Kumar et al. (2022) Liu et al. (202aa).
Experiment Design Similar to the sentiment control experiment,
19

---

Published as a conference paper at ICLR 2025
External Constraint We represent the external constraint by RoBERTa model with GPT-2 large embeddings that was fine-tuned on the Jigsaw dataset, following the same training protocol as used Liu et al. (2023a). The final classifier achieves accuracy of 94% on the hold-out dataset.
Examples
In Table 6 we show examples of generations for the various samplers we examine.
Table Examples for Language Detoxification. While all methods seem to have similar probabili- the qualitative impact of our method having lower average
ties of generating toxic text, here max toxicity: while other methods can end up producing very offensive generations, our method is able to ensure that the generation is not extremely toxic.
Prompt
One student's s complaint; however, has led to the logo being associated with
COLD
One student's complaint; however, has led to the logo being associated with the school. The school has since been forced to take the logo and the school's
MuCOLA
One student's complaint; however, has to the logo being associated various East Semitic cultures. What can you do to help save the culture?
BOLT
One student's complaint: however, has led the logo being associated with a negative image. The school chose to remove it from their website after com- plaints from students and parents
LM-Steer
One student's complaint; however, has led to the logo being associated with the campaign. And Turner had previously written Turner School Turner" Turner School Turner Turner
DAB (Ours)
One student's complaint; however, has led to the logo being associated with a site that is not affiliated with the University of California. The university's website and its social
Prompt
During show in Sydney in 2013, Tyler went tirade against member of group.
COLD
During show in Sydney in 2013, Tyler went on tirade against a member of the group. calling him "fucking faggot" and "faggot" and
MuCOLA
During show in Sydney in 2013, Tyler went tirade against a member of the group, telling him: "I want you to go back to your f***ing country." He also apologised
BOLT
During show Sydney in 2013, Tyler went on a tirade against member the group, referring to him as "a piece of shit." The singer also publicly criticized the band's decision to
LM-Steer
During a show in Sydney in 2013, Tyler went on a tirade against member of the group, writing.As for the woman who called Tyler slut, I' m afraid you don't
DAB (Ours)
During show in Sydney in 2013, Tyler went on tirade against a member of the group, who considered to be his friend. The man had been with the band at a party and
D.4 KEYWORD CONTROLLED GENERATION
Experiment Design We largely follow the experimental design introduced Liu et al, (2023a) We list the topics and keywords in
Constraint Function Following Liu et al. (023a),, Qin et al. (2022). we use the differentiable BLEU score introduced Liu et al. (2022). This function measures the uni- uni-gram similarity between the generated sentences and the target words,,,s, using an operation very similar to convolution.
Reference Text Generation We use GPT-4o to generate high-quality reference text to use in the BertScore computation. For a given topic t and keyword k, we query GPT-4o with the following prompt:
20

---

Published as a conference paper at ICLR 2025
Table 7: List of topics and correspending keywords.
Topic
Keywords
computer
router, Linux, keyboard, server
legal
plea, subpoena, tran- script, bankrupt
military
torpedo, headquarters, infantry, battlefield
politics
court, culture, commu- nism, capitilism
religion
Bible, church, priest, saint
science
microscope, mass, min- eral, scientist
space
meteor, planet, satellite, astronaut
Given the topic and the keyword k, write 30 different, unique sentences using the keyword and relevant to the topic.
We do this for each topic and for every keyword for that topic. This produces 120 different, unique sentences use reference text in the BertScore computation.
BertScore Computation Details We use the BertScore computation introduced in Zhang et al. (2020) to evaluate the topicality of the generations. Since BertScore relies on the contextualized embedding of the candidate generations and the reference text, this provides insight into how well methods use the keyword in the desired context.
For each generation, we compute the BertScore against all the 120 reference sentences for the cor- responding prompt and keyword. Because some of the reference text will not contain the keyword used in the generation, overall Fl score, as the precision metric matches tokens in the candidate generation to tokens in the reference text. This is preferable as we want to assess whether the generation is similar to any of the reference texts, as opposed to measuring whether all the reference texts are similar to the candidate generation.
use report the precision metric calculated in BertScore instead of the
Implementation Details We found that in order to obtain good results with DAB on this task, it was necessary to include string containing the keywords prior the prompt. More specifically, we included the following string before the initial prompt for keywords K and topic t:
Include the following keywords: K relevant t.
By including the target keywords and topic before the prompt, this increases the probability of these words and similar words in the underlying language model distribution. This enables the bias vectors computed in our method to have a more impact auto-regressive generation process and thus satisfy the external constraint.
In order to ensure that this was not providing our method with an unfair advantage, we applied the same trick to BOLT in order to determine whether this would improve the performance of BOLT as well. We provide results in Table
As visible, while the prompt does improve the success rate marginally, does not improve any other metrics for BOLT. In fact, we see that this degrades BOLT's fluency slightly through higher perplexity value.
Examples
In Table 9 we show examples of generations for the various samplers examine.
21

---

Published as a conference paper at ICLR 2025
Table Comparison on topic-guided generation between the original BOLT method, the prompted BOLT method, and DAB. As visible, even if the prompt manages to improve the success rate by .7,, this comes at the cost of worse fluency and slightly worse topicality. Furthermore, our method still outperforms this baseline.
Control
Fluency
Topic
BertScore
Success Rate
CoLA 1
REP-3gram |
PPL
BOLT
.2911 00033
99.1%
.705 .006
.005 .005
32.019 1.593
BOLT (Prompted)
.8123 .0002
99.7%
.705
.005
.005 + 0001
38.22 1 .951
DAB (Owrs)
.8303 = .0003
99.0%
.726 = .005
.004 + 0001
23.424 = .317
Table Examples for Topic-Constrained Generation. As visible, while previous methods include the keyword, they tend to either repeat the keyword too many times or misuse the keyword. In contrast, our method is able to include the keyword a meaningful way relevant to the given topic.
Prompt Topic
Once upon time Military
Keywords
torpedo, headguarters, infantry, battlefield
COLD
Once upon time, the world peaceful place. People headquarters of the world headquarters the world torpedo-
MuCOTA
Once upon time, the world was a world of the great battlefield the powerful headquarters torpedo of the good and infantry
BOLT
Once upon a time, there was man named John Smith who had a dream that he would be able to infantry his
DAB (Ours)
Once upon a time, there was a small group of officers who were in charge of the modemn infantry and logistics. They
Prompt
The book
Topic
Science
Keywords
microscope, mass, mineral, scientist
COLD
The book is scientist-driven, and is scientist mineralogist, microscope, micro- scope, mineral microscope,
McCOLA
The book also has massive properties, like the Alabaster House, which features extensive characters from Alabaster
BOLT
The book is divided into three parts, each of which contains chapter mass mineral scientist relevant to science. scientist
DAB (Ours)
The book is good introduction the field of mass spectrometry and is an excellent resource for hands-
22


Reviews:
[{"review_id": "5Zxc6GjyNG", "reviewer": "Reviewer_3EGw", "summary": "This paper introduces DAB (Discrete Auto-regressive Biasing), an algorithm for controlled text generation with large language models (LLMs).  Previous methods often use energy-based decoding in continuous space, which struggles to balance fluency and constraint satisfaction. DAB addresses this by operating entirely in the discrete space of text tokens.\n\nDAB samples from the joint distribution of generated sequence Y and an auxiliary bias sequence B and alternate between biased auto-regressive generation and discrete gradient-based sampling. Specifically, given a generated text Y, the gradient-based discrete sampling is used to maximise constraint satisfaction. Then B is fixed, and biased auto-regressive generation is used to sample Y. A penalization is applied based on sampled tokens' distance from B in embedding space.\n\nExperiments show DAB outperforms baselines such as BOLT and LM-Steer on sentiment-controlled generation, language detoxification, and keyword-guided generation.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper identifies previous work's deficiency in balancing fluency and constraint satisfaction and proposed a method that maximises both (according to several benchmark numbers).\n\n* DAB seems to be more stable and robust than other continuous methods.", "weaknesses": "Nothing major as I can see.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HnD91hf5nz", "reviewer": "Reviewer_w6Rd", "summary": "The paper proposes an approach to controlled text generation --- DAB (Discrete Auto-regressive Biasing) --- that exploits the DLP (Discrete Langevin Proposal) technique from (Zhang et al. 2022) for efficiently sampling inside a discrete space while still being able to exploit gradients of an energy function over this space. The DLP technique is not used directly over the output sequence, but over an auxiliary \"bias sequence\" that is coordinated with the output sequence through a Gibbs-Sampling-like alternation. The experiments show competitive results with a few baselines in terms of efficiency as well as balance between constraints and fluency of the obtained results.", "questions": "Questions/suggestions (in addition to those implicit in the previous section):\n\n- Lines 238-243 seem problematic, as they introduce a notation $P(Y|B)$ that is not conditioned by $X$. Are they correct and/or needed later?\n\n- In Lines 494-496, you mention that a good metrics for the keyword-guided generation should consider the meaning similarity of the produced sentence to the constraining keywords, not the actual presence of the keywords. I was not fully convinced by this remark, and was wondering whether considering the actual presence of the keywords should be seen as a more important metrics. More generally, I wondered whether it would be worth reporting, among the metrics, the value $f(Y)$ itself as this seems to be the main driver of the approach.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The main strength of the paper lies in its innovative application of the DLB technique to the general problem of controlled text generation (Zhang et al. did touch on text generation but only in a very limited way).\n\nThe paper is also creative in the way it uses an auxiliary bias sequence to steer the generation of the actual output sequence.\n\nThe experiments demonstrate certain advantages in terms of control (i.e. constraint satisfaction) and fluency over a number of baselines, in particular some based on gradient techniques over continuous relaxations of EBMs over discrete spaces.", "weaknesses": "The main weaknesses of the paper lie on two dimensions: (A) omission of significant related work, (B) lack of discussion/clarity about certain key aspects and modelling decisions in the paper. \n\n(A) The Related Work section 2.1 devoted to Language Models as EBMs, totally ignores a substantial line of work specifically devoted to *discrete* sampling from EBMs, either (i) with focus on training autoregressive approximations to these EBMs (exemplified by [1] and a number of more recent publications at ML conferences (see references in [3])), or (ii) with focus on decoding-time techniques [2]. This line of work, like the present paper, is concerned with discrete (as opposed to continuous) sampling, and is not limited to encoder-based architectures.\n\n[1] Khalifa et al. A distributional approach to controlled text generation. ICLR 2021.\n\n[2] Eikema et al. An approximate sampler for energy-based models with divergence diagnostics. TMLR 2022.\n\n[3] Kruszewski et al. disco: a toolkit for Distributional Control of Generative Models. ACL 2023.\n\n\n(B.1) Concerning the core Equation (5). First, in the expression $P^{LM}(Y|X,B)$, what you seem to mean is that you concatenate the input $X$ with the bias sequence $B$ and then apply the LM on this new input, but it would be worth discussing this assumption. Second, and more importantly, while it is not clear at this point in the paper, it seems that in Algorithm 1 you actually need to compute $f(Y)$, and not only $f(B)$. Then a pretty obvious question for the reader is, why not simply define $P(Y|X) \\propto P^{LM}(Y|X) \\exp(f(Y))$ ? That would be much more direct than what the paper does, would directly define $P(Y|X)$ as an EBM, and then presumably the DLB technique could be directly applied to this EBM. It is not clear to me why the authors do not consider and discuss this possibility. \n(Of course, several other techniques for sampling from this EBM would be possible, including those mentioned in (A) and techniques related to RLHF/PPO where $f(Y)$ might be seen as a reward.)\n\n(B.2) There are several other points in the paper that are kept implicit and would need more discussion. To give a few examples: \n- The fact that the length $n$ of the response $Y$ needs to be specified in advance, which appears to be a limitation of the approach, is kept implicit.\n- The important DLB-based equation (7) should be described in a more self-contained way (perhaps using the Appendix), in particular the way the gradient is actually computed.\n- The fact that step 6 in Algorithm 1 is actually deterministic should be mentioned, as this detracts from standard Gibbs-sampling practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1eNjFTgVlE", "reviewer": "Reviewer_HCm1", "summary": "This paper proposes a new controlled text generation approach called DAB. DAB happens at decoding time, and it consists of two alternating steps: updating the bias term based on the gradient, and updating the response sequence conditioned on the bias term. This approach is designed to address the trade-off between fluency and control satisfaction. They experiment on sentiment, toxicity, and lexical control benchmarks.", "questions": "see the weakness section.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- good summary of prior work, it nicely summarizes the field of controllable text generation from AR to NAR, with or without gradient guidance, etc. \n- the method makes intuitive sense, but the details of the methods seem very unclear.", "weaknesses": "- Controllable text generation is important, but the benchmarks this paper tested have been mostly saturated by prior approaches, so further testing on these benchmarks can no longer demonstrate the goodness of the newly proposed approach. Furthermore, prompting could solve these problems, so can this approach solve even harder problems, like model jailbreaking via such MCMC type of approach? \n\n- I think the math is not very solid in the paper. equation 6 seems wrong. It's a circular definition. The equation (not numbered) immediately after 6 is also strange, should it be P(B | Y, X)? There are also unclear notations in the method section. \n\n- The method section is very badly written. I don't understand many technical details in the method section. Why eqn 7 has the (1-bij) term? Why is eqn 10 argmax for a sampling distribution? If B is the bias term, intuitively initializing B at 0 seems more reasonable than initializing B at Y?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E8WcvLSukp", "reviewer": "Reviewer_RyTz", "summary": "The paper presents an algorithm for constrained autoregressive decoding where the constraint is defined by a distribution function. This is a common approach using energy-based models.\n\nEven though BOLT does propose similar algorithm, the main difference in this paper is that instead of sampling the bias in the continuous domain, the bias token sampling in this paper happens in discrete space. Author claim, and substantiate with experiments, that this produces sequences that not only follow the constraint better, but also the LLM policy model leading to more fluent outputs.", "questions": "In equation 8, the bias value b_{i,j} corresponding to any bias token b_{i} is calculated as the L2 distance between their corresponding embeddings. LLM loss functions use inner product between the embeddings as the distance metric while calculating logits. Any reason to use L2, and were other options tried?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The authors propose DAB - Discrete Autoregressive Biasing, a modification of BOLT where both output sequence Y and the biasing B are always sampled in discrete space. This allows the model to always remain in the discrete space leading to not just lower computational cost compared to other energy based decoding methods, but also improve the generation with respect to fluency and constrain.\n\nAs part of DAB:\n1. A joint distribution over the output Y and Bias tokens B is proposed.\n2. Similar to Gibbs sampling, the proposed sampling algorithm (Langevin within Gibbs) alternates between sampling better Y and B while at the same time using Langevin for prediction the distribution to sample the tokens from.\n3. Uses MCMC to sample the bias tokens, and then add bias from those while sampling the response token.\n\nAlbations are performed on both hard and soft constrains to show the effectiveness of the model.", "weaknesses": "- Even though the method is fast compared to other energy based models, the method is still slow because of the autoregresive sampling happening inside the step for loop as shows in Algorithm 1, line 3-7.\n- Results on Sentiment are mixed with decreased Fluency compared to BOLT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)