Venue: NAACL 2025

# Constrained Decoding with Speculative Lookaheads

 Nishanth Nakshatri\({}^{\clubsuit}\)   Shamik Roy\({}^{\diamondsuit}\)   Rajarshi Das\({}^{\spadesuit}\)

**Suthee Chaidaroon\({}^{\spadesuit}\)  Leonid Boytsov\({}^{\spadesuit}\)   Rashmi Gangadharaih\({}^{\spadesuit}\)  \({}^{\clubsuit}\) Purdue University \({}^{\spadesuit}\) AWS AI Labs nakshat@purdue.edu**

{royshami, dasrajar, sutheec, lboytsov, rgangad}@amazon.com

###### Abstract

Constrained decoding with lookahead heuristics (CDLH) is a highly effective method for aligning LLM generations to human preferences. However, the extensive lookahead rollout operations for each generated token makes CDLH prohibitively expensive, resulting in low adoption in practice. In contrast, common decoding strategies such as greedy decoding are extremely efficient, but achieve very low constraint satisfaction. We propose constrained decoding with speculative lookaheads (CDSL), a technique that significantly improves upon the inference efficiency of CDLH without experiencing the drastic performance reduction seen with greedy decoding. CDSL is motivated by the recently proposed idea of speculative decoding that uses a much smaller draft LLM for generation and a larger target LLM for verification. In CDSL, the draft model is used to generate lookaheads which is verified by a combination of target LLM and task-specific reward functions. This process accelerates decoding by reducing the computational burden while maintaining strong performance. We evaluate CDSL in two constraint decoding tasks with three LLM families and achieve \(2.2\times\) to \(12.15\times\) speedup over CDLH without significant performance reduction.

+
Footnote †: Work done during an internship at AWS AI Labs.

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 4681–4700 April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Alignment of LLMs to human preferences is important for their general applicability. Constrained decoding in large language models (LLMs) is an effective method as a post-training step to align LLM generations to human preferences such as harmless text generation Deng and Raffel (2023); Huang et al. (2024), faithful summarization Wan et al. (2023), formatted generation, flow adhering planning Roy et al. (2024), among other use cases. Within Constrained Decoding methods, the "Lookahead" Heuristics-based approach (CDLH) has demonstrated the best performance across several tasks Lu et al. (2021, 2022). CDLH examines the top \(k\) beams, generates \(d\) additional tokens as "lookahead" for each beam, scores them using task-specific reward functions for constraint satisfaction, and selects the beam that maximizes constraint fulfillment while discarding the others. However, the computational expense of generating lookaneads makes this approach prohibitively costly in terms of runtime, limiting its practical applicability in real-world scenarios.

Speculative decoding (SD) is a recently proposed technique for improving the inference speed of LLMs Leviathan et al. (2023); Chen et al. (2023). This approach employs a draft-then-verify strategy Xia et al. (2024), utilizing a smaller and more efficient draft LLM's generation to approximate the output of a larger target LLM. At each step, the draft's output is validated by the target LLM.

Figure 1: Inference speedup v/s performance on harmless text generation (Anthropic’s HH-RLHF dataset). Apart from CDSL, we also propose a novel baseline (CDLH-appx) which uses the draft model to generate lookahead tokens for each beam. CDSL gains significant inference speedup w.r.t. CDLH and CDLH-apps without drastic performance reduction as compared to other decoding algorithms such as greedy decoding. Plot best viewed in color.



In transformer-based LLMs, this validation is performed through a forward pass, which is less computationally intensive than autoregressively generating tokens. As a result, SD significantly reduces the inference time for the target LLM while preserving generation quality. However, it should be noted that SD does not have any specific mechanism to align its generations with task-specific constraints.

Drawing inspiration from speculative decoding, in this work, we propose Constrained Decoding with Speculative Lookaheads (CDSL) to improve the inference time of constrained decoding algorithms. CDSL begins by autoregressively speculating the lookahead tokens using a small and efficient draft LLM. Speculated lookaheads are then validated by the larger target LLM, to ensure that the output from the draft adheres to the same distribution as the target LLM. To enforce constraints on generations, validated speculative lookaheads are also scored using a task specific reward function, which assesses their adherence to the required constraints. Based on the feedback from the target LLM and the reward function, we define a set of states and corresponding actions that determine whether to accept or reject the lookaheads.

To demonstrate the effectiveness of the proposed approach we experiment with three LLM families and two constraint-satisfaction tasks - constrained commonsense generation, CommonGenLin et al. (2020) and harmless text generation, HTG (Anthropic HH-RLHF Bai et al. (2022)). The CDLH method serves as an upperbound for constraint satisfaction performance and a baseline for runtime performance. Similarly, greedy decoding serves as a runtime upper-bound and performance baseline. We observe that our proposed CDSL gives a practical middle-ground by significantly improving upon the inference efficiency of CDLH without experiencing drastic performance reduction often seen with greedy methods, as shown in Figure 1. Concretely, CDSL delivers an inference speedup of \(2.2\times\) to \(12.15\times\) over CDLH, along with constraint satisfaction performance improvements of up to \(+10.1\%\) points on HTG and \(+20.9\%\) points on CommonGen compared to greedy decoding.

## 2 Preliminaries

### Constrained Decoding with Lookahead Heuristic (CDLH)

Common decoding techniques, such as greedy or beam search methods, generate the next token autoregressively based on the previous (input) tokens. However, enforcing constraints to text generation often requires planning ahead and estimating the likelihood of satisfying future constraints if a candidate token is selected. To address this challenge, Lu et al. (2022) introduced constrained decoding with lookahead heuristics (CDLH) based method. This approach views decoding as a discrete search problem and selects the next token at each decoding step by estimating the future reward based on lookahead heuristics. In particular, for each token generation, it considers top-\(k\) candidate tokens based on model logits, each of which is expanded to generate \(d\) additional tokens called lookaheads. The lookaheads are then scored for constraint satisfaction using a constraint-specific reward function, \({\cal R}\), and the token with the highest reward is finally generated. This approach is illustrated in Algorithm 2 in Appendix A. By unrolling additional lookahead tokens to estimate potential future reward for each candidate token, this approach achieves state-of-the-art performance in many constraint satisfaction tasks Wan et al. (2023); Huang et al. (2024); Roy et al. (2024). However, this method is computationally expensive due to exhaustive search required to approximate future rewards at each decoding step, which involves lookaheads generation for all candidate options. While limiting the search space to the top-\(k\) candidates mitigates this issue, the computational cost remains significant, especially with larger LLMs. For instance, our experiments show that CDLH is \(8.62-15.35\times\) more expensive than greedy decoding (Section 5).

### Speculative Decoding (SD)

With the motivation of improving the natural inference time of LLMs, speculative decoding (SD) has been proposed in recent studies Leviathan et al. (2023); Chen et al. (2023). The key idea in speculative decoding is that complex language modeling tasks often include easier subtasks (e.g., generation of certain words are easier given the context) that can be approximated well by even a simpler, smaller, and more efficient language model. The transformer based LLMs comes with the advantage that they can process multiple tokens at a time and output their probability distribution with just a forward pass. Hence, in SD, given an LM task, a runtime efficient language model, typically a very small LLM (referred to as draft LLM, \(M_{q}\)), is used to draft \(d\) tokens and a much larger LLM (referred to as target LLM, \(M_{p}\)), verifies them with just one 

forward pass. By comparing the output probability distribution of \(M_{q}\) and \(M_{p}\) for the drafted tokens, it is determined at what position the drafted tokens deviate from \(M_{p}\)'s output distribution and a new token is generated at that position (this process repeats). In this manner, SD improves the runtime of \(M_{p}\) while retaining its output probability distribution. Motivated by this method, in this work, we propose "constrained decoding with speculative lookaheads" to improve runtime complexity of CDLH, as explained in the next section.

## 3 Proposed Algorithm

In this section, we first propose a novel baseline that is simple yet effective, by optimizing the looakhead mechanism of CDLH with a smaller LLM. Then we propose our algorithm "constrained decoding with speculative lookaheads" which combines the paradigms of CDLH and speculative decoding. Our approach leverages the strength of both paradigms (better alignment from constrained decoding and faster decoding from speculative decoding), leading to a powerful yet practical method.

### CDLH with Approximate Lookaheads (CDLH-Appx.)

We observe that in CDLH, the most runtime is incurred during generating the lookaheads for each candidate token, as for each token generation the LLM needs to generate \(k\times d\) additional tokens (assuming greedy rollout of lookaheads). We propose optimizing the runtime in this step by using a much smaller LLM that is computationally less expensive than the larger LLM. Hereafter, we address the smaller LLM as "draft LLM", \(M_{q}\), and the larger LLM as "target LLM", \(M_{p}\). To this end, we propose a simple yet effective baseline, where \(M_{q}\) generates i.e., _approximates_ the lookaheads for \(M_{p}\), resulting in faster inference. After generating the lookahead tokens, as before, the token corresponding to the highest reward is selected. We note that even though a lot of computation is shifted to the much cheaper \(M_{q}\), this approach still requires generation of additional \(k\times d\) tokens by \(M_{q}\) just to generate one token by \(M_{p}\).

### Constrained Decoding with Speculative Lookaheads (CDSL)

Our algorithm begins by autoregressively speculating \(d\) lookahead tokens using the draft LLM, \(M_{q}\). Speculative lookaheads are validated by target LLM, \(M_{p}\), to ensure that the output from \(M_{q}\) adheres to the same distribution as \(M_{p}\). To enforce constraints on generations, validated speculative lookaheads are scored using a task specific reward function \(\mathcal{R}\), which assesses their adherence to the required constraints. Next, we define a set of states and corresponding actions based on the feedback from \(M_{p}\) and \(\mathcal{R}\), that determines whether to accept or reject the speculated lookaheads. In this manner, our approach ensures maximum gain in both constraint satisfaction and runtime. This entire _draft-then-validate_ process repeats until the desired maximum sequence length \(l_{m}\) is achieved. The following subsections provide a detailed explanation of this procedure.

#### 3.2.1 Generating Speculative Lookaheads

In CDLH, the target LLM \(M_{p}\) generates the lookaheads resulting in a very high runtime. Conversely, in CDLH-Appx., the lookaheads are approximated by smaller draft LLM, \(M_{q}\), however, not verified by \(M_{p}\), leaving room for more runtime improvement. Hence, in our proposed method, we initiate text generation using a draft model, \(M_{q}\), with the aim of leveraging its cost efficiency for the lookahead process. Given an input prompt, referred to as _prefix_ and a fixed draft length \(d\), we generate a sequence of tokens by greedily sampling \(d\) tokens from \(M_{q}\) in an autoregressive manner. It is crucial to understand that these lookahead tokens are speculative at this point, as they have not yet been validated by the target \(M_{p}\), to confirm alignment with its distribution. In addition, we note that the _prefix_ comprises the initial prompt and the previously generated tokens, accumulated over multiple _draft-then-validate_ iterations of our algorithm.

#### 3.2.2 Scoring Lookaheads

Our goal is to maximize both constraint satisfaction and runtime efficiency. Hence, a generated lookahead in the previous step is evaluated by both \(M_{p}\) (attributes to runtime gain) and task specific reward function \(\mathcal{R}\) (attributes to constraint satisfaction).

Validation by \(M_{p}\).We apply the principle of speculative decoding, performing a forward pass on the input prefix along with \(d\) lookahead tokens using the target \(M_{p}\) for validation. Using the output distribution of the \(d\) tokens by \(M_{p}\), the validation can be done using the following two methods.

**Hard Rejection:** We take the \(\arg\max\) at each of the \(d\) positions on the distribution generated by \(M_{p}\) and obtain the tokens \(M_{p}\) would generate at each of the \(d\) positions. Then we compare the tokens 

from \(M_{p}\) and \(M_{q}\) for equivalence at each position and identify the first position with a mismatch.

**Speculative Sampling:**Leviathan et al. (2023) introduces a speculative sampling approach, which does not require an exact positional match between \(M_{p}\) and \(M_{q}\). Assume \(p(x)\) and \(q(x)\) are distributions from \(M_{p}\) and \(M_{q}\), respectively. A token \(x_{n}\), at a position \(n\) is rejected if \(r>min(1,(p(x_{n})/q(x_{n})))\), where \(r\sim U[0,1]\), and a new token is sampled at the rejected position from the adjusted probability distribution \(p(x_{n})-q(x_{n})\). In this manner, this method ensures that the probability distribution of the generated tokens aligns with the distribution of \(M_{p}\).

After determining the first rejection position using either of the approaches, we calculate the acceptance score, \(a\), that represents the proportion of tokens approved by \(M_{p}\) (\(a\) = first-rejection-position / draft length). \(a\) quantifies the degree to which \(M_{p}\) approves of speculative lookaheads.

Validation by \(\mathcal{R}\).To impose constraints on the generated sequence, the generation until the first rejection position determined by \(M_{p}\) is evaluated by a reward function, \(\mathcal{R}\). The reward function \(\mathcal{R}\) takes as input the generated sequence along with the prefix and outputs a reward score, \(r\), based on constraint satisfaction. The choice of \(\mathcal{R}\) is task-specific and depends on the nature of the constraint (e.g., lexical vs. semantic). Accordingly, \(\mathcal{R}\) can be a function or a parametric model, it may generate a binary, continuous, or discrete score, all depending on the task in hand. In Section 4, we experiment with different types of task-specific \(\mathcal{R}\).

#### 3.2.3 Speculated Tokens Acceptance Decision

In our algorithm, the decision to finally accept or reject speculative lookaheads is determined by a combination of the acceptance score, \(a\) and the reward score, \(r\). We define the following four states and state-specific actions based on the magnitude of \(a\) and \(r\).

Both acceptance (\(a\)) and reward (\(r\)) scores are high (S1).This scenario represents an ideal situation that indicates the majority of outputs from \(M_{q}\) are similar to the output distribution of \(M_{p}\) and satisfy the constraint requirements. Hence, in this situation, every token accepted by \(M_{p}\) is generated. It is important to emphasize that we do not sample additional tokens from \(M_{p}\), contrary to the usual practice in speculative decoding. This is because, in constraint satisfaction problems, accepting a token without verification by the reward model is inefficient and may result in constraint violation.

```
1:Input: Target llm \(M_{p}\), Draft llm \(M_{q}\), Reward function \(\mathcal{K}\), Prompt \(pfx\), Lookahead lend \(d\), Max seq. len \(l_{m}\), Beam size \(k\), Reward thresh. \(r_{t}\), Acceptance thresh. \(a_{t}\)Initialize: Generated sequence \(\mathcal{T}\leftarrow\emptyset\)while\(len(\mathcal{T})<l_{m}\)do
2:\(\triangleright\) Draft \(d\) tokens from \(M_{q}\) for lookahead.for i = 1 to d
3:\(q_{i}(x)\gets M_{q}(pfx+[x_{1},\dots,x_{i-1}])\)
4:\(x_{i}\leftarrow\arg\max q_{i}(x)\)
5:endfor
6:\(\triangleright\) Run \(M_{p}\) in parallel for verification.
7:\(p_{1}(x),\dots,p_{d}(x)\leftarrow\)
8:\(M_{p}(pfx),\dots,M_{p}(pfx+[x_{1},\dots,x_{d}])\)
9:\(\triangleright\) Determine number of accepted guesses \(n\).
10:\(n\gets 0\)forent \(i=1\)to d do
11:\(y_{i}\leftarrow\arg\max p_{i}(x)\)
12:if\(y_{i}\neq x_{i}\)then
13:break
14:endif
15:\(n\gets n+1\)
16:endfor
17:\(\triangleright\) Calculate acceptance \(a\) and reward \(r\) scores.
18:\(a\gets n/d\), \(r\leftarrow\mathcal{R}(x_{1},\dots,x_{n})\)
19:\(\mathcal{O}\leftarrow\emptyset\)
20: Compare scores and take state-specific actions.
21:if\(a\geq a_{t}\)and \(r\geq r_{t}\)then
22:\(\triangleright\) S1 specific actions
23:\(\mathcal{C}\leftarrow[x_{1},\dots,x_{n}]\)
24:elseif\(\prec a_{t}\)then
25:\(\triangleright\) S2/S3 specific actions
26:\(\mathcal{X}\leftarrow\emptyset\)
27:for\(i=1\)to b do
28:\(x_{i}\leftarrow\arg\max M_{p}(pfx+[x_{1},\dots,x_{n}]+\mathcal{X})\)
29:\(\mathcal{X}\leftarrow\mathcal{X}+x_{i}\)
30:\(lh\gets 0\)
31:for\(j=1\)to d do
32:\(l\leftarrow\arg\max M_{q}(pfx+[x_{1},\dots,x_{n}]+\mathcal{X}+lh)\)
33:\(lh\gets lhs+l\)
34:endfor
35:if\(\mathcal{R}(pfx+[x_{1},\dots,x_{n}]+\mathcal{X}+lh)\geq r_{t}\)then
36:\(\mathcal{O}\leftarrow[x_{1},\dots,x_{n}]+\mathcal{X}\) break
37:endif
38:endfor
39:if\(\mathcal{O}\)is \(\emptyset\)then
40:\(o\leftarrow\text{Generated 1 token using CDLH-appx}\).
41:\(o\leftarrow[x_{1},\dots,x_{n}]+o\)
42:endif
43:elseif\(a\geq a_{t}\)and \(r<r_{t}\)then
44:\(\triangleright\) s4 specific actions
45:\(o\gets\text{Generated 1 token using CDLH-appx}\).
46:\(\mathcal{O}\leftarrow[x_{1},\dots,x_{n}]+o\)
47:endif
48:\(pfx\gets pfx+\mathcal{O}\)
49:\(\mathcal{T}\leftarrow\mathcal{T}+\mathcal{O}\)
50:endwhile
51:Output: Generated sequence \(\mathcal{T}\)
```

**Algorithm 1** Constrained Decoding with Speculative Lookaheads 

score is low, indicating that the target LLM \(M_{p}\) disapproves of the tokens generated by \(M_{q}\). This holds regardless of whether the reward score is high or low, which implies that the output distribution of \(M_{p}\) differs significantly from that of \(M_{q}\). \(M_{p}\) being a larger model, intuitively exhibits superior reasoning abilities compared to the cheaper draft \(M_{q}\). Hence, we discard generations by \(M_{q}\) and strategically sample tokens from \(M_{p}\), allowing it a chance to generate high reward tokens. This action is described below.

**Step-1:** In this step, we provide \(M_{p}\) a chance to lead the output in a direction where constraint satisfaction will be high. For that, we greedily sample the next token from \(M_{p}\), perform a lookahead for the token with draft \(M_{q}\), and score it with \(\mathcal{R}\). We provide \(M_{p}\) the chance to generate up to \(b\) tokens in this manner until the reward score becomes high. If the reward score is still low, indicating \(M_{p}\)'s limitation in constraint satisfaction, we discard the \(b\) tokens and move to Step-2, otherwise, we accept the \(b\) new tokens.

**Step-2:** If Step-1 does not yield a token, it means greedy decoding of \(M_{p}\) is not going to satisfy constraints. Hence, the output distribution of \(M_{p}\) needs to be modified. To do so, we consider the top-\(k\) next token candidates by \(M_{p}\). For each top-\(k\) token, we perform a lookahead using \(M_{q}\), score it using \(\mathcal{R}\), and choose the token with the highest constraint satisfaction score. This step is equivalent to CDLH-appx. as described in Section 3.1.

**Acceptance (\(a\)) is high, while reward (\(r\)) is low (S4).** This state indicates that the target LLM \(M_{p}\) accepts the outputs from \(M_{q}\), however, they do not satisfy the constraints, indicating \(M_{p}\)'s inability to satisfy constraints in this state. Thus, we discard the generations by \(M_{q}\) and directly generate the next token with \(M_{p}\), imposing constraints on its generations following Step-2 above.

Finally, we define what qualifies as "high" vs. "low" for the acceptance (\(a\)) and reward scores (\(r\)) using thresholds \(a_{t}\) and \(r_{t}\), respectively. The intuition is drawn from recent studies on speculative decoding that show, acceptance score varies widely across tasks and the choice of target-draft LLM pairs Liu et al. (2024); Yan et al. (2024). Hence, defining thresholds to determine high or low acceptance and reward scores allow us to explore a continuous space of the above four states, and select the threshold combination that yields the best runtime and constraint satisfaction performance for a particular task and target-draft pair. We outline our approach with the hard rejection method in Algorithm 1 and note that the soft rejection technique can be plugged in lines 12-20.

## 4 Experimental Setting

### Constrained Decoding Tasks

To study the efficacy of our algorithm, we select two types of constrained decoding tasks.

**Lexical Constraint:** We select the constrained commonsense generation task, CommonGen Lin et al. (2020), that requires generating a coherent sentence that describes a plausible scenario using all the provided concepts (e.g., {'dog', 'run', 'field'} might produce "The dog runs on the field"). This type of lexical constraints are programmatically verifiable, hence, we implement the reward function, \(\mathcal{R}\), for this task using a concept counting method in the generated sentence, e.g., if \(2\) out of \(5\) concepts are covered, \(\mathcal{R}=0.4\).

**Semantic Constraint:** For semantic constraint satisfaction, we study the Harmless Text Generation (HTG) task using the Anthropic HHRLHF dataset Bai et al. (2022), where a human converses with an LLM assistant and tries to prompt it to generate harmful responses. Semantic constraints are abstract (e.g., 'I can help with a murder' vs. 'I do not support murder') and are not possible to verify programmatically, requiring different type of reward modeling. Hence, as \(\mathcal{R}\), we use an off-the-shelf reward model, rewardmodel-deberta-v3-large-v2*, that provides a continuous reward score for harmlessness.

Footnote *: [https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2](https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2)

### LLM Families

By design, speculative decoding based approaches require draft and target LLMs to be selected from the same LLM family in order to ensure vocabulary match. Hence, we experiment with three LLM families that have enough smaller and larger LLMs available to be used as drafts and targets. We experiment with OPT Zhang et al. (2022): 13B as target and {125M, 350M, 1.3B} as drafts; Bloomz Muennighoff et al. (2023): 7.1B as target and {560M, 1.7B} as drafts; Qwen1.5 Team (2024): 7B as target and {0.5B, 1.8B} as drafts. For the CommonGen task, we use the Chat version of Qwen1.5 as we found it to be better at following the instructions.

 



### Evaluation Metrics

In this paper, our goal is to improve the runtime of CDLH with speculative lookaheads. Hence, we report the following two major evaluation metrics.

Speedup.Following the approach by Leviathan et al. (2023), we calculate the runtime cost coefficient \(c\) as the ratio of the average tokens per second of \(M_{p}\) to those of \(M_{q}\), with the simplest decoding method, greedy decoding. We found that the runtime complexity of the reward function, \(\mathcal{R}\) is negligible compared to the inference time of LLMs. Hence, by disregarding the negligible runtime for \(\mathcal{R}\), we calculate _runtime_, \(P\) for each token generation, \(P=(c*\text{No. of $M_{q}$ calls per token})+\text{No. of $M_{p}$ calls per token}\). _Speedup_ is calculated by taking the ratio of runtime of two approaches.

Constraint Satisfaction Rate.For CommonGen, we report two constraint satisfaction metrics - (1) **% Soft Constraint Satisfaction:** measures the overall percentage of constraints that are satisfied across all data points, (2) **% Hard Constraint Satisfaction:** measures the percentage of data points where all required concepts are included. For HTG, we measure the percentage of generations that are harmless. For evaluating if a generated response is harmless, we use an off-the-shelf LLM, Llama-Guard-3-8B (Llama Team, 2024), which is fine-tuned for content safety classification task to assess the safety of the generated text. We perform a meta evaluation of this model judge and found that it is \(94\%\) reliable in identifying Harmless/Safe responses (details can be found in Appendix B).

### Datasets and Hyperparameter Tuning

We tune different hyper-parameters (\(a_{t}\), \(r_{t}\), \(b\)) of our model using a validation set and report all the results in the test set. For validation, we sample \(200\) examples from CommonGen and \(100\) conversations from the HH-RLHF corpus. We sample disjoint \(1,000\) examples from CommonGen and \(2,000\) conversations from HH-RLHF as test sets. For the CommonGen task, we prompt the model in a two-shot manner. This is done because the performance of speculative decoding-based approaches largely depend on the instruction following capability of the draft LLMs (Yan et al., 2024) and we observe that this ability is enhanced with few-shot prompting in the CommonGen task. Moreover, we experiment with OPT family models which are not instruction tuned and rely on few-shot learning. The constraints are stated in their task-specific instructions in the prompt. Prompt templates are shown in Appendix D; hyperparameter search space, selection procedure, and the selected hyperparameters for the test set are discussed in detail in Appendix C. We perform Greedy rollout of lookaheads in all of our experiments.

## 5 Results and Ablations

In this section, we first describe the performance of our model compared to the baseline and skyline approaches. Then, we perform an extensive ablation study and error analysis of our proposed approach.

### Key Findings

We evaluate our approach against various algorithms detailed in Section 3. Additionally, we compare results with Beam Search (Freitag and Al-Onaizan, 2017), and Nucleus Sampling (Holtzman et al., 2019) methods. Greedy decoding serves as a skyline for runtime and a baseline for performance. CDLH represents the performance skyline and runtime baseline, while CDLH-appx. serves as a strong baseline for both runtime and performance. Results are summarized in Table 1.

Standard Decoding Methods.When comparing greedy decoding with nucleus sampling, we observe that nucleus sampling does not improve constraint satisfaction performance. This method was proposed to increase diversity in generation (Holtzman et al., 2019) and we observe increased diversity does not contribute to better adherence to constraints. For a threshold of \(p=0.9\), the increased diversity in model responses leads to a significant drop in constraint satisfaction performance. Specifically, on the CommonGen task, performance decreases by \(\sim 3.6-17.2\%\) compared to using a lower threshold of \(p=0.5\). We observe that the performance remains lower than greedy decoding even with \(p=0.5\). In contrast, beam search (\(beam\_width=3\) for CommonGen and \(beam\_width=5\) for HTG) achieves higher constraint satisfaction in many cases, benefiting from broader search space exploration relative to greedy decoding. However, it incurs a significant runtime overhead compared to greedy decoding and consistently underperforms compared to the lookahead-based methods. Hence, greedy decoding serves as an ideal skyline for runtime efficiency and a baseline for performance

CD Methods.We first observe, our proposed baseline CDLH-Appx achieves better performance compared to other baselines. In constraint satisfaction, it outperforms the greedy baseline by \(\sim 4-27\%\) and \(\sim 5-14\%\) in CommonGen and HTG tasks, respectively. It also gains \(\sim 2.3-5.4\times\) and \(\sim 2.2-7.2\times\) speedup compared to CDLH, the runtime baseline, in CommonGen and HTG tasks, respectively, overall serving as a strong baseline.

Next, we observe that in both tasks, our proposed approach, CDSL, comfortably outperforms the constraint (greedy) and runtime (CDLH) baselines across all LLM pairs and families. It achieves \(\sim 4-21\%\) hard constraint satisfaction gain in CommonGen and a gain \(\sim 4-10\%\) in HTG, compared to greedy decoding. Our algorithm achieves \(\sim 3-5.5\times\) speedup in CommonGen and \(\sim 3.2-12.15\times\) speedup in HTG, compared to the runtime baseline CDLH. The constraint satisfaction performance of our method is slightly lower in most model pairs compared to our proposed baseline CDLH-Appx. However, in the case of almost all LLM pairs, our approach outperforms CDLH-Appx. in runtime by a good margin, proving the efficacy of our approach. We observe that our approach achieves speedup by reducing the overall number of LLM calls per token (detailed statistics are shown in Tables 5, 6 in Appendix E). In essence, our approach achieves a significant runtime reduction compared to the state-of-the-art CDLH method, maintaining a balanced trade-off between constraint satisfaction and runtime efficiency.

### Ablation Study

In this section, we discuss the effect of different hyperparameters in our approach using the target-draft pairs (Bloomz-7.18, Bloomz-1.7B), in the CommonGen task. The learning curves are generated using the validation set as described in Section 4.4 and are presented in Figure 2. Other learning

\begin{table}
\begin{tabular}{l|c|c|c c||c|c} \hline \multirow{2}{*}{**Target**} & \multicolumn{1}{c|}{} & \multicolumn{2}{c||}{**CommonGen**} & \multicolumn{2}{c}{**Harness Text Generation**} \\ \cline{3-6}  & **Approaches** & **Speedup** & **\% Constraint Satisfaction** & **Speedup** & **\% Harness** \\  & **LLM** & & **Soft** & **Hard** & & **Response** \\ \hline \multirow{9}{*}{**Cbl.**} & **Greedy Decoding** & \(8.62\times\) & \(86.12\%\) & \(68.08\%\) & \(14.72\times\) & \(80.30\%\) \\  & **Beam Search** & \(2.98\times\) & \(87.93\%\) & \(63.7\%\) & \(3.02\%\) & \(68.8\%\) \\  & **Nucleus Sampling (\(p=0.9\))** & \(8.62\times\) & \(72.17\%\) & \(37.1\%\) & \(14.72\times\) & \(75.05\%\) \\  & **Nucleus Sampling (\(p=0.5\))** & \(8.62\times\) & \(82.91\%\) & \(54.3\%\) & \(14.72\times\) & \(74.89\%\) \\  & **CD with Lookahead Heuristic (CDLH)** & \(1\times\) & \(97.94\%\) & \(93.10\%\) & \(1.00\%\) & \(91.75\%\) \\  & **CDLH with Approximate Lookahead** & & & & & \\  & **Lookhead with OPT-12SPR** & \(5.40\times\) & \(93.87\%\) & \(81.07\%\) & \(7.17\%\) & \(89.65\%\) \\  & **Localized with OPT-3SPR** & \(4.10\times\) & \(95.24\%\) & \(84.40\%\) & \(4.90\%\) & \(90.35\%\) \\  & **Cooked with OPT-1.3B** & \(3.90\times\) & \(96.07\%\) & \(87.5\%\) & \(4.34\times\) & \(90.60\%\) \\  & **Oursy CMW Speculative Lookahead** & & & & & \\  & Draft with OPT-12SPR** & \(5.54\times\) & \(93.35\%\) & \(79.80\%\) & \(12.15\times\) & \(87.85\%\) \\  & **Draft with OPT-1.3B** & \(4.51\times\) & \(94.07\%\) & \(81.20\%\) & \(9.05\times\) & \(85.65\%\) \\  & **Draft with OPT-1.3B** & \(4.97\times\) & \(94.38\%\) & \(81.70\%\) & \(8.90\times\) & \(86.05\%\) \\ \hline \multirow{9}{*}{**Cbl.**} & **Greedy Decoding** & \(8.91\times\) & \(96.42\%\) & \(88.30\%\) & \(13.98\times\) & \(72.10\%\) \\  & **Beam Search** & \(3.08\times\) & \(94.60\%\) & \(82.6\%\) & \(2.83\times\) & \(78.8\%\) \\  & **Nucleus Sampling (\(p=0.9\))** & \(8.91\times\) & \(91.20\%\) & \(73.3\%\) & \(13.98\times\) & \(72.6\%\) \\  & **Nucleus Sampling (\(p=0.5\))** & \(8.91\times\) & \(95.01\%\) & \(83.9\%\) & \(13.98\times\) & \(72.85\%\) \\  & **CD With Lookahead Heuristic (CDLH)** & \(1\times\) & \(98.97\%\) & \(96.40\%\) & \(1.00\times\) & \(87.6\%\) \\  & **CDLH with Approximate Lookahead** & & & & \\  & Lookahead with B100mz-56MM & \(2.50\times\) & \(97.76\%\) & \(92.30\%\) & \(2.77\times\) & \(85.8\%\) \\  & **Lockboard with B100zmz-1.7B** & \(2.30\times\) & \(98.19\%\) & \(93.90\%\) & \(2.48\times\) & \(86.9\%\) \\  & **(Oursy CMW Speculative Lookahead** & & & & & \\  & Draft with B100nz-56MM & \(3.49\times\) & \(97.82\%\) & \(92.50\%\) & \(3.24\times\) & \(82.20\%\) \\  & **Draft with B100nz-1.7B** & \(3.49\times\) & \(97.68\%\) & \(92.10\%\) & \(3.42\times\) & \(82.15\%\) \\ \hline \multirow{9}{*}{**Cbl.**} & **Greedy Decoding** & \(9.0\times\) & \(95.18\%\) & \(88.70\%\) & \(15.35\times\) & \(91.75\%\) \\  & **Beam Search** & \(3.11\times\) & \(96.07\%\) & \(88.0\%\) & \(3.10\%\) & \(91.75\%\) \\  & **Nodeus Sampling (\(p=0.9\))** & \(9.0\times\) & \(94.21\%\) & \(82.1\%\) & \(15.35\times\) & \(87.85\%\) \\  & **Nucleus Sampling (\(p=0.5\))** & \(9.0\times\) & \(95.50\%\) & \(85.7\%\) & \(15.35\times\) & \(89.05\%\) \\  & **CD with Lookahead Heuristic (CDLH)** & \(1\times\) & \(98.85\%\) & \(96.20\%\) & \(1.00\times\) & \(97.15\%\) \\  & **CDLH with Approximate Lookahead** & & & & \\  & Lookahead with Opennet \(5.-9.5\)B & \(2.50\times\) & \(98.02\%\) & \(94.30\%\) & \(2.40\times\) & \(97.3\%\) \\  & **Lookahead with Opennet \(5.-1.88\)** & \(2.30\times\) & \(98.68\%\) & \(95.60\%\) & \(2.22\times\) & \(96.7\%\) \\  & **(Oursy CMW Speculative Lookahead** & & & & & \\  & Draft with Opennet \(5.-9.5\)B & \(3.01\times\) & \(97.65\%\) & \(93.00\%\) & \(5.38\times\) & \(95.55\%\) \\  & Draft with Opennet \(5.-1.8\)B & \(2.96\%\) & \(98.11\%\) & \(93.70\%\) & \(6.25\times\) & \(95.65\%\) \\ \hline \end{tabular}
\end{table}
Table 1: Runtime and constrained satisfaction performance of different baselines and our proposed approach in the CommonGen and Harmless Text Generation tasks. Here CD stands for “Constrained Decoding”. The skyline and baseline runtime and performance are highlighted in blue and red, respectively. The speedup is calculated with respect to the runtime baseline, CDLH. Hard rejection method is used in our approach.

 

curves and ablations can be found in Appendix E.

**Hard Rejection vs. Speculative Sampling: Across all learning curves, we observe that the speculative sampling method achieves slightly better speedup, however, the performance is almost always lower when compared to hard rejection. In speculative sampling, the token rejection decision is made with some confidence by relying on the probability distributions of \(M_{p}\) and \(M_{q}\). Hence, although a good approximation, it slightly deviates from the target LLMs actual output distribution, consequently performance decreases.**

**Effect of the reward score threshold, \(r_{t}\): We observe, as the reward score increases, the speedup decreases and the performance increases with hard rejection. This is expected by design, because a stricter reward score threshold ensures more rejection of drafted tokens, hence, lower speedup, however, higher constraint satisfaction.**

**Effect of the approval score threshold, \(a_{t}\): As we increase the approval score threshold, the speedup and performance decrease a bit. This indicates that increasing reliability only on the approval by the target LLM does not improve the constraint satisfaction performance, rather the reward function ensures high performance.**

**Effect of \(b\): The parameter \(b\) controls how many tokens are generated by the target LLM in the case when the drafted tokens yield low acceptance rate. We found that the speedup is highest with \(b\)=1 and goes down at \(b\)=2. Performance goes down as we increase the value of \(b\). These trends are expected because higher value of \(b\) indicates more reliance on the target LLM which is both time consuming and sub-optimal for performance.**

**Effect of LLM pre-training and size: We observe, speedup and performance varies across LLM families, precisely, they depend on the acceptance rate between the target and draft LLMs and the expertise of the target and draft LLMs in the task (e.g., when decoded using greedy decoding). It indicates that the pretraining mechanism of the LLMs plays a role in speculative decoding based approaches which is in line with existing research Zhou et al. (2024). For example, as shown in Table 1, the greedy decoding performance of OPT-13B and Bloomz-7.18 are significantly low compared to the other LLMs in the CommonGen and HTG tasks, respectively; as verified by qualitative assessments in Appendix F (the greedy performance for the draft LLMs are reported in Tables 7 and 8 in Appendix E). As a result, with our approach, the constraint satisfaction performance decreases across both tasks in these LLM families. Additionally, as the size difference between the draft and target LLMs grows, we observe greater speedup (e.g., in

Figure 2: Effect of different hyperparameters on **runtime** ((a), (b), (c)) and **constraint satisfaction performance** ((d), (e), (f)) in the **CommonGen** task, for the model pairs **(Bloomz-7.18, Bloomz-1.7B)** as (target, draft). Approval, reward thresholds, and b values are kept as \(0.3\), \(0.3\), \(0\), respectively, when they are fixed.

 

the OPT family models), verifying the principle of speculative decoding.

## 6 Related Work

Speculative decoding (SD) has emerged as a technique for reducing LLM inference time with a faster small LM Leviathan et al. (2023); Chen et al. (2023); Kim et al. (2024); Xia et al. (2024). Different optimizations and variations have been further proposed on top of SD. For example, usage of a layer-skipped lighter version of the target LLM as a draft Zhang et al. (2024b), using a segment of the model as draft Liu et al. (2024a), retrieval as draft He et al. (2024), recurrent drafter Zhang et al. (2024a), tree-based drafting Jeon et al. (2019), graph structured SD for managing more than one hypothesis Gong et al. (2024), etc. Other approaches includes incorporation of contrastive learning is SD Yuan et al. (2024), focusing on more effective tokens during SD Lin et al. (2024), knowledge distillation in drafts for better acceptance Zhou et al. (2024), online SD Liu et al. (2024), etc. Another line of research focuses on scalability, robustness Chen et al. (2024), and parallelism for SD Qian et al. (2024); Sun et al. (2024); Spector and Re (2024).

In a different paradigm, constrained decoding based approaches are used as an alignment technique for LLMs at post-training phase Mudgal et al. (2024). The most effective approaches Cheng et al. (2022); Roy et al. (2024) for constrained decoding rely on lookahead-based approaches Lu et al. (2022). As a result, controlled generation Deng and Raffel (2023); Beurer-Kellner et al. (2015) with lookahead heuristic is prohibitively expensive in terms of runtime. In this paper, we combine the two paradigms, constrained decoding with lookahead heuristic and speculative decoding, with a goal to improve the runtime of such approaches while preserving the performance.

## 7 Conclusion

We propose constrained decoding with speculative lookaheads, a novel algorithm that combines the principles of constrained decoding with lookahead heuristics and speculative decoding, in order to improve the runtime efficiency of constrained decoding approaches. Our approach yields significant speedup over the constrained decoding with lookahead heuristics (CDLH) approach without major drop in performance and contributes to adaptability of such approaches in real-world use cases.

### Limitations

We identify the following limitations in our study.

**Runtime-Performance Tradeoff:** We note that our algorithm shows a runtime vs. performance tradeoff when compared to the performance of the skyline approach for performance, constrained decoding with lookahead heuristics. However, as shown in section 5, the performance tradeoff is not drastic if compared to the skyline for runtime, greedy decoding. We believe our approach will still increase the adaptability of the constrained decoding with lookahead heuristics based approaches in many real world online applications, despite this slight tradeoff in performance.

**Target-Draft Pairs:** In our study, we experiment with both target and draft models belonging to the same model family. This approach follows the principles of speculative decoding, which depend on achieving high acceptance rates between the target-draft model pair to realize enhanced run-time gains. Different LLM families use varying types of tokenizers, preventing the straightforward combination of the target and draft models from different LLM families. Since the output logits of both target and draft models are crucial for token verification, the presence of different tokenizers complicates this process, limiting our choice of target-draft model pairs. We note that this is a general limitation of state of the art speculative decoding-based approaches.

**Reward Function:** In our proposed algorithm, the reward function is distinctly isolated from other components, allowing a straightforward adaptation to different types of constraint. That requires constructing task specific reward functions and if a downstream task requires a resource-intensive reward function, the runtime will increase in constrained decoding with lookahead heuristic based approaches including our proposed algorithm.

**Parameter Optimization:** Our algorithm requires tuning different hyperparameters (e.g., acceptance and reward thresholds \(a_{t}\) and \(r_{t}\), \(d\), etc) because of the varying capabilities and acceptance rates among different target-draft pairs. As a result, a task specific validation set is required in order to get the best out of our approach, however, it facilitates task specific adaptability of our approach.

**LLM Size:** Due to computation resource limitations, we apply our approach on target LLM sizes up to \(13B\). However, we note that our approach is readily applicable in even larger LLMs 



**Open Source LLM Usage:** The CDLH and SD approaches in principle, require access to the LLM logits. As a result, we apply our approach on open-source LLMs only. Moreover, it is difficult to find closed source LLM families with models in varying parameter sizes so that they can be used as drafts in the speculative decoding based approaches.

**Batch Implementation:** Application of speculative decoding in a batched setting is a research area on its own (Qian et al., 2024). Hence, adaptation of our algorithm in a batched setting is out of scope for this paper and we leave it as our future work.

## Ethics Statement

We present all implementation and dataset details for reproducibility of our study (some parts are in the Appendix). The datasets and LLMs used in this paper are publicly available and permitted for scientific research. We perform meta evaluation of a model judge and the human evaluators are compensated in accordance with the standards in such tasks. The human evaluation details are presented in Appendix B. We perform extensive ablation studies in order to provide the readers an idea about potential error patters and risks associated to the proposed methods.

## Acknowledgements

We gratefully acknowledge Haifeng Qian, Sujan Kumar Gonugondla, Vinayshekhar Bannihatti Kumar, Sailik Sengupta, Daniel Melcer, Ting-Yun Chang, Anoop Deoras and the members of the AWS AI Labs for providing valuable feedback at different stages of this work. We would also like to thank the anonymous reviewers for their insightful comments.

## References

* Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. _arXiv preprint arXiv:2204.05862_.
* Beurer-Kelner et al. (2019) Luca Beurer-Kelner, Marc Fischer, and Martin Vechev. Guiding llms the right way: Fast, non-invasive constrained generation. In _Forty-first International Conference on Machine Learning_.
* Chen et al. (2023) Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. _Preprint_, arXiv:2302.01318.
* Chen et al. (2024) Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. 2024. Sequoa: Scalable, robust, and hardware-aware speculative decoding. _arXiv preprint arXiv:2402.12374_.
* Cheng et al. (2022) Yi Cheng, Wenge Liu, Wenjie Li, Jiashuo Wang, Ruihui Zhao, Bang Liu, Xiaodan Liang, and Yefeng Zheng. 2022. Improving multi-turn emotional support dialogue generation with lookahead strategy planning. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3014-3026.
* Dekoninck et al. (2022) Jasper Dekoninck, Marc Fischer, Luca Beurer-Kellner, and Martin Vechev. Controlled text generation via language model arithmetic. In _The Twelfth International Conference on Learning Representations_.
* Deng and Raffel (2023) Haikang Deng and Colin Raffel. 2023. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 11781-11791, Singapore. Association for Computational Linguistics.
* Freitag and Al-Onaizan (2017) Markus Freitag and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. In _Proceedings of the First Workshop on Neural Machine Translation_, pages 56-60, Vancouver. Association for Computational Linguistics.
* Gong et al. (2024) Zhuocheng Gong, Jiahao Liu, Ziyue Wang, Pengfei Wu, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. 2024. Graph-structured speculative decoding. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 11404-11415, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* He et al. (2024) Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. 2024. REST: Retrieval-based speculative decoding. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 1582-1595, Mexico City, Mexico. Association for Computational Linguistics.
* Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. _arXiv preprint arXiv:1904.09751_.
* Huang et al. (2024) James Y. Huang, Sailik Sengupta, Daniele Bonadiman, Yi an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. 2024. Deal: Decoding-time alignment for large language models. _Preprint_, arXiv:2402.06147.
* Jeon et al. (2017) Wonseok Jeon, Mukul Gagrani, Raghavy Goel, Junyoung Park, Mingu Lee, and Christopher Lott. Recursive speculative decoding: Accelerating llm inference via sampling without replacement. In _ICLR

2024 Workshop on Large Language Model (LLM) Agents_.
* Kim et al. [2024] Sehoon Kim, Kartikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. 2024. Speculative decoding with big little decoder. _Advances in Neural Information Processing Systems_, 36.
* Leviathan et al. [2023] Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In _International Conference on Machine Learning_, pages 19274-19286. PMLR.
* Lin et al. [2020] Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. Commongen: A constrained text generation challenge for generative commonsense reasoning. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1823-1840.
* Lin et al. [2024] Chi-Heng Lin, Shikhar Tuli, James Smith, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. 2024. SLiM: Speculative decoding with hypothesis reduction. In _Findings of the Association for Computational Linguistics: NAACL 2024_, pages 1005-1017, Mexico City, Mexico. Association for Computational Linguistics.
* Liu et al. [2024a] Jiahao Liu, Qifan Wang, Jingang Wang, and Xunliang Cai. 2024a. Speculative decoding via early-exiting for faster LLM inference with Thompson sampling control mechanism. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 3027-3043, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Liu et al. [2024b] Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding. In _Forty-first International Conference on Machine Learning_.
* Liu et al. [2024c] Zhuorui Liu, Chen Zhang, and Dawei Song. 2024c. How speculative can speculative decoding be? In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_, pages 8265-8275, Torino, Italia. ELRA and ICCL.
* de Meta Lama Team [2024] AI @ Meta Lama Team. 2024. The llama 3 herd of models. _Preprint_, arXiv:2407.21783.
* Lu et al. [2022] Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowal Zellers, Noah A. Smith, and Yejin Choi. 2022. NeuroLogic a*esque decoding: Constrained text generation with lookahead heuristics. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 780-799, Seattle, United States. Association for Computational Linguistics.
* Lu et al. [2021] Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Neurologic decoding: (un)supervised neural text generation with predicate logic constraints. _Preprint_, arXiv:2010.12884.
* Mudgal et al. [2023] Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, at al. Controlled decoding from language models. In _F forty-first International Conference on Machine Learning_.
* Muennighoff et al. [2023] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, et al. 2023. Crosslingual generalization through multitask finetuning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15991-16111.
* Qian et al. [2024] Haifeng Qian, Sujan Kumar Gonugondla, Sungsoo Ha, Mingyue Shang, Sanjay Krishna Oouda, Ramesh Nallapati, Sudipta Sengupta, Xiaofei Ma, and Anop Deoras. 2024. BASS: Batched attention-optimized speculative sampling. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 8214-8224, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Roy et al. [2024] Shamik Roy, Salilik Sengupta, Daniele Bonadiman, Saab Mansour, and Arshit Gupta. 2024. FLAP: Flow-adhering planning with constrained decoding in LLMs. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 517-539, Mexico City, Mexico. Association for Computational Linguistics.
* Spector and Re [2023] Benjamin Frederick Spector and Christopher Re. Accelerating llm inference with staged speculative decoding. In _Workshop on Efficient Systems for Foundation Models@ ICML2023_.
* Sun et al. [2024] Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beiranmi, Himanshu Jain, and Felix Yu. 2024. Spectr: Fast speculative decoding via optimal transport. _Advances in Neural Information Processing Systems_, 36.
* Team [2024] Owen Team. 2024. Introducing qwen1.5.
* Wan et al. [2023] David Wan, Mengwen Liu, Kathleen Mckeown, Markus Dreyer, and Mohit Bansal. 2023. Faithfulness-aware decoding strategies for abstractive summarization. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 2864-2880.
* Xia et al. [2024] Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. 2024. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 7655-7671, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Yan et al. [2024] Minghao Yan, Saurabh Agarwal, and Shivaram Venkataraman. 2024. Decoding speculative decoding. _Preprint_, arXiv:2402.01528 

Hongyi Yuan, Keming Lu, Fei Huang, Zheng Yuan, and Chang Zhou. 2024. Speculative contrastive decoding. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 56-64, Bangkok, Thailand. Association for Computational Linguistics.
* Zhang et al. [2024a] Aonan Zhang, Chong Wang, Yi Wang, Xuanyu Zhang, and Yunfei Cheng. 2024a. Recurrent drafter for fast speculative decoding in large language models. _arXiv preprint arXiv:2403.09919_.
* Zhang et al. [2024b] Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2024b. Draft& verify: Lossless large language model acceleration via self-speculative decoding. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11263-11282, Bangkok, Thailand. Association for Computational Linguistics.
* Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models. _Preprint_, arXiv:2205.01068.
* Zhou et al. [2022] Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-Francois Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. In _The Twelfth International Conference on Learning Representations_.

## Appendix A Constrained Decoding with Lookahead Heuristic (CDLH)

The constrained decoding with lookahead heuristic approach is illustrated in Algorithm 2.

```
1:Input: LLM \(M_{p}\), Reward function \(\mathcal{R}\), Prompt \(pfx\), Lookahead len \(d\), Max sequence len \(l_{m}\), Beam size \(k\)
2:Initialize: Generated sequence \(\mathcal{T}\leftarrow\emptyset\)
3:while\(len(T)<l_{m}\)do
4:\(\triangleright\) prompt \(M_{p}\) with \(prefix\) to get the next token probability distribution, \(s(x)\) and select top-k.
5:\(s(x)\gets M_{p}(pfx)\)
6:\([x_{1},\dots,x_{k}]\leftarrow\operatorname*{argmax\,topk}s(x)\triangleright \text{select top-k}\)
7:\(\triangleright\) roll out lookahead for each top-k candidate, score using \(\mathcal{R}\), and select the best
8:\(best\_new\_token,best\_reward\leftarrow\emptyset,-\infty\)
9:for\(i=1\)to k do
10:\(lh\leftarrow\emptyset\triangleright\) initialize lookahead
11:\(\triangleright\) autoregressively generate lookaaheads
12:for\(j=1\)to d do
13:\(\triangleright\) l CDLH-appx., below generation is done using a smaller LLM, \(M_{q}\), instead of \(M_{p}\).
14:\(l\leftarrow\arg\max M_{p}(pfx+x_{i}+lh)\)
15:\(lh\gets th+l\)
16:endfor
17:\(r\leftarrow\mathcal{R}(pfx+lh)\triangleright\) scoring the lookaheads
18:if\(r>best\_reward\)then
19:\(best\_new\_token\gets x_{i}\)
20:\(best\_reward\gets r\triangleright\) selecting the best
21:endif
22:\(pfx\gets pfx+best\_new\_token\)
23:\(T\gets T+best\_new\_token\)
24:endfor
25:endwhile
26:Output:Generated sequence \(\mathcal{T}\)
```

**Algorithm 2** Constrained Decoding with Lookahead Heuristic (CDLH)

## Appendix B Meta Evaluation of the Model Judge

To evaluate the harmless text generation task, we use an external LLM, Llama-Guard-3-8BLlama Team (2024), which is especially fine-tuned for content safety classification task. To assess the reliability of this LLM judge we perform a meta evaluation.

First, we collect the model generations in the validation set using all the baselines. Then we prompt Llama-Guard for evaluating if these responses are safe or unsafe. We use the same prompt that was proposed in the source paper (Llama Team, 2024). Then we randomly sample data points from each of the safe and unsafe buckets and use them for human evaluation. In this manner, we sample \(95\) (potentially) unsafe and \(105\) (potentially) safe responses, summing up to \(200\) total data points. Initially, we use Amazon SageMaker Ground Truth for the evaluation, where the annotators are presented with the same prompt as Llama-Guard (as shown in Figure 

5) along with some positive and negative examples and are instructed to mark the Assistant response as safe or unsafe. Note that, the annotators are not presented with the Llama-Guard generated labels. The annotators are warned that the contents may contain harmful responses. They are US-based and aged 18+. They are paid \(\$1.080\) for each annotation and each data point is annotated by three different annotators.

We find very low inter-annotator agreement in the \(200\) data points. For example, only \(46\%\) of the data points had a perfect agreement among all three annotators, indicating the high noise in the Amazon SageMaker Ground Truth annotations. As a result, one of the authors of this paper performs a re-work on the data points. For that, we filter for the data points where at least one of three human annotators or the LLM judge disagrees, and we manually annotate them. In this manner, we annotated \(134\) data points, which results in \(200\) responses that are annotated for safe (\(99\) responses) vs. unsafe (\(101\) responses).

We evaluate the efficacy of the LLM judge against the gold labels in these \(200\) responses and found an F1 score of \(94\%\), indicating a reliable judgement by this LLM judge.

## Appendix C Hyperparameter Tuning

We tune several hyperparameters for our model: draft length \(d\), acceptance threshold \(a_{t}\), reward score threshold \(r_{t}\), \(b\), and the sampling method.

The search space for these hyperparameters in the two tasks are summarized in Table 2.

We determine the draft length, \(d\) empirically in our initial stages of experiments. We determined the best values of draft length for the CommonGen and HTG tasks to be \(3\) and \(5\), respectively. For a fair comparison, we report the results with the baselines/skylines, CDLH and CDLH-appx. with the same lookahead (or draft) length in the test and validation sets.

We set the value of \(k=3\) (top-k) across all models while choosing a path with the highest constraint satisfaction. The top-k tokens are selected based on LLM logits.

For determining the best values of the other hyperparameters in our model, we keep the value of \(d\) fixed to the best values in the corresponding tasks. We benchmark the validation set with all combinations of the other three hyperparameters \(a_{t}\), \(r_{t}\), \(b\). We determine the best value of the hyperparameters for a (draft, target) LLM pair by considering both the speedup and constraint satisfaction performance. For this, we first select the best \(10\) hyperparameter combinations based on speedup. From the best \(10\), we select the one with the highest performance as the final hyperparameter. The best hyperparameters following this selection criteria for all LLM pairs are summarized in Table 3. We also find that across all model pairs, hard rejection performs the best in terms of constraint satisfaction performance, whereas, speculative sampling shows slightly better speedup (discussed in details in Section 5). As a result, we report the results on the test set using the hard rejection method in Table 1.

## Appendix D Additional Experimental Settings

In this section, we provide additional experimental settings.

**Beam search implementation:** For the Beam Search algorithm, we maintain the beam width, denoted by \(w\), to match the lookahead length specified in the CDSL method for both tasks. Specifically, for the CommonGen task, the beam search is executed with \(w=3\), and it is applied with \(w=5\) for the HTG task, to ensure a fair comparison.

**Prompts templates:** The prompt templates for the CommonGen and HTG tasks are shown in Figures 3, 4, respectively. The prompt template for Llama-guard evaluation is presented in Figure 5.

\begin{table}
\begin{tabular}{c|l c} \hline \multicolumn{1}{c|}{**Task**} & **Hyperparameter** & **Search Space** \\ \hline \multirow{7}{*}{**CommonGen**} & Draft length \(d\) & \(3,5\) \\  & Acceptance threshold \(a_{t}\) & \(0.3,0.6,0.9\) \\  & Reward score threshold \(r_{t}\) & \(0.3,0.5,0.6,0.9\) \\  & \(b\) & \(0,1,2\) \\ \hline \multirow{7}{*}{**HTG**} & Draft length \(d\) & \(3,5\) \\  & Acceptance threshold \(a_{t}\) & \(0.3,0.4,0.6,0.8\) \\ \cline{1-1}  & Reward score threshold \(r_{t}\) & \(0.05,0.1,0.15,0.2,0.3,0.4\) \\ \cline{1-1}  & \(b\) & \(0,1,2\) \\ \hline \end{tabular}
\end{table}
Table 2: Hyperparameter tuning.

\begin{table}
\begin{tabular}{c l r} \hline \multicolumn{1}{c|}{**Task**} & **(Dark, Large LID pairs** & **Selected Hyperparameters** \\ \hline \multirow{3}{*}{**Comment**} & (97-10, 97-129) & \(d=3,0.3,0.6,0.8\) \\  & (97-10, 97-129) & \(d=3,0.3,0.6,0.8\) \\  & (97-10, 97-129) & \(d=3,0.3,0.6,0.8\) \\
**Consection** & Bloom-17-11, Bloom-5-70 & \(d=3,0.3,0.6,0.8\) \\  & Bloom-17-11, Bloom-7-70 & \(d=3,0.3,0.6,0.8\) \\  & Quan-17-80, Qu 



**Infrastructure:** We ran all of our experiments in machines containing 8 NVIDIA(r) V100 Tensor Core GPUs with 32 GiB of memory each, 96 custom Intel(r) Xeon(r) Scalable (Skylake) vCPUs, and 1.8 TB of local NVMe-based SSD storage. Our codebase is implemented using PyTorch.

## Appendix E Ablation

In this section, we report additional ablation studies and statistics.

**Runtime cost coefficient, \(c\):** We report the runtime cost coefficient, \(c\) between different target and draft pairs in the two tasks in Table 5.

Figure 4: Prompt template used for harmless text generation task.

Figure 5: Prompt template used for scoring the generations using Llama-Guard-3-8B in the harmless text generation task.

Figure 3: Prompt template used for CommonGen task.





**Number of LLM calls statistics per token: Speedup and number of LLM calls for generating one token in different baselines and our proposed approach in the CommonGen and Harmless Text Generation tasks are reported in Table 6.**

**Ablation on speculative decoding: We present ablation on speculative decoding using different rejection techniques and LLM families in the CommonGen and Harmless Text Generation tasks in Tables 7 and 8.**

**Learning curves on validation set: Learning curves for different hyperparameters for OPT and Owen1.5-Chat family models in the CommonGen task are shown in Figures 7 and 8, respectively. Learning curves for different hyperparameters for Bloomz, OPT, and Owen1.5-Chat family models in the Harmless Text Generation task are shown in Figures 9, 10, and 11, respectively.**

## Appendix F Qualitative Evaluation

In this section, we present a few qualitative instances that showcase both the benefits and drawbacks of our method.

We observe that when applying hard-rejection, the outputs from our method are quite similar to those obtained using the CD with Approximate Lookahead approach. The similarity is seen because in our approach, the draft model generates tokens, which are then verified by both the target LLM and the reward-function. Additionally, we discover that models that are instruction-tuned deliver higher quality outputs compared to those that are not instruction-tuned, such as OPT-13B. An example illustrating this for the CommonGen task can be found in Table 4.

In the Harmless Text Generation task, we utilize the system instruction depicted in Figure 4. Through manual inspection, we see that the Qwen models deliver good responses compared to other LLM families, reinforcing the quantitative assessments from the Llama-guard evaluation. We also observe that the Bloomz model pairs provide poor responses, as demonstrated by an example in Figure 6.



Your task is to provide appropriate response to the given question. Please refrain from giving any legally questionable or dangerous advice in your responses.

Human: Where can I dump toxic waste? Assistant: There are special places where you can dispose of certain kinds of waste. What are yout trying to dump? Human: Old electronics, medical equipment, drugs, dead bodies Assistant:

# Greedy: - Yes, that's right.

CDLH with Exact Lookahead: - No. You should call your city hall to get the appropriate place to dispose your trash.

# CDLH with Approx. Lookahead (Draft: bloomz-1.7b): - No. You can't dump anything like that.

# Ours CD with Spec. Lookahead (Draft: bloomz-1.7b): - No, I'm not.

Figure 6: We note that responses from bloomz models tend to be poor overall. Nevertheless, upon manual examination, we see that results from our approach are usually better than greedy decoding and are on par with CDLH with Approximate Lookahead, if not slightly inferior.

Figure 7: Effect of different hyperparameters on **runtime** ((a), (b), (c)) and **constraint satisfaction performance** ((d), (e), (f)) in the CommonGen task, for the model pairs (OPT-13B, OPT-1 .3B) as (target, draft). Approval, reward thresholds, and b values are kept \(0.3\), \(0.3\), \(0\), respectively when they are fixed.

 



\begin{table}
\begin{tabular}{c|l|c|c||c|c|c c} \hline  & & \multicolumn{3}{c||}{**Commonen**} & \multicolumn{2}{c}{**iaralless Text Generation**} \\
**Target** & **Approaches** & **Speedup** & **Number of LLM Calls Per Token** & **Speedup** & **Number of LLM Calls Per Token** \\
* & 1.0 & 14.72\(\times\) &
- & 1.0 \\  & **CD with Lookahead Heuristic (CDLI)** & \(1\times\) &
- & 8.62 & 1.00\(\times\) &
- & 14.72 \\  & **Children with Approximate Lookahead** & & & & & & \\  & Lookhead with OPT-129K & \(5.40\times\) & \(7.65\) & 1.0 & 7.17\(\times\) & 13.69 & 1.0 \\  & Lookhead with OPT-158W & \(4.10\times\) & \(7.62\) & 1.0 & 4.90\(\times\) & 13.63 & 1.0 \\  & Lookedable with OPT-138 & \(3.90\times\) & \(7.6\) & 1.0 & 4.34\(\times\) & 13.6 & 1.0 \\  & **(Ours CD with Speedative Lookahead** & & & & & \\  & Draft with OPT-129K & \(5.54\times\) & \(8.79\) & 0.88 & 12.15\(\times\) & 8.51 & 0.56 \\  & Draft with OP-158W & \(4.51\times\) & \(7.27\) & 0.85 & 9.05\(\times\) & 7.19 & 0.57 \\  & Draft with OPT-138 & \(4.97\times\) & \(6.13\) & 0.78 & 8.90\(\times\) & 6.47 & 0.51 \\ \hline \multirow{10}{*}{**Constration**} & _Greedy Decoding_ & \(8.91\times\) &
- & 1.0 & 13.98\(\times\) &
- & 1.0 \\  & **CP with Lookhead Heuristic (CDLI)** & \(1\times\) &
- & 8.91 & 1.00\(\times\) &
- & 13.98 \\  & **CDLI with Approximate Lookahead** & & & & & & \\  & Lookahead with Bloom-564W & \(2.50\times\) & \(8.04\) & 1.0 & 2.77\(\times\) & 13.11 & 1.0 \\  & Lookedable with Bloom-17B & \(2.30\times\) & \(8.26\) & 1.0 & 2.48\(\times\) & 12.98 & 1.0 \\  & **(Ours CD with Speedative Lookahead** & & & & & \\  & Draft with Bloom-568W & \(3.49\times\) & \(5.62\) & 0.79 & 3.24\(\times\) & 11.72 & 0.7 \\  & Draft with Bloom-17B & \(3.49\times\) & \(5.33\) & 0.73 & 3.42\(\times\) & 9.68 & 0.63 \\ \hline \multirow{10}{*}{**Constration**} & _Greedy Decoding_ & \(9.0\times\) &
- & 1.0 & 15.35\(\times\) &
- & 1.0 \\  & **CDLI with LowLoadheuristic (CDLI)** & \(1\times\) &
- & 9.0 & 1.00\(\times\) &
- & 15.35 \\  & **CDLI with Approximate Lookahead** & & & & & \\  & Lookedable with Open1, \(5^{+}\)-5.08 & \(2.50\times\) & \(7.72\) & 1.0 & 2.40\(\times\) & 14.36 & 1.0 \\  & Lookedable with Open1, \(5^{+}\)-1.80 & \(2.30\times\) & \(8.26\) & 1.0 & 2.22\(\times\) & 14.47 & 1.0 \\  & **(Ours CD with Speedative Lookahead** & & & & & \\  & Draft with Open1, \(5^{+}\)-5.08 & \(3.01\times\) & \(6.36\) & 0.84 & 5.38\(\times\) & 6.04 & 0.59 \\  & Draft with Open1, \(5^{+}\)-1.80 & \(2.96\times\) & \(6.44\) & 0.81 & 6.25\(\times\) & 4.8 & 0.49 \\ \hline \end{tabular}
\end{table}
Table 6: Speedup and number of LLM calls for generating one token in different baselines and our proposed approach in the CommonGen and HarmlessTextGeneration tasks. Here CD stands for “Constrained Decoding”. The skyline and baseline runtime and performance are highlighted in blue and red, respectively. The speedup is calculated with respect to the runtime baseline, “CD with exact lookahead”. Hard rejection method was used in our approach. Our proposed approach achieves the best speedup in all cases by reducing the number of target and draft LLM calls per token.




\begin{table}
\begin{tabular}{l c c c|c c|c c}  & **Acceptance rate** & **Avg. draft LLM calls per token** & **Avg. target LLM calls per token** & \multicolumn{3}{c|}{**\% Const. Satisfaction**} \\
**LLMs** & **Acceptance rate** & **Avg. draft LLM calls per token** & **Avg. target LLM Folded by Decoding** & **Speculative Decoding** \\  & & & & & **Hard** & **Soft** & **Hard** & **Soft** \\ \hline

- & 1 & 52 & 83.85 &
**Draft LLMs** & 46.99 & 1.639 & 0.5466 & 1.5 & 21.23 & 54 & 83.31 \\
**OPT-35BM** & 54.26 & 1.4427 & 0.481 & 14 & 56.48 & 51.5 & 82.51 \\
**OPT-1.3B** & 63.86 & 1.227 & 0.409 & 26 & 66.36 & 55.5 & 83.97 \\ \hline

- & 1 & 83.5 & 95.33 &
**Draft LLMs** & 54.50 & 1.462 & 0.488 & 40.5 & 74.90 & 84 & 95.33 \\
**floor-1.7B** & 63.31 & 1.252 & 0.418 & 72.0 & 91.05 & 83.5 & 95.19 \\ \hline

- & 1 & 84.5 & 95.86 &
**Draft LLMs** & & & & & & \\
**Qwen1.5-0.5B-Chat & 48.66 & 1.589 & 0.53 & 72.0 & 92.26 & 85 & 95.99 \\
**Down1.5-1.8B-Chat** & 52.51 & 1.47 & 0.489 & 33.5 & 73.16 & 86.5 & 96.39 \\ \end{tabular} (a) Hard Rejection.

\begin{tabular}{p{85.4pt} p{85.4pt} p{85.4pt} p{85.4pt}|p{85.4pt} p{85.4pt}} \hline
**LLMs** & **Acceptance rate** & **Avg. draft LLM calls per token** & **Avg. target LLM calls per token** & \multicolumn{3}{c|}{**\% Const. Satisfaction Greedy Decoding Hard Soft**} & \multicolumn{3}{c}{**Specificity Decoding Soffe**} \\

- & 1 & 52.0 & 83.85 &
**Draft LLMs** & & & & & & \\
**QPT-125M** & 55.61 & 1.385 & 0.462 & 1.5 & 21.23 & 47 & 80.106 \\
**OPT-35BM** & 65.28 & 1.208 & 0.402 & 14.0 & 56.48 & 55.5 & 84.77 \\
**OPT-1.3B** & 73.29 & 1.059 & 0.353 & 26.0 & 66.36 & 51.5 & 81.97 \\ \hline

- & 1 & 83.5 & 95.33 &
**Draft LLMs** & & & & & & \\
**Sloom-56MM** & 63.71 & 1.237 & 0.412 & 40.5 & 74.90 & 69.5 & 91.05 \\
**Draft LLMs** & & & & & & \\
**Slomo-1.7B** & 73.41 & 1.06 & 0.353 & 72.0 & 91.05 & 68.5 & 90.65 \\ \hline

- & 1 & 84.5 & 95.86 &
**Draft LLMs** & & & & & & \\
**Open1.5-0.5B-Chat & 52.45 & 1.476 & 0.492 & 72.0 & 92.26 & 86 & 95.99 \\
**Down1.5-1.8B-Chat** & 58.09 & 1.333 & 0.445 & 33.5 & 73.16 & 84 & 95.46 \\ \hline \end{tabular} (b) Speculative Sampling.

\end{table}
Table 7: Ablation on speculative decoding using different rejection techniques and LLM families in the CommonGen task.



* [14] M. C. Bader, A. Krizhevsky, and D. V. Kingma, "Adam: a method for stochastic optimization of the problem," in _Advances in neural information processing systems_, vol. 39(2): PMLR, 2015, pp. 768-772.
* [15] J. L. Friedman, R. G. Hinton, and S. H. Zisserman, "A review on machine learning with applications to image recognition," _IEEE Transactions on Image Processing_, vol. 10(1): IEEE, 2015, pp. 1-12.
* [16] T. F. Goodfellow, I. Sutskever, and E. Rabinovici, "Learning from tiny images by reducing internal covariate effects," _arXiv preprint arXiv:1412.6090_, 2014.
* [17] Y. Bengio, P. Salakhutdinov, and N. Srivastava, "Dropout normalization: A simple approach to deep belief networks," _Journal of Machine Learning Research_, vol. 10(1): MIT press, 2015.
* [18] W. Tehgaler, "The impact of deep belief nets on the classification of natural data," _Neural Computation_, vol. 10(1): Springer, 2015.
* [19] X. Chen, Q. Li, and J. Sun, "Deep residual learning for image recognition," _CoRR_, abs/1412.0338v1, 2014.
* [20] Y. Deng, J. Dong, and J. Jia, "Image denoising via deep convolutional network," _ACM Trans. Mach. Learn. Res. Lett._, vol. 10(1): Springer, 2015.

[MISSING_PAGE_POST]

 

Figure 11: Effect of different hyperparameters on **runtime** ((a), (b), (c)) and **constraint satisfaction performance** ((d), (e), (f)) in the **Harmless Text Generation** task, for the model pairs (**Qwen1.5-7B,****Qwen1.5-1.8B**) as (target, draft). Approval, reward thresholds, and b values are kept \(0.3\), \(0.01\), \(0\), respectively when they are fixed.

Figure 10: Effect of different hyperparameters on **runtime** ((a), (b), (c)) and **constraint satisfaction performance** ((d), (e), (f)) in the **Harmless Text Generation** task, for the model pairs (**OPT-13B,****OPT-1.3B**) as (target, draft). Approval, reward thresholds, and b values are kept \(0.3\), \(0.01\), \(0\), respectively when they are fixed.