Fast Large Language Model Collaborative Decoding via Speculation
Authors: Jiale Fu
Yuchu Jiang
Junkai Chen
Jiaming Fan
Xin Geng
Xu Yang
Venue: ICML 2025 poster (2025)
Keywords: LLM Collaborative Decoding
Inference Acceleration
Speculative Decoding
Primary area: deep_learning->large_language_models
TL;DR: We propose a new framework that can accelerate any form of LLM collaborative decoding—including model ensembling, contrastive decoding, and decoding-time realignment—without compromising performance.

Abstract: Large Language Model (LLM) collaborative decoding techniques improve output quality by combining the outputs of multiple models at each generation step, but they incur high computational costs. In this paper, we introduce **Collaborative decoding via Speculation (CoS)**, a novel framework that accelerates collaborative decoding without compromising performance. Inspired by Speculative Decoding—where a small proposal model generates tokens sequentially, and a larger target model verifies them in parallel, our approach builds on two key insights: (1) the verification distribution can be the combined distribution of both the proposal and target models, and (2) alternating each model as the proposer and verifier can further enhance efficiency. We generalize this method to collaboration among *n* models and theoretically prove that CoS is never slower than standard collaborative decoding, typically achieving faster speed. Extensive experiments demonstrate CoS is **1.11x–2.23x** faster than standard collaborative decoding without compromising generation quality. Our code is available at https://github.com/Kamichanw/CoS/.

Fast Large Language Model Collaborative Decoding via Speculation
Jiale Fu 12
Yuchu Jiang
Junkai Chen
Jiaming Fan
Xin Geng 12 Xu Yang
Abstract
(a) Vanilla Collaborative Decoding: (4 M,+ 4x)
dHt (e)
Large Language Model (LLM) collaborative de- coding techniques improve output quality by com- bining the outputs of multiple models at each gen- eration step, but they incur high computational costs. In this paper, we introduce Collahorative decoding via Speculation (CoS), novel frame- work that accelerates collaborative decoding with- out compromising performance. Inspired by Spec- ulative Decoddnngwwher where a small proposal model generates tokens sequentially, and a larger tar- get model verifies them in parallel, our approach builds two key insights: (1) the verification distribution can be the combined distribution of both the proposal and target models, and (2) al- ternating each model as the proposer and verifier can further enhance efficiency. We generalize this method to collaboration among n models and the- oretically prove that CoS is never slower than stan- dard collaborative decoding, typically achieving faster speed. Extensive experiments demonstrate CoS is 1.11x-2.23x faster than standard collab- orative decoding without compromising genera- tion quality. Our code is available at https: / //gitttut com chanw/Cos
Base Model MM,
Combine
Token x, 7.--T, (z)
dill (x
Base Model MM,
x4
(b) Speculative Decoding: (4x,M, +1XM)
P Proposal Model M,
Mll
Verify & R Resample
Target odel MMr
ditl
dill
dill
dil
(c) Collaborative Decoding Speculation: (4x (4x,M,+1xM.)
M,+
flm (y(e)
n Gu
dlmm SxCr)
Vertly & Resample
Combine
M,
f
fth Pole)
drh t.
drfh Nele)
T
Figure I. Comparison of (a) vanilla collaborative decoding, (b) speculative decoding, and (c) collaborative decoding via specula- tion. In (b) and (c), cach discrete blue block represents a probability calculated by one forward pass of MMg while the continuous green block indicates the joint distribution requires only one forward pass of Mp.
to improve next-token prediction and capitalize on the com- plementary strengths of different models. For instance, en- sembling methods (Yu et al., 2024; Huang et al., 2024; Yao et al., 2024) average prediction distributions from multiple models; contrastive decoding (Li et al., 2023; O' O'Brien Lewis, 2023) improves generation quality and reduces hal- lucinations by subtracting the outputs of a smaller model from larger one; and decoding-ting realignment (Liu et al., 2024; Shi et al., 2024) jointly uses an aligned and an un- aligned model during decoding to enable flexible control over aligmment. In this paper, we refer to such approaches collectively as collaborative decoding.
1. Introduction
Recently, large language models (LLMs)) have demonstrated impressive performance across wide range of tasks. Be- yond advances in individual models-such as architectural innovations and training techniques there is increasing in- terest in collaborative approaches involving multiple L.MSs (Lu et al., 2024; Chen et al., 2025). A key class of these methods combines information from multiple models (e.g.. probability distributions or logits) during token generation
'Equal contribution 'Southeast University "Ky Labora- tory of New Generation Artificial Intelligence Technology and Its interdisciplinary Applications (Southeast University), Min- istry of Education, China. Correspondence to: Xu Yang <xuyang-palmel seu.edu.cn..
Despite the significant progress in collaborative decoding, key challenge persists: combining outputs from multiple models requires each model to perform separate forward pass, which substantially slows down inference compared to using a single model. This raises a crucial question: can we speed up collaborative decoding without compromis- ing quality? To address this, we propose Collaborative
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Fast Large Language Model Collaborative Decoding via Speculation
decoding via Speculation (CoS), novel framework that accelerates any form of collaborative decoding while main- taining output quality. The core idea of CoS comes from Speculative Decoding (SD) (Xia et al., 2023; Leviathan et al., 2023).
Sttp 1: Base Models: Myr M,
Step Base Models: Ma M,
Token id
3
4
Taken id 1
2
M,
(xx
M
9: (xt)
M,
M,
(x)
Speculative Decoding technique designed to acceler- ate LLM inference without sacrificing performance. As depicted in Figure 1 (b), uses smaller but more efficient proposal model MM to rapidly generate proposal tokens, which are then verified in parallel by a larger target model MM. The target model accepts a subset of these proposal tokens, enabling the generation of multiple tokens a sin- gle forward pass, thus significantly accelerating inference. Moreover, by employing specific acceptance-rejection cri- teria, the generated tokens can be considered as samples drawn from the distribution of the target model, thus ensur- ing the generation quality. In this paper, we extend specula- tive decoding to LLM collaborative decoding based on the following two observations.
Token
Token By
Step 3: Base Models: M, M,
Step Base Models: M, M,
Token id
4
Token
M,
(x)
(-(x)
M,
(x(z)
(ct
M,
(x)
M,
(x
(>())
Tokan 21-13
Token 17 13--1
Standard Collaborative Decoding
Step 1: Proposer. M, Verifer: M,,
Step Proposer. Verifer: M,
Token id
2
3
4
Token id 1
2
3
M,
(x)
M,
(x)
M,
M,
(x)
(x
First, SD allows not only sampling from the target model's distribution, but also sampling from any com- bined distribution of the proposal model and target model. In vanilla SD, the target model's distribution is directly em- ployed for token verification and resampling, ensuring that the generated tokens align with the target model's 's distribu- tion. Similarly, we find that if the combined distribution is used for verification and resampling, as illustrated in Fig- ure (c), the generated tokens will follow the combined distribution. We refer this generalization of SD to col- laborative decoding as Naive-CoS. Naive-CoS significantly reduces the number of model invocations required. For in- stance, as shown in Figure (a). generating four tokens with the vanilla collaborative decoding necessitates four in- vocations to both Mg and Mp. while Naive-CoS, in the optimal case, requires only four invocations to M, and a single invocation Mp.
Token 21-
Token
Step 3: Proposer: M,,, Verifer: M,
Step 4: Proposer: M, Verifer: M,
Tokan id
4
Token id 1
M,
(.
(x(x)
g.(x))
M, (x)
(x) (x(z)
M2
(x())
(x)
M,
(x)
p=(*)
P=(x) (xt
Token
Token i. 4y-f2 24-1s
(b) Alternate Proposal Framework
The sketch Altemate Proposal Framework. A contin- colored block indicates single model invocation, with the bonus token highlighted in a red rounded box. Beginning from Step 2, .M, and MD are invoked alternately. Each invocation in- volves both the verification of the current token and the generation of bonus token. For elarity, assume that the proposal length for each model is and that all proposed tokens are accepted.
Figure 2.
Secondly, alternating each model proposer and veri- fier can further accelerate the collaboration process. In standard SD, the proposer and verifier are fixed, with one model consistently serving as the proposer and the other as the verifier. However, we observe that in the collaborative decoding setting, this static assignment is suboptimal, as fails to fully leverage the bonus token. In SD, when all tokens from the proposal model are accepted by the target model, the target model will naturally generate an additional token, referred to as the bonus token. However, since the bonus token is drawn from the target model's distribution rather than the combined distribution, it cannot be directly appended to the output of collaborative decoding. A naive solution might be to discard the bonus token or to re-query the proposal model and compute the combined distribution. Instead, we propose more efficient approach: treating the
bonus token as a proposal from the target model, which then verified by the proposer model. This insight leads to the Alternate Proposal Framework, illustrated in Figure 2. Combined with the Naive-CoS, the alternate proposal frame- work forms the proposed CoS. We further extend CoS to the general case of n-model collaboration in Section 3.4.
As shown in Figure 2(a), in standard collaborative decoding, each model invocation can generate only one probability distribution, so generating n tokens requires 2n model in- vocations. With the alternate proposal framework (Figure Figure 2(b)), each invocation can generate two distributions in the optimal case, thereby doubling the generation effi- ciency.
We establish the effectiveness of CoS through both theoreti- cal and experimental perspectives. Theoretically, we derive
2

---

Fast Large Language Model Collaborative Decoding via Speculation
an expected improvement factor quantify its acceleration and prove that CoS is guaranteed to be at least as efficient as the standard collaborative decoding, typically achieving greater speed. Additionally, in the weighted ensemble set- ting-the most common collaborative decoding setting-ws we demonstrate that CoS maintains a provable lower bound on the acceptance rate, ensuring consistently high efficiency. Experimentally, we conduct extensive experiments across various tasks, including code generation, mathematical rea- soning, multi-task understanding, and text summarization. Our evaluation covers multiple L.LM pairs, including Llama, Vicuna, and Qwen series, under both two-model and three- model configurations. The results show that CoS consis- tently achieves the highest acceleration, with speedups of 1.34x-1.85x for weighted ensemble and 1.11x-2.23x for contrastive decoding.
Our proposed method, CoS, represents an orthogonal ap- proach to existing collaborative decoding techniques. It is designed to substantially improve inference speed while preserving the benefits of collaboration. Importantly, CoS is not restricted to accelerating the three methods discussed above; its generality allows it to enhance the efficiency of any collaborative decoding approach, including those yet to be developed.
Speculative Decoding. Speculative decoding (Xia et al., 2023; Leviathan et al., 2023; Chen et al., 2023) can be categorized into two main areas: proposal model design and verification design. In the first category, proposal models are designed to generate tokens that are more likely to be accepted by the verifier. This includes independent proposal models, such as distillation-bast method (Zhou et al., 2024) and sarget-informet verifier (Zhang et al., 2024; Elhoushi et al., 2024; Monea et al., 2023; Yi et al., 2024; Monea et al., 2023; Li et al., 2024b; Sun et al., 2024b).
models that incorporate information of
In summary, our key contributions are as follows: (1) We extend speculative decoding to the collaborative decoding setting by refining its verification mechanism, introducing a Naive-CoS that significantly improves efficiency. (2) We incorporate an alternate proposal framework into the Naive- CoS to get the final CoS, further boosting inference speed. (3) Through extensive theoretical analysis and experimen- tal evaluation, we demonstrate that our method achieves substantial acceleration while maintaining lower bound, ensuring it never underperforms compared to standard col- laborative decoding.
The second category optimizes target model's verification process to improve decoding efficiency, following two main research directions. The first one increases proposal to- kens and uses structured attention mechanisms (Miao et al., 2024; Cai et al., 2024; Li et al., 2024c; Gong et al., 2024) to validate multiple candidates simultaneously. The second direction modifies the verification strategy itself, employing methods like joint probability density estimation (Anony- mous, 2025a), Monte Carlo tree search (Hu & Huang, 2024), and a linear binary classifier (Anonymous, 2025b).
2. Related Work
LLM Collaborative Decoding. In this paper, LLM collabo- rative decoding refers to the integration of information from multiple LLMs when generating the next token. This typi- cally involves combining the output probabilities or logits to compute a final probability distribution. This method can be used to improve overall performance, especially solving complex tasks(Han al., 2025; 2024). A common approach is model ensembling, which averages applies weighted averaging to the probability distributions of multiple LLMs to derive the final sampling distribution (Yu et al., 2024; Huang et al., 2024; Yao et al., 2024). Studies have shown that this technique can enhance both performance and safety (Li et al., 2024a). Another method, contrastive decoding, is based on the observation that smaller models tend to pro- duce noisier outputs. By subtracting the logits of a smaller model from those of a larger one, a cleaner and more reli- able set of logits can be obtained, resulting in higher-quality outputs (Li et al., 2023; O'Brien& & Lewis, 2023). Finally, decoding-time realignment enables flexible alignment with human preferences during decoding by linearly combin- ing the logits of human- aligned and an unaligned model, thereby balancing performance with alignment objectives (Liu et al., 2024; Shi et al., 2024).
The most closely related work is Speculative Contrastive De- coding (SCD) (Yuan et al., 2024), which combines outputs from both large and small models during the verification phase to form a contrastive decoding distribution. Opera- tionally, SCD can be seen as a special case of Naive-CoS in the contrastive decoding setting. The method proposed in this paper differs from SCD in three key ways: (1) CoS is more broadly applicable and can accelerate any collab- orative decoding approach; (2) it introduces an alternative proposal framework that further improves decoding speed; and (3) it provides a complete theoretical analysis, ensuring inference efficiency comparable to standard collaborative decoding methods.
3. Collaborative Decoding via Speculation
3.1. Speculative Decoding
Unlike other acceleration methods (Sun et al., 2024a), spec- ulative decoding (SD) is a technique designed to speed up inference while maintaining the quality of generated out- puts. It involves two phases: the proposal phase and the verification phase. During the proposal phase, lightweight proposal model sequentially generates proposal tokens. In
3

---

Fast Large Language Model Collaborative Decoding via Speculation
the verification phase, a larger target model verifies these tokens in parallel. Furthermore, by incorporating appropri- ate acceptance-rejection criteria, the technique ensures that the generated tokens align precisely with the target model's distribution, thus maintaining high-quality results.
while contrastive decoding can be represented as
rz(2) = CL(,, Softmax pll)).
(5)
We note that by making slight modifications to the vanilla SD, the generated tokens align with the combined dis- tribution. Specifically, before verification, we first compute the combined distribution r. rx()) and update the verification formula in Equation (2) as follows:
Specifically, in the proposal phase, the proposal model Ma generates a sequence of length y. denoted as:
P142-.. - T1tT ~ II (+))()). 5=1
uy min (I, Ti+j(()
(6)
(z)
Here, Jitj represents the token generated at position ++ j. and (1) & q( I Ægi+j-1) is the conditional probability distribution computed by MMg over T4ty. given the previously generated sequence -1-
Then, if the token rejected, we resample Tits from the distribution norm(max(0,,) rits - 9:4))).
We theoretically prove the correctness of Naive-CoS, that is, the tokens generated by the above sampling process precisely align with the combined distribution, with an ac- ceptance rate a of
In the verification phase, the target model M, executes a forward pass, producing Y + target distributions: Pi+1 (a), ... PPt-- Pitr)++ (z)). The first y distributions are subsequently used validate the proposal tokens gen- erated in the proposal phase. Specifically, proposal token I+ts is accepted if the following condition holds:
a a= -=1-2pivgg
(7)
where DTV (g) ) is the total variation distance, defined as Drv(g, r) = Erey lq(x) - -r(x)), where V is the set of all tokens. The proof is provided in Appendix A.
U S min (1, ((tty())
where uj (O, 1) represents a uniformly distributed ran- dom variable. If the token ritj is rejected, the subsequent tokens #(+5+1:-... #i+t++ are discarded, and Kizs is sam- pled from the distribution norm(max(U (0. Pi+j qi+))). If all y tokens are accepted, an additional token directly sam- pled from P++++1 and appended to the generated sequence, referred to as the bonus token in our paper.
Analysis of speed improvement. In speculative decoding, inference speed is predominantly influenced by the accep- tance rate a, with a higher acceptance rate leading to more substantial speed improvements. In this part, we first ana- lyze the theoretical speed improvement when is known. When is unknown, we focus on weighted ensemble see- nario and provide lower bound for a. With this bound, we derive a series of favorable acceleration properties.
By iteratively alternating between the proposal and verifica- tion phases, SD improves inference speed while ensuring the generated tokens align with the target model's distribution.
Theorem 3.1. Let be the proposal length and c be the cost coefficient, defined as the ratio between the time for a single invocution of the proposal model and the target model. Then, the expected speed improvement factor is (1-o)(1-e7)
(1=o))(++e)
3.2. Naive-CoS
As discussed above, vanilla SD can only accelerate the infer- ence of a single model. In this subsection, we will introduce how to apply SD to scenarios involving an arbitrary com- bination of two models. Specifically, let qi (xxx) and pz( denote the distributions of token x; given by the proposal model and the target model, respectively, and let 1द and 1 be the corresponding logits. Then, the combined distribution ri rx(x) can be expressed as
The proof of Theorem 3.1 is in Appendix A.2. Theorem 3.1 provides the speed improvement factor when a is known. However, in most cases, a is unknown and requires exten- sive experiments to estimate. Nevertheless, we find that in the weighted ensemble scenario, which is the most common collaborative decoding, a has lower bound.
Theorem 3.2. Cf q) >q(x)) (1 A)p(z)) and ((x) is the proposal model. Then has lower bound of X.
Ta (x: (())()) C7)),
(3)
Proof. We have = Exey ((x) min (1, r(m) then a
q()))
where C(-) represents the combination function at the prob- ability level, while C'(-) is at logits level. For example, the common weighted ensemble that uses probability for weighted summation can be expressed as
Σ Σxey ((x) min (1, + (1 x)p((a) and then we get a > Erey >q(x) = X.
The equality holds if and only if (()) = 0 for all x E V. which means that p(x)) and g(x) do not overlap.
r((x) = (x). = (x) + (1 -x)p (4)

---

Fast Large Language Model Collaborative Decoding via Speculation
Corollary 3.3. Assume that C(p,g) (1-x))( and that Mg and Mp have comparable parameters. Then, by selecting an appropriate proposal model, a has a lower bound of at least 0.5.
can introduce some performance degradation. This trade- off provides mechanism to balance quality and speed in speculative decoding.
In contrast to existing approaches, this method improves interpretabilityy. This is because we know the distribution of the generated tokens after relaxation, i.e. combined distribu- tion defined in Equation (4). This allows us to design pro- posal models that accelerate inference without compromis- ing performance, and potentially even enhance the model's capabilities in specific areas. For instance, some research suggests that ensembling a smaller model appropriately can improve safety ( (Wang et al., 2024; Li et al., 2024a). The experimental results are shown in Appendix C.I.
Proof. Since Ma and Mp have comparable parameters, either can serve as the proposal model. Therefore, has lower bound of max(X, 1 - X), which is at least 0.5. D
By utilizing the lower bound property, we demonstrate that the proposed Naive-Cor CoS is guaranteed to be no slower than the weighted ensemble approach, and it is typically faster.
Corollary 3.4. Assume that C(p, g) >g(xr) + (1 - A)p(2)), then if X > Itc. there exists a value of y that enhances the inference speed.
3.3. Alternate Proposal Framework
In Section 3.2, we explore the application of speculative de- coding to LLM collaboration. However, we don't 't consider the bonus token, that is, the additional token generated when all proposal tokens are accepted. This is because the bonus token follows the distribution of the verifier rather than the combined distribution and can not be directly appended to the output sequence. In this subsection, we introduce a collaboration framework, termed the alternate proposal framework, which effectively leverages the bonus token and demonstrates superior performance.
Proof: Consider 7 = 2, and solve the inequality (1- (1+t))
(1+o) 1. Solving this inequality yields a > The- Since a 2 A follows, establishing the corollary.
O
Corollary 3.5. Assume that C(p. g) = >q(x) + (1 -))(z).. Then, for any X, there exists a value of y such that the speed of the Naive-Cos. CoS is not slower than the vanilla weighted ensemble, and it is almost always faster.
Proof. As stated in Corollary 3.4, if Itc. there exists value of y that enhances the inference speed. If we swap the proposer and the verifier, the condition for acceleration changes to
As shown in Figure 2, in the alternate proposal framework, the generation of bonus token is treated proposal from the current verifier, which is subsequently verified by the current proposer. Specifically, let the proposer be denoted as M, and the verifier as M, with proposal lengths 7, and 7p- respectively. If all tokens proposed by Mg are accepted, a total of Y4 + tokens will be generated. The first 7g tokens follows the distribution ri+)(()) = C(g++ (x+5 for 1,.... 74: while the 7i + 1-th token is drawn from P++7,+1(2)) (x). At this stage, the 74 + 1-th token, referred to as bonus token, is treated as the initial token in MP 's proposal. Subsequently, MM will generate an additional 7p - tokens to complete its proposal.
A> >>
1/c which simplifies to itc 1+1/c-
If + - either X > T or x < e must hold, which ensures a speedup. Otherwise, if X Itc then a 2 A (1+a))(++)) (1-r) which ensures that the speed does not decline. Moreover, equality only holds when the two distributions
I+c
do not overlap, which is almost impossible in practice. D
Interpretable quality-speed tradeoff in vanilla SD. In SD, some studies focus on relaxing the acceptance crite- ria for a higher acceptance rate to achieve faster inference, such lossy SD (Zhou al., 2024) and typical acceptance (Cai et al., 2024). However, these methods often lack in- terpretability, that is, we do not know which distribution the generated tokens will follow. We find that Naive-CoS can naturally be an interpretable strategy for adjusting the quality-speed tradeoff in vanilla SD.
If any proposed tokens are rejected and no bonus token is generated, the default proposal model will take over as the proposal model. This default model is predefined and fixed. As outlined above, the two models alternate as proposers during the decoding process, which is why this approach is called the alternate proposal framework. The pseudocode for this framework is provided in Algorithm I.
Specifically, we apply a weighted ensemble using the pro- posal and target models in SD. In this setup, when A= 0, the combined distribution aligns exactly with the target dis- tribution, reducing the method to standard SD. For À >>0 0, the acceptance rate is guaranteed to have lower bound and greater than that of vanilla SD, as demonstrated in the proof of Theorem 3.2, leading to greater acceleration. However, incorporating less precise information from a smaller model
Analysis of speed improvement. We now analyze the speed improvement achieved by the alternative proposal framework. For the sake of clarity, we focus cycle, which encompasses one proposal and one verification. In this cycle, both the proposer and verifier are fixed. Given that the decoding process is composed of multiple such
single
5

---

Fast Large Language Model Collaborative Decoding via Speculation
cycles, the overall decoding performance can be inferred from the behavior of single eycle.
Step 1: Current Model: M,
Step Current Model: MMs
Taken id
id
M,
A (4)
M
First, similar to Theorem 3.1, we provide the expected speed improvement factor for the alternate proposal framework.
M2
M₂
(=)
Theorem 3.6. Let My be the proposer and Mp be the verifier, then the expected speed improvement factor of the alternate proposal framework is
Ma
Ma
Token 5; p.
Taken s; A 5y
(1 (1-0"9
[I-o][[+eng-a'p]]
Step Current Model: Ma
Step Current Model: Ms
Taken id 1
4
Token id
1
3
4
M
M
8 (x)
Proof. When the bonus token is generated, the proposer only needs to generate 7, new tokens; otherwise, it must generate "g tokens. The probability of generating the bonus token is the probability that all proposal tokens in the last cycle were accepted, which is a² Therefore, the expected time spent on proposal and verification is a (1 + c(7g - 1)) + (1 - (1 (1 + o%g). Then the factor can be derived following the process in Appendix A.2. D
Ma
p (+)
p. (e)
Ma
A (G)
(+)
Ma
Ga)
p! Ge)
GC)
Ma
(
(+) (
Mumty Token io pe"
sy pf
Token 11-1)
Step S: Current Model: M,
Step Current Model: M
Takan
Token Id
M,
(x)
Hf (4)
Pf' (x)
pf (x)
M
(x)
( (a
M2
(x)
p! (z)
M2
(+)
In Corollary 3.5, we proved that in the weighted ensemble scenario, the Naive-CoS is never slower than the vanilla collaborative decoding and is typically faster. In this subsec- tion, with the alternate proposal framework, we extend this conclusion to any form of two-model collaboration.
Ma
(+)
(2)
pi Go)
Ma
(+)
(P) (P)
Varty xy-pp
Taken 21-1
Taken 21-
Figure 3. The sketch of CoS in three-mode collaboration scenario. The colored boxes represent the stored probability distributions, while the gray boxes represent the discarded ones. Each invocation involves scoring the current proposal tokens and generating bonus token. For clarity, we assume that the proposal length for each model is and that all proposed tokens are accepted.
Corollary 3.7. For any two models, there exist values of 7g and 1ip such that the speed of the alternate proposal framework is never slower than the vanilla collaboration and is almost always faster.
(1=o2))(1ee)
Proof. Consider 7/ = 7/- = 1, then [1-a 2 holds universally. The equality holds only when c = 0 or a = O. However, c > because the execution time of the proposal model is non-negligible, and a > 0 holds unless the proposal distribution and the combined distribution do not overlap, which is almost impossible.
ity distribution of proposal from other models.' Second, during scoring, a model can naturally generate a bonus to- ken, which further improves efficiency. We illustrate the CoS process in the n-model scenario with a simple example, while detailed pseudocode and a general visualization are provided in Appendix B.2.
D
As shown in Figure 3, the process begins in step with the default proposal model, Mi. generating proposal token in In step 2, M2 scores J1 while simultaneously generating a bonus token Ø2. Similarly, in step 3, MM scores both Di and 22 in parallel and produces another bonus token, wg. At this point, 31 has been scored by both M2 and M3. enabling the computation of its combined distribution
An intuitive interpretation of Corollary 3.7 is that when Yg = Tpp = 1, even in the worst-case scenario-vy whhere all proposal tokens are rejected-
token generation still requires only one proposal and one verification. This results in the same number of model invocations as the standard collaborative decoding. In practice, however, it is rare for all proposal tokens to be rejected. Once token is accepted, the collaboration process becomes more efficient.
ri (x) for verification. The associated distributions yi (Ir),
(1) P2 (z)). Ps (1) (Ir) are no longer needed and are discarded.
3.4. Generalize to More Models
If I is accepted, Mi computes (1) (x), (1) (I)) Pa (x) in parallel as shown in step 5, allowing verification of xz. Oth-
(1)
In this subsection, we extend CoS to the n-model collabo- ration scenario. The core principles remain similar to the tooe case, with acceleration driven by two key factors. First, each model can score the proposals of other models in parallel, where scoring refers to computing the probabil-
We use the term "scoring" rather than "verification" because, unlike in the two-model case, scoring does not immediately trigger verification; instead, verification occurs only after all models have scored a token.
6

---

Fast Large Language Model Collaborative Decoding via Speculation
erwise, if 22 is rejected, all stored distributions are cleared, and Mi generates new proposal, similar to step 1.
Table I. Model pair configuration. The first column represents the name of the corresponding model pair for simplicity.
Name
Mp
4. Experiments
Weight Ensemble (WE)
4.1. Experimental Setups
Lluma- Vicuna
Llama-2-7B
Vicuna-BB-VLs
Qwen-3b
Qwen2.5-3B-Instte
Qwen2. S-Cods---B--nnnseee
Datasets and evaluation. We test CoS across multiple tasks including code generation, mathematical reasoning, multi- task understanding, and text summarization on HumanEval (Chen al., 2021), GSM8K (Cobbe et al., 2021), MMLU (Hendryeks et al., 2021). and CNNDM (See et al., 2017), respectively. We measure each method's speed by the aver- age tokens generated per second and compute the speedup ratio relative to the standard collaborative decoding. All experiments are conducted on RTX 3090, except for evalu- ations involving the Llama-
Qwen-11.5b
Qwen2.5- 1-1.5.55-1---mmmm 1-B--ttttt
Qwen2 Sove225-Coder-1..5 1.B-ul--u,B---tttuu
Contrastive Decoding (CD)
IJama-3
Llama-3.2-11 1B
Llama-3.1 -.B-tttttet
Ilama-2
Llama- 68M
1-7ma-2-7B
OPT
OPT-125M
OPT-133
reporting the optimal results. 7 = is the common setting, while ? = 1 ensures acceleration (Corollary 3.7). In the thrce-model CoS, all models have a proposal length of I.
model pair, which use the A6000 GPU. Additionally, we also test on the Ascend 910B3 NPU; the corresponding results are shown in Table 9 and Table 10.
4.2. Main Results
Combination fuucctions and methods. We experiment with two combination functions: weighted ensemble (WE) at the distribution level (Equation (4)) and contrastive decoding (CD) at the logits level (Equation (5)). For WE, in the two- model case, we set À = 0.5 and temperature T 1; in the three-model case, each model's coefficient was set to 1/3. For CD, we set and set T to both 0 and 1. WE with 7' = 0 is not tested due to its uncommon use, as it leads to a one-hot distribution, reducing information. Among two combination functions, four methods are compared: (1) the standard collaborative decoding (WE, CD); (1) parallel collaborative decoding (WE-P, CD-P); (2) an accelerated version with speculative decoding (SD), using the smallest model as the proposal and the combined distribution as the target (WE-SD, CD-SD); and (3) CoS (WE-CoS, CD-CoS). Since SCD is equivalent to CoS, its results are included in our ablation on alternative proposal frameworks (Appendix C.4).
Table 2 and Table 3 display the speedup ratios for each method relative to the standard collaborative decoding in the WE and CD settings, respectively.? From these two tables, we have the following findings. First, CoS not only consistently achieves the highest speedup in all settings, it also gets speedup across all settings, which supports the findings in Corollary 3.7. In contrast, SD may reduce the collaboration speed in some cases. For example, when using the Llama-2 model pair with = 1 on HumanEval in Table 3, applying SD reduces the speed to 0.94x of the standard collaboration. A similar speed reduction was also observed in the model scenario in Table 2. This is because vanilla SD does not inherently ensure acceleration. When the acceptance rate is low, SD may perform slower than standard decoding.
0.1, which is the most common setting,
Second, compared to the CD scenario, the WE scenario ensures higher minimum speedup for CoS. In the two- model case, CoS achieves minimum speedup of 1.34x, while in the three-model case, it reaches at least 1.27x. In contrast, the CD scenario has a speedup as low as 1.1 This difference arises because CoS maintains a consistently high acceptance rate in the WE scenario, as outlined in Corollary 3.3.
Model pair configuration. We experiment on different types of LLMs, including Llama-2 (Touvron et al., 2023; Miao et al., 2024), Vicuna (Zheng et al., 2023), Llama-3 (Dubey et al., 2024), Qwen-2.5 (Team, 2024), and OPT (Zhang et al., 2022). Model pair configurations for each combination function are in Table 1. We also test a three- model collaboration using Oween2.5-1.5B-Instruu code and math versions in the WE setting.
Third. the speedup varies across tasks and is influenced by the determinism of task outputs. For example, in the WE scenario, CoS achieved the highest speedup on HumanEval, averaging 1.65x, as code generation demands strictly for- matted outputs. Conversely, CoS has a lower speedup of 1.36x on a text summarization task, where output flexibility is higher. This difference stems from the alignment between the proposal and target models: in highly deterministic tasks, their outputs exhibit greater similarity, leading to higher
and its
Configuration of 7.. The proposal length y is the only hyperparameteriin in SD, affecting the al algorithm's 's acceleration. In the two-model CoS setting, 7 corresponds to the proposal length of the smaller model, with the larger model fixed at For simplicity, we refer to the smaller model with 7 >> as the proposal model of CoS, since it typically serves this role. We tested 7 5 and 7 for CoS and SD speeds,
'hh results of OPT model pair are shown Appendix C.2.
7

---

Fast Large Language Model Collaborative Decoding via Speculation
20
2,0
2.0
2.0
18-
1/8
1.8
1.8
Speedup
1.6
Speedup
1.8
Speedup
1.6
Speedup
1.6
Optimal
Optimal LlamaZ Vicuns
Optimal
Liama2
Owen
Owen
Vicuna
1.00x
Qwen--Coder
Qwen-Coder
0.8
0.8
0.8
0.2
0.4
0.8
0.9
0.2
0.4
0.6
0.8
0.2
0.4
0.8
0.2
0.4
0.6
0.8
lambda
lambda
lambda
lambda
(a) Llama Vicuna (HumanEval)
Llama- Vicuna (GSM8K)
(c) Qwen-3b (HumanEval)
(d) Qwen-3b (GSM8K)
Figure 4. Comparison of speedup ratios for different À in WE across diverse setings. The blue and green lines represent the speedup ratios when the corresponding models serve as proposal model, while the shaded region highlights maximum speedup between two.
2.25
HumanEval (T=0)
Table 2. The speedup ratio cach method in WE setting. The method with the optimal speedup is highlighted in bold.
HumenEval (T=1)
GSMBK (T=0)
2.00
GSM8K (T=1)
Method HumanEval GSM8K MML.U CNNDM
Speedup
1.75
WE
1.00x
1.00x
1.00x
1.00x
1.50
Llama Vicuna
WE-P
0.69x
0.73x
0.70x
0.75x
1.25
SD
1.27x
1.21x
1.19x
1.15x
CoS
1.58x
1.52x
1.41x
1.46x
1.00
1,00x
WE
1.00x
1.00x
1.00x
1.00x
0.1
0.2
0.3
0.4
0.5
Qwen-33 3b
WE-P
0.74x
0.79x
0.79x
0.77
SD
1.13x
1.06x
1.09x
1.08x
Figure 5. Comparison speedup ratios for different j CD across different temperatures and datasets.
CoS
1.62x
1.52x
1.42x
1.38x
Qwen-1 1.5b
WE
1.00x
1.00x
1.00x
1.00x
WE-P
0.63x
0.62x
0.64x
0.63x
acceptance rate and, consequently, stronger acceleration.
SD
1.11x
1.13x
1.08x
1.10x
CoS
1.56x
1.46x
1.34x
1.35x
4.3. Analysis
Qwen- 1.5b .5b Model)
WE
1.00x
1.00x
1.00x
1.00x
Impact of proposal length y. We evaluate the influence of various 7 values, ranging from to 5, on speedup across both the WE and CD scenarios, utilizing the HumanEval and GSMSK datasets. The results in Figure 6 show that when the
WE-P
0.54x
0.73x
0.80x
0.82x
SD
0.96x
0.92x
0.98x
0.95x
CoS
1.85x
1.53x
1.38x
1.27x
models similar in size, the speedup ratio remains stable across values, as seen in the WE scenario Figure 6 (a). This is because the high cost of invoking the proposal model offsets the speedup from increasing ?. However, when the models differ significantly in size, the speedup ratio varies considerably with 7, as shown in the CD scenario Figure (b) and (c).
tokens, which improves acceleration; while later proposal tokens depend on earlier, unverified tokens, making them less accurate and more likely to rejected, which wastes computation. Thus, the optimal speedup is achieved at specific In some cases, however, speedup either monoton- ically increases or decreases due to high or low acceptance rates. For instance, this is observed in the experiment exper- iments with the Llama-3 pair on HumanEval (Figure 6 (b)) and the Llama-2 pair on HumanEval (Figure 6 (c)).
Additionally, observe that speedup initially increases with 7 before decreasing. For example, in the experiment with the Llama-3 model pair on GSM8K (Figure 6 (b)), speedup improves as y rises from to 5, peaks at 7 = 5, and then declines. This behavior is explained by two fac- tors: increasing 7 boosts the expected number of accepted
Speedup ratio for different weight À in WE. We exam-
8

---

Fast Large Language Model Collaborative Decoding via Speculation
2.25
Jamaa-/ (Humanbval)
2.25
Liamed (NamarEbvd))
2.25
Liered (fLmentive)
2,00
Gwc1-25
2.00
(GSMBK)
2.00
1.75
1.75
1.75
1.90
1.50
1.50
1.25
1:25
1.00
1.00
1.00
2
2
2
gamma
(a) WE (7= À 0.5)
(b) CD (T = 0, 0.1)
(c) WE =T= 0.1)
Figure Comparison speedup ratios for different yy across different settings.
process across all tested μ..
Table 3. The speedup ratio of each method in CD setting,
T Method HvmanEval GSM8K
MMLU CNNDM
Furthermore, we find that an increase in μ results in re- duced speedup. This occurs because CD computes the com- bined distribution by subtracting the proposal model's in- formation from the target model's. As μr grows, the gap between these distributions widens, lowering the acceptance rate (Equation (7)). Despite this, the speedup ratio remains above 1.00x, confirming that CD-CoS always accelcrates, consistent with Corollary 3.7.
1.00x
1.00x
1.00x
1.00x
CD-P
0.41x
0.40x
0.41x
0.41x
0
SD
2.04x
1.81x
1.52x
1.58x
Llama-3
CoS
2.23x
2.00x
1.77x
1.61x
CD
1.00x
1.00x
1.00x
1.00x
CD-P
0.39x
0.41x
0.42x
0.41x
SD
1.55x
1.21x
1.20x
1.07x
CoS
1.65x
1.44x
1.31x
1.18x
5. Conclusion
CD
1.00x
1.00x
1.00x
1.00x
CD-P
0.59x
0.50x
0.54x
0.48x
This paper introduces Collaborative Decoding via Specula- tion (CoS), an extension of speculative decoding that acceler- ates LLM collaborative decoding while maintaining output quality. CoS refines the verification mechanism for direct collaborative sampling and introduces an alternate proposal framework to further boost efficiency. We demonstrate the effectiveness of CoS through both theoretical analysis and empirical validation.
SD
1.15x
1.62x
1.08x
0.93x
Llama-2
CoS
1.26x
1.65x
1.68x
1.30x
CD
1.00x
1.00x
1.00x
1.00x
CD-P
0.56x
0.51x
0.53x
0.49x
SD
0.94x
1.16x
1.23x
1.10x
CoS
1.15x
1.20x
1.37x
1.11x
A Acknowledgements
ine the speedup effect of CoS when À takes values other than just 0.5. Specifically, we conduct experiments with values ranging from 0.1 0.9, using the Llama-Vicuna and Qwen-3b model pairs the HumanEval and GSM8K datasets. The results, presented in Figure 4, show that CoS consistently achieves a high speedup of at least 1.5x across all tested À values. This consistent speedup occurs because, when the two models are of similar sizes, for any X, an appropriate proposal model can be selected to maintain a high acceptance rate during CoS process (as explained in Corollary 3.3), ensuring the observed speedup.
This work supported by the National Science Foundation of China (62206048), the Natural Science Foundation of Jiangsu Province (BK20220819) and the Fundamental Re- search Funds for the Central Universities (2242024k30035), Additional support was provided by the Big Data Comput- ing Center of Southeast University and the SEU Kunpeng & Ascend center of cultivation.
Impact Statement
This paper presents work whose goal is to advance the field of Deep Learning. There are many potential societal conse- quences of our work, none which we feel must be specifi- cally highlighted here.
Speedup ratio for different weight values of y in CD. Sim- ilarly, we examine the speedup effect of CD when takes other values. Specifically, we conduct experiments with μ ranging from 0.1 to 0.5, using the Llama-3 model pair on the Eval and GSM8K datasets. The results, presented Figure 5, show that CoS consistently accelerates the CD

---

Fast Large Language Model Collaborative Decoding via Speculation
References
Roman, A., Aly, A., Chen, B., and Wu, C.-J. Layerskip: Enabling early exit inference and self-speculative decod- ing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), Bangkok, Thailand, 2024.
Anonymous. Optimized multi-token joint decoding with auxiliary model for Ilm inference. In The Thirteenth International Conference on Learning Representations (ICLR 2025), 2025a.
Gong, Z., Liu, J., Wang, Z., Wu, P., Wang. J., Cai, X., Zhao, D., and Yan, R. Graph-structured speculative decod- ing. In Findings of the Association for Computational Linguistics (ACL 2024), Bangkok, Thailand, 2024.
Anonymous. Judge decoding: Faster speculative sam- pling requires going beyond model alignment. In The Thirteenth International Conference on Learning Representations (ICLR 2025), 2025b.
Cai, T., Li, Y., Geng, 2., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple IIm inference acceleration frame- work with multiple decoding heads. In Proceedings of the st International Conference Machine Learning (ICML 2024), Vienna, Austria, 2024.
Han, S., Liu, T., Li, C.,, Xiong, X., and Cohan, A. Hybrid- mind: Meta selection of natural language and symbolic language for enhanced Ilm reasoning. arXiv e-prints, pp. arXiv-2409, 2024.
Han, S., Gomez, F. P., Vu, T., Li, Z., Cer, D., Zeng, H., Tar, C., Cohan, A., and Abrego, G. H. Ateb: Evaluating and improving advanced nlp tasks for text embedding models. arXiv preprint 1Xiv:2502.116666 2025.
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D.., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J.. Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R.. Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray. S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E.,, Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S. Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J.. Achiam, J.. Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M.. Murati, M.. Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. 2021.
Hu, Z. and Huang, H. Accelerated speculative sam- pling based tree monte carlo. In Forty-first International Conference Machine Learning (ICML 2024), 2024. URL h http://///peeeeeeee///o forumidd-stMii1snn2
Huang, Y., Feng. X., Li, B., Xiang, Y., Wang, H., Liu, T., and Qin, B. Ensemble learning for heterogencous large language models with deep parallel collaboration. Advances in Neural Information Processing Systems, 37: 1 119838-1 19860, 2024.
Chen, Z., Li. J., Chen, P., Li, Z., Sun, K., Luo, Y., Mao, Q. Yang, D., Sun, H., and Yu, P. S. Harnessing multiple large language models: A survey on Ilm ensemble. arXiv preprint arXiv:2502.18036 2025.
Leviathan, Y., Kalman, M., and Matias, Y. Fast infer- ence from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), Honolulu, Hawaii, USA, 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, I.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:21 141168 2021.
Li, T., Liu, Q., Pang. T., Du, C., Guo, Q., Liu, Y., and Lin, M. Purifying large language models by ensembling a small language model. arXiv preprint arXiv:2402. 14845, 2024a.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llaaa 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T. B., Zettlemoyer, L.., and Lewis, M. Con- trastive decoding: Open-ended text generation as opti- mization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 12286 126661,22,,,.-1 2023.
Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, I., Mahmoud, A., Acun, B., Agarwal, S.,
10

---

Fast Large Language Model Collaborative Decoding via Speculation
Li, Y., Wei, F., Zhang, C., and Zhang, H. Eagle: Specu- lative sampling requires rethinking feature uncertainty. In Proceedings of the s1st International Conference on Machine Learning (ICML 2024). Vienna, Austria, 2024b.
cache in shadows for high-throughput long-context I1m inference. arXiv preprint arXiv:2410.21465, 2024a.
Sun, H., Chen, Z., Yang. X., Tian, Y., and Chen, B. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. In First Conference on Language Modeling, 2024b.
Li, Y., Wei, F.. Zhang. C., and Zhang, H. Eagle- 2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), Miami, Florida, USA, 2024c.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https: http///wwnnmmmmmmmm github. io/ blog/ ht/p////n...........
Liu, T., Guo, S., Bianco, L., Calandriello, D., Berthet, Q., Llinares, F., Hoffmann, J., Dixon, L., Valko, M., and Blondel, M. Decoding- time realignment of language models. In Proceedings of the International Conference on Machine Learning, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09285 2023.
Wang. C.,, Chen, X., Zhang, N., Tian, B., Xu, H., Deng, S., and Chen, H. MIlm can see? dynamic correction decoding for hallucination mitigation. arXiv preprint arXiv:2410.1177 2024.
Lu, J., Pang, Z., Xiao, M., Zhu, Y., Xia, R., and Zhang, Merge, ensemble, and cooperate! survey on collabora- tive strategies in the era of large language models. arXiv preprint 20x.:2407.06089, 2024.
Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., and Sui, Z. Speculative decoding: Exploiting speculative execu- tion for accelerating seq2seq generation. In Findings of the Association for Computational Linguisties: EMNLP
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang. Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., Shi, C., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2024), 2024.
2023, pp. 3909-3925, 2023.
Yao, Y., Wu, H., Liu, M., Luo, S., Han, X., Liu, J., Guo, Z. and Song, L. Determine-then-ensemte top-k union for large language model ensembling. arXiv preprint arXiv:2410.03771 2024.
Necessity of
Monea, G., Joulin, A., and Grave, E. Pass: Parallel specu- lative sampling. arXiv preprint arXiv:2302.01318, 2023.
Yi, H. Lin, F., Li, H., Ning P., Yu, X., and Xiao, R. Generation meets verification: Accelerating large lan- guage model inference with smart parallel auto-correct decoding. arXiv preprint arXiv:2402. 18899, 2024. URL https: /aapss///arxv...// http://wwwww..g///////...... wttpoo/////......a..../
URL http:: //tp://rxxiii v. orgo abtp//bs/231111 13581.
O' Brien, S. and Lewis, M. Contrastive decoding improves reasoning in large language models. arXiv preprint arXiv:2309.0911 2023.
Yu, Y.-C., Kuo, C. C., Ziqi Y., Yucheng, C., and Li, Y-S. Breaking the ceiling of the LLM commu- nity by treating token generation as a classification for ensembling. In Al-Onaizan, Y., Bansal, M., and Chen, Y.NN (eds.). Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 1826- 1839, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18653/v1/2024.
See, A., Liu, P. J., and Manning, C. D. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1073-1083 1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 18655/v URL https: // //www//www////////////// www.wwwwwwbbb... aalaaaattooooyyy11a11ee
URL https: / ac aclanthoooy...... org/2024. findings - - emnlp 1p.1pp... 99/.
Shi, R., Chen, Y., Hu, Y., Liu, A., Hajishirzi, H., Smith, N. A., and Du, S. S. Decoding-time language model alignment with multiple objectives. Advances in Neural Information Processing Systems, 37:48875- 48920, 2024.
Yuan, H. Lu, K. Huang. F. Yuan, Z., and Zhou, C. Speculative contrastive decoding. In Ku, L.- W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp- 56 64, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1
Sun, H., Chang, L.W,., Bao, W., Zheng, S., Zheng, N., Liu, X., Dong, H., Chi, Y., and Chen, B. Shadowkv: Kv

---

Fast Large Language Model Collaborative Decoding via Speculation
acl-short.5. URL https:://cllatthlloloooo org/ 2024.acl-short.55
Zhang, J., Wang, J., Li. H., Shou, L., Chen, K., Chen, G., and Mehrotra, language model acceleration via self-speculative decod- ing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), Bangkok, Thailand, 2024.
verft:: Lossless large
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mi- haylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer,
L. Opt: Open pre-trained transformer language models, 2022.
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, 7., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt mt-bench and chatbot arena, 2023.
Zhou, Y., Lyu, K., Rawat, A. S., Menon, A., Ros- tamizadeh, A., Kumar, S., Kagy, J.-F., and Agarwal, R. Distillspec: Improving speculative decoding via knowledge distillation. In The Thirteenth International Conference on Learning Representations (ICLR 2024), 2024. URL id-rsY6J3ZaTF.
net/ forum?
12

---

Fast Large Language Model Collaborative Decoding via Speculation
A. Mathematical Proofs
A.1. Correctness CoS
Assume that in speculative decoding, the proposal distribution is ((z), the target distribution is p(x), and the combined distribution is r(æ:) C(g(x), p(a)). Referring the proof of speculative decoding correctness (Leviathan et al., 2023), we now prove that the sampling method described in Section 3.2 ensures that the generated tokens align with the combined distribution r(z).
First, we have:
P(r=x) x') = P(proposal x' accepted) P(proposal x') + P(proposal rejected) P(resampled = x')
(8)
By definition, P(proposal æ' accepted) = min (1,
r(x') ((x")
and P(proposal x') q(z'.
min 1, (127 )
Ppproposal rejected)
= Σ ((x)
[((())
zEV
Σ (1(x)) min (1(x), r(x))
(9)
2EV
Σ max (r(x) q((),
ZEV
max(r(x') {(x'), o()
and by definition, P P(resampled = x') Evev max (r(x) g()),)) Substituting these into Equation (8), we get:
P(r z')= P(proposal x' accepted): P(proposal x') P proposal rejected) Preesampld x')
r(a') -min (1, ('g(xT) ((x') Σ max (r(x) q(>), 0) ΣvEV max (r(x) - ((x).0)
max(r(x') - g(x'),
(10)
sEV
= min((((2))) r(x)), + max(((x') -(rx')))(xr))))))) -
= =r(x').
The acceptance rate P(proposal accepted) is computed as:
P(proposal accepted)
P(proposal rejected)
Σ max (r(x) - 0)
zEV
(1I)
In(x) 9(x)
Drv(r.g).
A.2. Proof of Theorem 3.1
Referring to the proof given by Leviathan et (2023), our proof is as follows:
First, after one proposal and one verification, the proposed method generates at least one token, P(#tokens 0) 0 and Po#looens = 1) 1. If the model generates i tokens (1 < i < 7), it means the first i 1 tokens are accepted and the i + 1-th token is rejected. Therefore, P(#tokens = a² 1- - -a). If the model generates tokens (i = 7). it means the first y tokens are accepted. Thus, P(4tokens = 7) = =-
E E(#tokens) = Σ PP(ttokens = i) = 1- a 1-6
(12)
1=0
13

---

Fast Large Language Model Collaborative Decoding via Speculation
Note this value differs from that given by Leviathan et al. (2023). This discrepancy is because, in Section 3.2, do account for the bonus token.
Assume that the time required to invoke the target model once is T, and the time required to invoke proposal model is cT' Therefore, one proposal and one verification together take (7c+ 1)T time. The time required to generate one token is (1 (1-a)(ret1) T. In the vanilla collaborative decoding, the time required generate one token is (c 1)T. Therefore, the 1-07
(1-a7)(++e) improvement factor in total walltime is (1-o)(ye+1))
B. Algorithm Details
B.1. Alternate Proposal Framework
We provide the detailed pseudo-code of the alternate proposal framework (see Algorithm 1), which introduced in Section 3.3.
Algorithm The alternate proposal framework. PROPOSE takes the current token sequence S and constant 7 inputs and generates y tokens T. SCORE feeds sequence to the current verifier to obtain logits L and bonus token t. VERIFY examines to decide whether it should be accepted according to combined logits.
input Models MMg proposal lengths Yr. 7pp prefix sequence prefix.
S - --refix
C 0 Initialize cached tokens
while not finish do
if C 0 then
> Standard speculative decoding step
proposer +- Ma
verifier < Mp
T + PROPOSE(S, proposer.y)
else
Alternate proposal decoding step
S SwAp(proposer, verifier)
T + C' PROPOSE(S + C, proposer.y C. C.length)
end if
L SCORE(T)
CLL VERIFY(T, L')
if all tokens in T are accepted then C+t
else
Some tokens are rejected, clear C and resample from T' + resample from residual distribution
C O+00
end if 5 < 5 + T
end while return S
B.2. CoS Framework
The CoS framework is a generalization of the alternate proposal framework to scenarios involving more than three models. Figure 7 illustrates our CoS in three-model scenario, where the models M2, have proposal lengths of 3.
72 = 2, and 73- = 1, respectively.
Specifically, in step 1, the default proposal model invoked to generate ?1 proposal tokens: Ø1, 22, Ty. In step 2, M2 invoked score 31, T2, F3 while naturally generating bonus token, 24. Since M2 has proposal length of 72
14

---

Fast Large Language Model Collaborative Decoding via Speculation
then invoked again generate additional 722 score -:,,-..; Iy in parallel and generate bonus token, 3t6--
proposal token complete its proposal. In step M3 is called
At this stage, I1, 22, 2g have been scored by all models, allowing the combined distributions ri (z), rz (a( to be computed and used for verification, illustrated in step 3. Assuming that 21, 23 are all accepted, their corresponding probability distributions are no longer needed and are discarded, as shown in step 4. Subsequently, in step 5, MM invoked again to score 24; 45, Z6 and generate new proposal tokens: 27, Tg.
Next, 24 and I5 undergo verification. x4 accepted while æs is rejected, 21 remains unchanged, whereas æ5 is replaced with a/g. which is generated through the resampling phase. Since the 5-th token has changed, all subsequent proposal tokens and probability distributions derived from it become invalid and are discarded, illustrated in step 6. Once these are removed, the default proposal model is invoked to generate 7/ new proposal tokens, similar to step 1.
The corresponding pseudocode provided in Algorithm 2.
Step 1: Current Model: MM,
Step 2: Current Model: M2
Token
1
2
7
Token id
2 3
5
9
M
(2)
(x)
(=)
(()
(s)
pf (2)
.Mg
M
(s)
(x) (s) (
(x)
.Mg
M
Taken pl
Token p.
2 of xy ax 2y-
Step 3: Current Model: M,
S teep 4: Current Model: Ms
M,
p (z)
(x)
(s)
M₁
(D)
(x)
M2
of (x)
MM₃
(a) (x)
pP(a)
p2 (x) (P()
M2
pi (z)
88
16 (2)
(x)
of (e) (x)
M2
(x)
(x) (1)
pr
(x)
(x)
mwtty
x. -.
pl
ne
Token x.
Step
M,
Step 6: Model: Mr
Token
Token id
3
9
M
p (x)
#f ())
(2)
1 (x)
(x) (x)
A (e)
pel
(x)
(x)
(2)
(x)
Go
(x) Decord
(2)
M2g
(z)
G0)
(s)
2
M2
(x)
(
60
M3
(x)
16 (z)
(2)
(G)
M,
(x)
(x)
(2)
(x)
(
Verty@ 2
Marty@
Racampla
Token 13
"
mypp?
Token 2:-1
26-rs
pl
amppl
Model: M,
Step 7: Model:
Taken
7
Taken
3
M,
(s)
Go) G))
60)) (()
M2
(e)
of (G)
(x)
Go (x)
M4
(a)
of (a)
(P)
M,
(x)
NF (P))))
(P)) (PG)
M2
(x)
Gz)
22 (x)
(e)
(x)
M,
(P
(P)(s
G)
(P)
xi-ry
x.-7
2
2i-ry
Figure 7. The sketch of CoS framework in three-model scenario. The colored boxes represent the stored probability distributions, while the grey boxes represent cleared ones. Each invocation involves scoring current proposal tokens and generating bonus token. The proposal length for model Mi, M2, M3 is 3, 2, 1, respectively.
15

---

Fast Large Language Model Collaborative Decoding via Speculation
Algorithm The CoS framework. PROPOSE employs model M to take the current token sequence S and a constant as inputs and generates y tokens T. ScORE employs model M to score sequence to obtain sequence probabilities P, bonus token and its probability p. VERIFY examines T to decide whether should be accepted according combined probability distribution.
input Models MI...., M, proposal lengths "..... Ine prefix sequence prefix.
S <- prefix
S. + () > Initialize cached sequence
C 0, 0, for 1...,, D Initialize cached probabilities for each model
while not finish do
if S. 9 then
P PROPOSE(MM, S, 71) If cached sequence, default proposer M₁ is invoked to generate proposal
S. +- T Cache proposal tokens T and corresponding probabilities P
C + P
else
i - arg min; |Cal
& Find the model with the shortest cached probabilities, P, t, p S., & Score the S., generating probabilities of S.. bonus token t and its probability
| represents the number of elements
S. + (UU
G. (p
while Vjj C do
> af Cj are nonempty, t (the first token of S.) must have been scored by all models, so verify it p/ + C(pn,....PPa)
Pij represents the j-th probability of C;
VERIFY('',t t1)
if f accepted then 5.
C CyPAp....... for I....,n
else
resample token from residual distribution
S. + 0) C' + 0, for j = I....
end if
t SU {t1}
end while
T, P PROPOSE(M, S So, 74 1)
Generate more 7 tokens finish the proposal
S. 6 S.UT
C C C
end if end while return
16

---

Fast Large Language Model Collaborative Decoding via Speculation
C. Additional Results
C.1. CoS for Quality-Speed Tradeoff
As outlined in 3.2, creating a weighted ensemble of the proposal and target models in CoS offers a way to balance the tradeoff between quality and speed. We conducted experiments with the Llama-3 model pair on four datasets, adjusting from 0.1 0.9. As shown in Figure increasing leads steady improvement in inference speed but gradual decline in performance, allowing users choose tradeoff that best suits their needs.
Pef
-04
Por!
-04
100
-04
-04
60-
60
Incestlec::
Iacantec
80-
80
Perf
80
Peef
-222
50
0.3
lambee
limoda
(a) HumanEval
(b) GSM8K
(c) MMLU
(d) CNNDM
Figure 8. CoS for quality-speed tradeoff
C.2. Speedup on OPT Model Pair
The results are shown in Table
Table 4. The speedup ratio of each method in CD setting.
T Method
HumanEval
GSM8K
MMLU
CNNDM
CD
1.00x
1.00x
1.00x
1.00x
0
CD-SD
0.97x
1.05x
1.47x
1.40x
CD-CoS
3.28x
2.61x
3.42x
3.95x
CD
1.00x
1.00x
1.00x
1.00x
1
CD-SD
1.47x
1.55x
2.11x
1.85x
CD-CoS
1.69x
1.76x
2.16x
1.85x
C.3. The Raw Speed
The results are shown in Table 5 and Table 6.
C.4. Ablation study on Alternate Proposal Framework
The results are shown in Table 7 and Table 8. Note that CD-CoS without APF is equivalent to SCD.
C.5. Speedup on NPUs
The results shown in Table 9 and Table 10.
17

---

Fast Large Language Model Collaborative Decoding via Speculation
Table The speed of each method under WE setting. The table reports the average number of tokens generated per second. Models are of comparable sizes.
Model Pair
Method
HvmanEval
GSM8K
MMLU
CNNDM
WE
22.617
22.054
20.459
20.782
Llama- Vicuna
WE-SD
28.723
26.685
24.346
23.899
WE-CoS
35.734
33.522
28.847
30.341
WE
49.624
49.025
47.764
46.966
Qwen-3b
WE-SD
56.075
51.966
52.062
50.723
WE-CoS
80.390
74.518
67.824
64.813
WE
82.584
77.941
79.631
76.625
Qwen-1.5b
WE-SD
91.668
88.073
86.001
84.287
WE-CoS
128.831
113.793
106.705
103.443
WE
57.048
53.869
53.742
56.286
Qwen-1.5b
WE-SD
55.050
51.082
52.666
51.927
WE-CoS
105.485
68.287
74.426
85.980
Table raw speed of each method under CD setting. Models are of different sizes.
Model Pair T
Method
HumanEval
GSM8K
MMLU
CNNDM
CD
39.387
38.735
38.969
38.100
CD-SD
80.349
70.110
59.233
60.198
CD-CoS
87.833
77.470
68.975
61.341
Llama-3
CD
38.981
38.585
39,028
38.124
CD-SD
60.421
46.688
46.834
40.793
CD-CoS
64.319
55.562
51.127
44.986
CD
49.314
48.129
49,700
46.215
CD-SD
56.711
77.969
53.676
42.980
CD-CoS
62.136
79.413
83.496
60.080
Llama-2
CD
47.325
47.015
47,492
43,512
CD-SD
44.486
54.537
58.415
47.863
CD-CoS
54.424
56.418
65.064
48,298
CD
23.525
23.525
19.576
19.692
0
CD-SD
23.211
24.701
28.776
27.568
CD-CoS
78.487
61.400
66.949
77.783
OPT
CD
23,934
23,422
19,492
19.671
CD-SD
35.182
36.304
41.128
36.391
CD-CoS
40.448
41.222
42.102
36.391
18

---

Fast Large Language Model Collaborative Decoding via Speculation
Table 7. Ablation study on alternate proposal framework (APF) under weighted ensemble (WE) setting (A = 0.5). Models are of comparable sizes.
Model Pair
Method
HumanEval
GSM8K
w/o APF
1.46x
1.39x
Llama- Vicuna
WE-CoS
1.58x
1.52x
w/o APF
1.40x
1.32x
Qwen-3b
WE-CoS
1.62x
1.52x
Table Ablation study on alternate proposal framework (APF) under contrastive decoding (CD) setting (μr 0.1). Models are different sizes.
Model Pair
T
Method
HumanEval
GSM8K
w/o APF
1.98x
1.72x
CD-CoS
2.23x
2.00x
Llama-3
w/o APF
1,41x
1.24x
CD-CoS
1,65x
1,44x
w/o APF
1.18x
1.59x
CD-CoS
1,26x
1.65x
Llama-2
w/o APF
0.89x
1.21x
CD-CoS
1.15x
1.20x
Table 9. The raw speed of each method under both WE and CD settings using the Ascend 910B3 NPU.
Model Pair
Method
HumanEval
GSM8K
MMLU
CNNDM
WE
7.60
5.83
2.60
2.35
Llama-Var Vieuna
WE-SD
13.59
8.73
3,81
3.55
WE-CoS
14.14
9.98
4.32
4.06
CD
11.98
10.63
4.96
4.51
Llama-2
CD-SD
13.00
16.28
7.098
4.83
CD-CoS
16.71
16.30
7.54
5.23
Table 10. The speedup ratio of cach method under both WE and CD settings using the Ascend 910B3 NPU,
Model Pair
Method
HumanEval
GSMSK
MMLU
CNNDM
WE
1.00x
1.00x
1.00x
1.00x
Llama- Vieuna
WE-SD
1.79x
1.50x
1.47x
1.51x
WE-CoS
1.86x
1.71x
1.66x
1.73x
CD
1.00x
1.00x
1.00x
1.00x
Llama-2
CD-SD
1.09x
1.53x
1.43x
1.07x
CD-CoS
1.39x
1.53x
1.52x
1.16x
19


Reviews:
[{"review_id": "lfucoaeHss", "reviewer": "Reviewer_hXqY", "summary": "This paper proposes \"Speculative Ensemble\", a method for speeding up auto-regressive generation from LLM ensembles using ideas from speculative decoding. For example, in the case of a two model ensemble, one can treat one of the models as the draft model, generate tokens with that model, and then process those tokens with the other model, with the important twist that during the verification algorithm the ensemble distribution is used as the target distribution (instead of directly using the \"verifier's\" distribution). This method can be extended to ensembles with > 2 models. Additionally, this method can be used to explore speed-quality trade-offs when using a weak model as the draft model and a strong model as the target model---as you assign larger weight to the draft model in the ensemble distribution, the acceptance rate and speed goes up, but the quality of the generation goes down.\n\nTheoretically, the paper proves that speculative ensemble is always at least as fast as the naive ensembling approach of running each model independently.\nEmpirically, the paper shows across a variety of tasks that the proposed method attains 1.11x-2.23x speedup over the naive ensembling approach.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the time and effort spent reviewing our submission and greatly appreciate your insightful comments and constructive suggestions. Below, we have done our best to address each of your concerns in detail.\n\n**Methods And Evaluation Criteria: report the \"raw speeds\"**\n\nWe sincerely thank you for your insightful suggestions. To better showcase SE's performance, we have reported raw generation speeds, as presented in [Raw Speed](https://anonymous.4open.science/r/SE-Rebuttal-Supplement-ICML25-667C/raw_speed.md).\n\nAs you correctly noted, SE is the first method proposed to accelerate any form of LLM ensemble. We have extended SD baselines to the ensemble scenario where possible and included them in our comparisons. Other SOTA methods in SD, such as EAGLE [1] and Medusa [2], might also be adapted to ensemble scenario, but doing so would require specific modifications, which are beyond the scope of this work.\n\n**Other Comments Or Suggestions: compare with the very strong baseline**\n\nThank you for your valuable suggestion. For clarity, we refer to the strong baseline you identified as the *parallel ensemble* (PE).\n\nFirst, we use the popular LLM ensemble repository yaoching0/GaC to test the speed of PE and implement a corresponding sequential ensemble to compute speedup. However, as shown in [Parallel Ensemble](https://anonymous.4open.science/r/SE-Rebuttal-Supplement-ICML25-667C/pe.md), PE is even slower than the sequential ensemble.\n\nThe inefficiency likely stems from overly frequent communication. In particular, during the generation of each token, PE requires communication between the main node and two GPUs. Because the time to generate a single token is very short (especially when kv cache is enabled) the communication overhead becomes significant, introducing noticeable latency. In contrast, standard sequence-level parallelism avoids this issue, as the time to generate an entire sequence is much longer than the communication time, rendering the overhead negligible. These findings further underscore the importance of SE. Note that the speed reported in the original GaC repository was measured with kvcache disabled. In contrast, our tests were conducted with the kvcache enabled, as it is commonly used in current LLM inference. Therefore, our setup provides a more realistic evaluation.\n\nAdditionally, the speedup achieved by PE is closely tied to its implementation and the underlying hardware. With improved engineering and more powerful hardware, PE could potentially attain greater speedup. Nonetheless, we maintain that SE remains a promising approach. For a detailed explanation, please see our response to Reviewer 3eWy under \"Claims and Evidence 1\".\n\n**Questions For Authors: the SD method in Tables 2 and 3**\n\nWe apologize for the earlier lack of clarity. \n\nWe use an example to clarify this SD process: In each cycle, suppose the proposal model $\\mathcal{M}_q$ sequentially generates 5 tokens, which are then verified in parallel by the target model $\\mathcal{M}_p$, resulting in 6 distributions, where the first 5 ones correspond to the verified distributions of 5 proposal tokens, while the 6-th is the bonus distribution. Since the 6-th bonus distribution lacks a corresponding distribution from $\\mathcal{M}_q$ to compute the ensemble distribution, we cannot directly sample the bonus token from the ensemble distribution. To resolve this, we invoke $\\mathcal{M}_q$ again to produce the 6-th distribution. Finally, we apply the standard accept-reject criterion in SD.\n\nCompared to this SD baseline, SE addresses this by treating the 6-th token as a proposal from $\\mathcal{M}_p$ and verifies it using $\\mathcal{M}_q$.  This verification not only obtains the 6-th distribution from $\\mathcal{M}_q$, but also generates an additional bonus token from $\\mathcal{M}_q$. In the next cycle, only 4 invocations to $\\mathcal{M}_q$ are needed to form the proposal, thus reducing one invocation of $\\mathcal{M}_q$ compared to the SD in Tables 2 and 3.\n\nAs described above, the SD in Tables 2 and 3 does not throw away the bonus token; instead, it utilizes the token in the same manner as the standard SD. However, as previously discussed, SE utilizes bonus tokens more efficiently. With the same number of model invocations per cycle, it can generate one additional bonus token, resulting in greater acceleration.\n\n\n\nIf you have any further questions or concerns, please feel free to let us know. We are committed to addressing any concerns to the best of our ability.\n\n\n\n[1] EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test\n\n[2] Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads", "claims_and_evidence": "Yes, the claim are support by clear and convincing evidence across numerous model pairs and tasks.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense. I found the \"alternate proposal framework\", as well as the extension of this method to N models, clever. The theoretical analysis is interesting and illuminates the speedups that should be expected from this method.\n\nIt would have been useful to report the \"raw speeds\" attained by the proposed method, to compare with SOTA in the literature.", "theoretical_claims": "I (relatively quickly) checked the proofs of the claims in the main paper, and did not find any issues. The results make sense to me.", "experimental_designs_or_analyses": "Yes, I did not see any issues.", "relation_to_broader_scientific_literature": "To the best of my knowledge, this paper is the first to show how to use speculative decoding to sample from a distribution which is computed by combining two or more other model distributions. The method relates to the broader literature on speculative decoding and on how to speed up autoregressive generation from LLMs.", "essential_references_not_discussed": "I am not aware of any key references that weren't discussed.", "other_strengths_and_weaknesses": "The method is clever, novel, and well-validated.  I see no glaring weaknesses.", "other_comments_or_suggestions": "It could be worth comparing with the very strong baseline of running the different models in the ensemble on difference devices, simply as a reference point (each model processes the last generated token in parallel, then the ensemble distribution is computed, and then a token is sampled from that distribution).", "questions_for_authors": "In Tables 2 and 3, what exactly is the SD method? Does it simply throw away the bonus token?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tf8NnpW6Pj", "reviewer": "Reviewer_A7ww", "summary": "This paper proposes Speculative Ensemble, accelerating ensemble speed without sacrificing the ensemble quality, inspired by speculative decoding. They theoretically prove the speed improvement over standard ensemble approaches. Experimental results also support their arguments and better ensemble efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments. Below, we address each concern in detail.\n\nFor clarity and brevity, we use the following abbreviations: Methods and Evaluation Criteria (MAEC), Theoretical Claims (TC), Experimental Designs or Analyses (EDOA).\n\n**MAEC1: contrastive decoding is the ensemble function in our setup when model sizes differ**\n\nAs discussed in Section 4.1 \"Ensemble Functions and Methods\" and summarized in Table 1, for model pairs like Llama-3, -2, and OPT, where the proposer and verifier differ a lot, we applied **contrastive decoding (CD)** [1] rather than the traditional weighted ensemble (WE).\n\nCD enhances token generation quality by subtracting the logits of a smaller model from those of a larger one, as defined in Equation (5). When combined with CD, LLMs generally achieve improved performance. To support this, we present results using Llama-3 and -2 with CD, and compare them to the large model baseline, as detailed in [CD](https://anonymous.4open.science/r/SE-Rebuttal-Supplement-ICML25-667C/cd.md).\n\nTherefore, when model sizes differ, our method is not intended to trade performance for speed. Instead, our focus is, when an ensemble method (e.g., CD) improves LLM performance, SE can further accelerate the ensemble while preserving these gains.\n\nAdditionally, it is important to point out that, as shown in Section 4.1 (\"Ensemble Functions and Methods\") and Table 1, we also evaluated the speedup of SE under the WE setting when model sizes are similar. The results are presented in Table 2.\n\n**MAEC2: about weighted ensemble when model sizes differ in Figure 8**\n\nAlthough this paper primarily focuses on CD when model sizes differ, we also introduce WE with different model sizes as an exploratory approach to the quality-speed tradeoff. This discussion appears on page 4, line 203 (right column), with the corresponding results shown in Figure 8. However, this is not a central focus of the paper; rather, we present it as a potential strategy for achieving faster acceleration in vanilla SD—specifically, by applying WE with the proposal model and using SE.\n\nIn this context, $\\lambda$ serves as a hyperparameter that governs the quality-speed tradeoff. One possible selection strategy is to choose the largest $\\lambda$ such that the ensemble performance exceeds a predefined threshold. As shown in Figure 8, a larger $\\lambda$ indicates greater acceleration. Therefore, this selection strategy ensures that the highest speed is achieved while meeting the performance requirements.\n\n**MAEC3: concerns about verifying bonus tokens from small model**\n\nFirst, as previously noted, when model sizes differ, our primary focus is on CD. This ensemble approach can outperform a single large model. Therefore, verifying bonus tokens is meaningful.\n\nIn the context of  WE with different model sizes, as you noted, verifying bonus tokens from the large model typically leads to reduced performance. Therefore, using the bonus tokens directly without verification is a more suitable approach. We appreciate your constructive feedback and will reflect it in the revised version. That said, our design prioritizes interpretability—specifically, ensuring that the distribution of generated tokens remains controllable and consistent. Consequently, in some cases, it remains beneficial to verify bonus tokens produced by the large model. For instance, as mentioned on page 5, line 231 (left column), some studies suggest that appropriately ensembling a smaller model can enhance safety.\n\n**TC: about Corollary 3.5**\n\nThe term \"swap\" in Corollary 3.5 does not refer to the dynamic role alternation between proposer and verifier in the alternate proposal framework.\n\nCorollary 3.5 is specifically formulated for the Speculative-based Ensemble, which includes only the methods introduced in Section 3.2 and excludes the alternate proposal framework. It states that when the proposer and verifier are fixed, choosing an appropriate proposer model before inference begins can ensure acceleration. For example, when ensembling $\\mathcal{M}_p$ and $\\mathcal{M}_q$, if using $\\mathcal{M}_p$ as the proposer does not guarantee acceleration, then $\\mathcal{M}_q$ must. The \"swap\" referenced in Corollary 3.5 refers to this initial selection at the start of inference and does not occur dynamically during the inference process.\n\nAdditionally, Corollary 3.7 guarantees the acceleration of SE under the alternate proposal framework. This corollary provides a more general result that ensures acceleration for any form of ensemble, not just WE.\n\n**EDOA & MAEC2: compare to single model SD**\n\nRegarding this, please refer to our response to Reviewer m8L7 under \"Q1, 3, 4 & W3, 4\".\n\n\n\nIf you have any further questions or concerns, please feel free to let us know. We are committed to addressing any concerns to the best of our ability.\n\n\n\n[1] Contrastive Decoding: Open-ended Text Generation as Optimization", "claims_and_evidence": "While vanilla ensemble approaches require sequential calls of all models, the proposed methods smartly leverage Speculative decoding to make it more efficient. Theoretical guarantee for speedup is also supporting the claim very well.", "methods_and_evaluation_criteria": "- If the proposal and verifier differ a lot (i.e., small $c$), setting $\\lambda$ larger than $\\frac{c}{1+c}$ will naturally increase the acceptance rate and speed obviously. However, we know that there's a trade-off where performance might degrade as it converges towards the proposal model's performance. Using an ensemble, there could be an optimal $\\lambda$ for peak performance, but this method doesn't seem to offer a way to find that optimal value. While ensemble intuitively speeds things up, uniform weights like 0.5 or 0.3 are likely near-optimal only when model sizes are similar. Combining small and large models might introduce inefficiency in finding the right trade-off.\n- Regarding this, Figure 8 suggests that with differing model sizes, there isn't a clear optimal $\\lambda$, and ensembling the proposal distributions doesn't seem to help performance. It's unclear if ensembling actually improved overall performance through this graph. In such settings (llama-3, -2, opt), a comparison of the accuracy-speedup trade-off with vanilla SD would be useful. Can you show a performance comparison between them given the same throughput?\n- Also, if ensembling small model distributions is less effective, I don't see the benefits for verifying bonus tokens. Of course, it makes sense if it helps performance with similar sized models, and I think that efficient refinement of bonus tokens could be possible during autoregressive generation after the swap.", "theoretical_claims": "I’ve reviewed the theoretical claims well. One question I have is regarding Corollary 3.5, which states that if the proposer and verifier are swapped (alternate proposal framework), there is acceleration when $\\lambda < \\frac{c}{1+c}$. Have these things been reflected upon or considered? It seems like the same $\\lambda$ is used even when switching.", "experimental_designs_or_analyses": "The validation is too focused on speed, and there's no performance trade-off from the ensemble. While it's clear that it's faster than existing ensembles, I'd like to see the performance gain from the ensemble as well. A clearer comparison of speed and performance with existing SD-based methodologies that used model ensembles would be helpful.", "relation_to_broader_scientific_literature": "I found the concept of using SD for model ensembling interesting. However, I'm not sure how much gain there is in terms of speed and performance compared to existing methodologies in the SD literature that used combinations of two distributions (small-large).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I’d be happy to discuss with the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ggPvFRBQ3M", "reviewer": "Reviewer_wgdK", "summary": "## Update after Rebuttal\n\nMy concern regarding the insufficient analysis of the quality–speedup trade-off was addressed by the authors’ rebuttal, therefore I have reflected this by increasing my score from 2 to 3, i.e., leaning towards acceptance.\n\nHowever, the results of comparing with non-ensemble baselines suggest that the proposed method is ~$2\\times$\n slower than standard speculative decoding (as also noted by Reviewer m8L7). This prevents me from giving a higher rating.\n\nIMHO, this actually raises concerns about the necessity of using the ensemble approach --- when comparable or even better performance might be achievable by pairing a small and a large model (e.g., 1B–14B in standard speculative decoding), using two models of similar size as in the ensemble setup (e.g., 7B–7B) seems unnecessary since the non-ensemble methods could potentially lead to both faster generation speed and better generation quality.\n\nOn a related note, this ICLR 2025 paper (SpecCascade [1]) appears to be a closely relevant work, which is able to provide better cost-quality trade-offs than their sequential cascade and speculative decoding counterparts. **Given the similarity between Speculative Ensemble and SpecCascade, it is suggested that the authors discuss this work and better position the unique contribution of the submission.**\n\n[1]. Faster Cascades via Speculative Decoding, ICLR 2025.\n\n## Original Summary\n\nThis paper introduces Speculative Ensemble (SE), a method designed to accelerate large language model (LLM) ensembles by leveraging speculative decoding principles. Instead of computing ensemble predictions independently for each model, SE employs a verification scheme where models alternate as proposers and verifiers. The paper claims that this design maintains ensemble quality compared to standard ensemble methods while achieving faster decoding speeds. Theoretical analysis demonstrates that the proposed SE is guaranteed to be no slower than a standard ensemble under certain assumptions. Empirical results on benchmarks such as HumanEval, GSM8K, MMLU, and CNNDM show speedups ranging from 1.11x to 2.23x over standard ensembles. However, no comparison of generation quality is provided.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments. Below, we address each concern in detail.\n\nFor brevity, we use the following abbreviations: Claims and Evidence (CAE), Methods and Evaluation Criteria (MAEC), Theoretical Claims (TC), Experimental Designs or Analyses (EDOA), Weaknesses (W), and Questions for Authors (Q).\n\n**CAE & MAEC1 & TC & EDOA1 & W2 & Q1: concerns about generation quality**\n\nWe apologize for the lack of clarity regarding generation quality. In the SD domain, performance is theoretically well-established, so researchers typically focus on comparing speed rather than performance [1] [2] [3]. Our approach follows this standard practice.\n\nFrom both theoretical and experimental perspectives, we confirm that SE consistently maintains ensemble quality.\n\nTheoretically, as shown in Appendix A.1, we proved that the generated tokens precisely follow the ensemble distribution—that is, they can be regarded as samples from this distribution. Reviewers hXqY, A7ww, and 3eWy also endorsed the correctness of our proof.\n\nTo further validate this, we conducted additional experiments. As shown in [Generation Quality](https://anonymous.4open.science/r/SE-Rebuttal-Supplement-ICML25-667C/generation_quality.md), when T=0, the performance is exactly the same (as proved in Section 3.2 of [4]), while when T=1, due to randomness, the performance shows slight differences but remain largely consistent. For the weighted ensemble (WE), we report only T=1 case, as T=0 is uncommon in WE setting, as discussed in Section 4.1 \"Ensemble functions and methods\".\n\n**MAEC1 : concern about Corollary 3.5**\n\nCorollary 3.5 states that for any given $\\lambda$, there exists a $\\gamma$ that guarantees an ensemble speedup. Notably, $\\lambda$ is not a hyperparameter of the SE algorithm; instead, the proposal length $\\gamma$ is. As an acceleration method, SE aims to speed up the ensemble under a given $\\lambda$. The selection of $\\lambda$ follows the same procedure as in standard ensembles—that is, it is set to the value that yields the best performance. For example, [5], [6], and [7] respectively set $\\lambda$ based on the trade-off between model purification and standard performance, the performance on a development set, and the perplexity.\n\nIn addition to accelerating common weighted ensembles (Equation (4)), the proposed SE can also speed up any form of probability- or logits-level ensembles (Equation (3)). In this more general setting, Corollary 3.7 provides a theoretical guarantee of SE's acceleration.\n\n**MAEC2 & EDOA3 & W1: compare with non-ensemble baselines**\n\nRegarding this, please refer to our response to Reviewer m8L7 under \"Q1, 3, 4 & W3, 4\".\n\n**Q2: the practicality of ensemble small and large models**\n\nRegarding this question, please refer to our response to Reviewer A7ww under \"MAEC1\".\n\n**EDOA2: concerns about ablations**\n\nFirst, the quality-speed tradeoff in Appendix C.1 is not a central focus of the paper, please refer to our response to Reviewer A7ww under \"MAEC2\" for a more detailed explanation.\n\nSecond, we did not separate the impact of speculative decoding and ensemble benefits because SE is designed specifically for ensemble scenarios, rather than treating ensemble as a separate component. While it is well known that ensemble can enhance LLM performance, it typically incurs slow inference. SE aims to accelerate inference while preserving these performance gains.\n\n**W3 & Q3: concerns about alternate proposal**\n\nFirst, please refer to our response to Reviewer hXqY under \"Questions For Authors\" for an example illustrating the impact of using the alternate proposal framework (APF). As shown, APF enables the generation of an additional bonus token within the same number of model invocation, thereby enhancing efficiency.\n\nSecond, our ablation study confirms the effectiveness of APF. The results are shown in [Ablation of APF](https://anonymous.4open.science/r/SE-Rebuttal-Supplement-ICML25-667C/ablation_of_apf.md).\n\nThird, as detailed in Section 4.1, \"Ensemble Functions and Methods\" and Table 1, we evaluated 4 groups of comparable model configurations (e.g., Llama-2-7B + Vicuna-7B). The results in Table 2 show that APF can still enhance efficiency if the drafter and verifier models are of comparable sizes.\n\n\n\nIf you have any further questions or concerns, please feel free to let us know. We are committed to addressing any concerns to the best of our ability.\n\n\n\n[1] Fast Inference from Transformers via Speculative Decoding\n\n[2] EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty\n\n[3] GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding\n\n[4] Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation\n\n[5] Purifying Large Language Models by Ensembling a Small Language Model\n\n[6] Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration.\n\n[7] Relative representations enable zero-shot latent space communication", "claims_and_evidence": "- The claim that Speculative Ensemble (SE) maintains performance compared to standard ensembles lacks empirical validation. In Section 4, the experiments primarily focus on speedup comparisons, while no quality metrics (e.g., accuracy) are provided to confirm that speculative ensembling does not degrade generation performance. As a result, it remains unclear whether SE preserves the benefits of ensembling.", "methods_and_evaluation_criteria": "- Lack of generation quality evaluations. While the speedup metric (tokens per second) is reasonable, the absence of generation quality evaluations is a significant concern. As stated in Corollary 3.5, there exists a hyperparameter $\\lambda$ such that the proposed Speculative Ensemble (SE) is guaranteed to be at least as fast as a standard ensemble. However, the choice of $\\lambda$ inherently impacts ensemble quality. Although certain values of $\\lambda$ may lead to speedups, they could also result in performance degradation, potentially making SE less effective than non-ensemble methods or standard ensemble methods. Without empirical evidence demonstrating that SE improves generation quality, the practical utility of the method remains uncertain.\n\n- Insufficient baseline comparisons. The authors do not compare SE against non-ensemble baselines to demonstrate its effectiveness in terms of generation speedup and quality. A basic evaluation setup should include the following typical baselines: 1) single-model baselines: using only the large or smaller model alone for token generation. 2) standard speculative decoding: employing a fixed small model as the drafter and a large model as the verifier for token generation.\n\nWithout such comparisons, it is difficult to assess whether SE provides meaningful improvements over simpler or more established approaches.", "theoretical_claims": "The mathematical proofs seem correct, but they do not address whether SE actually improves generation quality.", "experimental_designs_or_analyses": "- The experiments demonstrate speedup but fail to evaluate generation quality.\n- The quality-speed tradeoff study in Appendix C.1 is insufficient. No ablations are provided to separate the impact of speculative decoding vs. ensemble benefits.\n- The paper does not compare SE against simpler baselines, such as using the large model alone or using a standard speculative decoding setup with one small drafter and one large verifier (w/o alternating the drafter and verifier).", "relation_to_broader_scientific_literature": "The paper borrows principles from speculative decoding and applies them to LLM ensembles.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Other Strengths\n- An interesting attempt to extend speculative decoding to ensemble settings\n\nOther Weaknesses\n- Missing comparisons with simpler baselines\n- No empirical validation of quality preservation—only speedup is measured\n- Alternating proposer/verifier is not clearly justified—it may degrade efficiency and quality", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How does Speculative Ensemble (SE) impact generation quality compared to baseline methods?\n2. The ensemble setup seems less practical. While pairing a smaller model with a large model is common in speculative decoding, ensembling is typically used with comparable models to achieve further performance gains. In what scenarios would it be necessary to ensemble a large model with a much smaller model? How does SE compare to simply using a single large model? Is ensembling actually necessary for performance gains in the setting described in this paper?\n3. Can the authors justify why alternating between proposer and verifier roles improves efficiency, rather than using the large model exclusively for verification? This finding appears to depend on the specific choice of drafter and verifier models. Would the same efficiency gains hold if the drafter and verifier models were of comparable sizes?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "q9hniaoORY", "reviewer": "Reviewer_m8L7", "summary": "The authors extend speculative decoding to ensemble models and demonstrate, through both theoretical analysis and empirical results, that their approach outperforms standard ensemble baselines.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments. Below, we address each concern in detail.\n\nFor clarity and brevity, we use the following abbreviations: Experimental Designs Or Analyses (EDOA), Weaknesses (W), and Questions (Q).\n\n**W1: Speculative Ensemble (SE) offers two non-trivial improvements over vanilla SD.**\n\nCompared to simply adapting vanilla SD to ensemble scenarios, SE offers two non-trivial improvements:\n\n1. Compared to vanilla SD, where one model consistently serves as the proposer and the other as the verifier, we introduce an alternating proposal framework (APF) in Section 3.3, as shown in Figure 2. This method switches the roles of proposer and verifier during decoding, and is specifically designed to enhance efficiency in ensemble settings. We also validate its effectiveness through ablation experiments shown in [Ablation of APF](https://anonymous.4open.science/r/SE-Rebuttal-Supplement-ICML25-667C/ablation_of_apf.md). For a case-specific explanation, please refer to our response to reviewer hXqY under 'Questions for Authors'.\n3. Unlike the vanilla SD, which employs two models, the proposed SE extends to an n-model scenario to accelerate the n-model ensemble in Section 3.4, as shown in Figure 3 and Figure 7.\n\nMoreover, Reviewer hXqY described our method as “clever, novel, and well-validated,” while Reviewer 3eWy referred to it as a “novel framework,” both highlighting the non-trivial improvements offered by our approach.\n\n**W1: concerns about general applicability**\n\nOur proposed SE not only accelerates traditional weighted ensemble (WE) methods (Eq 4), but also accelerates ensembles of any form, size, or number of LLMs at the probability or logits level. This includes techniques such as contrastive decoding (CD) [1] (Eq 5) and decoding-time realignment [2], demonstrating its broad applicability. As Reviewer 3eWy noted, SE “can be useful in various application,” further supporting this point.\n\n**W2: concerns about theoretical novel insights**\n\nWhile our theoretical analysis of correctness (i.e., losslessness) and speedup follows original SD paper, we introduce two novel theoretical insights tailored to the ensemble scenario:\n\n1. As discussed on page 4, line 205 (left column), in vanilla SD, estimating the expected speedup requires extensive experiments to estimate the acceptance rate $\\alpha$. However, in WE setting, the parameter $\\lambda$ (Eq 4) naturally serves as a lower bound for $\\alpha$. This allows us to estimate the speedup in advance and choose a suitable proposal model, such as those described in Corollary 3.5.\n2. Vanilla SD does not guarantee acceleration; in fact, it may result in slower speed when $\\alpha$ is small. In contrast, SE is theoretically proven to be no slower than the standard ensemble, even in the worst case, as established in Corollary 3.7.\n\n**Q1, 3, 4 & W3, 4: compare with more baselines**\n\nThank you for your valuable suggestions. To better demonstrate SE, we conducted experiments comparing SE with three non-ensemble baselines: Large Model, vanilla SD, and Medusa. The results are shown in [Non-ensemble](https://anonymous.4open.science/r/SE-Rebuttal-Supplement-ICML25-667C/non_ensemble.md). Note that for Medusa, we only reported the results for Vicuna, because it only provides the pretrained draft model for Vicuna. For Qwen-3B, we did not report results for vanilla SD, because there is no suitable draft model for it. \n\nAdditionally, it is important to note that SE, as an acceleration algorithm, is a line of research parallel to SD. It does not focus on finding a quality-speed trade-off. Instead, SE focus on accelerates inference while preserving the performance gains provided by ensemble.\n\n**EDOA1 & Q2: results for models larger than 7B and the impact of model size**\n\nAs described in Section 4.1, \"Model Pair Configuration\" and in Table 1, we evaluated the CD setting using the model pair (OPT-13B, OPT-125m). The corresponding results are presented in Table 4.\n\nIn addition, we also tested a larger model pair (Llama-2-13B, Vicuna-13B) under the WE setting. The results are shown in [Larger Model](https://anonymous.4open.science/r/SE-Rebuttal-Supplement-ICML25-667C/larger_model.md).\n\nTogether with the results in Tables 2 and 3, we observe that SE achieves greater speedup as model size increases in both the WE and CD scenarios, with a more pronounced effect in CD. In WE, this trend may be due to improved model performance with larger sizes, which increases similarity between models and leads to a higher acceptance rate. In the CD scenario, the growing speedup may stem from the increasing cost of invoking the larger model.\n\n\n\nIf you have any further questions or concerns, please feel free to let us know. We are committed to addressing any concerns to the best of our ability.\n\n[1] Contrastive Decoding: Open-ended Text Generation as Optimization\n\n[2] Decoding-time Realignment of Language Models", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The theoretical analysis largely follows the original speculative decoding framework, but appears to be correct.", "experimental_designs_or_analyses": "1. Do you test larger models, larger than 7B?\n2. Could you design experiments to compare with more baselines/related works? Only comparing with pure ensemble models and the original speculative decoding seems insufficient to fully evaluate the contribution of the proposed method.", "relation_to_broader_scientific_literature": "Speculative decoding is important for efficient AI.", "essential_references_not_discussed": "Not that I'm aware of.", "other_strengths_and_weaknesses": "Strengths:\n1. The proposed method shows improvements over standard ensemble models, demonstrating its practical value within that setting.\n\nWeaknesses:\n1. The method appears to be a relatively minor adaptation of standard speculative decoding applied specifically to ensemble models, which raises concerns about its general applicability beyond this narrow use case.\n2. The theoretical analysis closely follows prior speculative decoding work and does not introduce significantly novel insights.\n3. It is unclear whether the authors have compared their approach against strong non-ensemble baselines. A comparison in terms of speedup and accuracy trade-offs with single-model speculative decoding would help contextualize the contribution.\n4. The paper includes limited baseline comparisons, making it difficult to assess the overall effectiveness and competitiveness of the proposed method.", "other_comments_or_suggestions": "No minor issues.", "questions_for_authors": "1. Could you compare the speedup and accuracy trade-offs with single-model speculative decoding, not just with ensemble models?\n2. Have you tested on larger models? How does the model size affect the results?\n3. How does the method compare with batch speculative decoding methods such as Medusa [1]?\n4. Could you design experiments to compare with more baselines/related works? Only comparing with pure ensemble models and the original speculative decoding seems insufficient to fully evaluate the contribution of the proposed method.\n\n[1] Cai, Tianle, et al. \"Medusa: Simple llm inference acceleration framework with multiple decoding heads.\" arXiv preprint arXiv:2401.10774 (2024).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JlGLvAqTjS", "reviewer": "Reviewer_3eWy", "summary": "This paper introduces \"Speculative Ensemble\" (SE), a novel framework that accelerates Large Language Model (LLM) ensembles without sacrificing performance. While ensemble methods enhance LLMs by combining multiple models, they suffer from high computational costs. The authors build on speculative decoding—where a small model generates tokens sequentially and a larger model verifies them in parallel—with two key insights: (1) the verification distribution can be the ensemble distribution of both models, and (2) alternating each model as proposer and verifier enhances efficiency. The approach generalizes to n-model ensembles and theoretical analysis proves SE is never slower than standard ensembles. Experiments across various tasks demonstrate speed improvements of 1.11x–2.23x over standard ensemble techniques without compromising generation quality.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments. Below, we address each concern in detail.\n\n**Claims And Evidence 1: compare to PE when model sizes are close**\n\nFirst, we evaluate the speedup of *parallel ensemble* (PE). However, PE is even slower than the sequential ensemble. For details on the experimental setup, results, and further discussion, please refer to our response to Reviewer hXqY under 'Other Comments or Suggestions'.\n\nMoreover, as you noted, when model sizes are comparable, SE is theoretically expected to perform similarly to or slower than PE. Nonetheless, we maintain that SE remains a promising and broadly applicable approach. Our reasoning is as follows:\n\n1. In practice, PE is limited by its reliance on complex engineering and powerful hardware. It also requires twice the throughput and more GPU memory. Consequently, sequential ensemble is also widely used, as seen in repositories like xydaytoy/EVA, starrYYxuan/UniTE, and cmavro/PackLLM. SE can greatly speedup sequential ensemble without requiring specialized engineering or additional throughput and GPU memory, highlighting its practical potential.\n\n2. Furthermore, if SE is equipped with the same throughput as PE (i.e., double its current throughput), it can be combined with PE to accelerate more models. For example, the 2-model ensemble in Figure 2 can be naturally extended to a 4-model ensemble. The procedural steps for this extension are detailed below, and a schematic illustration is provided in [SE with_PE](https://anonymous.4open.science/r/SE-Rebuttal-Supplement-ICML25-667C/se_with_pe.pdf).\n\n   - Step 1: Models $\\mathcal{M}_1$ and $\\mathcal{M}_2$ are invoked in parallel to produce distributions $p_1^{(1)}(\\cdot)$ and $p_1^{(2)}(\\cdot)$. A token $x_1^{(1)}$ is sampled from $p_1^{(1)}(\\cdot)$ as the proposal, and its score under $\\mathcal{M}_2$, given by $p_1^{(2)}(x_1^{(1)})$, is saved.\n   - Step 2: Similarly, $\\mathcal{M}_3$ and $\\mathcal{M}_4$ are invoked in parallel to score $x_1^{(1)}$ and generate bonus distributions $p_2^{(3)}(\\cdot)$ and $p_2^{(4)}(\\cdot)$. A bonus token $x_2^{(3)}$ is sampled from $p_2^{(3)}(\\cdot)$, and its score under $\\mathcal{M}_4$, i.e., $p_2^{(4)}(x_2^{(3)})$, is saved.\n   - At this stage, $x_1^{(1)}$ has been scored by all 4 models and proceeds to verification. If accepted, $x_1^{(1)}$ is treated as a sample from the ensemble distribution, as illustrated in Step 3.\n   - Step 4: The process is repeated with $\\mathcal{M}_1$ and $\\mathcal{M}_2$ invoked again to score $x_2^{(3)}$ and generate new bonus distributions, mirroring the procedure in Step 2.\n\n   The algorithm retains the properties discussed in the paper, including losslessness, and it is never slower than the 4-model PE.\n\n   Note that the method described here is a preliminary version and can be further optimized. For instance, in Step 1, we can sample two tokens from both  $p_1^{(1)}(\\cdot)$ and $p_1^{(2)}(\\cdot)$, and verify them in parallel using a tree attention to further improve the acceptance rate. We will explore this in future work to further enhance the performance of SE.\n\n**Claims And Evidence 2: compare to PE when model sizes differ**\n\nWhen the model sizes differ greatly, speedup is mainly coming from reducing the number of large model invocations.\n\nIn the optimal case, PE reduces only the computational time of the small model, while the large model remains a major time bottleneck. In contrast, SE—under setting shown in Figure 2—can halve the number of large model invocations, resulting in a substantially more effective speedup.\n\nOn the other hand, the speedup comes from allowing the smaller model focuses on sequential generation, while the larger model focuses on parallel verification. This shares a common acceleration principle with SD.\n\nAs noted on Page 5, Line 251 (left column), two hyperparameters, $\\gamma_p$ and $\\gamma_q$, control the proposal length for each model when acting as the proposer. When model sizes differ greatly, the larger model’s $\\gamma$ is set to 1, and the smaller model is assigned a higher value (e.g., 5), as described in Section 4.1, under “Configuration of $\\gamma$”. As a result, the smaller model is invoked more frequently—about 5 times as often as the larger model—rather than \"half\" of the total invocations. This setup ensures that the smaller model focuses on sequential proposal generation, while the larger model conducts parallel verification, echoing the acceleration strategy of vanilla SD. This is further supported by the ablation study in Figure 6(b), which demonstrates that speedup consistently increases with higher $\\gamma$ values of the small model.\n\n**Experimental Designs Or Analyses:**\n\nWe also provide a case to better illustrate how SE achieves greater speedup compared to SD; please see our response to Reviewer hXqY under \"Questions for Authors\".\n\nIf you have any questions or concerns, please let us know. We’re committed to addressing them.", "claims_and_evidence": "The paper's claims are generally well-supported by both theoretical analysis and empirical evidence. They prove and experimentally show that the speculative ensemble improves speed without sacrificing performance, and it is not slower than an ensemble approach.\n\nOne main concern I have is that based on practical intuition, the approach does not seem to have a clear speedup over the normal ensemble. In the case of 2 models alternating, in Figure 2, at best, each model processes their tokens, and then they are verified by another model. If they are close in size, the expectation is that the parallel run of these models (normal ensemble) will be equal or better than the current proposal. This is because, in the current proposal, we have the autoregressive steps at every step (except for verification), which can run the models in parallel, and avoid extra time for verification. If the models differ in size greatly, then in half of the steps, we are doing better than the parallel ensemble, and in the other half we are not. Hence, it is not clear where the speedup is coming from.", "methods_and_evaluation_criteria": "The paper uses appropriate methodologies and evaluation criteria for assessing the proposed Speculative Ensemble framework. The authors evaluate their approach across diverse tasks (code generation, mathematical reasoning, multi-task understanding, and text summarization) using established benchmarks (HumanEval, GSM8K, MMLU, and CNNDM), which effectively represent a range of LLM applications. Their evaluation metrics focus on tokens generated per second and speedup ratios relative to standard ensemble methods. The experimental design includes various model pairs (Llama-Vicuna, Qwen, Llama-3, OPT), testing both two-model and three-model configurations, and comparing multiple ensemble functions (weighted ensemble and contrastive decoding). Additionally, the comprehensive ablation studies examining the impact of various parameters (λ, μ, and γ) further strengthen the evaluation by providing insights into the factors affecting performance.", "theoretical_claims": "The paper provides well-structured and logical proofs for its key theoretical claims. The authors offer detailed proofs for the correctness of speculative-based ensemble (showing tokens align with the ensemble distribution), acceptance rate calculation, and the speed improvement factors.", "experimental_designs_or_analyses": "The experimental design in the paper is generally sound, with a comprehensive evaluation across multiple model architectures, tasks, and ensemble functions. However, the concern mentioned above, should be addressed in some experimental results, by showcasing the profile of each run (parallel ensemble, and SD ensemble), so we can have a better understanding of how the model achieve this speedup.", "relation_to_broader_scientific_literature": "The proposal is an interesting approach for speeding up the ensemble methods, and can be useful in various application.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "mentioned above.", "other_comments_or_suggestions": "NA", "questions_for_authors": "mentioned above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Ensembling a set of LLMs has often shown to yield better quality than the individual LLMs in the pool. This paper proposes the use of speculative decoding (SD) to speed-up an ensemble of multiple models, with the smallest model used as the drafter and the ensembled distribution used to the verify the draft tokens. When comparing to sequential ensembles, the authors show notable speed-ups, while maintain quality neutrality.

The reviewers liked the application of SD to ensembles, but had concerns about the lack of comparisons to non-ensemble methods and questions about the relative sizes of the LLMs in the ensemble needing to be similar to provide quality gains. The authors provided additional experiments to partially address this concern. 

While we are happy to accept the paper, we encourage the authors to include additional discussions contrasting their proposal with prior work:

**Contrasting with non-ensemble methods**: Based on the additional results, it appears that speculative ensembles (e.g. with weighted ensembling for similar sized models, and contrastive decoding for different sized models) offer ***better quality* but at a *lower speed-up* compared to non-ensemble methods** (e.g. vanilla SD, Medusa). We strongly urge the authors to include these additional results in the paper, and include an elaborate **discussion on comparison to non-ensemble methods and the role of model sizes**.

**Contrasting with similar prior work:** The authors need to differentiate their work from two closely related papers which came up during the post-rebuttal discussions:
- **Speculative Contrastive Decoding**, ACL 2024: The authors cite this paper but state that it differs in the use of "contrastive decoding for (only the) rejected tokens". I am not sure if this distinction is entirely accurate - my understanding is that this prior work also makes very similar changes to SD's acceptance criterion and residual distribution via contrastive decoding (lines 6 and 8 in their Alg. 1), but may differ in how it handles the bonus token. The main difference seems to be that the current paper handles more general ensembling functions.
- **Faster Cascades via Speculative Decoding**, ICLR 2025: This paper was not cited, and also makes a similar proposal as the present paper, except that the ensembling is done via a cascading rule instead of a simple weighting or contrastive decoding.

Both these papers need to be discussed more elaborately in the main text. In particular, it is important for the authors to differentiate their results with CD from the previous "Speculative CD" method, and consider including direct empirical comparisons to this method.
Decision: Accept (poster)