Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding
Authors: Ziyao Wang
Muneeza Azmat
Ang Li
Raya Horesh
Mikhail Yurochkin
Venue: ICML 2025 poster (2025)
Keywords: Large language model
Knowledge fusion
Speculative decoding
Primary area: deep_learning->large_language_models

Abstract: Large Language Models (LLMs) often excel in specific domains but fall short in others due to the limitations of their training. Thus, enabling LLMs to solve problems collaboratively by integrating their complementary knowledge promises to improve their performance across domains. To realize this potential, we introduce a novel Collaborative Speculative Decoding (CoSD) algorithm that enables efficient LLM knowledge fusion at test time without requiring additional model training. CoSD employs a draft model to generate initial sequences and an easy-to-learn rule or decision tree to decide when to invoke an assistant model to improve these drafts. CoSD not only enhances knowledge fusion but also improves inference efficiency, is transferable across domains, and offers greater explainability. Experimental results demonstrate that CoSD improves accuracy by up to 10% across benchmarks compared to existing methods, providing a scalable and effective solution for LLM-based applications. Our code has been released at https://github.com/ATP-1010/CoSD.

Speculate, then Collaborate:
Fusing Knowledge of Language Models during Decoding
Ziyao Wang' Muneeza Azmat
Ang Li
Raya Horesh 2 Mikhail Yurochkin 2
Abstract
mentary knowledge, with each model specializing in certain areas. For example, one model may be more proficient in technical writing, while another may outperform in creative tasks. This heterogeneity has led to an increasing interest in developing methods that can fuse the knowledge of multiple LLMs, enabling users to harness their collective strengths for more robust and versatile applications.
Large Language Models (LLMs) often excel in specific domains but fall short in others due to the limitations of their training. Thus, enabling LLMs to solve problems collaboratively by inte- grating their complementary knowledge promises to improve their performance across domains. To realize this potential, we introduce novel Collab- orative Speculative Decoding (CoSD) algorithm that enables efficient LLM knowledge fusion test time without requiring additional model train- ing. CoSD employs a draft model to generate initial sequences and an easy-to-learn rule or de- cision tree to decide when to invoke an assistant model to improve these drafts. CoSD not only en- hances knowledge fusion but also improves infer- ence efficiency, is transferable across domains and models, and offers greater explainability. Experi- mental results demonstrate that CoSD improves accuracy by up to 10% across benchmarks com- pared to existing methods, providing a scalable and effective solution for L.LM-base applica- tions. Our code has been released https: / httggghu.ccom///oo
To address these challenges, recent research has shifted fo- cus to test-time knowledge fusion, which eliminates the need for retraining by combining model outputs during inference. This approach allows users to leverage the complementary knowledge of LLMs without the training overhead. For example, Wang al. (2023) proposed a method that selects expert models dynamically at inference time using super- vised learning, while Ong et al. (2024) introduced router model that optimizes the selection of models based per- formance and cost. Other approaches focus on integrating outputs through the decoding process, such as token-wiss de- coding (Shen et al., 2024) and character-wise decoding (Gu et al., 2024), which combine outputs at fine-gr ine-ggrined level. Although these methods offer potential, they often strug- gle to balance strong knowledge integration with efficiency, which limits their practicality in real- world applications.
In response to these limitations, we propose Collaborative Speculative Decoding CoSD, a novel algorithm designed to efficiently fuse the knowledge of multiple LLMs at inference time. CoSD builds upon recent developments in Specula- live Decoding (Leviathan al., 2023; Xia al., 2023) to create efficient system where multiple LLMs collabo- rate during the inference process. As shown in Figure 1, CoSD consists of two models: a draft model that generates an initial sequence of tokens and an assistant model that verifies these tokens in parallel. When the assistant model predicts a token different from that of the draft model, a comparison of their token probabilities is used to determine whether to replace the draft token. This decision-makinn process can be guided by either predefined rule set (Rule- Based CoSD) or pre-trained decision tree (Tree-Basse CoSD). The sequence is then regenerated and re-verified iteratively until accepting all tokens, ensuring both accuracy and computational efficiency.
1. Introduction
State-of-the-art large language models (LLMs), such as GPT-4 (Achiam et al., 2023) and Llama-3 (Dubey et al., 2024), have demonstrated impressive capabilities in gener- ating high-quality text across a variety of domains. These models are trained vast datasets, allowing them to per- form well on wide range of tasks. However, despite their general effectiveness, no single LLM excels uniformly across all domains. Different models tend to have comple-
'Department of Electrical and Computer Engineering, Uni- versity of Maryland, College Park, USA. 'IBM Research, Yorktown Heights, USA. Correspondence to: Ziyao Wang <ziyaow@uma @dd..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
CoSD presents several notable advantages over existing

---

Collaborative Speculative Decoding (CoSD)
test-time fusion methods. First, by leveraging speculative decoding, CoSD improves inference efficiency, relying token probabilities rather than more complex and resource- intensive representations like embeddings or hidden states. Second, CoSD demonstrates superior knowledge fusion due to the carefully designed decision-man maainn process, which can be optimized for specific domains. Third, Rule-Based CoSD is highly transferable across tasks and model pairs: once the rules are established with optimal hyperparameters, they can be applied to a broad range of tasks. Similarly, the decision tree-based approach exhibits strong transferabilityy, even when trained on domain-specific data. Finally, CoSD offers an interpretable framework, i.e., its use of human- readable rules or decision trees provides transparency, mak- ing it easier to evaluate, optimize, and understand compared to less transparent deep learning systems.
desired model performance. Towards mitigating these flaws, new wave of works adopt decoding methods to fuse LMs.. Gu et al. (2024) propose a character-wise ensemble decod- ing method to fuse two LLMs' outputs. Shen et al. (2024) and Wang et al. (2023) fuse model knowledge by training to choose between the generation of different LLMs. In our experiments, we consider several baselines from the latter group of works and observe gains in either efficiency or performance when using our method to merge cross- cross-domain knowledge from different LMs when decoding.
Speculative Decoding is an efficient decoding paradigm for LM inference (Xia et al., 2024; Stern et al., 2018; Xia et al.. 2023). It accelerates the inference process by first generating draft tokens efficiently, and then using an LLM to verify draft tokens in parallel and correct them if needed (Leviathan et al., 2023), which avoids the autoregres- sion process. In practice, the draft generator in speculative decoding could be a small LM (Chen et al., 2023; Miao et al., 2023; Zhou et al., 2023), a sub-model of an LLM (Zhang et al., 2023; Yang et al., 2023; Elhoushi et al., 2024), or text database retriever (He et al., 2023; Li et al., 2024). The final generation of speculative decoding will be similar to the autoregressive generation of the target LLM, which is only acceptable when the target LLM has much better performance but is less efficient than the draft generator. No previous work focuses on using speculative decoding to approach the model fusion problem.
We validate the effectiveness of CoSD through extensive experiments on standard benchmarks and multiple model pairings. Our results show that CoSD not only significantly enhances the fusion of LLM knowledge but also improves efficiency and transferability across various domains. The key contributions of this work are as follows:
We introduce CoSD, a novel algorithm that enables effi- cient test-time fusion of LLM knowledge without requir- ing retraining.
CoSD's efficiency and transferability make it practical for wide range of users, facilitating its implementation through both models and APIs.
3. Collaborative Speculative Decoding
Our experimental results demonstrate that CoSD im- proves overall accuracy by up to 10% across benchmarks, surpassing the state-of-the-art methods.
In our Collaborative Speculative Decoding system, our pur- pose is to fuse the predicted sequences of two LLMs effi- ciently. We define our problem as follows: given an input sequence Ti,.... Itt, CoSD uses a draft model Mp and an assistant model MMy to collaboratively generate an out- put sequence 1t+1,..., Tt+ K that integrates both models' knowledge and expertise.
2. Related Work
Language Model Fusion from multiple LMs aims at en- hancing the cross-domain performance of the resulting model and reducing bias. The primary efforts for such integration include model merging (Goddard et al., 2024), such as model weight averaging (Wortsman et al., 2022) and linear mode connectivity (Ainsworth et al., 2022; Ito et al., 2024; Wang et al., 2020). Another series of works is called model stacking, which refers to concatenating mod- els along the depth dimension. Wu et al. (2024) and Kim et al. (2023) stack the decoder blocks to expand the depth of Llama models. For large language models, some other re- search proposes knowledge fusion (Wan et al., 2024). They combine the capabilities of existing LLMs and transfer them into a single LLM. Another important trend of work called Mixture of Expert (MoE) (Zhu et al., 2024; Xue et al., 2024) builds sparse neural networks and only activates a subset of parameters (i.e., experts) for each input. However, these methods either require the fused models to have the same structure or require fine-tuning after fusing to achieve the
As Figure I illustrates, the process begins with the draft model Mp generating draft sequence T.+1,..., Te+K K in an autoregressive manner. Subsequently, the assistant model My verifies the draft tokens by producing an assistant se- quence ft-1,... Li4K. After both sequences are gener- ated, CoSD iterate through the tokens and their correspond- ing probabilities
verify whether to accept draft token Eits or replace it with the corresponding assistant token Re+ Both rule-based or tree-based verification strategies, use token probabilities to determine whether replacement is necessary. When replacement occurs, all subsequent draft tokens are discarded, and a new draft sequence is generated starting from the replaced token. This process continues until the output reaches the maximum length or an EOS token is generated. The full generation and veri- fication process is elaborated in Algorithm I and described
2

---

Collaborative Speculative Decoding (CoSD)
Draft Model
The
cost
of subs is
4
Draft
token
Rale/ Bale/Tree-BBsee
CoSD verificution
Assist. Model
Auforegresivve generation
The
cost
of
subs
are
tokens
Input
Generation
purallel
F(P(draft), P(assistant)) + {0,1} 0 Accept;
Rule-Based CoSD: F(.) are several pre-designed rules.
Reject
Tree-Based CoSD: F(+) pre-trained decision tree.
CoSD
Verification
Reject the draft token. Replace with the assistant token.
X
Accept the draft token to the final output
Regenerate all the following draft tokens.
Final Output
The
cost
of subs
is
2
Figure
The workflow collaborative speculative decoding.
in following sections.
3.2. Verification
After the generation, we have a draft sequence E141....,āt4: and an assistant sequence fc+1...., L along with the corresponding probabilities Mp(3(+s) and (ā We then use this information verify whether to keep the draft token ã; replace it with the assistant token 2; and thus ensemble the model knowledge. In or- der to make CoSD suitable for a wider range of tasks, we propose two strategies for verification. The first strategy, called Rule- Bule-Based Verification, applies clear rules to decide whether to select the draft token or the assistant token. The second strategy, i...,
3.1. Generation.
The generation process follows the principles of Speculative Decoding. First, the draft model Mp generates a sequence of tokens autoregressively:
for i do Tet M, T11511)
Here, T5अ. represents the token predicted by the draft model position selected the token with the highest probabil- ity. The sequence Fe41....,5 Tt is generated autoregres- sively and produced sequentially.
Verification, involves train- ing decision tree to classify and select between the draft and assistant tokens.
After the draft sequence is generated, the assistant model Mg is used to verify these tokens. The assistant model generates tokens in parallel:
Rule-Based Verification. In Rule-Based Verification, the system applies simple yet general rules to determine whether the draft token Zz+; should replaced by the assistant token L.+s. The intuition behind these rules that if the draft model predicts token with low confidence and the assistant model offers a higher-confidencee alternative, the draft token should be replaced. The following rules define the verification process:
i = K in parallel do
fett (g Te+111).
Note that we already have all the draft tokens Ti-1...., LitK when we generate the assistant tokens. Thus, all the Z in Eq. (2) can be generated in paral- lel. The process can also handle cases where the draft and assistant models use different tokenizers. In such cases, the draft sequence is first decoded by the draft model's 's tokenizer and then encoded by the assistant model's tokenizer:
Jiti # Metis Mn(T++s) <
(4)
(5)
Mg(E++)) > 3 Mp
(6)
=1, ==,,... K in parallel do
These conditions check whether (1) the draft and assistant tokens differ, (2) the draft token has probability below a threshold a, and (3) the assistant token has probability sufficiently higher than the draft token's probability by factor of B. If all conditions are met, the draft token is replaced with the assistant token.
decode
encode
[11 (
Texts
Tp
T2
Gets M, 3
(3) where Tp and T, are the tokenizers of the draft model and the assistant model respectively. The draft sequence is first decoded into texts by Tр and then encoded by T, to fit the assistant model.
Intuitively, the Rule- Kule-Based Verification can be explained as follows: if the draft model is uncertain and the assistant model provides better alternative, the system opts for
3

---

Collaborative Speculative Decoding (CoSD)
Algorithm 1 Workflow of CoSD
Input:
Draft model Mp. assistant model M,. input sequence 21..... 2y, predefined hyperparameters a, 3 and trained decision tree T:
Output: Output sequence .
Generation
1: for i 0, 1....., K do
.....,KK.
2: Tets ~ MMM (a 8141-1) # Generate draft in an auto-regressive manner.
3: end for
4: Verify the draft in parallel:
5: i -,..... K in parallel do
6: ăeti ~
I1) # Generate the assistant sequence in parallel.
7: Send both T1, KK. f1....,,xx and all related probabilities Mp(T;), M, to verification.
Verification
8: for in 0, 1,........ ................................ K do
9:
if Tr.i + 8. and (Z a and Mg(ā(44) B M, then or
10:
if Tcti # 214s and T (MM,
then
11:
Erti < Itti
12:
i+t+i
13:
Exit loop, go to Generation
14:
end for
the assistant's prediction. replacement made, the sequence is updated, and the draft model regenerates from that point onward.
is one training sample the decision tree. As for the related ground-truth label, we have three rules:
If Te = Te+-is assign the label y; = 0 to encourage the decision tree to select the draft token.
Tree-Based Verification. For domain-specifee applica- tions, Rule-Based Verification may not always be optimal. It necessary to improve performance in specialized domains, such as healthcare (Poonia Al-Alshaikh, 2024), smart home (Amru al., 2024), or math (Mazraeh et al., 2024). Therefore, we design the Tree-Based Verification method, which involves training decision tree to decide when to re- place draft token with an assistant token. Training the deci- sion tree on specific domain data allows for a more accurate assessment of knowledge fusion performance within those particular contexts. Specifically, our decision tree T takes two probabilities, (T++++)) and (ā(++.) inputs. The decision tree's output (TM indicates whether to the draft token (z/ 0) replace it with the assistant token (y/= 1).
If Ze4s * Z++i and le+; = It+i, we assign the label
Vi to encourage the decision tree to select the assistant token.
If neither ãe+t nor êi+i match the target, we drop the sample and continue the loop with i+
We iterate through all the input-output pairs and fi- nally construct the training data sample in the form of (M,, M, M.). In the training process, we aim to train the decision tree classifier T R22 + {0, 1 to minimize the difference between the predicted label and the ground truth:
E 1)
U: log Mg,)) min Σ
To train decision tree suitable for specific domains, we first select a commonly used benchmark dataset D for this domain (e.g., GSM8K (Cobbe et al., 2021) math) with several input and ground-truth output pairs, i.e., Z1. 2t and 1t+1..... . Kt+K. We iterate through all the tokens in the ground-truth output in each pair. For the i-th token, we concatenate the input sequence and the first i - tokens of output sequences. Then, we feed the concatenated input Ii. JI...., It+i-1 into the two models separately to obtain the predicted next token Te4s, 2.4s and their corresponding probabilities (T(+++)) (5+++)) This probability pair
+(1 - Bi) log(1
((M, (MM M,))))
After training, our decision tree can predict whether to choose the draft token the assistant token based on the two input probabilities. If the decision tree predicts the same as the rule-based verification, update the accepted token number, and send the new input sequence back to the generation. Since the decision tree is trained on a dataset specific to the corresponding domain, us- ing this decision tree to fuse the model outputs can achieve better results that domain.
replace the token,

---

Collaborative Speculative Decoding (CoSD)
Table LLM pairs in the experiments.
Methods
Draft Model
Assist. Model
Simulated Scenario
Pair
Mistral 7B DARE
Mistral 7B Mixed
Complementary Knowledge Fusion
Pair
Llama 3 Wissenschaft 8B
Llama 3 Bophades 8B
Complementary Knowledge Fusion
Pair
Mistral 7B (Jiang al., 2023)
Mistral Math 7B
Catastrophic Forgetting Recovery
Pair
Lima (Zhng et al., 2024)
Llama 2 Chat
Capacity Imbalance
Pair
Llama 2 Chat (Touvron al., 2023)
WizardMath (Luo et al., 2023)
Different Tokenizers
Pair
Llama 2 Chat
DeepSeek Coder (Guo al., 2024)
Different Tokenizers
4. Experiment
M, (2a) (O, Average Decoding: It averages the predicted probabilities of the draft model and the assistant model and chooses the final output from the averaged prob- abilities. Co-LLM (Shen et al., 2024): It trains a single layer to classify the hidden state of a base model. The out- put probability of the layer decides to use the base model generation evoke assistant model to help generation.
Mp
4.1. Experimental Settings
Scenarios, Models, and Benchmarks. We evaluate CoSD and compare it against several baselines in scenarios that reflect common use cases where users may seek to fuse the knowledge of multiple LLMs. These scenarios include: (i) Complementary Knowledge Fusion: The fused LLMs have complementary knowledge, and users hope that the knowledge fusion system can perform as well as the best model for each task across all tasks; (ii) Catastrophic For- getting Recovery: The fused models are one base model and a model fine-tuned from the base model. Fine-tuning improves performance in certain domains but reduces the performance in other domains due catastrophic forget- ting. Users expect to heal the catastrophic forgetting by fusing the knowledge of the two LLMs; (iii) Capacity Im- balance:
Hy alpeepaarametes. tings. For Rule-Based CoSD, we set a = 0.5 and / = 0.5, which were determined to be the optimal and most trans- ferable parameters based on our analysis in Figure 2. For Tree-Based CoSD, we randomly select three samples from the AlpacaEval dataset to train the decision tree. It is impor- tant to note that we use MMLU, GSM8K, and HvaanEval as our benchmarks. Consequently, the training data for the decision tree do not overlap with the test data, creating a more realistic scenario to evaluate the decision tree's trans- ferability across different tasks and domains.
We run CoSD with the following set-
Users use a small draft model and adopt an API of the assistant model with a much larger capacity. The fusion system is expected to perform similarly to the as- sistant model; (iv) Different Tokenizers: Fuses the LLMs with different tokenizers. To simulate these scenarios, we carefully selected six pairs of LLMs from the HuggingFace repository (Jain, 2022), representing each of the four use cases outlined above. Table I lists the model pairs and the corresponding simulated scenarios.
4.2. Experimental Results
Fusing LLMs with Complementary Domain Knowledge. We first evaluated the performance of different methods for fusing LLMs with complementary knowledge, with results shown in the pair 1 and pair 2 columns of Table 2. Both CoSD-Rule and CoSD-Tree consistently outperformed the baseline methods in terms of averaging performance. For instance, in pair 1, CoSD-Rule and CoSD-Tree achieved averaged scores of 52.41 and 51.24 on, surpassing all the baselines and both the draft model and the assistant model. Besides, CoSD-Rule also achieves the best performance on MMLU, HumanEval, and Hellaswag. In pair 2, CoSD matches the performance of the better model for each task across all tasks. For example, CoSD achieves a similar MMLU performance to the draft model and a similar perfor- mance on GSM8K and HumanEval to the assistant model in pair 2. Compared with CoSD, Speculative Decoding only performs similarly the assistant model, thus will be more suitable to the scenario when the assistant model is much stronger than the draft model. Average Decoding fuses model knowledge but only achieves an average accuracy across tasks, unlike CoSD, which integrates the strengths of different L.MMs. Co-LLM performance is the closest to
For all the scenarios and model pairs, we use MMLU (Hendrycks et al., 2020), GSM8K (Cobbe et al., 2021), HumanEval (Chen et al., 2021), Hellaswag (Zellers al., 2019), and TruthfulQA (Lin et al., 2021) as the evaluation benchmark. We use rinyBenchmarks (Polo et al., 2024) for all the benchmarks except HumanEval to further increase the efficiency of experiments. These benchmarks test general question-answering mathematical reasoning, and coding capabilities, providing a comprehensive assessment of the models' abilities across different domains. By using these benchmarks, we can evaluate the effectiivness of CoSD and the baselines in fusing complementary knowledge across diverse tasks and model configurations.
Baselines. We use tree basclines in the experiment. Spec- ulative Decoding: It also uses a draft model and an assistant model to generate the output. However, it adopts a different verification algorithm that replaces the draft token when
5

---

Collaborative Speculative Decoding (CoSD)
Table The results of fusing L.LMs with complementary knowledge and the tokenizer. Pair I and pair 2 complementary knowledge fusion results. Pair 3 simulates a catastrophic forgetting healing scenario, and pair 4 is disparate capacity LLM fusion result.
Spec. Decoding
Avg. Decoding
Models
Benchmarks
Draft
Assist.
Co-LLM
CoSD-Rule CoSD-Tree
MMLU
65.82
59.26
59.33
62.22
60,40
65.06
63.71
GSM8K
31.20
42.19
33.36
38.33
38.85
36.81
37.24
HumanEval
28.66
31.10
14.02
25.60
29.91
31.34
28.29
Pair 1
Hellaswag
84.35
79.71
80.25
81,44
82.58
85.19
83.17
TruthfulQA
43.92
40.62
44.18
40.52
42.78
43.65
43.77
AVG.
50.79
50.58
46.23
49.62
50.90
52.41
51.24
MMLU
54.81
52.02
53.20
52.31
55.25
56.97
58.37
GSM8K
39.79
51.02
43.85
43,89
41.04
45.72
41.89
EvmanEval
21.34
43.90
39.02
38.41
37.25
39.10
36.22
Pair 2
Hellaswag
86.41
82.36
82.52
86.39
85.64
86.96
86.84
TruthfulQA
56.90
49.54
51.29
54.45
55.07
55.09
55.40
AVG.
47.85
55.77
53.98
55.09
54.85
56.77
55.74
MMLU
61.45
46.59
43.39
56.60
58.78
62.41
63.87
GSM8K
25.01
35.43
33.10
36.61
37.15
45.47
33.85
HumanEval
27.44
9.76
10,97
18.90
21.88
25.61
23.17
Pair 3
Hellaswag
82.71
73.86
80.67
75.73
79.03
82.81
81.95
TruthfulQA
35,45
48.50
36.23
41.88
44.60
44.49
46.01
AVG.
46.37
42.83
40.87
45,94
48.29
52.16
49.77
MMLU
32.13
47.65
47.30
42.62
47.47
47.84
48.15
GSM8K
3.36
15.63
14.63
12.12
11.97
12.52
12.29
HumanEval
8.53
12.20
10.39
12.55
11.73
12.80
10.54
Pair 4
Hellaswag
60.76
77.95
75.94
68.77
73.19
73.23
74.08
TruthfulQA
30.47
32.09
29.98
32.35
30.64
32.17
32.58
AVG.
27.05
37.10
35.65
33.68
35.00
35.71
35.53
Table Fusing LLMs with different tokenizers.
Models
Benchmarks
Draft
Assist.
Char-ED
CoSD-Rule
CoSD-Tree
MMLU GSM8K
47.65
40.61
44.29
50.65
52.
Pair 5
15.63
51.13
37.54
44.88
37.01
MMLU
47.65
59.63
52.51
57.33
55.20
Pair 6
HumanEval
8.53
73.17
59.04
59.88
51.42
CoSD, but since it requires training on specific datasets, its transferability across different datasets is inferior to CoSD.
models on MMLU, GSM8K, and Hellaswag. These results suggest that CoSD further enhance the origi- nal performance of both models by enabling collaboration between them.
These results highlight the effectiveness of CoSD, particu- larly the superior fusion capabilities across multiple bench- marks and model pairs. With the support of experimental results, we state that if users have multiple language models with different specialties but are unsure when to use each model, CoSD always provides better overall performance.
Fusing LLMs with disparate capacity. When the assis- tant model has a much larger capacity than draft model, the model fusion system supposed to achieve a similar performance the draft model. Speculative Decoding more suited for this task because its verification strategy tends to replace more draft tokens with assistant tokens. However, CoSD results in pair 4, Table 2 are still compara- ble to Speculative Decoding. For instance, CoSD-Rule has higher HumanEval scores than Speculative Decoding, while CoSD-Tree has higher MMLU, Hellaswag, and TruthfulQA
Catastrophic Forgetting Recovery. We select Mistral base model and fine-tuned math Mistral model for pair 3 in Table 2 to simulate the catastrophic forgetting recovery. We found that CoSe-Rule performs particularly well on this type of task. not only recovers from forgetting across all benchmarks but also outperforms both the draft and assis-
6

---

Collaborative Speculative Decoding (CoSD)
scores than Speculative Decoding. These results LLMs with disparate capacities indicate that CoSD is not only ap- plicable to complementary knowledge LLM fusion but also to efficient inference tasks. When the draft model is smaller and the assistant model is larger, our CoSD can achieve per- formance similar to the assistant model. At the same time, since the assistant model only performs parallel verification, CoSD still has more efficient inference compared to using the assistant model alone.
0
63.75
207.60
$6.12
102.86
102.49
101.93
200
382
100.37
91.43
90.00
95:93
200.65
101.41
100.66 6G
87.39
97.05
91.36
99.83
100.00
97.68
0.25
0.5
0.75
0.25
0.75
α
α
Fusing LLMs with Different Tokenizers. Although CoSD needs to decode and then encode the sequences dur- ing the verification when the models have different tokeniz- ers, which sacrifices some efficiency, it can still effectively fuse the model knowledge. In the experiments, a Llama 2 Chat and a WizardMath to evaluate the CoSD performance on MMLU and GSM8K. We fuse a Llama 2 Chat and a Deepseek Coder to evaluate CoSD on MMLU and Hvla Results are shown in Table 3. CoSD out- performs the character-wis averaging method CharED (Gu et al., 2024) in both model pairs and benchmarks. We do not include other baselines since they are not applicable to the different tokenizer settings.
Figure 2. The sum score of MMLU and GSM8K with various B settings on pair (left figure) and pair 2 (right figure).
to train the decision tree. This would result in a domain- adapted tree that better suited to the specific task. In addition, as mentioned in the table title, we use very few samples to train the decision tree, thus training decision trees introduces almost no additional computational overhead.
fuse
Case Studies. We use an example in GSM8K to demon- strate how CoSD effectively combines the knowledge of two models in Table 4. CoSD replaces the red tokens generated by the draft model with the green tokens from the assistant model. We display the answer from the single draft model and the assistant model in Table 10 in the appendix. Neither the draft model nor the assistant generates the correct result when used alone. The main issue with the draft model is its weak mathematical calculation ability (e.g.. in the fourth line, it calculates the tax as 20% of 20 to be 10, instead of the correct answer 4). On the other hand, the assistant model performs well in terms of mathematical calculations but lacks the logical rigor of the draft model. It fails com- pute the subtotal without the tip first, leading to the incorrect final calculation. CoSD effectively integrates the strengths of both models. For instance, in Rule, in the fifth line, the assistant model rejects the draft model's 's incorrect computation of 20% of 20 = 10 and instead uses the correct calculation of 20 0.2 = 4, successfully avoiding the error in the draft model's tax calculation. In the sixth line, the draft model correctly leads to generate the subtotal of $24, so in the final step, CoSD-Rule computes the simpler 24 + 5 instead of the more complicated 15 + in the correct answer.
Ablation Studies. We have several tunable hyperparam- eters in COSD. In Rule-Based CoSD, we have a and 3 that determine the rules to replace the draft tokens. In Tree- Based CoSD, the training data and hyperparameters influ- ence the performance of the decision tree. Thus, we use ablation experiments to identify the impact of these hyper- parameters on the final model performance, allowing us to determine the optimal and transferable hyperparameter set- tings. Figure 2 shows the relationship between a, B values in Rule- Based CoSD and model performance. The x-axis represents the values of a, and the y-axis represents the values of 3. The numbers in the small squares represent the sum score of MMLJU and GSM8K, which reflect the overall model performance of COSD. We can see that with = 0.5, 0.75 and 3 0.5, 0.75, Rule- Based CoSD per- form consistently well in the two model pairs. We ultimately selected a 0.5, 3- = 0.5 the general hyperparameters in our experiments. We believe this setting effectively inte- grates the knowledge of the models.
3 + 2 + 5, resulting
Table 5 displays the impact of the tree training dataset on Tree-Based CoSD. The decision tree trained on different datasets performs relatively consistently, even when the training set is not in the same distribution with any bench- mark (e.g., AlpacaEval achieves good results across all three benchmarks). When the decision tree's training set shares the same distribution as a particular benchmark, CoSD-Trea tends to perform slightly better on that benchmark. There- fore, if users are aware of the model's 's application scenario, they can use the corresponding benchmark from that task
Also, there are situations that CoSD makes wrong decisions. As shown in Table in Appendix B, CoSD does not always select the correct answer. In the above example, the draft model made the correct choice with high confidence, so the final generation retained the correct answer. However, in the example below, while the draft model also made the correct choice, the assistant model provided an incorrect answer with higher confidence, leading to the final output being changed to the wrong answer. This demonstrates that using confidence as the criterion does not guarantee selecting the
7

---

Collaborative Speculative Decoding (CoSD)
Table An example of how CoSD polish the draft generation in GSM8K dataset. The table shows the outputs for GSM8K question generated by CoSD- Rule and CoeD-Tree. Ouiput tokens that are not highlighted represent accepted draft tokens, while tokens marked in pink are rejected draft tokens, followed by the assistant tokens that replace the rejected ones highlighted green.
Model
Example
Rory orders subs for $7.50 each, 2 bags chips for S1.50 each and 2 cookies for $1.00 each for delivery. There's s a 20% delivery fee added at check out and she wants to add $5.00 tip. What will her delivery order cost? Correct answer: 29
Question
The cost of the subs 2
$7.50 <007.0==5)
The cost of the bags of chips is $2
We
The
delivery
total
finll
sub
+
before
t*x
the
t**
delivery
fee
CoSD Rule
15
15 + + =2 15++
The delivery fee 20% of S20
so the delivery fee is S20 0.20 <:00====200000300
The total eest before the tip is $20 $4 The tip is $5.00
The total of delivery is order
will
cost
$24 $5
The cost of the subs is $7.50=
The cost of the bags chips
The cost of the cookies: $1.00 =2"==2222
CoSD Tree
The cest subtotal ef before the delivery fee: 2
The 20% delivery fee: 20% of 20 = <0 % 2 2=424
The total cost of the order before the tip: 20 + 4= 220+4=24324
The total cost of the order is by adding all the tip: 24 + 2445=299229
has the lowest latency among all methods, since it makes the least token replacement. However, although CoSD meth- ods replace a few more tokens, the increase in total latency is almost negligible. Considering that CoSD has the best knowledge fusion performance, we have achieved better balance between efficiency and effectiveness.
Table 5. Training the decision tree with different datasets in pair 3. Each column represents decision tree trained by the dataset in the column header. We use 10 samples of MMLU, 3 samples of each other datasets to train the decision tree.
Benchmarks
MMLU
GSMSK
HumanEy.
AlpacaEv.
MMLU
63.94
60.88
61.23
63.87
GSM8K
35.04
37.17
30.08
33.85
HumanEy. 25.62
23.04
23.09
23.17
5. Conclusion
In this paper, we fuse the LLM knowledge in a simple yet effective way. Our algorithm CoSD takes the probabilities of predicted tokens from two LLMs as the feature ver- ify whether to keep the draft token adopt the assistant token. The verification strategy can be either rule- rule-based or pre-trained decision tree. Our extensive experiments show that CoSD performs better than the state-of-the-ar methods across LLM pairs and 5 benchmarks. Compared to previ- ous works, CoSD has superior knowledge fusion ability, a broader range of application scenarios, and comparable effi- ciency. It works well in scenarios including complementary knowledge fusion, catastrophic forgetting recovery, knowl- edge fusion with disparate model capacity, and knowledge fusion with different tokenizers. CoSD makes it possible for ordinary users to fuse the L.M knowledge with only the API queries, without any training or fine-tuning of LLMs, or requirements of white-box LLM information such as hid- den states. It provides users with better tools manipulate LLMs in wider application scenarios.
Table Efficiency of LLM Knowledge Fusion. Token latency represents the averaged time to generate one token, and acceptance rate refers the proportion of draft tokens that not replaced. Typically, the higher the latter, the lower the former. as fewer tokens require regeneration. Experiments are done by pair 3.
Token
Acceptance Rate
Methods
Latency (ms)
Spec. Decoding
131.22 132.31
0.89
CoSD-Rule
0.81
CoSD-Tree 135.82
0.77
correct option but only aim to choose the correct answer with higher probability.
Efficiency. Since we perform fusion during the inference, efficiency is a major advantage of our approach. We com- pared the time overhead of CoSD with the baselines. We use token latency and acceptance rate as the metrics for efficiency. As displayed in Table 6, Speculative Decoding
8

---

Collaborative Speculative Decoding (CoSD)
Impact Statement
Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L.., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., et al. Layer skip: Enabling early exit inference and spefuupecu decoding. arXiv preprint ar arXiv:2404.1677 2024.
This paper presents work whose goal is to improve the test-time knowledge fusion algorithm for language models. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Goddard, C., Siriwardhana, S., Elghaghi, M., Meyers, L.,, Karpukhin, V., Benedict, B., McQuade, M., and Solawetz, J. Arcee's s mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403. 13257, 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I,, Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Gu, K., Tuecke, E., Katz, D., Horesh, R., Alvarez-Melis, D., and Yurochkin, M. Chared: Character- wise ensem- ble decoding for large language models. arXiv preprint arXiv:2407. 1009, 2024.
Ainsworth, K., Hayase, J., and Srinivasa, S. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.0448 0.006, 2022.
Guo, D.. Zhu, Q. Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deepseek-coder: When the large language model meets programming the rise of code intelligence. arXiv preprint arXiv:2401.14199 2024.
Amru, M., Kanman, R. J., Ganesh, F. N., Muthumarilak- shmi, S., Padmanaban, K., Jeyapriya, J., and Murugan, S. Network intrusion detection system by applying en- semble model for smart home. International Journal of Electrical & Computer Engineering (2088-8708), 14(3), 2024.
He, Z., Zhong, Z., Cai, T., Lee, J. D., and He, D. Rest: Retrieval-based speculative decoding. arXiv preprint arXiv:2311 .08252, 2023.
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.08 003300, 2020.
Ito, A., Yamada, M., and Kumagai, A. Analysis of linear mode connectivity via cermutation-based weight match- ing. arXiv preprint arXiv:2402.0405 2024.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P.. Kaplan, J.. Edwards, H., Burda, Y.. Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder. N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J. Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., MeGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code, 2021.
Jain, S. M. Hugging face. In Introduction to transformers for NLP: With the hugging face library and models to solve problems, pp. 51-67. Springer, 2022.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.., et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
Kim, D., Park, C., Kim, S., Lee, W., Song. W., Kim. Y., Kim, H., Kim, Y., Lee, H., Kim, J., et al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.
2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, I.., Plappert, M., Tworek, J., Hilton, J., Nakano, R., al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1416 2021.
Leviathan, Y., Kalman, M., and Matias, Y. Fast in- ference from transformers via speculative decoding. International Conference on Machine Learning, pp. 19274-19286 PMLR, 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama herd of models. arXiv preprint MxXi:2240..21783, 2024.
Li, M., Chen, X., Holtzman, A., Chen, B., Lin, J., Yih, W.-t,, and Lin, X. V. Nearest neighbor speculative de- coding for 1lm generation and attribution. arXiv preprint arXiv:2405. 10.4405.1922 19325, 2024.

---

Collaborative Speculative Decoding (CoSD)
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint 10w.21999.7958 2021.
Wang, H., Polo, F. M., Sun, Y., Kundu, S., Xing, E., and Yurochkin, M. Fusing models with complementary ex- pertise. arXiv preprint
2023.
Luo. H., Sun, Q.. Xu, C., Zhao, P., Lou, J., Tao, C.,, Geng, X., Lin, Q. Chen, S., and Zhang, D. Wizard- math: Empowering mathematical reasoning for large lan- guage models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583 2023.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roclofs, R., Gontijo-Lopss, R., Morcos, A. S., Namkoong. H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965- 23998. PMLR, 2022.
Mazraeh, A., Bagherifar, M., Shahanlou, S., and Ekhlas- mand. R. A novel committee-based framework for modeling groundwater level fluctuations: A combina- tion of mathematical and machine learning models us- ing the weighted multi-model ensemble mean algorithm. Groundwater for Sustainable Development, 24: 24:1010 2024.
Wu, C., Gan, Y., Ge, Y., Lu, Z., Wang. J., Feng, Y., Luo, P., and Shan, Y. Llama pro: Progressive Ilama with block expansion. arXiv preprint arXiv:2401.02413 2024.
Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., and Sui, Z. Speculative decoding: Exploiting speculative execu- tion for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3909-3925, 2023.
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang. Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., et al. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arXiv preprint XXXXx:2333509788, 2023.
Xia, H., Yang, Z., Dong, Q. Wang, Li, Y., Ge. T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.077 007551, 2024.
Ong. I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gon- zalez, E., Kadous, M. W., and Stoica, I. Routellm: Learning route Ilms with preference data. arXiv preprint arXiv:2406.18666 2024.
Xue, F., Zheng, Z., Fu, Y., Ni, J., Zheng, Z., Zhou, W., and You, Y. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739, 2024.
Polo, F. M., Weber, I.,, Choshen, I.,, Sun, Y., Xu, G., and Yurochkin, M. tinybenchmarks: evaluating Ilms with fewer examples. arXiv preprint arXiv:2402. 14992. 2024.
Yang, S., Lee, G., Cho. J., Papailiopoolos, D., and Lee, K. Predictive pipelined decoding: A compute- latency trade-off for exact Ilm decoding. arXiv preprint NVV:2307.05903 2023.
Poonia, R. C. and Al-Alshaikh, H. A. Ensemble approach of transfer learning and vision transformer leveraging explainable ai for disease diagnosis: An advancement towards smart healthcare 5.0. Computers in Biology and Medicine, 179:108874, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? 2019.
Shen, S. Z., Lang, H., Wang, B., Kim, Y., and Sontag, D. Learning to decode collaboratively with multiple lan- guage models. arXiv preprint arXiv:2403.03870, 2024.
ar preprint
Zhang. J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. Draft & verify: Lossless large language model acceleration via seff-specul decoding. arXiv preprint arXiv:2309.08163 2023.
Stern, M., Shazeer, N., and Uszkoreit, J. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018.
Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model. arXiv preprint EXXi:2401.022883 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0m 220.02230..09288 009288, 2023.
Zhou, Y., Lyu, K., Rawat, A. S., Menon, A. K., Ros- tamizadeh, A., Kumar, S., Kagy, J.-F., and Agarwal, R. Distillspec: Improving speculative decoding via knowl- edge distillation. ar preprint wxiv:23110004444 2023.
Wan, F., Huang, X., Cai, D., Quan, X., Bi, W., and Shi, S. Knowledge fusion of large language models. arXiv preprint arXiv:2401.1049) 2024.
Zhu, T. Qu, X., Dong, D., Ruan, J., Tong, J., He, C., and Cheng, Y. Llama-moe: Building mixture-of-experts from 1lama with continual pre-training. arXiv preprint arXiv:2406. 16554, 2024.
Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. Federated learning with matched averaging. arXiv preprint 200::20002.00400000 2020.
10

---

Collaborative Speculative Decoding (CoSD)
A. Limitations
While CoSD demonstrates strong performance across various scenarios, is important to acknowledge its limitations. This section highlights cases where CoSD may not be applicable and tasks that it fails address, Identifying these constraints provides clarity on its scope of use and helps guide future improvements. Below, we outline two specific limitations:
(1) When the two collaborating models are of similar size and one significantly outperforms the other, CoSD offers no advantage over using only the better model. In this case, using the better model only is sufficient. This also requires the user to have prior knowledge of the performance of the two models on different benchmarks and to determine that one model is significantly better than the other. If the user is uncertain, we still recommend using CoSD to ensure the best results.
(2) Another limitation of CoSD is that cannot guarantee the replaced assistant token is always better than the discarded draft one. It relies on the confidence scores of the models, which are not always perfectly aligned with token quality. The algorithm selects the output of the more confident model, aiming to maximize the likelihood of choosing better token, but this approach may occasionally lead to suboptimal results.
B. Additional Experiments and Discussion
The Average Iterations CoSD. The number of iterations required during collaborative decoding depends on the maximum length of the model output. Table reports the average number of iterations on the GSM8K dataset for different maximum lengths.
Max Length
CoSD-Rule
CoSD-Tree
Spec. Dec.
128
11.41
13.58
9.77
256
15.29
16.01
14.20
512
21.23
21.95
18.51
Table 7. Average number of iterations for different maximum output lengths.
Although the number of iterations scales with the output length, it does not directly imply proportional increase in generation time. As the number of accepted tokens grows, the number of tokens requiring regeneration decreases significantly. For instance, with a maximum output length of 128, the average number of iterations is 11, but the total generated output length remains around 300 tokens. This highlights the efficiency of our approach in reducing redundant generation.
Collaborate with More LLMs. Our CoSD also supports multiple collaborating models. Table 8 presents the results when three models are used for collaboration:
Dataset
Draft
Assist. 1
Assist. 2
CoSD-Rule
CoSD-Tree
MMLU
32.13
47.65
35.62
44.14
46.48
GSMSK
3.36
15.63
8.33
15.85
14.02
Table 8. Performance of three collaborator I.LMs.
In this setup, the draft model is TinyLlama, while the assistant models are Llama 2 Chat 7b and Llama-7b. Our findings demonstrate that involving additional models improves prediction accuracy. Table 8 demonstrates that when three models collaborate if one significantly outperforms the other two, the final system will achieve performance close to that of the best model. This indicates that our algorithm is effective when applied to more than two models. With sufficient LLMs, we can also better utilize training data, even when certain samples are excluded.
The Case Study of MMLU. While CoSD is effective in many cases, there are instances where it makes incorrect decisions, highlighting its limitations. As shown in Table 9, CoSD does not always select the correct answer when the draft model and the assistant model disagree. In the first example, the draft model correctly identified the answer with high confidence, which allowed the final output to retain the accurate result. This showcases the potential of CoSD to preserve correct answers when confidence aligns with accuracy.
However, in the second example, the draft model once again made the correct prediction, but the assistant model, despite being incorrect, provided an answer with higher confidence. Consequently, the final output was altered to the wrong answer,

---

Collaborative Speculative Decoding (CoSD)
Example
The Yang-shao culture gave way the Lung-Shan sometime after:
Question
A. 6,000 B.P. B. 5,000 B.P. C. 4,000 B.P. D. 3,000 B.P.
Correct answer: B
Draft
Assist.
CoSD-Rule
CoSD-Tree
Answer
(wrong)
Rowena can paint a room in 14 hours, while Ruby can paint it in 6 hours. If Rowena paints for hours and Ruby paints for hours, they will finish half of the painting, while Rowena paints for hours and Ruby paints for x hours they will paint the whole room. Find the ordered pair (zz y). 21
Question
A. [11 TO B.
20 20
C. 2311
40
D. (1,1)
Correct answer: C
Draft
Assist.
CoSD-Rule D (wrong)
CoSD-Tree (wrong)
Answer
D (wrong)
Table 9. Two examples of how CoSD modify the generation in MMLU dataset. The example above demonstrates how CoSD helps improve generation quality, while the example below shows instances where CoSD sometimes selects incorrect answers.
overriding the draft model's correct prediction. This illustrates a shortcoming of the CoSD approach: relying solely on confidence scores as the decision maaiin criterion does not guarantee correctness. Confidence may reflect certainty but not necessarily accuracy, leading to situations where errors from the assistant model dominate the final outcome.
This limitation suggests that while CoSD can improve generation quality by prioritizing higher-confidence predictions, it does with the assumption that confidence correlates with correctness. In practice, this assumption does not always hold, especially when the assistant model is overconfident in its incorrect predictions. To address this, future improvements could explore additional heuristics or cross- cross-validation mechanisms to better balance confidence with accuracy, ensuring that correct answers are more consistently selected.
Model
Example
Rory orders 2 subs for $7.50 each, 2 bags of chips for $1.50 each and 2 cookies for $1.00 each for delivery. There's a 20% delivery fee added check out and she wants to add a $5.00 tip. What will her delivery order cost? Correct answer: 29
Question
The cost of the subs is $7.50 = 25
The cost the chips is * $1.50 2
add the total cost the food items by $15 + + $2 $
Draft Model
The cost of the delivery fee 20% of the total cost of the order,
20% of 20 <220% of 20=10>>>1
The total cost of the food items and delivery fee is $20 $10 $0=
The cost of the tip is $5.00
The total cost of the order is 15 10+ (50
The cost of the subs is x $7.5=
The cost of the chips is 2 x $1.5 $1.5=
Assist. Model
The total cost of her order without any fees is $15 + $3 +
The delivery fee is 20% of the total order, so $20 0.20 = 5=0===2000000--
The tip is an additional $5
Therefore, her delivery order will cost $15 + $3 + + $4 + $4=
Table 10. The answers from the single draft model and assistant model with the question in Table 4.
12


Reviews:
[{"review_id": "IvPX8Rh0RW", "reviewer": "Reviewer_cV7E", "summary": "The paper introduces an algorithm called Collaborative Speculative Decoding (CoSD) that is designed to efficiently fuse the knowledge of multiple Large Language Models (LLMs) together at inference time, without requiring any additional model training. The key idea is to leverage the same inference paradigm followed in standard speculative decoding, except that a general decision rule (instead of the typical rejection sampling) is used to reconcile  differences in token predictions (and as such proceed efficiently in the cases where there is no discrepancy). Experimental results support the effectiveness of this model-merging approach.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's recognition of the strengths of our work. Regarding the weaknesses and questions raised, we address all concerns in detail below.\n\n> However, I was surprised that no evaluation of speed/efficiency was done, as this seems to be a critical claimed strength over other work that does not take advantage of blockwise parallel decoding.\n\nWe would like to clarify that our method has indeed been evaluated in terms of speed and efficiency. Specifically, **Table 6** in our paper reports latency results, showing that our method achieves nearly the same runtime performance as standard speculative decoding. This confirms that CoSD maintains comparable decoding efficiency. However, we agree that additional efficiency experiments would further highlight our contributions, and we will include them in the revised version of the paper.\n\n> Both the \"drafter\" and the \"assistant\" model are not necessarily computationally that much different, so decoding from each model in parallel (vs. one first, and then the other blockwise parallel) might not have as much benefit?\n\nThis is an excellent point — In our case, the drafter and assistant models have similar capacities, so such gains might not be expected. However, **our task is fundamentally different**. While speculative decoding aims to approximate the behavior of a large model efficiently, **CoSD is designed to combine complementary knowledge from two models**, rather than mimicking one with the other. Therefore, the appropriate baseline is not a single model, but a **naive two-model decoding** setup where both models generate token-by-token and decide on the final output via a selection mechanism (e.g., our baseline *Avg Decoding*).\n\nFor clarity, we provide the comparison below:\n\n| Method    | CoSD-Rule | CoSD-Tree | Avg Decoding | Co-LLM |\n|------|-----|-----|-----|-----|\n| Token-Wise Latency  | 132.31  | 135.82  | 212.73  | 254.16 |\n\nNote that *Avg Decoding* could also be implemented using a speculative-style process, but we adopt standard two-model decoding (i.e., two LLMs generating autoregressively) to emphasize the efficiency gain introduced by speculative decoding in CoSD. In this setting, CoSD is at least **1.5× faster** than baselines that do not use speculative decoding. We will include these results in Table 6 of the revised paper.\n\n> One weakness which sticks out to me is that this method is unlikely to scale well to multiple collaborating LLMs? \n\nWe agree that collaborative generation involving more models may reduce the acceptance rate. However, our experiments suggest that the drop is not as drastic as one might expect. Our key observation is that for many common, non-domain-specific token sequences (e.g., “I am”, “This is”), well-trained models tend to produce highly consistent predictions. As shown below, the acceptance rate remains relatively high even as more models are added:\n\n| # of LLMs | Acceptance Rate (%) |\n|----|----|\n| 2   | 81  |\n| 3   | 79  |\n| 4   | 76  |\n| 5   | 77  |\n\nThe experiment settings follow Table 6.\n\nAdditionally, we can also use a lower number of generated token per step (i.e., lower **K** in Algorithm 1) to drop less tokens, thus ensuring the efficiency when we have a lower acceptance rate. We will clarify and expand upon this discussion in the revised version of the paper.\n\n>From just looking at Fig. 1 it is unclear what the main differences are between CoSD and standard speculative decoding.\n\nWe agree with the reviewer that Fig 1 focuses more on the speculative decoding part than the knowledge fusion part. We will modify the figure in the revised paper and emphasize the knowledge fusion ability of the “CoSD Verification” part.\n\n>L138: \"generated autoregressively and produced sequentially\" is redundant.\n\nThanks for pointing out the redundant part. We will modify it in the revised paper.\n\n>The results should have error bars... Speculative Decoding should have the same average performance as the assistant model, but exhibits a substantial amount of deviation in the table.\n\nWe will add the error bars in the revised paper. The speculative decoding algorithm we use (Miao et al., 2023) has a soft verification strategy, that uses a random number as a threshold to decide whether to accept the draft token for efficiency. In this case, the average performance of speculative decoding will not be exactly the same as the assistant model.\n\n>The assistant model is always \"invoked\" question.\n\nYou're absolutely right that in our current method, the assistant model is always invoked via the speculative decoding. What we intended to convey in the abstract is that the token replacement is invoked when the assistant token differs from the draft model's token and pass the decision rule.\nWe agree that the original wording may misleadingly suggest that the assistant model is not used at all unless triggered by a separate process. We will revise this in the final version for clarity, to better reflect the actual mechanism.", "claims_and_evidence": "The claims are supported. For the most part, the results of CoSD-Tree are somewhat disappointing relative to the naive CoSD-rule approach. However, the results of CoSD-rule are encouraging relative to the evaluated baselines.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria make sense. However, I was surprised that no evaluation of speed/efficiency was done, as this seems to be a critical claimed strength over other work that does not take advantage of blockwise parallel decoding. Furthermore, this *should* also be a claim that should be empirically supported, since the typical assumption in speculative decoding is that the drafter model is much more efficient to run sequentially than the verifier model. In this setting, however, both the \"drafter\" and the \"assistant\" model are not necessarily computationally that much different, so decoding from each model in parallel (vs. one first, and then the other blockwise parallel) might not have as much benefit?", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The experiments are sound.", "relation_to_broader_scientific_literature": "The paper introduces a simple method for merging token predictions from different models, that is made efficient by leveraging the same blockwise parallel decoding strategy of speculative decoding. The approach makes sense and is well presented, albeit with fairly limited novelty.", "essential_references_not_discussed": "The related work is adequately discussed.", "other_strengths_and_weaknesses": "One weakness which sticks out to me is that this method is unlikely to scale well to multiple collaborating LLMs? With multiple models, the likelihood that a discrepancy arises within just one or a few steps will quickly grow, and speculative decoding will cease to continue to be an efficient strategy (e.g., vs a strategy such as the on in Shen et. al., 2024).", "other_comments_or_suggestions": "- From just looking at Fig. 1 it is unclear what the main differences are between CoSD and standard speculative decoding, apart from generalizing the acceptance / rejection mechanism for draft tokens to incorporate an arbitrary rule or decision tree. I think what you want to emphasize more clearly throughout the paper is that the primary goal is to be able to efficiently merge the predictions of two LLMs that are on equal footing (i.e., vs standard speculative decoding where the verifier is considered to be the better, but slower model). Rather, if I understand correctly, here the speculative decoding aspect is mainly used as a trick to speed up inference and merging of the two models.\n- L138: \"generated autoregressively and produced sequentially\" is redundant.\n- The results should have error bars to reflect the effects of randomness in the generations. For example, in theory, Speculative Decoding should have the same average performance as the assistant model, but exhibits a substantial amount of deviation in the table.", "questions_for_authors": "The abstract says \"CoSD employs a draft model to generate initial sequences and an easy-to-learn rule or decision tree to decide when to invoke an assistant model to improve these drafts.\" However, from the method description it seems that the assistant model is always \"invoked\", but just via the speculative decoding procedure. So I am a bit unsure what this is referring to.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CO6GxqkihP", "reviewer": "Reviewer_VZYJ", "summary": "This paper addresses the challenge of language model knowledge fusion, aiming to effectively integrate complementary knowledge from multiple LLMs while maintaining efficiency. The authors propose CoSD, a method that classifies output tokens based on their probabilities to achieve fusion and leverages a speculative decoding framework to enhance generation speed. Experimental results demonstrate that CoSD successfully merges knowledge from two or three LLMs while maintaining flexibility, eliminating the need for explicit model selection. The approach is straightforward and empirically effective, offering a practical solution for multi-LLM integration. A deeper analysis of its impact on generation quality and potential trade-offs in diverse scenarios would further strengthen the contribution.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's recognition of the strengths of our work. Regarding the weaknesses and questions raised, we address all concerns in detail below.\n\n>What will happen if we swap the draft model and the assistant model? Since the authors claim that users don’t need to choose between LLMs, they may also not want to determine which LLM to be the draft/assistant model. Therefore, the question is will the performance drop significantly if we swap the two models? What will happen if we swap the two LLMs?\n\nWe performed experiments on swapping the draft model and the assistant model. Here are the results of pair 2:\n\n| Benchmarks | Draft | Assist. | Spc. Dec. | CoLLM | CoSD-R | CoSD-T |\n|------------|-------|---------|-----------|--------|--------|--------|\n| MMLU       | 52.02 | 54.81   | 54.23     | 53.92  | 54.17  | 56.18  |\n| GSM8K      | 51.02 | 39.79   | 41.28     | 45.75  | 49.52  | 48.88  |\n| HumanEval  | 43.90 | 21.34   | 25.17     | 36.90  | 42.31  | 43.62  |\n| **Avg**    | 48.98 | 38.65   | 40.23     | 45.52  | 48.67  | **49.56** |\n\nThe results show that CoSD still outperforms all the baselines. We will add these results to the revised paper.\n\n>I’m also very curious about the real samples of the HumanEval dataset. Since the authors already showed the real samples of MMLU (QA) and GSM8K (Math), what the sample will look like in code generation will be interesting. Will the assistant model help repair bugs in the draft generations? A table similar to Table 4 can be helpful.\n\nThanks for the valuable suggestion. We will add a HumanEval real sample to the revised paper. Our sample shows that the assistant model can modify the draft code to another style (e.g., with more function), which sometimes improves the accuracy and reduces the number of bugs.\n\n>The knowledge fusing and catastrophic forgetting healing seem similar, might need some explanations.\n\nThe knowledge fusing task aims to fuse the knowledge of two models with complementary knowledge. The catastrophic forgetting task often fuses the knowledge of one LLM that is only good at one specific task (through fine-tuning) and another model good at all other tasks.\n\n>The hyperparameters seem to be determined by Figure 2. Do other benchmarks follow the same pattern? Need to be explained.\n\nYes, all the experiments of CoSD-Rule follow the same hyperparameters. We found that the optimal $\\alpha$ and $\\beta$ values are transferable between models and tasks.\n\n>Comments and Suggestions\n\nThanks for the suggestions of some details in the paper. We will further polish the paper according to the suggestions.\n\n>If the two models are in the same size, can CoSD still speed up the inference?\n\nYes, although speculative decoding cannot speed up inference with two models with the same size, our CoSD is used for a different task. CoSD is designed to combine complementary knowledge from two models rather than mimicking one with the other. Therefore, the appropriate baseline is not a single model but a naive two-model decoding setup where both models generate token-by-token and decide on the final output via a selection mechanism (e.g., our baseline Avg Decoding). In this case, the token latency will be more than 1.5x times higher than CoSD:\n\n| Method         | CoSD-Rule | CoSD-Tree | Avg Decoding | Co-LLM |\n|----------------|-----------|-----------|---------------|--------|\n| Token-Wise Latency | 132.31    | 135.82    | 212.73        | 254.16 |\n\nTherefore, CoSD still benefit from the speculative decoding algorithm in this task.", "claims_and_evidence": "The claims about the paper’s main contribution include the knowledge fusing performance and the effficiency. The paper provides algorithm and evaluations for both points. The evidences are clear to support the claims.", "methods_and_evaluation_criteria": "This paper evaluate CoSD on 6 LLM pairs and 5 common benchmarks. The experiments use different scenarios such as complementary knowledge fusion and catastrophic forgetting healing. Experiment results delivered the basic ideas. However, the following experiments are not included:\n\n1)What will happen if we swap the draft model and the assistant model? Since the authors claim that users don’t need to choose between LLMs, they may also not want to determine which LLM to be the draft/assistant model. Therefore, the question is will the performance drop significantly if we swap the two models?\n\n2)I’m also very curious about the real samples of the HumanEval dataset. Since the authors already showed the real samples of MMLU (QA) and GSM8K (Math), what will the sample look like in code generation will be interesting. Will the assistant model help repair bugs in the draft generations? A table similar to Table 4 can be helpful.", "theoretical_claims": "The paper have no strict theoretical proofs. I have checked the algorithms and the equations and there is no significant flaws.", "experimental_designs_or_analyses": "I have checked the following experimental parts\n\n(1)The scenarios defined by the paper, including knowledge fusing, catastrophic forgetting healing, capacity imbalance, and different tokenizers. The settings are overall valid, but the knowledge fusing and catastrophic forgetting healing seem similar, might need some explanations.\n\n(2)The picked baselines are valid. The difference between CoSD and baselines is clear.\n\n(3)The hyperparameters seem to be determined by Figure 2. Do other benchmarks follow the same pattern? Need to be explained.", "relation_to_broader_scientific_literature": "The key contributions are related to the LLM knowledge fusion and the speculative decoding. LLM knowledge fusion aims to fusing the knowledge of different LLMs with complementary knowledge. Speculative decoding focus on speeding up the inference with collaboration between large annd small LMs. CoSD in this paper seems to combine the advantage of the two fileds by specifically designed algorithm.", "essential_references_not_discussed": "Some new papers in this field:\n\n[1]Wan, F., Zhong, L., Yang, Z., Chen, R., & Quan, X. (2024). Fusechat: Knowledge fusion of chat models. arXiv preprint arXiv:2408.07990.\n\n[2]Liu, L., Zhang, D., Li, S., Zhou, G., & Cambria, E. (2024, October). Two Heads are Better than One: Zero-shot Cognitive Reasoning via Multi-LLM Knowledge Fusion. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (pp. 1462-1472).", "other_strengths_and_weaknesses": "N/A.", "other_comments_or_suggestions": "If you have any other comments or suggestions (e.g., a list of typos), please write them here.\n\nThe algorithm part might be too big (Can it be put in one column?).\n\nThe ‘texts’ in Eq. (3) should be inside a bracket. E.g., [x.....x].", "questions_for_authors": "If you have any important questions for the authors, please carefully formulate them here. Please reserve your questions for cases where the response would likely change your evaluation of the paper, clarify a point in the paper that you found confusing, or address a critical limitation you identified. Please number your questions so authors can easily refer to them in the response, and explain how possible responses would change your evaluation of the paper.\n\n1)What will happen if we swap the two LLMs?\n\n2)If the two models are in the same size, can CoSD still speed up the inference?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MYA08U6mT7", "reviewer": "Reviewer_sKWh", "summary": "This paper introduces Collaborative Speculative Decoding (CoSD), a new inference-time algorithm designed to fuse complementary knowledge from multiple LLMs without additional model training or fine-tuning. CoSD leverages a draft model to autoregressively generate initial tokens, which an assistant model then verifies in parallel. A simple, interpretable rule-based or decision-tree-based strategy then determines whether draft tokens should be retained or replaced by tokens proposed by the assistant model, thus effectively integrating diverse knowledge sources.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s recognition of the strengths of our work. Regarding the weaknesses and questions raised, we address all concerns in detail below.\n\n> The number of samples for the decision tree. The hyperparameters of the decision tree.\n\nWe would like to clarify that using 3 dataset samples (e.g., 3 samples from GSM8K) does not mean the decision tree is trained on only 3 tokens. Each GSM8K sample typically contains more than 50 tokens and can have up to hundreds of tokens. Therefore, 3 samples from GSM8K can provide over 150 training points for the decision tree, which is sufficient given the data sample size (1×2). The sensitivity of CoSD-Tree to the training dataset is reported in Table 5. The relationship between the number of training samples and CoSD-Tree performance is shown below:\n\n| # of Samples | MMLU   | GSM8K  | HumanEval |\n|-----------------------|--------|--------|-----------|\n| 1                     | 62.68  | 35.84  | 22.19     |\n| 3                     | 60.88  | 37.17  | 23.04     |\n| 5                     | 62.34  | 37.62  | 20.94     |\n| 10                    | 61.01  | 36.46  | 20.04     |\n\nWe also conducted experiments varying the max depth of the decision tree:\n\n| Max Depth | MMLU   | GSM8K  | HumanEval |\n|-----------|--------|--------|-----------|\n| 3         | 61.56  | 34.72  | 21.61     |\n| 5         | 63.00  | 36.52  | 22.34     |\n| 10        | 60.88  | 37.17  | 23.04     |\n| 20        | 60.88  | 37.17  | 23.04     |\n\n(10 and 20 get the same decision tree)\n\nThese results show that CoSD-Tree is not sensitive to the number of training samples or the tree’s hyperparameters. However, it is more sensitive to the type of dataset, as illustrated in Table 5.\n\n> Statistical significance testing\n\nWe will report mean ± standard deviation over 5 runs in our experiments in the revised paper.\n\n> It would be valuable to include a detailed error analysis to identify exactly when and why the decision rules or decision tree sometimes select incorrect tokens.\n\nWe include illustrative examples in Table 4 and Table 9. For MMLU samples, where a single token determines whether an answer is correct, the decision rule or tree may choose incorrect tokens when the model assigns higher confidence to an incorrect answer. This typically occurs due to hallucination by the assistant model.\n\nFor benchmarks requiring chain-of-thought outputs (e.g., GSM8K in Table 4), the replaced tokens are often not directly tied to the final answer (e.g., replacing “we” with “the” in CoSD-Rule). Thus, it can be challenging to pinpoint exactly when and why the decision rule/tree fails. A more informative analysis is to compare CoSD outputs with the outputs of the individual draft and assistant models.\n\nAs shown in Table 4, in the fifth line, the assistant model corrects a draft model’s incorrect calculation of 20% of 20 = 10 by replacing it with the correct calculation 20 * 0.2 = 4, avoiding an error in tax computation. In the sixth line, the draft model correctly computes the subtotal $24, leading CoSD-Rule to choose the simpler 24 + 5 over the more verbose 15 + 3 + 2 + 5, producing the correct final answer.\n\nFor the tinyMMLU dataset, our experiments show that the wrong replacement rate is around 2%–3% across different model pairs. For Capacity Imbalance tasks (i.e., pair 4), this rate is 0%.\n\n> Have you considered incorporating more advanced uncertainty estimation methods or ensemble learning approaches (such as Bayesian ensemble methods)? Would this significantly alter CoSD’s complexity or performance?\n\nThank you for the thoughtful suggestion.\n\nWe agree that advanced uncertainty estimation techniques and ensemble learning methods, such as Bayesian ensemble models, are valuable and may be worth exploring in future work. However, in this paper, our primary goal is to develop a simple, lightweight, and efficient collaborative decoding framework. To this end, we intentionally adopt minimal decision strategies — a rule-based filter and a shallow decision tree — which are fast, interpretable, and require minimal supervision.\n\nWhile more advanced methods like Bayesian ensembles may provide stronger theoretical guarantees, we believe they are unlikely to yield significant performance improvements over CoSD in practice, especially considering the strong results we already observe across benchmarks. Moreover, such methods typically involve additional complexity, including performing posterior inference, and evaluating model-specific performance. These extra steps would compromise one of CoSD’s key advantages — being plug-and-play, without requiring per-model evaluation on multiple benchmarks.\n\nWe will clarify this design choice and trade-off more explicitly in the revised version. We sincerely thank the reviewer for bringing up this important point.", "claims_and_evidence": "NA", "methods_and_evaluation_criteria": "NA", "theoretical_claims": "This paper does not provide theoretical justification or analysis to support the claim that training the decision tree with very limited data (eg: only three samples) is sufficient for effective generalization across different tasks or domain. Additionally, while the proposed rule-based mechanism is intuitive, this paper does not include adequate theoretical or statistical analyses examining how well model confidence aligns with correctness in varying scenarios.", "experimental_designs_or_analyses": "The selection of diverse models (e.g., Mistral, Llama, TinyLlama, WizardMath, DeepSeek Coder) effectively demonstrates the generalizability and practical applicability of the proposed method. The authors also used several popular benchmarks such as MMLU, GSM8K, HumanEval, Hellaswag, and TruthfulQA, ensuring comprehensive evaluation.", "relation_to_broader_scientific_literature": "Extend the speculative decoding research.", "essential_references_not_discussed": "I'm not familiar with related works in this field.", "other_strengths_and_weaknesses": "**Strengths**\n\n1.  This paper clearly defines scenarios where its method performs well and where it may struggle.\n2.  Transparent and interpretable methods (rule-based and tree-based verification) enhance the trustworthiness in practice.\n\n**Weaknesses**\n\n1. The training set for the decision tree method is extremely limited (e.g., only 3 samples in some scenarios), raising concerns about robustness and generalization.\n2. This paper lacks in-depth statistical significance testing.", "other_comments_or_suggestions": "1. It would be valuable to include a detailed error analysis to identify exactly when and why the decision rules or decision tree sometimes select incorrect tokens.\n2. It would be better to analyze the complexity or depth of the decision tree to achieve optimal performance.", "questions_for_authors": "1. Can you clarify why you chose only three samples for training the decision tree? How sensitive is CoSD's performance to increasing the number or diversity of training samples?\n2. Have you considered incorporating more advanced uncertainty estimation methods or ensemble learning approaches (such as Bayesian ensemble methods)? Would this significantly alter CoSD's complexity or performance?\n3. Could you provide additional details or analyses on cases where the confidence-based verification fails? What percentage of incorrect token replacements are caused by the assistant model's overconfidence?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uErzFhvS8m", "reviewer": "Reviewer_fAfk", "summary": "This paper tackles the problem of knowledge fusion across language models while aiming to balance effectiveness and efficiency. The paper proposes CoSD that integrates speculative decoding to accelerate generation and employs probability-based classification for token selection. CoSD provides a flexible and adaptive approach to multi-LLM fusion. The empirical results demonstrate its ability to effectively merge knowledge from multiple models. Comparing to baselines, CoSD has advantages on efficiency and fusion performance.\n\n## update after rebuttal\nThe authors address most of my concerns. So I keep my positive score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s recognition of the strengths of our work. Regarding the weaknesses and questions raised, we address all concerns in detail below.\n\n> However, some questions remain in the hyperparameter part. What is the hyperparameter for the decision tree? Do we need to adjust the parameters during the inference stage, and how can we do this?\n\nWe perform the decision tree training with max depth=10. The results of using different max depths can be found here:\n\n| Max Depth | MMLU   | GSM8K  | HumanEval |\n|-----------|--------|--------|-----------|\n| 3         | 61.56  | 34.72  | 21.61     |\n| 5         | 63.00  | 36.52  | 22.34     |\n| 10        | 60.88  | 37.17  | 23.04     |\n| 20        | 60.88  | 37.17  | 23.04     |\n\nSince the performance is mainly influenced by the training dataset rather than the hyperparameters of the decision tree, so it is not necessary to repeatedly tune the hyperparameters — setting a reasonable value is sufficient.\n\n>Essential References Not Discussed\n\nWe will add the references to the revised paper. Thanks for pointing out valuable papers to cite and discuss.\n\n>I suggest to show more interesting samples like Table 4 in the appendix.\n\nWe will add more samples, especially for the coding samples (i.e., HumanEval test samples) to the appendix and further discuss them in the revised paper.\n\n>What is the token latency for only use one LLM in Table 6?\n\nThe token latency of using only one LLM will be around 30-50 ms in Table 6. However, since CoSD is designed to combine complementary knowledge from two models, rather than mimicking one with the other. Therefore, the appropriate baseline is not a single model, but a naive two-model decoding setup where both models generate token-by-token and decide on the final output via a selection mechanism (e.g., our baseline Avg Decoding). In this case, the token latency will be more than 1.5x times higher than CoSD:\n\n| Method         | CoSD-Rule | CoSD-Tree | Avg Decoding | Co-LLM |\n|----------------|-----------|-----------|---------------|--------|\n| Token-Wise Latency | 132.31    | 135.82    | 212.73        | 254.16 |\n\n>It seems that the CoSD does not directly modify the answers in Table 4. So how does CoSD improve the score on GSM8K? More explanations are needed.\n \nWe discuss how CoSD improves the score in the Case Study section. In CoSD-Rule, in the fifth line, the assistant model rejects the draft model’s incorrect computation of 20% of 20 = 10 and instead uses the correct calculation of 20 * 0.2 = 4, successfully avoiding the error in the draft model’s tax calculation. In the sixth line, the draft model correctly leads to generate the subtotal of $24, so in the final step, CoSD-Rule computes the simpler 24 + 5 instead of the more complicated 15 + 3 + 2 + 5, resulting in the correct answer.", "claims_and_evidence": "The paper claims that CoSD improves knowledge fusion performance, enhances efficiency through speculative decoding, and provides flexibility by eliminating explicit model selection. Empirical results basically support these claims, showing multi-LLM integration and faster inference.", "methods_and_evaluation_criteria": "The method involves running a draft model to generate candidate tokens and verifying them with assistant models before finalizing outputs. Evaluation is conducted on standard LLM benchmarks, measuring fusion performance via accuracy and perplexity while assessing efficiency through decoding speed. Comparisons are made against individual LLM baselines and naive ensemble methods.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiment part focuses on setting up several model pairs to mimic the real-world scenarios when we need to fuse LLM knowledge. CoSD is tested on 5 benchmarks. The experiment settings are valid. However, some questions remain in the hyperparameter part. What is the hyperparameter for the decision tree? Do we need to adjust the parameters during the inference stage, and how can we do this?", "relation_to_broader_scientific_literature": "The paper is related to the LLM knowledge fusion field and the efficient inference field. It is an application and augmentation of both fields.", "essential_references_not_discussed": "An early work on model knowledge fusion (not LLMs):\nDong, X. L., & Srivastava, D. (2015, May). Knowledge curation and knowledge fusion: challenges, models and applications. In Proceedings of the 2015 acm sigmod international conference on management of data (pp. 2063-2066).", "other_strengths_and_weaknesses": "N/A.", "other_comments_or_suggestions": "I suggest to show more interesting samples like Table 4 in the appendix.", "questions_for_authors": "What is the token latency for only use one LLM in Table 6?\n\nIt seems that the CoSD does not directly modify the answers in Table 4. So how does CoSD improve the score on GSM8K? More explanations are needed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes to integrate complementary knowledge from multiple LLMs for model knowledge fusion. The proposed method classifies output tokens based on their probabilities to achieve fusion and leverages a speculative decoding framework to enhance generation speed. 

In general, the proposed method is intuitive and elegant. Extensive evaluation on 6 LLM pairs and 5 common benchmarks demonstrates the effectiveness of the proposed method, and the major concerns have been addressed by the rebuttal. Therefore, I recommend to accept this paper.
Decision: Accept (poster)