Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding

Authors: Ziyao Wang, Muneeza Azmat, Ang Li, Raya Horesh, Mikhail Yurochkin
Primary area: foundation or frontier models, including LLMs
Keywords: Large language model; Knowledge fusion; Speculative decoding

Abstract:
Large Language Models (LLMs) often excel in specific domains but fall short in others due to the limitations of their training. Thus, enabling LLMs to solve problems collaboratively by integrating their complementary knowledge promises to improve their performance across domains. To realize this potential, we introduce a novel Collaborative Speculative Decoding (CoSD) algorithm that enables efficient LLM knowledge fusion at test time without requiring additional model training. CoSD employs a draft model to generate initial sequences and an easy-to-learn rule or decision tree to decide when to invoke an assistant model to improve these drafts. CoSD not only enhances knowledge fusion but also improves inference efficiency, is transferable across domains, and offers greater explainability. Experimental results demonstrate that CoSD improves accuracy by up to 10% across benchmarks compared to existing methods, providing a scalable and effective solution for LLM-based applications.

Decision: reject
Reviewer scores: 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730172037727

thread_type:
official_review

<thread_object>
creator: reviewer_1scr

<content>
summary:
This paper introduces the Collaborative Speculative Decoding (CoSD) algorithm that enables efficient LLM knowledge fusion during decoding. Upon the idea of speculative decoding, the paper proposes two novel decision-making rules: Rule-based and Tree-Based. The method features 1) efficiency (parallel generation, no additional model training), 2) transferability across different domains and models with different tokenizers, and 3) interpretability. CoSD successfully improves baselines by up to 10%.

strengths:
- interesting and inspiring idea on fusing knowledge at the decoding time
- The algorithm is clearly presented in this paper through both a workflow diagram and mathematical expressions.
- Both Rule-Based verification and Tree-Based verification are well-designed and both make sense to me.

weaknesses:
- I'm not sure if the goal of this algorithm is to A) achieve performance comparable to the assistant model but in a more efficient way, or if it's aimed at B) outperforming both the draft model and the assistant model individually (1+1>2). How do these objectives apply to the four scenarios of knowledge fusion discussed in section 4.1? If the goal is A, since the draft models in complementary knowledge fusion and catastrophic forgetting recovery scenarios are about the same size as the assistant model, and the algorithm involves autoregressive generation of the draft model, I doubt the algorithm improves efficiency.  If the goal is B, I can't see improvement based on Table 2.

questions:
- "the algorithm regenerate and re-verify iteratively until all tokens are accepted" How many iterations does it take on average? 
- during the training process of the decision tree, if neither the draft model's generation nor the assistant model's generation match the target, you drop the sample and continue the loop with i &larr; i+1. Any ideas of improvement other than simply dropping these samples?
- typos: line 287, "tree" to "three", "drat" to "draft"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730589843390

thread_type:
official_review

<thread_object>
creator: reviewer_hds6

<content>
summary:
This paper introduces a novel collaborative speculative decoding algorithm which can efficiently fuse the knowledge from different LLMs during inference. The experiment setting is quite interesting and includes different types: complementary knowledge fusion, catastrophic forgetting recovery, capacity imbalance and different tokenizers. The results are better than different baselines.

strengths:
1. This paper provides an interesting perspective to fuse knowledge between LLMs using speculative decoding, which leverages the strengths of different LLMs while still keeping the efficiency.
2. The experiment setting is interesting, which tries complementary knowledge fusion, catastrophic forgetting recovery, capacity imbalance and different tokenizers.

weaknesses:
1. The paper only does the experiment in each pair of the LLMs. It would be interesting to see more LLMs collaboratively fuse knowledge.
2. It would be better to show more details about the limitations of the proposed method and show some error analysis.

questions:
1. Is the proposed algorithm suitable for collaboration among multiple LLMs? What will be the potential challenges?
2. Can you explain more about the limitations of the current method? I'm curious when it doesn't work well.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690281051

thread_type:
official_review

<thread_object>
creator: reviewer_jwxz

<content>
summary:
This paper proposes Collaborative Speculative Decoding (CoSD) that fuses LLM knowledge at test time. The algorithm employs a draft model to generate initial response sequences and a rule-based or decision tree to decide when to leverage an assistant model to improve the drafts.
The authors have conducted experiments using different pairs of LLMs and under various experimental setups.

strengths:
- The authors have put effort in experimenting their proposed framework under different setups, including various draft and assistant models, different simulated scenarios, etc.
- The proposed framework gains some advantage over the existing framework of Co-LLM in certain scenarios.

weaknesses:
- In Table 2, I notice that in most cases, the fused model underperforms the draft model and the assistant model.
For instance, for Pair 1, none of the fusion methods outperform both draft and assistant model for GSM8K, HumanEval; for Pair 2, none of the fusion methods consistently outperform both draft and assistant model for GSM8K, and MMLU.
Then I wonder what is the point of fusing knowledge in these cases if we can simply adopt one model instead of the other?

- It seems that for Pair 3, CoSD-Rule performs exceptionally well on GSM8K, yielding 45.47 while the draft and assistant models yield 25.01 and 35.43, which is very different from the performance patterns for this same pair on other datasets such as MMLU and also other pairs. Could you give more insights into such a result? Could you present some examples that CoSD-Rule excel at under this situation that cannot be addressed by either the draft nor the assistant model?

questions:
- In 3.2 Verification, for Tree-Based Verification, you claim to use benchmark datasets such as GSM8K to train the classifier, but then in your test, you incorporate the GSM8K dataset as well. Is there any information leakage in terms of that you are training your verifier on the test set so that it gains advantage over other models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732533406322

thread_type:
official_comment

<thread_object>
creator: area_chair_rm3n

<content>
title:
Action Required: Respond to Author Rebuttals - Nov 27

comment:
Dear ICLR Reviewers,

The author discussion phase is ending soon. Please promptly review and respond to author rebuttals for your assigned papers. Your engagement is critical for the decision-making process.

Deadlines:
- November 26: Last day for reviewers to ask questions to authors.
- November 27: Last day for authors to respond to reviewers.
- November 28 - December 10: Reviewer and area chair discussion phase.

Thank you for your timely attention to this matter.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes Collaborative Speculative Decoding (CoSD), a method for efficient knowledge fusion between LLMs during inference using rule-based and tree-based verification strategies. The approach aims to both improve efficiency when combining models of different sizes and enhance performance through complementary knowledge fusion.

The reviewers value the comprehensive experimental analysis across different model pairs and scenarios. Through extensive discussion, the authors clarified several reviewers’ questions. However, some concerns remain: 1) performance improvements are not consistent across all benchmarks, with only three test datasets making it difficult to draw robust conclusions, 2) efficiency claims compared to baselines need better substantiation, and 3) the averaging of scores across benchmarks may mask important per-task performance variations.

While the authors have engaged constructively with reviewer feedback and provided additional experiments and analysis, I recommend rejection as more comprehensive evaluation across a broader set of benchmarks is needed to convincingly demonstrate the method's effectiveness and generality.