SpecEM: Training-Free LLM Ensembling via Iterative Drafting, Verification, and Online Feedback
Authors: Bo Lv
Nayu Liu
Chen Tang
Xin Liu
Yue Yu
Ping Luo
Venue: NeurIPS 2025 poster ()
Keywords: Ensembling language models.+Ensemble method.+Generative.+instruction response
Primary area: deep_learning
TL;DR: SpecEM is a plug-and-play LLM ensemble that boosts strong models and enables fast, segment-level collaboration. It outperforms prior methods across six tasks and five LLMs.

Abstract: Ensembles of generative large language models (LLMs) are a promising way to compensate for individual model limitations, integrating the strengths of different LLMs. Existing LLM ensemble methods, however, face limitations such as first-token delay and challenges in long-range semantic collaboration between models, Moreover, they typically assume equal voting weights for all models during ensemble, ignoring performance differences between models for a given task. In this work, we propose SpecEM, a training-free, plug-and-play LLM ensemble framework that dynamically adjusts each model's model contribution in real time based on task performance. Inspired by speculative decoding, SpecFuse iteratively performs drafting and verification, allowing models to collaborate semantically at the segment level for integrated output. Furthermore, we introduce an online feedback mechanism with multiplicative weight updates, where each model's voting weight is adjusted on-the-fly according to how often it "outperforms" others during verification stage, ensuring that stronger models exert greater influence on the ensemble during generation. Experimental results on five popular LLMs (ranging from 7B to 72B parameters) and six benchmark tasks, spanning instruction following, reasoning, commonsense, and general instruction response, demonstrate consistent performance improvements compared to state-of-the-art LLM ensemble methods.

SpecEM: Training-Free LLM Ensembling via Iterative Drafting, Verification, and Online Feedback
Bo Lv Lv3, Nayu Liu"*, Chen Tang", Xin Liu', Yue Yu' Ping L,2.
1,2.3
Peng Cheng Laboratory
2 Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)
University of Chinese Academy of Sciences
4 TTinjin Laboratory Autonomous Intelligence Technology and Systems, School of Computer Science and Technology, Tiangong University Institute for Advanced Algorithms Research, Shanghai 1vbo190mails ucas ac.
Abstract
Ensembles of generative large language models (LLMs) are a promising way compensate for individual model limitations, integrating the strengths of differ- ent LLMs. Existing LLM ensemble methods, however, face limitations such as first-token delay and challenges in long-range semantic collaboration between models, Moreover, they typically assume equal voting weights for models during ensemble, ignoring task-specific performance differences among models. In this work, we propose SpecEM, training- free, plug-and slug-and-play LLM ensemble framework that dynamically adjusts each model's model contribution in real time based on task performance. Inspired by speculative decoding, SpecEM iteratively performs drafting and verification, allowing models collaborate semantically the segment level for integrated output. Furthermore, we introduce an online feed- back mechanism with multiplicative weight updates, where each model's voting weight is adjusted on-the-fly according to how often it outperforms others during verification stage, ensuring that stronger models exert greater influence during ensembling. Experimental results on five LLM families (ranging from 7B to 72B parameters) and six benchmark datasets. spanning open-domain instruction follow- ing, reasoning, commonsense, demonstrate consistent performance improvements compared state-of-the-art LLM ensemble methods. Our code available https///gthub.coom///vveeeeee///
1 Introduction
Generative large language models (LLMs) (et@ 2024 Yang et al. 2024 have been widely adopted due to their impressive performance across broad range of domains. Owing to differences in training data and model architectures, off-the-shelf generative LLMs often exhibit strengths in different areas. Consequently, ensembling multiple LLMs inference time help mitigate individual biases and errors, resulting in more robust and reliable user experience.
Existing LLM ensemble approaches [Chen et al. 2025 Jiang al. 2023b Huang et al. 2024 Lv
ct al. 2024a| can be broadly categorized into generate- generate-then-ene tne---neenbbbleee Jiang ef al. 2023b Lv et al.
[Huang al. 2024 Yao et al. 2025 paradigms. The former typically generates full responses from all base models for given query, then leverages an additional
2024b and
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

fusion model to summarize select the best outputs. The latter adopts more interleaved approach, greedily aggregating the output probabilities from different models at certain timesteps to decide next tokens, which is then broadcast to all models.
Despite promising progress, senerate-then-enseee must wait until all models complete their responses before receiving integrated output. In contrast, methods mitigate this latency but may fall short in enabling long-range
methods suffer from first-token delay, users
ensemble-whie wwile-generatee semantic communications across models. Moreover, existing methods focus solely on aggregating model outputs, typically assuming equal contribution from all models, and overlooking the fact that different models may perform differently depending on the task. We argue that incorporating an online learning mechanism to dynamically assign higher weights down-won wwwn-weigtting weaker ones can lead to higher-quality ensemble outputs.
setter-performing models while
Based the ahove observations, we propose SpecEM, training-free, plug-and-play L.LM ensemble framework that performs segment-level fusion of model outputs and dynamically adjusts model weights on-the-fly based on task-specific performance. Inspired by speculative decoding (Xia et al.) 2023 Leviathan et al., [2023a] SpecEM iteratively executes two key stages; drafting and verification. In the drafting stage, each base LLM generates candidate text segment given the prior context, with a predefined maximum length per iteration. In the verification stage, all LLMs receive these candidate segments with prior context and mutually evaluate them in parallel based on their output logits. The top-ranked segment is then broadcast to all models, guiding them to generate higher-quality text in subsequent rounds. This iterative drafting-verificatinn verification process eliminates the need for training fusion modules or selection aggregators, and allows for effortless integration of base LLMs without any fine-tuning-
Furthermore, we introduce an online feedback mechanism in SpecEM to dynamically adjust each model's influence during the verification stage. It is based on the assumption that models capable of generating higher-quality segments in drafting are also better at evaluating others in verification. Specifically, we treat the number of times a model outperforms its peers in verification as a reward signal to update its voting weight using a multiplicative weight update algorithm. This ensures that stronger models progressively exert greater influence during generation.
We evaluate SpecEM on five popular L.LM families (ranging from 7B to 72B parameters) across six benchmark datasets, covering open-domain instruction following. reasoning, and commonsense. Experimental results demonstrate consistent performance improvements over state-of-the-art LLM ensemble methods. In summary, our contributions are as follows:
We propose SpecEM, a training-fres and plug-and-play ensemble framework that integrates outputs by iteratively coordinating drafting and verification across multiple LLMs.
We propose an online feedback mechanism that dynamically adjusts each model's contri- bution to inference and verification during generation, ensuring that stronger models exert greater influence the ensemble.
We conduct comprehensive evaluations on five LLM families and six benchmarks, showing that SpecEM consistently outperforms state-of-the-ar ensemble methods.
2
Related Work
2.1 LLM Ensembling
Recent efforts in LLM ensembling have shown that combining multiple models can enhance perfor- mance by leveraging their complementary strengths. These methods can be broadly categorized into mnneraat-then-ensemie and ensemble-while-gemnee generation, depending on when and how the ensembling of model outputs occurs.
Generate-ther seetratetthen-enseee then-ensemble methods first let each base LLM generate complete response, and then aggregate the outputs through selection [Freitag et al. 2023 or fusion [Lv et al.] 2024b). Selection- based methods, such as MBR [Freitag et al.) 2023 and PairRank [Jiang et al.. 2023b), rank or compare candidates among all outputs to select the best one as the output. Fusion-based methods, such as GenFuse [Jing et al. 2023b) and MOA [Wang et al, 2024), generate new outputs by using base model responses input a fusion model or aggregator.

---

Drafting Stage
Verification Stage
Finol score
Candiiiones
5)
Harelt ttep-by--tap
1-11
Working computer mouse
Madel
50
PA
Select Top Condidote
Online Feedback
Condidater
compster mooos is simple input device
Rewrodd
Here's
Gr
Wording computer mouse
Prior contest (0-11)
Updette voding welights
Single www--oooodddooo Sound--uuputt
shep-by-ttep guide
Convotenste candidore C2"
Figure I: Overview of SpecEM, training-free free blug-and-ple LLM ensemble framework with three components: drafting, verification, and online feedback. The blue solid lines indicate single iteration; dashed lines denote input refreshing for the next round.
methods Yu et al. 2024 Huang et al. 2024 Xu et al. 2024 aggregate model outputs during the generation process, typically by fusing output probability distributions to produce ensembling results incrementally. Due to vocabulary mismatches across different LLMs that hinder the combination of multiple probability distributions, Yu al. (2024) construct new union vocabulary by combining the vocabularies of multiple models to include all tokens from each model. They then project the distribution information from each model onto this merged vocabulary for averaging aggregation. Similarly, DeePEn [Huang et al. [2024] and EVA [Xu et al. 2024] project the output distributions of multiple models into shared relative/pivot space, followed by averaging aggregation. While, these methods operate over all LLM's vocabulary at each timestep, which may leads to some computational overhead. More efficiently, UniTe (Yao al., 2025) focuses only on the top-K K portion of each model's 's output distribution and uses the union vocabulary strategy (Yao) ct al. 2025 to reduce alignment costs while maintaining good performance.
While recent progress has been made, challenges remain in balancing efficiency with effective cross- model collaboration. In this work, introduce SpecEM, a plug-and-play training-free free framework that performs segment-level collaboration via iterative drafting and verification. Unlike previous methods that rely on static model contributions, SpecEM incorporates an online feedback mechanism to dynamically adjust each model's 's influence based on performance during generation, promoting more adaptive and effective ensembling.
2.2 Speculative Decoding
Speculative Decoding Xia et al. 2023 Chen al. 2023 Sun 2024 aims accelerate inference in LLMs by leveraging a lightweight draft model to propose multiple candidate tokens, which are then verified by larger target model [Leviathan al. 2023b Concretely, at each decoding step, the draft model efficiently generates sequence of potential tokens, and the target model accepts only those that match its own predictions [Miao et al. 2024 This significantly reduces the number of expensive forward passes through the larger model without compromising output quality.
Inspired by this idea, propose SpecEM, which reimagines speculative decoding for model ensembling rather than acceleration. Instead of a small model drafting for a large one, multiple LLMs iteratively generate draft segments and verify each other's outputs in parallel. This collaborative refinement allows stronger models to guide weaker ones. In addition, SpecEM incorporates an online feedback mechanism that dynamically adjusts each model's influence during verification.
Methodology
Figure presents an overview SpecEM. SpecEM performs L.MM ensembling through iterative drafting. verification, and online feedback, which are described in detail in the following subsections.

---

Original Attention Mook Mlennt
Kerigy
E
G
G
X
X
X
X
Original Position Ae
Jartjp-dedinn Pootion of
C
X
G
Port Ad
X
Figure 2: An overview of verify-in- attention mask and postion id.
3.1 Drafting Stage
During the drafting stage, in each generation round, all models are simultancously activated to perform parallel inference based the task query and the best candidate segment broadcast from previous rounds. Formally, let the ensemble consist of models (M,),, where M; denotes the i-th base LLM. In the k-th iteration round, the draft candidate segment generated by M, is denoted
as (((e) (1(x) ...,,4 (1,,11,) 1(44), where the number generated tokens is constrained by a predefined maximum segment length L.
ci M, (T (T",,...,, I(k-1))
(1)
Here, 1( denotes the best candidate segment broadcast in the k-th round, and ((0) corresponds to the initial task query.
3.2 Verification Stage
During the verification stage, all models perform mutual evaluation the candidate segments generated during the drafting stage. At iteration k, each model receives the full set candidate segments {Cr J.. along with the prior context (I(")...,, ((-.))) Each model scores the candidates, and the one with the highest aggregated score is selected as the output the current round. This selected segment is then broadcast to all models part of the context for the next generation round.
As illustrated in the mutual evaluation matrix Figure let [ denote the score assigned by model to candidate j generation round Concretely, for each candidate segment, we compute the average of the logits produced by the model over the tokens in the segment, which serves as the
model's s evaluation score [Lv et al. 2023 Varshney et al. 2023
Lv al. 2025 for that candidate:
Σ
(()())
where p(t denotes the logit score by model in round & for the u-th token of candidate To mitigate scoring bias caused by some models producing systematically higher or lower logits, we normalize the scores that the model assigns all candidates before aggregation:
Ej=1
Finally, the overall score {uy h of each candidate is computed as a weighted sum of the scores from all verifier models:
(k)
with weights {wi} x dynamically updated by the online feedback mechanism described in Section 3.3
Specially, we introduce a verify-in- innf-ein-line mechanism, avoiding redundant attention computations over the prior context and the increased time complexity caused by serial model-wise scoring. Concretely, at the k-th generation iteration, we concatenate the prior context and all candidate segments along the sequence dimension construct unified input sequence:
LINE ( (k)
(5)

---

where I[(=1) denotes the prior context [1(0] :..: [[[=1)] for brevity, and [:] denotes sequence wise concatenation. We then modify the attention mask and position IDs in the Transformer such that each model can process LINE and output scores for all candidates in parallel, as shown in Figure 2
Verify-in-line attention mask. During verification, each candidate segment should only attend to the shared prior context I(-=1), without accessing information from other candidate segments. To enforce this, we augment the standard triangular attention mask of the decoder-only Transformer with additional masking scheme that blocks attention across candidate segments. This ensures that tokens c( can only attend to [(k-1), not to tokens in other candidates Cj for j + As result,
although all candidates are concatenated into LINE, model M, effectively "sees" only [ c during scoring, enabling efficient and parallel evaluation.
Verify-in-line position IDs. While the modified attention mask guarantees the correct visibility, the default position encoding would still reflect the segments' physical positions in the concatenated LINE, which may distort modeling. For instance, the actual input sequence for scoring C/&²) is [[(+1) [mask] C2(a) [mask] : ...] rather than [[(+ cg To address this, we further adapt LLMs' relative positional encoding so that each candidate segment is positioned if it were
immediately following the prior context. That is, the position IDs for each ( are reset to be consecutive with eenuuinn context.
ensuring the position modeling remains consistent with the actual evaluation
3.3 Online Feedback Mechanism
is generally difficult to anticipate which model performs best on given query. Due to differences in model architectures and training corpora, different models may exhibit varying strengths across domains, and may produce low-quality outputs when encountering unfamiliar challenging topics. Since the verification stage relies on each model's scoring of candidate segments, models with weaker generation capabilities may also produce unreliable evaluations when acting as validators.
We propose a core assumption: models that perform better in generation tend to offer more reliable judgments during verification. This assumption empirically supported Appendix A Building on this, we introduce an online feedback mechanism based on the multiplicative weights update algorithm. dynammically adjusts each model's Is contribution in the verification stage by tracking and weighting its validation performance during the generation process. As a result, decisions from better-perforee perter-performing models are prioritized, enhancing the overall ensemble effectiveness.
Formally, let there be N models. Denote the verification weight of model M; at generation round k wh( All models initially assigned uniform weights: w/ N At round k model M;
receives feedback reward 7(x), and its weight is updated according to:
wi (N)
wi (k-1) en!
66)
To address the fact that more models (i.e. N) lead to smaller initial weights, and to ensure that updates become more stable over time (i.e. k), we define the learning rate 7) in Eq. Bl.
11/k
N
where a is hyperparameter. Then all weights are normalized as:
W
(8)
Σイ=1
This beedback-driver reweighting ensures that more credible validators exert greater influence on the selection process, leading progressively improved collective decisions over time.
Reward Definition. For each model M,, define its reward (k) in Eq. 6 based on how often its generated segment is preferred over others in the evaluations conducted by the remaining models.
Specifically, count the number of times model M, 's candidate ((()) is scored higher than another
5

---

candidate c,a0 by third model M,. where j * and r + i, and normalize this count as reward (k),
7,
(1) Ejjt En bool( (k) > (1))
E bool(s), ().))
(2)
Here, (-ol(-) returns 1 if the condition inside is true, and O otherwise. Intuitively, if model's candidate frequently outperforms others in peer evaluations, it is considered better suited to the current task. In Section 4.3 we empirically compare this reward formulation against alternative segment
selection strategies. Finally, the comprehensive score for each candidate {y
(y( )N is computed as
the weighted sum of its scores assigned by all validators, following Equation 4 The highest-scoring
candidate is selected as the best output 7 7(6), and it is appended to the prior context for use in the next round of drafting and verification.
Experiments
4.1 Experimental Setup
Datasets. We evaluate SpecEM on six datasets that reflect key capabilities of LLMs, including open- domain instruction following, commonsense, and reasoning. FuseEval: A multilingual instruction response benchmark we construct by combining Dolly-15k [Conover et al. 2023 and Alpaca- GPT4 Peng et al., 2023) for English, and Human-Valle and Ruozb from COIG-CQIA [Bai et al..
2024 for Chinese. FEval [Zhou et al. 2023]: Evaluates instruction adherence under four granular
settings, prompt-strict/looss instruction-stre -triiot---srrretooeee.. AlpacaEval 2.0 (Dubois al. 2024]: Measures alignment with human preferences via GPT-4 based pairwise comparisons against GPT-4 outputs. MMLU (5-shot) [Hendrycks al., 2021] and ARC-C (5-shot) (Clark et al. 2018): Multiple-choice benchmarks that test factual know ledge and general commonsense, GSM8K (3-shot) [Cobbe et al.)
2021]: Focuses on arithmetic and multi-step reasoning through grade-school math problems. x-shot refers to providing examples as in-context during inference, Please refer to Appendix B.] for detailed description of datasets.
Base LLMs. We use top-performing open-source instruction- tuned models (7B-9B) as base LLMs in our ensemble, including -Llama---B-iistruu [AT@Metz 2024 -iistral-7B-v0.--nntttt
Jiang
ct al. 2023a), Qwen2-7B-instruct (Yang et al.) 2024], Glm-4-9binnsruuct [GLM et al. [0224] and
Jemma-2-9b-instruct [Gemma et 2024]. Moreover, assess scalability, we also evaluate
SpecEM with larger base models, including Qwen2-72B-instrrud Llama3-70B-instruee Qwen2.5-
32B-instruct (Qwen al. 2025 and Mistraal-24B-insstuut--25111
TTem. 2025
Metrics. We follow standard evaluation protocols for each task in previous works. For FuseEval, use BARTScore Yuan al. 2021] BERTScore (Zhang al. 2019]. BLEU et al.) 2002)
[OpenAl et al. [024] to assess reference-based generation quality. MMLU and ARC-C report accuracy by selecting the option with the highest likelihood. For GSMSK, we compute exact match accuracy based on the predicted answer. For IFEval, we use the provided evaluation files to test under prompp/istrructionsttiie and -loose conditions. AlpacaEval 2.0 reports length-controlled (LC) win rates against GPT-4 outputs using gpt-4-1106-previee
ROUGE [Li 2004), and
Comparative methods. We compare our proposed SpecEM with several strong recent LLM ensemble methods, including PairRank [Jiang et al. 2023b), Minimum Bayes Risk (MBR) [Freitag et al. 2. Generation Fusion (GF) Jiang et al. 2023b), Mixture-of-Agemi (MOA) [Wang et al.
2024 Majority Voting [Davani et al. 20222, and Unite Yao et al. 2025 Please refer to Appendix
B.3 for a detailed description of these baseline methods.
Implement details. SpecEM requires no training and operates purely during inference. All models are loaded using 6 precision, with do tassamlleee--eeee True, temperature = 0.6, and top_p 0.9 generation settings. For experiments with 7B-9B models, we use A100 GPUs, while larger models (24B-72B) are evaluated on H200 GPUs. The maximum number of candidate segments is set to == 10, and the online feedback hyperparameter set to a All reported results are averaged over three independent runs to ensure stability.

---

Table 1: Results on the English and Chinese subsets of the FuseEval benchmark. Pink highlights the
best overall result, and Blue marks the best result among base LLMs. The upward arrow means higher is better, and the downward arrow J means lower is better.
Model
ROUGE-11
ROUGE-21
ROUGE-LI
BLEU
BARTScore"
BERTScore
GPT4-Rank
English Sceuario
Base LLMSs
Llama 3-DD instree
AttMeta
2024
25.16
9.77
23.31
3.57
-2.98
69.99
9.52
Glm-4-9B instract
GLM et
2024
25.85
10.26
23.90
3.48
-2.96
70.51
9.24
Boen2-7B-instmun
Yang et al.
X04
26.63
10.81
24.49
-2.94
Gemma:
Gennaa
2024
-2.9
Mistml-7BBinssmmst-0..
27.75
10.75
-2.44
7.62
Larger LLMs
Llama-3-708. -aam----70--nsstuue
Alta
2024
26.77
10:87
24.56
4,10
-2.84
70.98
5.22
Qwen2 -inimnoe-i
instuct
Yang ml
2023
27.26
11.23
25.11
4.29
-2766
71.73
4,21
Mistral- 8x7B -iintrrct
Bung et
2023g
29.04
12:25
26.75
4.08
-2.81
72.19
4.16
Methods Ensembling Base
L.MMs
(Qwen2)
Hong at.
-2.95
(Ciemmn-3
Daagg
-3.02
GF (Mistra))
Jang
X0.36
24.92
9.58
22.97
-2.93
70.38
8.56
MBR
Frorap et
2025
27.12
10:40
25.33
4,56
-2.89
71,63
7.66
PairRank
Vinn ei
10.36
28.21
10.86
25.94
4,99
-2.86
72.09
6.84
MOA
Wing et al.
A024
27.61
11.30
25.47
5.12
-2.88
71.90
7.48
-2.00
Speck
Scenario
I.M.s
Gemman2 2-98 -nstinntout
Genmn
2024
29.15
7.65
18.35
3,36
-4.28
68.73
8.58
(ikaml-7B-insttcii-a
Diang
D023g
30.99
8.65
20.66
4,42
-4.45
70.10
6.55
Qwen2-7RH Quen2-7B-irstmuc
Yang of al.
20.93
8.09
20.03
3.62
-4.3
69.09
6.42
GLM et ml
2024
30.88
8.71
20.42
4.47
4.300
70.25
5.18
Larger L.MSs
Llama:
Qwen2-73B-insmmuc
Yang
Methods Ensembling
Base
LLMs
GP (Mistral)
Diang al.
20236
30.20
8.12
20.33
3.88
-4.54
70.04
7.28
GF (Qwen2)
Jiang et al. 20236
28.69
7.87
18.93
3.32
-4.4
69.81
8.41
GP (Glm=4)
Diang et al.
11230
30.26
8,70
20.51
4.27
-4.3
70.23
5.40
MBR
Fretay al
30.93
8.71
20.63
-4.3
5,33
Yao
2074
SpecEM (Ours)
9.94
24.00
4.75
4.29
71.03
3.25
4.2
Main Results
Results on Diverse Evaluation Benchmarks. Table pressens results on the English and Chinese subsets of the FuseEval benchmark. SpecEM, built on 7B-9B base models, outperforms all individual LLMs and existing ensemble methods across all metrics.
Notably, it achieves over 3- point point average gains in ROUGE- 1/2/L and ranks highest on GPT4-Rank. Despite using only several 7B-9B LLMs, SpecEM performs compara- bly to 70B-scale single mod- els while remaining more pa- rameter efficient. The improve- ments are consistent in the En- glish and Chinese scenarios, indicating the generalizability of SpecEM across languages. We further assess SpecEM on MMLU, ARC-C, GSM8K, and IFEval benchmarks using base
Table 3: Performance on FuseEval and AlpacaEval 2.0 bench- marks. Win rates on FuseEval are measured relative to the outputs of Owen2-72b-instee
English FuseE'val Chinese FuseEval AlpacaEval (I.C-wlarate)
Avg
Model
(winrate)
(winrate)
Base LLMs
Quen2-72b-insrret
Seken2.--3b-iinete
43.82
Slams--70biinstuuc
43.10
10.66
34.40
29.39
Mistral 2bbb instruct 2501
52.55
31.79
48.46
44,27
Methods of Envembligg Baye LLMS
(3.12
46.98
GenPPase
$1.06
$2.41 (+144933)
49 (s0.60)
50.84
UniTE
54.79 (+2.24)
19.13 (-18.35)
49.20 (+0.74)
41.04
SpecEiM
15:46 (43.91)
56.77 (419.29)
32 (42.86)
$4.52

---

Table 2: Results on MMLU, ARC-C, GSM8K, and IFEval benchmarks. Values in parentheses indicate performance difference from the best-performing base model in ensemble each column.
IFEval
MMLJ
ARC-C
GSMSK
prompt-avg
instruct-avg
Base LLMss
Qwea2-7B-instruct
68.23
84.73
74.22
41.70
53.88
GLM-I-9B-instee
67.16
85.15
71.80
56.01
67.14
Gemma-2 2-9B-instruce
71.51
88.14
77.26
61.64
72.26
Methods of Ensembling Base LLMs
Majority-Votiin
71.78 (+0.27)
(+0.24)
77.29 (+0.03)
MBR
76.98 (-0.28)
54.96 (-6.68)
66.21 (-6.05)
MOA
70.43 (-1.08)
88.28 (+0.14)
77.30 (+0.04)
60.80 (-0.84)
68.81 (-3.45)
UnITE
71.94 (40.43)
88.54 (-0.40)
76.52 (-0.74)
56.72 (-4.92)
62.08 (-10.18)
SpecKM (Qwen2+GLM4)
(42.30))
87.34 (-2.39)
75.44 (+1.22)
$1.13 (-4.86)
63.07 (-4.07)
SpecEM Genumaa)
72.18 (40.67)
88, 40.40 (+0.26)
78.70 (+1.44)
56.01 (-5.33)
67.39 (-4.77)
SpecEM Gemma2))
71.82 (+0.31)
88. (+0.60)
75.82 (-1.44)
66.89 (45.25)
75.52 (43.26)
SpecEM (All)
73.01 (+1.50)
(+0.94)
77.41 (+0.15)
62.11 (+0.47)
71.56 (-0.70)
LLMs with varied strengths. As shown in Table SpecEM consistently surpasses all baseline ensemble methods across these benchmarks. In particular, SpecEM (Qwen2 + GLM4) achieves +2.5 and +2.4 improvements on MMLU and ARC-C, respectively, leveraging complementary model capabilities. These results demonstrate the effectiveness of SpecEM across diverse task formats beyond open-ended generation.
Scaling to Larger Models. To further evaluate the scalability of our framework, we conduct experiments by integrating four larger LLMs ranging from 24B to 72B parameters on FuseEval and AlpacaEval 2.0. As shown in Table SpecEM consistently outperforms all base models and prior ensemble baselines, achieve the best performance. In particular, surpasses the strongest single base model by an average win rate margin of 10.3 points. These results demonstrate that SpecEM generalizes robustly across model sizes scale.
Table 4: Win-rate comparisons between SpecEM and ablations on English (EN) and Chinese (CN) FuseEval. Each cell shows the win/loss percentage judged by GPT-40-2024-11--0 A denotes average win-rate improvement over ablations, and Avg reflects the mean win rate across EN and CN.
Comparison
EN (win/lose)
CN (win/lose)
A (EN/ CN)
Avg A
SpecEM w/o Online feedback
53.66 / 46.34
$2.98 / 47.02
+7.32/+5.99
+6.64
SpecEM vs w/ Feodback (Score-bused reward)
52.17 / /4788
51.63 /48.37
+4.34 /3.26
+3.80
SpecEM w/o Feedback, Top-win selection
53.40 / 46.60
$2.35 / 47.65
+6.80 / 44.70
+5.75
4.3 Analysis
Ablation analysis. We perform ablation studies to assess the effect of the online feedback mech- anism in the verification stage. Using GPT-4 as the evaluator, we compare the full SpecEM with three variants: (a) w/o online feedback: Removes feedback; final output is selected solely based on verification scores. (b) w/ score-based reward: Replaces win count 7/ with the normalized average verification score the reward. (c) w/o feedback, top-win selection: Selects the candidate with the highest win count without reward accumulation. As shown in Figure 4 the full SpecEM outperforms all variants. Online feedback yields a 6.6-point average gain; using win count as the reward offers an additional 2.8 2.8-point improvement over score-based reward. Directly selecting the top-win segment leads to 5.8-point drop, likely due to close or tied win counts, which are more reliable as soft rewards than as decisive selection criteria.
Inference Latency Analysis. We evaluate the inference efficiency of SpecEM against other en- semble methods and base models, focusing on two key metrics shown in Figure 3 (1) First Token Latency, the time from user input to the generation of the first token, which is crucial for interactive user experience; (2) Total Generation Time, the time to generate a complete response across varying output lengths. SpecEM achieves the lowest total response time among all ensemble methods, with

---

Lama-340
250
Gmm--56
100
Qmen2-18
RBdval-78-v0.3
Matral 70-00)
80
Gerfuse
MDA
Total Time
150
Unfia
SpacEM
SpecEM
First
50
250
300
730
1000
1230
1300
1730
2000
250
500
750
1000
1250
1500
1750
2000
Number Generated Tokens
Number of Generated Tokens
Figure 3: Comparison of inference latency performance across methods. Left: total generation time (seconds); Right: first-token latency (seconds), both plotted against the number of generated tokens.
only 20% overhead compared the slowest single model. This is because, under parallel inference settings, the ensemble latency is bottlenecked by the slowest model's output time.
Crucially, SpecEM maintains consistently low first token latency (under 0.6s) across all lengths, en- abling fast feedback. In contrast, methods that wait for full outputs before fusion suffer from rapidly increasing first token latency, making them unsuitable for real-time applications.
27.52
SprtEM
05
Base Model Number Analysis. We evaluate how SpecEM scales with the number of integrated base LLMs on the English FuseEval dataset. As shown in Figure g performance consistently improves as more base models added. The improvements are more pronounced when stronger models are in- troduced, while
23.63
Linan Ap
HDde at
#peeddh mi
MMteinl- 7M
Comma
weaker models still contribute positively. These results highlight the flexibility and scalability of SpecEM, where new models can be seamlessly integrated without additional train- ing or adaptation, making the system robust and extensible in real-world deployment.
Figure 4: The variation in SpecEM's ROUGE-
L score as the number of base LLMs increases. [model] indicates the incremental addition of a specific model to the ensemble.
Candidate Segment Length Analysis. We study the impact of the maximum generation length of candidate segments SpecEM's performance using the English FuseEval development set. As shown Figure both BERTScore and ROUGE-L improve as increases, peaking at T. 10, then gradually decrease. This trend arises because shorter segments lack sufficient information, which weakens the judgment of the verification component and limits the mutual inspiration between models. Conversely, overly long segments reduce the frequency of cross-model interactions, hindering effective knowledge fusion and ultimately degrading the final output quality.
28
20.03
377
12 7347
7225
13344
1349
37333
20.25
21.0
22337
72.00
03
1.5
Figure 5; Performance trends of SpecEM. Left: Varying maximum candidate segment length. Right: Varying hyperparameter in the online feedback.
Online Feedback Hyperparameter Analysis. To investigate the impact of the hyperparameter in learning rate in online feedback mechanism, vary its value from 0.1 to 2 and evaluate the

---

results on the English FuseEval development set. As shown in Figure both BERTSccor and ROUGE-L scores initially increase with larger a, stabilize around the range of 0.8-1.3, and then begin to decline. A very small leads to insufficient updates, limiting the effectiveness of online feedback, while an overly large a causes instability and degrades performance.
Conclusion
We propose SpecEM, a training-free and alug-anddplay ensemble framework for generative LLMs. SpecEM integrates model outputs through an iterative crafting-verifiattinn process at the segment level, enabling semantic collaboration across models without additional training. Further, in- troduce an online feedback mechanism that dynamically adjusts each model's influence during generation based on real-time performance. Experiments across six datasets and five LLM families ranging from 7B to 72B parameters show that SpecEM can effectively coordinate multiple LLMs, demonstrating strong generalization across model scales, task types, and languages.
Limitations
Although SpecEM enables flexible integration of new models without additional training, it still faces challenges in the ensemble process. Specifically, introducing model that performs poorly on the current task can degrade the overall performance. sometimes even falling below that of an ensemble excluding the weaker model. For instance, as shown in Table 2 on the IFEval dataset, SpecEM integrating Qwen2, GLM4, and Gemma2 underperforms compared to the ensemble of only GLM4 and Gemma2. While we adopt an online feedback mechanism to dynamically adjust each model's 's contribution, low-quality outputs from weaker models can still negatively affect the initial generation phase. In future work, we plan to explore rejection sampling and resampling strategies to more effectively identify and amplify stronger models in the generation process, while suppressing the influence of weaker ones.
7 Acknowledgmentss
This work was supported by the National Key Research and Development Program of China (No. 20227D01 15301). Major Key Project of PCL via grant No. PCI.2025AS11, National Natural Science Foundation of China (NSFC) via grant 62206140,,2406223,, Thanks for the support provided by OpenI Community (https://openi, .ww.apeem.pc....cn.....
References
AI@ Meta. Llama 3 model card, 2024. htg.lltpp//gggttuu.co/////////a/ main/KODEL CARD md.
Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Wang, et al. Coig-eqia: Quality is all you need for chinese instruction fine-tuning, 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Man-iaatiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, 2023, URL
btttp:://arxiv.orgge.. wwwp///////aaaaaaaa//a//eaeeeee/ /00p./ 01318
Zhijun Chen, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Dingqi Yang, Hailong Sun, and Philip Yu. Harnessing multiple large language models: A survey on Ilm ensemble, 2025. URL https: // aaxxivv org/abs. abs/abbs/ 22022 18036
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. https://:
org/ abs/ 1803. 05457
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñichrro Nakano, Christopher Hesse, and John
10

---

Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv abs/2110.14168
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi. Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tunnd 1lm. 2023, URL https: wwwp://wwwww.oo databricks com http://ooo/222/////////
Solll-first-open- www.eeccmommrrcallco....a..aaa.e wi-iiaalllvvbblllee.e. t
Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. Dealing with disagreements:
Looking beyond the majority vote in subjective annotations. Transactions of the Association for
Computational Linguistics, 10:92-1 10, 01 2022. ISSN 2307-387X.
doi: 10.1 10.1162/ a 00400,
URL
1162/tac1ccc../ a_0499
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. (RRLattps://rxxi.orrg
1810.04805
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint 0rXiv:2404.04475, 2024.
Yu Fei, Yasaman Razeghi, and Sameer Singh. Nudging: Inference-time alignment of Ilms via guided decoding, 2025.
http::////wxxwv.ooaaaaaa arg/abs//// 241///bss../..../// 2410. 09300
Markus Freitag, Behrooz Ghorbani, and Patrick Fernandes. Epsilon sampling rocks: Investigating sampling strategies for minimum Bayes risk decoding for machine translation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9198-9209 Singapore, December 2023. Association for Computational org/
Linguistics. doi: 10. 10611/v1112222 2023. findiffinding-eeii emn 617
http://ww00200..innnneeemmm.ppp findings-emmll.611. URL http::///caaaattto/l/o////
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings, 2022. URL httt:httts//a//te wwtp:/o////bbbb/////// 08821
Team Gemma, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, and Pouya Tafti et al. Gemma 2: Improving open language models a practical size, 2024. URL httht:ttt////aittt orw. hwwgg/bbs24408... 2/0///00000000188
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng. Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang. Jiadai Sun, Jiajie Zhang, Jiale Cheng, and Jiayi Gui al. Chatglm: A family of large language models from glm-1300 glm-4 all tools, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL httpp::////et //raxiv org/abs: 2ts//bb//009 .33000
Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Bing Qin, and Ting Liu. En abling ensemble learning for heterogeneous large language models with deep parallel collaboration, 2024.
Albert Q. Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Maric-Ame Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed al. Mistral 7b, 2023a. URL hhttps://arxx: org/ h/tg/ab///0 /abs/2310... 20111.082000.....1111
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blenders Ensembling large language models with pairwise ranking and generative fusion, 2023b. URtssttss//arxii.vo
wwt..orgg..../.......p// 2306. 02561
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. 2023a. URL https: //tp///rxvvvv///////a/ org/abs// http//bb/221.... hbss/221..7799 17192

---

Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. 2023b. URL hRttps::://axx http://rtwi..oog////////11/ atggaba//211..22. 17192
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising seuuuence-tosequeeee pre- pre-training for natural language generation, translation, and comprehension, 2019. URL http::httt:/rrtt org/ abs 13461
Yew Lin. Rouge: A package for automatic evaluation of summaries. Text summurization branches oul, pages 74-81, 2004.
Bo Lv, Xin Liu, Shaojie Dai, Nayu Liu, Fan Yang, Ping Luo, and Yue Yu. DSP: Discriminative soft prompts for zero-shot entity and relation extraction. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 5491-5505, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 10.18653/v 1/2023. 1/2002..nnnnn.n.-s-ea....a 339/
URL hatpa////o//aaaoloooooooo hrtg:.orr///0/ 2023. wtt.0....inniiio findings- acl
Bo Lv, Xin Liu, Kaiwen Wei, Ping Luo, and Yue Yu. TAcKD: Teacher assistant enhanced knowledge distillation for closed-source multilingual neural machine translation. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Joins International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15530-15541, Torino, Italia, May 2024a. org/ 2024. wrec-mainn..... 13tee-/0aan..100//
ELRA and ICCL. URL httpstts::// ttttpp::/aclaanthollo
org/2024.
Bo Lv, Chen Tang, Yanan Zhang. Xin Liu, Ping Luo, and Yue Yu. URG: A unified ranking and generation method for ensembling language models. Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors. Findings of the Association for Computational Linguistics ACL 2024, pages 4421-4434, Bangkok, Thailand and virtual meeting, August 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2022 finiiiiiinnngaa..2.... URL atttps:/acllatthlllee org. 2024 findings-acl.22 261
Bo Lv, Nayu Liu, Yang Shen, Xin Liu, Ping Luo, and Yue Yu. Whether LLMs know if they know: Identifying knowledge boundaries via debiased historical in-context learning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 19516- 19528, Vienna, Austria, July 2025, Association for Computational Linguistics. ISBN 979-8-89176-255-3 doi: 18.18653/v1/ 2025. 2025.findings-al..og
URL llttts::/aalannthooo.g org/ http://222....fffid wwt.:r/r//.......inneeooo.e findings--cll..... 999/
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang. Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedingy of the 29th ACM Interna- tional Conference on Architectural Support for Programuning Languages and Operating Systems, Volume 3, ASPLOS 24, page 932 949. ACM, April 2024. doi: 10.1145/3620666.365111 3651335
URL
http:// dx. doi orgo 10.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, and Sam Altman et al. Gpt-4 technical report, 2024.
URL tattts://arzzen org. /233. 087744
Kishore Papineni. Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of ACI., pages 311331, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.33155/10733033..77333 URL
.rg.ey...rrr/ee... wwt. 1040
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with 03277
gpt-4, 2023. wwkllattpp://arxtt.orro////
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin
12

---

Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang. and Zihan Qiu. Qwen2.5 technical report, 2025. URL
https://arxiv. wwgp://ii.rrreeooo abs /tabs//411. 15115
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport, 2024. URL httph::tt//arttt org/ abs/ 2310. 15141
Mistral Al Team. Mistral small 3: Apache 2.0, 81 URL
aestralllmaall
Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. A stitch in time saves nine: Detecting and mitigating hallucinations of Ilms by validating low-confidence generation,
2023. Uttpttss:://art //axxivv///xxv.../// org/ abs abs/2307. .39877
Junlin Wang. Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabiliiies, 2024. URL https://arxiv.. wwwpa/wwwwww...a///a/aaaaa/a///e 04692
Heming Xia, Tao Ge, Peiyi Wang. Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding:
Exploiting speculative execution for accelerating seq2seq generation, 2023. UtLhhtts:://axx org/abs 2203. 16487
Yangyifan Xu, Jinliang Lu, and Jiajun Zhang. Bridging the gap between different vocabularies for Ilm ensemble, 2024. URL htttps://aaxii wwtrrxivv.vooroee org abs/ 2404./444..00 09492
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, and Haoran Wei et al. Qwen2 technical report, 2024. httpp:t:s///ars
org/ hts://bs2220 2200 10671
Yuxuan Yao, Han Wu, Mingyang Liu, Sichun Luo. Xiongwei Han, Jie Liu, Zhijiang Guo, and Linqi Song. Determine-tthen-ensmmbbee Necessity of top-k union for large language model ensembling, 2025. UtL/attss://aaxi/ org abs httpg///aa//21.21/1111
Yao-Ching Yu, Chun-Chih Kuo, Ziqi Ye, Yu-Cheng Chang, and Yueh-Se Li. Breaking the ceiling of the Ilm community by treating token generation as a classification for ensembling, 2024. URL
https://arxiv. www.aaiwww...aaaaaaaaaaaaaoooeae org.abs..rrr/ba//b 2b4/ 12585
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances Neural Information Processing Systems, volume 34, pages 27263-27277 Curran Associates, Inc., 2021. URL https:::::: httt:p:://proceeding ottp///atgg///ppppeeenonnnooooo. www.nnnnrii... wttp://nww........e.appeeeeeeeee wttpc///pppp//////11222
@812566ffca30000ff2216666666 Paper .dff]
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. CoRR, abs/1904.09675, 2019. URL http:///aaxxvv.og/abbba
09675
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followii evaluation for large language models, 2023. URL
https://arrx: wwwpl//..rg/bbb///aeee htgg/abb/2311.099.1
13

---

A Empirical Validation of the Correlation Between Generation and Verification Capabilities of LLMs.
Compartson Win Against LLaMAS
asdead MaMMA Wis flane 2.MMA
(19)
30,9
Qweel-B
Crams 2-88
4-8
Figure 6: Win rates of four LLMs compared -iama---BB-instruet,, evaluated by GPT-4. Orange bars indicate generation performance (as response generators); Blue bars indicate verification performance (as candidate scorers for LLaMA). The consistent ordering supports the hypothesis that stronger generators are also stronger verifiers.
we hypothesize that models with stronger generation capabilities also possess stronger verification abilities, to conduct online feedback mechanism in SpecEM. To verify the assumption, we design an empirical experiment. Specifically, we use GPT-4 as an external evaluator, and evaluate four models, 1estral-7B-v03-instttt by comparing their responses against those generated by Llama-3-88 --iitinsttuct on FuseEval. The 6 report the win rates of these models over LLaMA, judged by GPT-4. The
Section 3.3
Qwen2-7B-instreee
Gemma-2- 2irmma-2-bb-instruct, and Glm-4-9b-instruct,
orange bars in Figure results indicate a consistent performance ranking: Mistral > Qwen > Genma GLM, with all models outperforming LLaMA.
To assess verification ability, we treat each of the four models verifier For every input example, we sample four candidate responses from LLaMA. Each verifier independently scores all candidates using the method described in Section 3.2,, and selects one with the highest score. The selected responses are then evaluated by GPT-4 against the original single-response baseline from LLaMA. The blue bars Figure 6 show the win rates the verifier-selected responses, again consistently ranked Mistral Qwen Genma GLM.
This alignment in performance ordering across generation and verification supports our central hypothesis: models with stronger generation capabilities are more competent as verifiers. For example, Mistral not only achieves the highest generation win rate, but also, when acting as verifier, selects the strongest responses for LLaMA, demonstrating its superior verification ability.
B Experimental Setup
B.1 Dataset Details
We evaluate all the models on six datasets that represent different core capabilities of L.LMs, including open-domain instruction-followine commonsense, and reasoning.
14

---

FuseEval: We evaluate the model's instruction-responee capability by constructing this category on both English and Chinese scenarios. For English, we choose the Dolly- 15k (Conover et al. 2023 and Alpaca-gpt4 (Peng et al. 2023] datasets for evaluation, both of which have inputs that consist of human instructions. We select these two datasets because their response sources differ: the Dolly-15k dataset features Himan--roovdded responses, while the Alpaca-GPT4 contains responses generated by the state-of-the-ar GPT- 4 [OpenAl et al. 2024] model, which provides neutral reference answers to each question and can refuse to answer inappropriate or harmful ones. Using both types of responses for scoring allows to more thoroughly compare the advantages of our ensemble system. Additionally. due to the large size of these datasets, we randomly sample portions from each to create a new test set and a development set. For Chinese,
utilize the Human- Value
and Ruozb datasets from the COIG-CQIA [Bai et al. [024] benchmark for testing. The instructions in these two datasets consist of human-posed questions, with answers provided either by humans or generated by GPT-4. The COIG-CQIA authors manually review and filter the responses, retaining only the correct answers generated by GPT-4
Al AlpacaEval 2.0 Dubois et al. 2024): This is an automated benchmark for evaluating large language models' instruction- wollowwnoooogggggoo capabilities. employs GPT-4 Preview (11/06) as an evaluator to compare model responses against a baseline (also GPT-4 Preview (1 (11/0))) computing win rates with length-controle
scoring mechanism to reduce verbosity bias. Given the strong performance of 24-72B foundation models, we further conducted direct comparisons between ensemble model outputs and GPT-4 generated responses.
MMLU (5-shot) [Hendrycks al. 2021]: A widely-used massive multitask language understanding benchmark for evaluating knowledge and commonsense reasoning across 57 subjects, including STEM, humanities, and social sciences. It assesses models' breadth of understanding across a diverse set of multiple-choice questions. x-shot refers to providing x examples as in-context during inference.
ARC-C (5-shot) [Clark et al. 2018]: A subset of the A12 Reasoning Challenge benchmark
consisting of grade 3-9 science exam questions that require non-trivial logical reasoning.
The task is formulated as a multiple-choice question answering problem.
GSM8K (3-shot) [Cobbe et al. 2021]: A high-quality dataset of linguistically diverse grade school math word problems, curated to evaluate arithmetic reasoning capabilities. Each question requires multi-step reasoning to arrive at the correct solution.
IFEval [Zhou et al., 2023): A targeted benchmark for assessing instruction Dastructionffolloiing following proficiency. It contains prompts with explicit directives and uses GPT-4 to evaluate how well model outputs comply with the given instructions.
B.2 Evaluation Methods
We use a variety of metrics for different tasks, following the test scripts from the Openlim leaderboard. For FuseEval, we apply BARTScore (Bart-S) Yuan et al. 202T], BERTScore (Bert-S) (Zhang et al. 2019), GPT4-Rank Rank (GPT4-R) [OpenAl et al. 2024]. B BLEU [Papineni et al. 2002]. and ROUGE (R-n) [Lin 2004]. For multiple-choice tasks such as MMLU and ARC-C, select the option with the highest likelihood to calculate accuracy (Acc). For the reasoning dataset GSM8K, we evaluate exact match (EM) accuracy. For IFEVAL, we rely on the evaluation files provided by the dataset creators [Zhou et al. 2023). testing under prompt-strict, instruction-stric strict, prompt-loose and instruction-loose conditions. For AlpacaEval 2.0, we use the official GPT-4-based pairwise comparison framework [Conover et al. 2023). where each model's 's output is evaluated against GPT-4 reference response, and the win rate is computed as the final metric. A detailed description of some evaluation metrics for FuseEval is as follows:
BLEU (B-n) (Papineni et al. 2002 and ROUGE (R-n) [Ln 2004] compare a generated response with a reference by calculating n-gram overlap. For the Chinese results, we use JiebaPto to split the text into words before calculating these two scores.
BERTScore [Zhang et al. 2019 (comprising Precision, Recall, and F1-score) measures the similarity between two texts based on the contextualized embedding from BERT (Devlin et al. 2 In this paper, we report the Fl score of ScoTe
15

---

BARTScore (Yuan et al. 2021] is a unified evaluator which evaluates with the average
likelihood of the pretramed encoder-decoder model, BART [Lewis et al. 2019]. It can predict different scores depending on the formats of the source and target.
The GPT4-Rank [OpenAl et al. 2024 evaluation utilizes the GPT-40-2024-11 1-20 model compare two different responses against ground-truth response. The model will select the better of the two responses. For each test sample, we pair the responses generated by different models and have GPT-4 determine which one is superior. Since the MBR and PairRank methods do not generate new responses, we do not re-rank the responses they select from the base LLMs. Instead, we use the average rankings the responses they select from the base LLMs to represent their BPT4-Rank. Once all comparisons are complete, we count the number of wins for each model. Based on these win counts, we rank the responses from the different models. The average ranking of each model across all data in the dataset is the value reported in our table. The evaluation instructions for GPT-4 are shown in Table
The win rate comparisons between models in this study were conducted using GPT-4o-2024- 11 11-20 the evaluator. Both Table 3 and Table 4 employ GPT-40-2024-11120 to compare outputs from different models, with the evaluation instructions for GPT-4 shown in Table S In Table 3: The "English FuseEval winrate" and "Chinese FuseEval winrate" metrics compare the outputs of various base models (and ensemble methods) against those generated by Qwen2_722 72b _nnstuuttt...
B.3 Baselines
Since our approach has not undergone any additional training, we selecte several types of untrained baseline models for comparison with our method:
PairRank: An English reward model introduced in the H.M--lender (Jiang et al., 2023b), which compares candidate results generated by different LLMs and selects the best candidate as the ensemble output.
Majority Voting [Dvann et al. 2022]: ]: each model provides a choice, and the final result is determined by the option with the most votes.
Minimum Bayes Risk (MBR) (Freitag et al. 2023] : Selects the answer with the highest lexical similarity to other candidate answers. In this paper, we use the SimCSE (Gao et al.. [022 model to calculate the similarity between candidate responses.
Generation Fusion (GF) [Dang et al. 2023b]: Uses the outputs of other models as context, passing them a new model, which generates a response based on this context. In our implementation. Mistral-7B-v0.3 is employed as the final model for the 7B-9B scale integration, and -n5stutt2----s for the 24B-72B scale integration, based on their performance advantages.
(MOA) Waag et al. 2024 Multi-layer fusion is applied, where the outputs of all base models concatenated and fed back into the models, with aggregator Mistral-7B-vit
outputting the final result. In this work, we adopt the grronger-performen the aggressor in the 7B-9B scale integration, and the better-performing Mistral-24BB instruct-2501 as the aggressor in the 24B-72B scale integration. The fusion process is repeated three times, consistent with the original MOA methodology.
Unite Yao et al. 2025]: Constructs a new union vocabulary by combining the vocabularies of multiple models to include all tokens from each model.
C Case Study
Table presents case from the SpecEM workflow where the user's request is "Write simile to describe person who is hard-working." The reasoning process goes through four iterations, and the verify model's selection of the best candidate is not always from the same model. In the first round, the best candidate is generated by Qwen2. In the second round, Mistral, after receiving Qwen2's output from the previous round, is inspired and generates response that better meets the user's needs, as using "farmer" to describe hard-working person is inappropriate. Additionally, the table shows
16

---

Table 5:
The template used for GPT-4 compares two models' responses.
Instruction:
S[instraction]
Ground-Truth Response:
Iruuh response)
S(A respouse]
Template
S/B response]
Given the User's Instruction Ground-Truthh response please compure Model's responses.
You only have choices to output:
If you think A is better, please output: 1, A better
you think better. please output:
Output your choice below:
A is better
Comparison Option
that through mutual inspiration between models, the final response more closely matches the user's expectations, thereby improving the overall quality of the reply.
D Discussion on Segment Length
SpecEM employs fixed-length candidate segments for model evaluation and routing, which simplifies the computation process but may occasionally cause semantic truncation, particularly when segment boundaries intersect with meaningful linguistic or contextual units. This truncation can weaken semantic coherence and reduce the accuracy of optimal segment selection. In contrast, adaptive seg- mentation approaches such as NUDGING Fei et al. (0255 dynamically determine token boundaries based uncertainty estimation across model families. However, these methods assume shared vocabulary and tokenization scheme, which restricts their applicability in heterogeneous large-model scenarios where model architectures and vocabularies differ. Future work could explore adaptive or semantics-awan -wware segmentation mechanisms that maintain context completeness while remaining compatible with multi-man model routing frameworks.
17

---

User instruction: Write simile describe person who hard-working.
Input for each round
Qwen2 Generation
Mistral
Gim
Candidate
Round I:
Write simile desemibe person who
who hand- working detemmined
This dodiseteed
their
A hand-working like
who lard- working like detemnined
hard working
Round 2:
simile
person Barmeer, tirelasily cullivating the and nurturing crops
hard- working penson hard-
horse pulling heavy through long Journey.
working g is like determined
stcadily curving path through the
river, stcadily curving its path through the
Round
Write a simile to deseribe person who hard working, penson who is hard- working is like determined river, stcadily curving path through
rockiest landscapes. <jm end?
rocks and obstaeles its way, nover giving up
rocks and obstacles way, never giving up
mountaiins
Round d:
Write a simile describe person who hard working. penson
working is like a determined river, steadily
carving path through obstacles in its way, never giving
until reaches its destination, +ig>
until renches the seu.
+jusce
until it reachus its destination.
Overall Output: A person who is hard-working like determined river, steadily carving its path through the rocks and obstacles in its
way, never giving until it reaches its destination.
Figure 7: Case study of SpecEM integrating the base LLMs Qwen2, Mistral, and GLM-4. The Best Candidate the top-ranked option determined by the verify component and is directly presented to the user. < lim [in__nnd] >, < /s and < |user| > are special end tokens for the three base LLMs, and generation halts when the best candidate includes any of these end tokens.
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information how each question in the checklist:
these questions. For
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use checklist as one the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] proovided proper justification is given (e.g., "error bars are not reported because would be too computationally
expensive" or "we were unable to find the license for the dataset we used"). In general, answering
18

---

"[No] or "[NA] " is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
.
Do not modify the questions and only use the provided macros for your answers.
1.
Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction provide a comprehensive overview of the background and motivation of this study, effectively outlining its main contributions point- by-point, thus accurately reflecting the paper's scope and significance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We primarily focused on discussing the limitations associated with this study in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
19

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The theories used in this paper are all based on well-established algorithms, with detailed derivations available in the cited literature.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All information regarding the key contribution of this paper have be fully disclosed (to the extent that it affects the main claims and/or conclusions of the paper). Fur- thermore, the implementation of other components within the proposed SpecEM framework is facilitated by the plenty of support available from existing open-source resources within the community.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
20

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The supplementary material accompanying the manuscript contains all source code and scripts necessary reproduce the main experimental results. Detailed instructions for execution are embedded within the scripts.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc)
guuides/CodeSubmisse
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ wttppccccpppcccccctpcccc.cc.cclt http://gweeeeeeeee http:///eeeeooosssssoooeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides detailed experimental configurations in Section and and Appendix B offering readers the necessary information to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
21

---

7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Since our framework requires no training, it avoids potential errors introduced by the training process. Moreover, all experimental results in our paper averaged over multiple runs to reduce randomness in model generation and ensure stable measurements.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: As mentioned in 4. for ensembles involving models with fewer than billion parameters, we use 8xA100 GPUs with 80 GB memory. For larger models with billion parameters more, utilize 4xH200 GPUs equipped with 140 GB memory.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: After carefully reviewing the referenced document, we certify that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
22

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper focuses on improving response quality and reducing bias via model ensembling, with a societal impact comparable to that of deploying a strong single LLM, while introducing no additional societal risks beyond those already associated with standard tigh-performing models.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification: The datasets used in this study, such as AlpacaEval, MMLU, and GSM8K, well the pretrained models such as Qwen and Llama, sourced from open-access platforms like Hugging Face. These resources have undergone comprehensive safety as- sessments and are widely utilized within the research community. All datasets and models employed in this work are publicly relcased, well-documented, and subject to extensive community serutiny, ensuring their reliability and appropriateness for academic research.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
23

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In the paper, we clearly specified the datasets and code sources used, and pro- vided appropriate citations in the reference section. Additionally, ensured transparency by including the original sources of any modified code files, making the changes traceable.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URI..
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcooeeeooo thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The supplementary materials contain the full source code and comprehensive usage instructions. Following the conclusion of the review process, the code will be made publicly available to support transparency and reproducibility within the research community.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]

---

This study does not involve any crowdsourcing experiments or research with human subjects.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing experiments research with human subjects were involved in this study. All experiments were conducted using code and GPU servers.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality the research, declaration is not required.
Answer: [Yes]
Justification: We did not design or train new large language models. Our method aims to improve system performance based on existing L.MMs. In Section 4.1 (Base L.Ms) and Appendix B we describe how use existing open-source large language models.
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs any important, original, non-standard components.
Please refer to LLM policy for what should or should not be described.
25


Reviews:
[{"review_id": "kiHuiT6sbC", "reviewer": "Reviewer_zJQy", "summary": "This paper proposes SpecFuse, a training-free, plug-and-play LLM ensemble framework designed to address several key limitations in current large language model (LLM) ensemble techniques. Rather than relying on token-level aggregation or post-generation fusion, SpecFuse employs a segment-level iterative process that includes drafting, verification and online feedback three stages. It is evaluated on six benchmarks (including FuseEval, MMLU, ARC-C, GSM8K, IFEval, AlpacaEval 2.0) and five model families (7B to 72B), consistently outperforming individual models and strong ensemble baselines (MOA, UniTE, MBR, PairRank, etc.). The framework offers both accuracy gains and latency improvements.", "questions": "See weaknesses.\n\n1. Equation 3 normalizes scores across models to mitigate bias (e.g., due to logit scale differences), but this assumes linear comparability. Is this sufficient when combining models with heterogeneous architectures, tokenizers, or calibration characteristics (e.g., mixing a 7B and 70B model)?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. SpecFuse is training-free and plug-and-play. It can directly incorporate diverse LLMs without any task-specific fine-tuning, making it highly accessible for practical use.\n\n2. Strong empirical validation: The method is tested on five major LLM families and six benchmarks, spanning open-domain instruction following, knowledge and reasoning (MMLU, ARC-C, GSM8K, IFEval), and multilingual contexts (English/Chinese). These results show consistent and sometimes large gains over both strong individual models and state-of-the-art ensemble approaches.\n\nWeaknesses:\n\n1. The verify-in-line mechanism requires direct control over attention masks and position IDs, which may not be feasible when using closed-model APIs (e.g., GPT-4, Claude).\n\n2. Most evaluation is conducted via automated metrics, with no human inspection of qualitative failure cases. It remains unclear how SpecFuse handles incoherent, contradictory, or hallucinated candidates, or whether it produces undesirable blends of inconsistent partial segments.\n\n3. As described in Figure 5, the choice of segment length L=10 is reported as optimal based on development set results. However, it remains unclear whether this setting was tuned primarily on short or mid-length inputs. In long-form generation tasks that require extended reasoning or discourse coherence, such a short segment length may lead to fragmented context understanding or premature truncation.", "was_revised": true, "final_justification": "Given the rebuttal and other reviews, I have updated scores and reviews accordingly.", "rebuttal": "Dear Reviewer zJQy,\n\n**Thank you for your thorough review and feedback on our manuscript. We truly appreciate the time and effort you dedicated to understanding our work, and your constructive comments. Below, we respond to each of the concerns you raised. We hope this addresses your concerns and helps further improve your evaluation of our work.**\n\n\n**[Weakness 1：The verify-in-line mechanism may not be feasible when using closed-model APIs (e.g., GPT-4).\n]**\n\n* Thank you for pointing this out. We acknowledge that the Verify-in-line mechanism requires access to token-level logits and attention masks, which may not be directly accessible when using closed APIs. However, we note that SpecFuse can still be applied to partially closed models like ChatGPT, which expose a limited number of output token-level logits via their APIs. In such cases, we can adopt LLM-as-a-judge strategies [1] to enable compatibility.\n\n\n  Specifically, the core idea of the Verify-in-line method is to obtain normalized token probabilities for all candidates in parallel, thereby improving inference efficiency. \n\n  Suppose we ensemble three models: A (ChatGPT), B and C (open-source models), which produce Candidate 1, Candidate 2, and Candidate 3, respectively. We can construct an input prompt that concatenates the candidates (e.g., [A.Candidate1; B.Candidate2; C.Candidate3]) and instruct ChatGPT to choose the best one from A/B/C. The logits over the choices (A, B, C) provided by ChatGPT's API can then be normalized and used as scores, which are compatible with the Verify-in-line framework and can be combined with scores from open-source models B and C. \n\nWe hope this clarification addresses your concerns, and we sincerely appreciate your insightful feedback!\n\n\n**[Weakness 2: How SpecFuse handles incoherent or hallucinated segments]**\n\n* Thank you for your insightful comments regarding the potential risks of incoherence or undesirable blending when combining candidate segments. In our observations, such issues are rare in practice. This is primarily because each segment is selected through an interactive scoring mechanism that considers the context formed by previously chosen segments. The mechanism scores candidates using logits-based values, which is strongly correlated with model perplexity, to evaluate each candidate’s semantic relevance to the preceding text and its overall fluency.\nIntuitively, this means the system favors segments that are more semantically coherent. In other words, if a candidate segment exhibits high perplexity, such as due to semantic contradiction, it is less likely to be selected.\n\n* To further illustrate the robustness of our method, we construct a controlled case. Specifically, we ensemble four candidates: three generated by Qwen2, Mistral, and GLM-4, and one intentionally crafted as a noisy, contradictory input.\n\n  |      | Qwen2  | Mistral  | Glm-4  | Human   Noise | Best   |\n  |-----------------|----------------------------------------------------|------------------------------------------------|---------------------------------------------------|------------------------------------------------------|----------------------------------------------------|\n  | Round   1 Text  | A   person who is hardworking is like a determined | This   person is as dedicated to their work as | A   hard-working person is like a well | Title:   The Cowardly Man| A   person who is hardworking is like a determined |\n  | Round   1 Score | 0.292 | 0.2535 | 0.259  | 0.1955|  |\n  | Round   2 Text  | seed   that refuses to give up until it grows      | river,  carving its path through the | oak,   roots sunk deep in the earth               | slacker   waiting for success without doing anything | river,   carving its path through the    |\n  | Round   2 Score | 0.2609 | 0.312| 0.2646     | 0.1717  |   |\n  | Round   3 Text  | rockiest   of landscapes.  | mountains.   | rocks   and obstacles in its way, never giving up | couch   cushions while watching TV.   rocks   and obstacles in its way, never giving up  |\n  | Round   3 Score | 0.2776     | 0.2388  | 0.312   | 0.1717 |                                                    |\n  | Round   4 Text  | until   it reaches the sea.  | until   it reaches its destination | until its strength fades away.  | …                                                    | until   it reaches its destination    |\n  | Round   4 Score | 0.2749                                             | 0.3765 | 0.2397   | 0.1089   |   |\n\n  As shown in the table above, across all decoding steps, SpecFuse consistently selects segments from high-quality candidates while assigning progressively lower scores to the human-injected noise. This demonstrates that our verification mechanism effectively filters out incoherent segments during the decoding process.\n\nWe hope this response addresses your concerns, and we are happy to discuss further if needed. We appreciate the reviewer’s helpful suggestion and will consider including such illustrative examples and options in future releases.\n\n**[Weakness 3: Risk of Fragmented Context Understanding or Premature Truncation]**\n\n* Thank you for raising this important point regarding the impact of segment length on long-form generation. We hope our response to Weakness 2 already addresses part of this concern by clarifying how SpecFuse maintains coherence across segments.\n\n* From a methodological perspective, SpecFuse can be viewed as a greedy algorithm that assembles the most probable output sequence by selecting among candidate segments generated by all ensemble models at each step. In contrast, a standard single-model decoding process constructs an output sequence by only its own produced candidates. As we explained in our earlier response of Weakness 2, previously selected segments are shared as context with all models. During the drafting stage, each model conditions on this shared context to produce its next candidate. In the verification stage, candidates that are semantically inconsistent with the preceding context are unlikely to be selected.\n\n* On the experimental side, our development set for FuseEval includes tasks requiring long-form generation, such as story writing prompts that ask for narratives exceeding 2,000 tokens. We found that a segment length of L=10 performs robustly across both short and long contexts and yields stable improvements over baselines.\n\n*  Furthermore, expanding the search space (e.g., via beam search) could help mitigate the risk of premature truncation. While our current implementation adopts a top-1 selection strategy, we plan to offer beam search as an optional feature in future versions of our code. This would allow users to balance computational cost against the potential for more globally optimal segment combinations.\n\nWe hope this clarification addresses your concerns, and we sincerely appreciate your insightful feedback!\n\n\n**[Questions 1：Is this approach sufficient when combining models with heterogeneous architectures, tokenization schemes, or calibration characteristics?]**\n\n* Thank you for your careful inspection. \n\n  First, we would like to acknowledge a typographical error in Equation (3): the index i=1 should actually be j=1, where j refers to the model that generated the j-th candidate. This correction is consistent with our implementation, as shown in the model_verify_in_line function in the provided utils/normal_utils.py script. We apologize for the oversight and appreciate your understanding.\n\n* Building on this correction, we would like to clarify that the normalization in Equation (3) is specifically designed to address potential logit scale disparities across different models. Without normalization, models that systematically produce higher or lower logit values may dominate the final ensemble score, skewing the selection unfairly toward one model. To mitigate this, equation (3) normalize the scores that each model assigns to all candidates before aggregation, effectively removing the impact of differing logit scales. For example, suppose three models a, b, and c produce scores for the current candidate set as follows: a: [2.3, 2.6, 3.2], b: [1.4, 1.6, 2.2], c: [0.6, 0.3, 0.5].\n\n\n  Though their score distributions differ in scale, equation (3) normalize each model’s scores over the candidates: a: [0.284, 0.321, 0.395], b: [0.269, 0.308, 0.423] , c: [0.429, 0.214, 0.357].\n\n  This approach ensures that no model disproportionately influences the final decision due to inherently larger or smaller logits. While this normalization assumes linear comparability, our experiments suggest it is sufficient in practice (24-72b LLM ensembling in Table 3), and is not affected by models with different architectures and tokenizers.\n\nWe thank the reviewer again for the valuable feedback, and we will make the clarification more explicit in the final version.\n\n\n**Thank you for taking the time to read our comments. If we have addressed your concerns, we sincerely hope you can reconsider and improve your evaluation of our work. If you have any other questions, we look forward to further discussions with you. We sincerely await your reply.**"}, {"review_id": "wsGhg1NuhZ", "reviewer": "Reviewer_tSy8", "summary": "This paper proposes a training-free, plug-and-play LLM ensemble method, SpecFuse, that integrates multiple models' outputs by iteratively coordinating drafting and verification with an online weights feedback mechanism to adjust the influence of each model based on their contributions. By interleaving drafting and verification at the segment level, it keeps first-token latency near that of a single model while adding only ~20 % runtime overhead. Tested on six instruction-following and reasoning benchmarks with 7-72 B models, SpecFuse outperforms base models and state-of-the-art ensemble baselines.", "questions": "1. Can you explain why to mitigate scoring bias caused by some models producing systematically higher or lower logits the scores are normalized across the **model dimension** (I think this might be a typo)? \n   1. Combining quation 3 and 4: $y^k_j = \\sum_i^N w_i^k \\frac{s^k_{i,j}}{\\sum_i^N s^k_{i,j}}$, so at the beginning when you have all weights are equal, $y_j^k = \\sum_{i=1}^{N} \\frac{1}{N} \\frac{s^k_{i,j}}{\\sum_{i=1}^{N} s^k_{i,j}}=\\frac{1}{N}$, meaning that all candidates will have the same score ignoring completely the scores. Is this expected?\n   2. If model i produce very high logits, i.e. $s_{i,j} \\rightarrow \\infty, \\forall j$, by equation 3, $s_{i,j} \\approx 1$ for i and other r, $s_{r,j}\\approx 0$ for all j, then $y_j$ would be dominated by the model i's judgement\n2. Why in table 3 SpecFuse improve Chinese FuseEval so much? A deep dive into this might help illustrate why SpecFuse is effective.\n3. Slight abuse of notation in equation (3), consider using hat: $\\hat{s}^{(k)}_{i,j} \\leftarrow ...$\n4. Duplicated reference line 378 380\n5. Other typos: end of line 4, beginning of line 230, 303", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Quality\n  - Overall solid work with comprehensive experiments on performance and efficiency, comparison with prior methods and ablation studies, clearly outlining the advantages of SpecFuse\n  - Although the authors claim that the online feedback mechanism ensures stronger models exert greater influence in the ensemble, this claim is only supported by final performance metrics, without any analysis or visualization of how model weights actually evolve during generation.\n  - The learning rate is designed to ensure that the updates become more stable over time (k). However, I am not sure if this would handle situations where there's code switching (First English, then later Chinese; or first English then code at some point). The concern is different models are good at different aspects and the design of weights might not allow it to adapt to changes of tasks during generation.\n- Clarity\n  - The paper is generally well written, logically organized, and easy to follow.\n  - The motivation behind multiplicative weight update is a bit unclear\n    - Why are small initial weights a problem at all? Line 175\n    - Why is the learning rate designed this way? Equation (7)\n  - There are a few typos, formatting or notation issues (see questions/suggestions).\n- Significance\n  - Interesting idea of combining models at the segment-level with unsupervised online weight adjustment, achieving strong performance with relatively low computational overhead compared to previous ensemble methods.\n  - Although the improvement on FuseEval is notable, the SpecFuse (All) for other benchmarks like MMLU, GSM8K (Table 2) is much smaller and can even hurt the model performance (IFEval). It's good that the authors touch on this in the limitation section. However, this slightly contradicts the discussion in line 288-290 \"The improvements are more pronounced when stronger models are introduced, **while even weaker models still contribute positively**.\" Given the fact that one has to host multiple models and sacrifice some inference overhead, it casts some doubt on the practical usefulness of the method.\n- Originality\n  - Fresh blend of segment-level speculative drafting and verification with online multiplicative-weights voting.", "was_revised": true, "final_justification": "I will maintain my score of 4: Borderline Accept.\nThe authors' rebuttal clarified key questions about the normalization formula and benchmark results. However, concerns remain. \n- The online weighting mechanism, due to its decaying learning rate, has limited ability to adapt to task changes mid-generation. \n- Furthermore, the learning rate's design feels heuristic and lacks an ablation study for justification. \n- The effectiveness of ensembling \"specialist\" models (where each model is strong on one sub-task but weak on others) remains a concern. \n\nHowever, overall I find the core idea of fusing models in the output space—without requiring access to model weights—to be very compelling. With the potential limitations properly discussed, I think the paper will make a meaningful contribution to the field.", "rebuttal": "Dear Reviewer tSy8,\n\n**Thank you for your thoughtful and thorough review of our manuscript. We truly appreciate the time and effort you dedicated to understanding our work, and your constructive comment. Below, we respond to each of the concerns you raised. We hope this addresses your concerns and helps further improve your evaluation of our work.**\n\n**[Weakness 1: Lack of Visualization of Model Weight Evolution During Generation]**\n\nThank you for this helpful suggestion, which will help clarify the role of the online feedback mechanism.\n\n* In response, we conducted an additional experiment to analyze how model weights evolve during generation. Specifically, we ensemble three models: Qwen2, GLM-4, and Gemma-2, and track their score-based weights at each decoding step. The task was to \"Describe the Forbidden City in Chinese,\" for which we deliberately selected a setting where one model (Gemma-2) performs relatively poorly. This is because Qwen2 and GLM-4 are pre-trained on large-scale Chinese corpora, while Gemma-2 is less proficient in Chinese.\nThe table below shows how the model weights adjust dynamically during the generation process:\n\n  | Round | Qwen2  | GLM-4  | Gemma-2 | Best Candidate Generator |\n  |-------|--------|--------|---------|----------------|\n  |0| 0.3333 |0.3333 |0.3334| |\n  |1| 0.4002 |0.2999 |0.2999|Qwen2|\n  |2| 0.3824 |0.3310 |0.2865|GLM-4|\n  |3| 0.3701 |0.3527 |0.2773|GLM-4|\n  |4| 0.3785 |0.3479 |0.2735|GLM-4 |\n  |5| 0.3708 |0.3612 |0.2680|GLM-4 |\n  |6| 0.3644 |0.3723 |0.2633|GLM-4|\n  |7| 0.3615 |0.3772 |0.2613|GLM-4 |\n  |8| 0.3591 |0.3814 |0.2595|Qwen2 |\n  |9| 0.3606 |0.3830 |0.2564|Qwen2 |\n  |10| 0.3566| 0.3898 |0.2536|GLM-4 |\n  |11| 0.3547| 0.3930 |0.2523|Qwen2 |\n  |12| 0.3575| 0.3913 |0.2512|GLM-4 |\n  |13| 0.3601| 0.3897 |0.2502|Qwen2|\n\n  As shown, the weights of Qwen2 and GLM-4 gradually increase, while the weight of Gemma-2, being the weaker model in this context, decreases over time. This demonstrates that the online feedback mechanism can effectively upweight stronger models and downweight weaker ones as generation progresses.\n\nWe sincerely appreciate your valuable question and will include a corresponding line plot of the weight evolution in the final version of the paper to better visualize this dynamic.\n\n**[Weakness2: Adaptation of the online feedback mechanism to task changes during generation.]**\n\nThank you for your insightful comment. You raise an important scenario where a single user instruction involves multiple tasks, such as switching between languages or modalities.\n\n* First, although the learning rate decays over time, model weights can still adjust meaningfully. This is because stronger-performing models continue to receive higher reward signals during the verification phase, and the final weight updates depend jointly on the reward and the learning rate. Thus, our mechanism still allows for adaptive reweighting, even in later steps.\n\n* To better handle such cases, a promising direction is to dynamically adjust the learning rate based on reward trends, increasing responsiveness when rewards shift consistently. We will continue exploring ways to improve the stability of the SpecFuse ensemble framework in complex situations.\n\nWe appreciate your thoughtful feedback and will include this in the paper’s future work section.\n\n**[Weakness 3: Why are small initial weights problematic (Line 175), and what is the rationale behind the learning rate design in Equation (7)?]**\n\nThank you for your insightful questions. We would like to take this opportunity to clarify the motivation behind our multiplicative weight update strategy.\n\n* **Regarding small initial weights :** Since the number of models N in the ensemble is not fixed, we initialize the model weights uniformly, i.e., each model starts with a weight of 1/N. This means that when N is large, each model begins with a relatively small weight. If we use a constant learning rate across different values of N, the magnitude of weight updates would differ significantly depending on the number of models. For example, with N=2, each model starts with a weight of 0.5, while with N=10, each starts at 0.1. Applying the same learning rate (e.g., 0.01) in both cases would result in disproportionately larger updates when N is small. To mitigate this, our learning rate is designed to scale with the number of models, ensuring that models with small initial weights are not over-updated in larger ensembles.\n\n* **Regarding the design of Equation (7):** The inclusion of N is motivated by the explanation above. The term k is introduced to gradually decay the learning rate over the course of generation. Since the number of generation steps k is typically larger than the number of models N, we apply a square root to k to suppress overly rapid decay and maintain stable updates in later stages.\n\nWe hope this clarifies the design choices in our update rule, and we appreciate the reviewer’s thoughtful feedback.\n\n**[Weakness 4: There somewhat contradict the discussion in lines 288–290.]**\n\nThank you for your thoughtful feedback. We would like to clarify that the results presented in Figure 4 and Table 2 are not contradictory.\n\n* Figure 4 shows that when model performance gaps are small (e.g., 23.31–25.57), even weaker models can complement others and enhance ensemble performance. In contrast, Table 2 illustrates that adding a significantly weaker model may hurt performance. For instance, on IFEval, Qwen, GLM, and Gemma score 41.7, 56.01, and 61.64, respectively. While GLM+Gemma reaches 66.89, adding Qwen drops the ensemble to 62.11.\n\n* In open-domain scenarios, the relative strengths of models are often unknown in advance. As illustrated in Table 1, Gemma performs well in English tasks, whereas Qwen2 is more effective in Chinese. This variation highlights the necessity of integrating multiple models, as doing so enhances the system’s robustness across diverse query types.\n\n  Moreover, recent advances in query-aware model selection [1], which dynamically select a small subset of base models with similar performance for each query, offer a promising way to mitigate this issue. Therefore, model ensembling remains a meaningful and practical approach.\n\nWe will revise the relevant statements in the final version to make this distinction clearer.\n\n**[Question1：Why normalize scores across the model dimension)? (I think this might be a typo)]**\n\nThank you for your careful observation and concern. \n\n* Upon review, you are correct, there is a typographical error in Equation (3): the index should be j=1, not i=1, where j refers to the j-th model that produced the candidate. This correction aligns with our implementation in the provided code (model_verify_in_line function in utils/normal_utils.py).  \nWe sincerely apologize for the oversight and truly appreciate your careful attention to detail. \n\n  \"model dimension\" refers to the normalization of the scores that a model produces for all model candidates. Based on the correction of the error in formula 3, the problems of Question1, Question1.1, and Question1.2 will not occur.\n\n* We would like to further clarify with an example:  suppose three models a, b, and c produce scores for the current candidate set as follows: a: [2.3, 2.6, 3.2], b: [1.4, 1.6, 2.2], c: [0.6, 0.3, 0.5]. \nThough their score distributions differ in scale, equation (3) normalize each model’s scores over the candidates: a: [0.284, 0.321, 0.395], b: [0.269, 0.308, 0.423] , c: [0.429, 0.214, 0.357].\nBy performing this normalization, we ensure that the ensemble decision is not biased toward models that produce higher-magnitude logits.\n\nThanks for your question again. We hope this clarifies the scoring normalization strategy and resolves your confusion.\n\n\n**[Question2: Why in table 3 SpecFuse improve Chinese FuseEval so much?]**\n\n* Thank you for the question.  We would like to clarify the large improvement of SpecFuse on the Chinese FuseEval benchmark as shown in Table 3. As noted in the table caption, the \"Base LLMs\" ( Qwen2.5-32B, LLaMA3-70B, Mistral-24B) are compared against Qwen2-72B on FuseEval. Given that Qwen2-72B is particularly strong on Chinese benchmarks, the other base models individually perform worse, leading to win rates below 50% (indicating they lose to Qwen2-72B).\nIn contrast, all ensemble methods (MOA, GenFuse, UniTE, SpecFuse) include all four base models, including Qwen2-72B. Thus, the ensembles leverage Qwen2-72B’s strengths, plus complementary knowledge from the other models, allowing SpecFuse to outperform Qwen2-72B (>50% win rate) on FuseEval (Chinese).    \n\n* Furthermore, the numbers in parentheses represent the relative improvement of the ensemble model over the best-performing single base model excluding Qwen2-72B. Since the next-best base model performs significantly worse than Qwen2-72B, the relative improvement of SpecFuse appears particularly large.\n\nWe appreciate the reviewer’s interest and will consider adding further clarification in the next version.\n\n**[Question3: Slight abuse of notation in equation (3), consider using hat]**\n\n* Thank you for your suggestion. Our initial intent was to keep the use of symbols minimal; however, we are happy to revise it according to your recommendation.\n\n**[Question4: Duplicated reference line 378 380]**\n\n* Thank you for your comment. We will review the references and remove any duplicates accordingly.\n\n\n**[Question5: Other typos: end of line 4, beginning of line 230, 303]**\n\n* Thank you for your careful review. We will go through the paper and correct all typos in the next version.\n\n\n**Thank you for taking the time to read our comments. If we have addressed your concerns, we sincerely hope you can reconsider and improve your evaluation of our work. If you have any other questions, we look forward to further discussions with you. We sincerely await your reply.**\n\nReferences： \n\n[1] SELECTLLM: Query-Aware Efficient Selection Algorithm for Large Language Models,  ACL 2025"}, {"review_id": "av0GJr0kI2", "reviewer": "Reviewer_VDu3", "summary": "This paper introduces SpecFuse, a novel, training-free framework for ensembling large language models (LLMs) at inference time. Unlike previous ensemble methods, SpecFuse enables segment-level collaboration among multiple LLMs by iteratively performing two main stages: drafting (where each model generates a candidate segment) and verification (where all models evaluate each other's segments). A central innovation is the online feedback mechanism, which dynamically updates each model’s influence via multiplicative weight updates, rewarding models whose candidates are more often preferred by peers. The authors show strong empirical results across six benchmarks and five model families (from 7B to 72B parameters), demonstrating that SpecFuse consistently outperforms both single LLMs and state-of-the-art ensemble baselines in instruction-following, commonsense, and reasoning tasks. The method is entirely plug-and-play, requiring no additional training or fusion modules.", "questions": "1. **On Weighting Dynamics and Diversity:**  \n   Given the online feedback mechanism’s tendency to upweight high-performing models, is there a risk of converging to a single-model regime, thus losing the core ensembling advantage? Have you considered or tested approaches that explicitly preserve some minimum influence for weaker or specialist models, such as entropy regularization or soft lower bounds on weights? Under what conditions do you observe diversity collapse in practice?\n\n2. **Failure Modes of Generation-Verification Alignment:**  \n   Can you elaborate on cases where a model may be strong at generation but unreliable at verification, or vice versa? For example, might there be scenarios where a model is verbose and persuasive but inaccurate, or vice versa? How robust is SpecFuse to such misalignments, and could alternative reward/weighting schemes improve resilience?\n\n3. **Calibration and Scoring Metric Choice:**  \n   Your method uses average logit (normalized across models) as the primary segment scoring metric in the verification stage. This appears to implicitly assume that all models are reasonably calibrated, such that a higher logit truly reflects better output quality. If a model is systematically miscalibrated (overconfident or underconfident), could this bias the verification and ensemble process? Have you considered or compared alternative scoring functions (e.g., mean log-probability, entropy, or learned reward models), and do you have empirical evidence that average logit is the most robust choice? A deeper discussion or ablation here would be valuable.\n\n4. **Long-Range Dependencies and Semantic Consistency:**  \n   Does segment-level iterative generation ever result in local coherence but global inconsistency or contradictions? Have you observed failure cases where context is not preserved well across rounds, and if so, are there modifications that can mitigate this?", "limitations": "Yes, but I think the paper would still benefit from an expanded discussion of both technical and societal limitations. In particular, a more detailed analysis of when and why the weighting mechanism may reduce diversity, deeper consideration of calibration and scoring metric choice, and a clearer articulation of possible negative societal impacts and mitigation strategies would strengthen the work. I think it's worth adding some discussions on the risks of value amplification or emergent behaviors when ensembling diverse LLMs. For example, could combining LLMs with divergent or poorly-aligned objectives lead to unpredictable or harmful outputs? What safeguards would you recommend? Not saying this should be a focus of the paper or be a basis for accept/reject decisions, but such discussions could be helpful.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- **Originality:**  \n  SpecFuse innovates by combining speculative decoding concepts with model ensembling, creating an entirely training-free and adaptable approach for integrating multiple LLMs. The segment-level iterative collaboration is a significant departure from generate-then-ensemble and ensemble-while-generation paradigms.\n\n- **Quality:**  \n  The methodology is clearly described, with thorough ablations and analysis. The attention to technical details, such as efficient mutual verification (with verify-in-line attention masks and position id remapping), is impressive. The authors also provide a thoughtful investigation into hyperparameters and scalability.\n\n- **Significance:**  \n  Ensembling is a key path to robust and safe LLM deployment, and a training-free, plug-and-play solution with strong empirical gains is both timely and impactful. The consistent improvements across diverse benchmarks and model scales underscore broad utility.\n\n- **Clarity:**  \n  The paper is clearly written, well-organized, and includes helpful figures to clarify the workflow (e.g., the iterative drafting-verification-feedback loop). The empirical validation of the link between generation and verification ability is a valuable addition.\n\n- **Efficiency:**  \n  Compared to previous methods, SpecFuse achieves lower first-token latency and faster inference, making it more suitable for real-time or interactive applications.\n\n### Weaknesses\n\n- **Potential for Dominance by a Single Model:**  \n  The multiplicative weighting scheme, while adaptive, risks allowing the \"strongest\" model to dominate over time, particularly in settings where one model is consistently preferred. This could diminish the diversity benefits of ensembling, especially in heterogeneous task environments where different models have specialized strengths. The authors mention and partially investigate this, but further analysis and possible mitigation would strengthen the work.\n\n- **Assumption of Generation-Verification Alignment:**  \n  The core assumption is that models strong at generation are also reliable as verifiers. While empirically supported, there may be edge cases (e.g., adversarial or highly specialized tasks) where this alignment breaks down. More discussion on potential failures of this assumption would be valuable.\n\n- **Implicit Assumption of Calibration in Logit-Based Scoring:**  \n  The segment verification process relies on the average logit (or normalized average logit) for scoring candidates. This approach implicitly assumes that all models are at least somewhat well-calibrated: that is, their token-level logits meaningfully reflect output quality and are comparable across models. However, if a model is systematically overconfident or underconfident, or if models are miscalibrated in different ways, the segment selection mechanism might be biased or unreliable—even with normalization. The paper does not discuss the sensitivity of the method to calibration errors, nor does it compare to alternative scoring functions (e.g., mean log-probability, entropy, or reference-based metrics). Deeper analysis or empirical study of this point would help clarify whether average logit is the best possible choice for robust ensembling.\n\n- **Limited Discussion of Negative Societal Impacts:**  \n  The paper focuses primarily on technical contributions and empirical results. Broader discussion on societal implications, such as ensembling models with conflicting value systems or potential misuse in amplifying harmful outputs, is not addressed in depth.", "was_revised": true, "final_justification": "Thanks authors for the thoughtful response and additional experiments. All my concerns are address, and I am thus raising my score and vote for acceptance.", "rebuttal": "Dear Reviewer VDu3,\n\n**Thank you for your thoughtful and thorough review of our manuscript. We truly appreciate the time and effort you dedicated to understanding our work. Below are our responses to your comments, which we hope address your concerns and and helps further improve your evaluation of our work.**\n\n**[Weakness1&Question1: Potential for Dominance by a Single Model and Weighting Dynamics on Diversity:]**\n\n* Thank you for your suggestion. We would like to clarify your concern regarding the diversity of the ensemble. We set the minimum reward count to 1, ensuring it is never zero (i.e., the numerator in Equation 9 is at least 1). This design aims to allow dominant models to leverage their strengths while ensuring that weaker models still retain some influence, thereby preserving the core advantage of the ensemble.\n\n* Furthermore, we calculate the percentage of times each base model produces the best candidate among all outputs on the FuseEval English set, as shown in the table below.\n\n  | Llama-3-8B | Mistral-7B | Qwen2-7B | Glm-4-9B | Gemma-2-9B |\n  |------------|------------|----------|----------|------------|\n  | 15.09%| 27.10%| 22.21%| 18.78% | 16.83% |\n\n  Experimental analysis shows that when the performance gap between models is not significant, thus allowing effective ensembling, the diversity advantage of the ensemble is preserved. Each model contributes a certain proportion of candidates that are selected as the best. \n\nWe sincerely thank you for your valuable suggestion, which greatly helps us improve the quality of the paper. We will include this additional experiment in the appendix.\n\n**[Weakness2&Question2:Assumption and Failure Modes of Generation-Verification Alignment]**\n\n* Thank you for question.\nExperiments presented in Figure 6 of Appendix A  show that models with stronger generation capabilities also tend to perform well on verification tasks. While this supports the alignment assumption, we have carefully considered the edge cases you mentioned, where generation and verification may diverge. \n\n  We hypothesize an extreme scenario in which a strong verifier model, due to do_sample, occasionally selects a low-probability token, leading to generation failure. In such cases, misalignment between generation and verification may occur. However, this misalignment is more likely to arise when a model acts solely as a generator. As a verifier, such failures are less likely due to multiple models verifying in parallel, which helps correct occasional errors from individual generators.\n\nThank you again for your valuable suggestion. We will include this discussion as part of Appendix A.\n\n**[Weakness3&Question3:Implicit Assumption of Calibration in Logit-Based Scoring and Metric Choice]**\n\nThank you for your question. We would like to address your concern with the following explanation.\n\n* First, the score of each candidate is computed by summing the scores assigned by all models. Even if a single model is overconfident (assigning consistently high scores) or underconfident (assigning consistently low scores), the normalization ensures that all models operate on a consistent scale. In this way, the influence of miscalibrated models is mitigated, as other models can effectively calibrate the final score by providing more balanced evaluations.\n\n  For example, suppose models a b and c generate candidate segments in one round. Model a assigns scores line_a = [2.3 2.6 3.2] for candidates a b c, model b gives line_b = [1.4 1.6 2.2], and model c gives line_c = [0.6 0.3 0.5]. Since these raw scores differ in scale, we normalize each model’s scores over the candidate set, resulting in line_a = [0.40 0.32 0.28], line_b = [0.42 0.31 0.27], and line_c = [0.27 0.34 0.40].Without online feedback, candidate a’s score = 0.40 + 0.42 + 0.27 = 1.09 > candidate c’s score = 0.28 + 0.27 + 0.40 = 0.95\nThis shows that even if model c is overconfident, its influence can be corrected by the other models.\n\n* Second, logit-based scoring is a widely adopted unsupervised approach for evaluating outputs, commonly used in metrics such as perplexity and confidence estimation. Since language models inherently generate tokens based on logits, inaccurate logits would also lead to poor generation quality.\nWe provide consistency experiments in Appendix A, showing that models perform consistently as both generators and verifiers, those with better generation quality tend to produce more accurate scores. For biased or low-quality models, their poor generation also reflects unreliable scoring. Our online feedback mechanism downweights such models in the verifier role, leading to more accurate overall scoring. \n\n* Additionally, in the scoring metric comparison, we evaluated several alternatives. Specifically, we applied softmax to the logits to obtain probabilities, then took the logarithm to compute the mean log-probability of the generated sequence. This was compared with directly using token-level logits as in our main approach. Experimental results on the FuseEval English dev set are as follows:\n\n  |Metric| rougel | bertscore |\n  |-------------------|--------|-----------|\n  | logits | 27.94| 72.66|\n  | log-probabilities | 27.47 | 72.32 |\n\n  The experimental results show that directly using the average of token-level logits yields better performance.\n\nWe sincerely appreciate your thoughtful question and hope that our response has clarified the issue.\n\n**[weakness4&Limitation： Discussion of Negative Societal Impacts]**\n\nWe sincerely thank the reviewer for raising these important concerns regarding both technical and societal limitations. \n\n* Our current submission primarily focuses on technical design and empirical evaluation, we fully recognize the relevance of broader implications, especially when ensembling large language models (LLMs) with diverse or potentially misaligned objectives. In future versions of this work, we are committed to including a dedicated section on ethical and societal considerations. This will include a more detailed analysis of when and how our weighting mechanism may inadvertently reduce output diversity and a reflection on the risks of value amplification or emergent behaviors when combining heterogeneous LLMs. We also intend to highlight the need for proactive safeguards, such as post-ensemble moderation, and to call on the community to further investigate robustness and safety in multi-model ensembling settings. \n\nWe appreciate the reviewer’s thoughtful feedback, which will help us broaden the impact and responsibility of our research.\n\n**[Question: Long-Range Dependencies and Semantic Consistency]**\n\n* Thank you for your question regarding long-range dependencies and semantic consistency. In our observations, such issues rarely occur. This is because each current segment is selected from candidate segments based on a voting mechanism, using a context composed of previously selected segments. The scoring is based on logits, which are strongly correlated with model perplexity, to evaluate each candidate’s semantic relevance to the preceding text and its overall fluency.\nIf a candidate segment contains semantic inconsistencies, leading to higher perplexity, it is unlikely to be selected. \n\n* To verify this, we constructed a case to simulate such a scenario. Specifically, we assembled four candidates—three normal outputs from Qwen2, Mistral, and GLM-4, and one manually crafted noisy candidate with semantic inconsistency. As shown in the table below, during each round of the ensembling process, SpecFuse consistently selected candidates from Qwen2, Mistral, or GLM-4, while assigning increasingly lower scores to the human-crafted noisy candidate. This demonstrates that SpecFuse effectively avoids semantically inconsistent candidates during the verification stage.\n\n  | | Qwen2| Mistral| Glm-4| Human   Noise| Best|\n  |-----------------|----------------------------------------------------|------------------------------------------------|---------------------------------------------------|------------------------------------------------------|----------------------------------------------------|\n  | Round   1 Text  | A   person who is hardworking is like a determined | This   person is as dedicated to their work as | A   hard-working person is like a well | Title:   The Cowardly Man | A   person who is hardworking is like a determined |\n  | Round   1 Score | 0.292 | 0.2535| 0.259| 0.1955||\n  | Round   2 Text  | seed   that refuses to give up until it grows | river,  carving its path through the | oak,   roots sunk deep in the earth | slacker   waiting for success without doing anything | river,   carving its path through the  |\n  | Round   2 Score | 0.2609 | 0.312 | 0.2646  | 0.1717 | |\n\n* Furthermore, your concern could be addressed by generating along the top-k best segments at each round, rather than using the current top-1 selection strategy adopted in this work. This is essentially a beam search approach, which explores a broader search space to approximate a more globally optimal solution. However, this also comes with increased computational cost. \nThank you for your suggestion，we plan to include beam search as an optional decoding strategy in future versions of our code, allowing users to choose the approach that best suits their computational resources.\n\n**Thank you for taking the time to read our comments. If we have addressed your concerns, we sincerely hope you can reconsider and improve your evaluation of our work. If you have any other questions, we look forward to further discussions with you. We sincerely await your reply.**"}, {"review_id": "29RxWwrdAQ", "reviewer": "Reviewer_KAZA", "summary": "The key idea of this paper is to use speculative decoding (which was used earlier for acceleration) to ensemble multiple LLMs at inference time. It targets an ensemble-while-generation approach for better inference latency as opposed to generate-then-ensemble.  The approach first drafts (all LLMs generate candidates as “segments” with predefined maximum length) and then verifies (all LLMs mutually evaluate all generated candidates in parallel; produce MxM scores for M ensemble models), followed by an online learning mechanism to adjust the weight for better-performing models over time dynamically. Next, the top-ranked segment is selected as the best output and forwarded to all models for subsequent rounds of drafting-verification.\nThe model’s evaluation score is the average of the logits produced by the model over the tokens in the segment. The final score for each candidate is a weighted sum of the scores from all verifier models, where scaler weights are dynamically updated by the proposed online feedback mechanism based on a simple reward (counting the number of times a model candidate is scored higher than another candidate by a third model).", "questions": "Q1) Focusing on widely known benchmarks of MMLU and GSM8k in Table 2, the proposed method by ensembling two or three LLMs (including Gemma-2) shows marginal score improvement (max 1.5%) compared to the best LLM, which is Gemma-2. This raises the concern whether simpler methods, such as self-consistency with majority voting on Gemma-2, would reach similar scores without necessarily using different LLMs?\n\nQ2) Please elaborate on this statement: \n> Despite using only several 7B–9B LLMs, SpecFuse performs comparably to 70B-scale single models while remaining more parameter efficient. \n\nIt seems that it doesn’t quite hold, because 70B-scale LLMs are consistently outperformed in the first place by smaller base LLMs in Table 1 and Table 3 (except on BARTScore on English Scenario in Table 1), and there are no large LLMs for Table 2.\n\nQ3) It would be great to see the impact of the method on more recent benchmarks such as GSM-hard, or MATH-500", "limitations": "No limitation is discussed. One possibility is to open up the limitation of he method compared to larger LLMs, or smaller LLMs when using a simple test-time scaling approach, on the studied benchmarks.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: The paper is clearly written, and the proposed method is positioned well wrt prior art in ensemble-while-generation. The online feedback mechanism seems to be novel in this context.\n\nThe weakest conceptual point is that this method is not a fusion in any form (as opposed to what has been called in the paper) but simply a selection (the highest-scoring candidate is selected as the best output). Hence, the title, abstract, and intro are misleading and should be adjusted accordingly. \n\nFor technical weakness, see questions.", "was_revised": true, "final_justification": "Thoughtfully addressed the fusion concept and showed willingness to revise the title to more clearly reflect the underlying technique, reducing potential confusion. I also appreciate the additional self-consistency results, which effectively demonstrate the cost-benefit trade-off of ensembling. These clarifications and additions have strengthened the work, and as a result, I am happy to increase my score to 4.", "rebuttal": "Dear Reviewer KAZA,\n\n**Thank you for your thoughtful and thorough review of our manuscript. We truly appreciate the time and effort you dedicated to understanding our work, and your constructive comment. Below, we respond to each of the suggestions you raised. We hope this addresses your concerns and helps improve your scoring of our work.**\n\n**[weakness1:Concerns regarding fusion concepts]**\n\n* Thank you very much for your insightful comments. We fully understand your concern, as we mentioned selecting the best model candidate at each iteration. We assume that by “fusion,” you refer to combining all models’ votes or weighted scores to select the optimal token at each round—that is, token-level fusion. However, we would like to clarify that in our work, fusion refers to fusion between segments. Although within each segment we select the single best candidate from one model, the final output is composed of segments generated by different models, **representing segment-level fusion**, which ensures longer semantic coherence and fewer fusion points. Furthermore, during each iteration’s segment generation, the best segment is broadcast to all models, meaning that the historically best segments chosen by other models are fused as semantic context in the current round. These models are thus inspired by the historical best segments to continue generation.\n\nWe sincerely hope the above explanation clarifies your concerns regarding the concept of fusion. If you still have concerns, we would be happy to further discuss the appropriateness of using the term “fuse” and are open to revising the wording to make the method description more precise and rigorous.\n\n\n**[Question1: whether simpler methods, such as self-consistency with majority voting on Gemma-2, would reach similar scores without necessarily using different LLMs?]**\n\n* We sincerely appreciate your constructive suggestions. Following your suggestion, we conducted additional self-consistency experiments with Gemma-2 under the same experimental setup as in Table 2, specifically on the MMLU and GSM8K datasets. For each sample, Gemma-2 was prompted to generate three predictions, and the final answer was selected by majority voting. If all three predictions differed, one was randomly chosen. The experimental results are as follows:\n\n  | | MMLU  | GSM8k |\n  |------------------------------|-------|-------|\n  | Gemma-2                      | 71.51 | 77.26 |\n  | Gemma-2（ self-consistency=3） | 71.74 | 77.75 |\n\n\n  As shown in the table, Gemma-2 (self-consistency) achieves an improvement of +0.23 on MMLU and +0.39 on GSM8K compared to the original Gemma-2. While this demonstrates some gains under the same computational budget, SpecFuse achieves a much larger improvement of +1.5. \n\n  Moreover, the applicability of the self-consistency method is relatively limited. In open-domain text generation tasks, the system typically cannot determine in advance which domain a user query belongs to, nor which model performs best for that particular query. As a result, it is difficult to apply self-consistency, an approach that relies on multiple independent generations and aggregation in such scenarios, especially in non-multiple-choice tasks where standard answers are unavailable. This highlights the strength of ensemble methods: by integrating multiple heterogeneous models, SpecFuse leverages their complementary strengths to enhance the stability of system outputs in open-ended tasks.\n\n\n**[Question2: Elaborate on the statement: \"Despite using only several 7B–9B LLMs, SpecFuse performs comparably to 70B-scale single models while being more parameter efficient.\"]**\n\n* Thank you for your question. We would like to clarify a possible misunderstanding that may have arisen from the pink and blue markings in the table. As described in the caption of Table 1, the blue marks indicate the best-performing 7–9B base LLMs (excluding 70B-scale larger LLMs), while the pink marks indicate the best results overall, including base models, larger models, and ensemble methods. Generally, aside from Mistral-7B-Instruct-v0.3, which achieves performance comparable to LLaMA-3-70B-Instruct, larger LLMs consistently outperform base LLMs. We have extracted the best-performing models from both the base model group and the larger model group, as shown below:\n\n  | Model | ROUGE-1↑ | ROUGE-2↑ | ROUGE-L↑ | BLEU↑   | BARTScore↑ | BERTScore↑ | GPT4-Rank↓ |\n  |--------------------------|----------|----------|----------|---------|------------|------------|------------|\n  |  | | | | English ||||\n  | Mistral-7B-instruct-v0.3 | 27.75    | 10.75    | 25.57    | 4.82    | -2.94 | 71.88| 7.62  |\n  | Mixtral-8x7B-instruct    | 29.04    | 12.25    | 26.75    | 4.08    | -2.81 | 72.19 | 4.16|\n  ||  |    |  | Chinese | |  | |\n  | Glm-4-9B-instruct | 30.88    | 8.71     | 20.42    | 4.47    | -4.33      | 70.25      | 5.18       |\n  | Qwen2-72B-instruct | 31.44    | 8.97     | 22.48    | 4.88    | -4.32      | 70.65      | 3.63       |\n\n  The upward arrow ↑ means higher is better, and the downward arrow ↓ means lower is better.\n\n   As shown in the table, except for the BLEU score on the English dataset, the larger models outperform the base models on all other metrics, which supports the validity of our statement.\n\n* Moreover, the comparison with larger models is intended as a reference rather than our primary focus. It aims to illustrate that, for some generation tasks, combining multiple smaller models can yield competitive performance compared to much larger models with more parameters and training data. Our main goal remains to compare against previous ensemble baselines.\n\nThank you for your question, we hope the above response addresses your concern.\n\n\n**[Question3: It would be great to see the impact of the method on more recent benchmarks such as GSM-hard, or MATH-500\n]**\n\n* Thank you very much for your constructive suggestion. Based on your advice, we added experiments on these two datasets, where SpecFuse integrates three base LLMs: GLM-4, Gemma-2, and Qwen2. All experiments were conducted under the same zero-shot setting. The results are as follows:\n\n  | Model    | GSM-hard | MATH-500 |\n  |----------|----------|----------|\n  | GLM-4| 50.03| 48.00 |\n  | Gemma-2| 55.02| 49.80|\n  | Qwen-2 | 52.55| 51.40|\n  | SpecFuse | 56.10| 53.40 |\n\n \n  The results show that SpecFuse achieved the highest accuracy, outperforming the best single model, Gemma-2, by 1.38 points on GSM-Hard, and surpassing the best single model, Qwen2, by 2 points on Math-500.\n\nThank you again for your valuable suggestion. We hope our response helps resolve your confusion.\n\n\n**[Limitations:No limitation is discussed.One possibility is to open up the limitation of the method compared to larger LLMs, or smaller LLMs when using a simple test-time scaling approach.]**\n\n* Thank you for your thoughtful suggestion. As noted in Appendix E of the original paper, we discuss some limitations of our method, particularly the potential impact when there is a significant performance gap among base models.\nRegarding your point about comparisons with larger LLMs or smaller ones using simple test-time scaling, we will include a detailed discussion on this aspect in the revised version of the paper.\n\nThank you for your suggestion, we hope the above response addresses your concern.\n\n\n**Thank you for taking the time to read our comments. If we have addressed your concerns, we sincerely hope you can reconsider and increase your score. If you have any other questions, we look forward to further discussions with you. We sincerely await your reply.**"}]

Author rebuttal:

Meta-review: This paper introduces SpecFuse, a training-free plug-and-play framework for ensembling LLMs. The method works by iteratively drafting candidate text segments from multiple LLMs and then scoring all candidates across all models and selecting (and broadcasting to all models) the segment with the highest combined score. An online feedback mechanism dynamically adjusts the influence of each model based on its performance, allowing stronger models to contribute more. This segment-level collaboration improves performance and efficiency compared to prior ensemble methods.

**Strengths:**
*   The method is novel, taking inspiration from speculative decoding, but applying these ideas in the model ensembling domain. Unlike Speculative decoding, there is no lossless speedup guarantee here, but the gains are by empirically getting better results via ensembling, without having to wait for all models to complete their full generations before fusing them.
* The online feedback mechanism is also a nice addition and provides additional gains.
*   The paper provides strong and comprehensive empirical validation, demonstrating consistent improvements over strong baselines across various tasks and model sizes. The authors provided additional experiments during the rebuttal that further strengthened these claims.

**Weaknesses:**
*   The method relies on logit access, making it difficult to apply to closed-source models. The authors propose to use LLM-as-judge approach with API calls to get scores, but that will probably make the whole process too slow and expensive.
*   Reviewers raised questions about the online weighting mechanism, such as its robustness to mid-generation task shifts and the justification for its specific design (tSy8, VDu3).
* Some terminology can be more precise to avoid confusions, like the use of “fusion” instead of selection, and clarifying that there is no speculative decoding guarantees.
Decision: Accept (poster)