Title: SpecEM: Training-Free LLM Ensembling via Iterative Drafting, Verification, and Online Feedback
Venue: NeurIPS 2025
Accepted: yes
Keywords: Ensembling language models.+Ensemble method.+Generative.+instruction response
Abstract: Ensembles of generative large language models (LLMs) are a promising way to compensate for individual model limitations, integrating the strengths of different LLMs. Existing LLM ensemble methods, however, face limitations such as first-token delay and challenges in long-range semantic collaboration between models, Moreover, they typically assume equal voting weights for all models during ensemble, ignoring performance differences between models for a given task. In this work, we propose SpecEM, a training-free, plug-and-play LLM ensemble framework that dynamically adjusts each model's model contribution in real time based on task performance. Inspired by speculative decoding, SpecFuse iteratively performs drafting and verification, allowing models to collaborate semantically at the segment level for integrated output. Furthermore, we introduce an online feedback mechanism with multiplicative weight updates, where each model's voting weight is adjusted on-the-fly according to how often it "outperforms" others during verification stage, ensuring that stronger models exert greater influence on the ensemble during generation. Experimental results on five popular LLMs (ranging from 7B to 72B parameters) and six benchmark tasks, spanning instruction following, reasoning, commonsense, and general instruction response, demonstrate consistent performance improvements compared to state-of-the-art LLM ensemble methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe key idea of this paper is to use speculative decoding (which was used earlier for acceleration) to ensemble multiple LLMs at inference time. It targets an ensemble-while-generation approach for better inference latency as opposed to generate-then-ensemble.  The approach first drafts (all LLMs generate candidates as “segments” with predefined maximum length) and then verifies (all LLMs mutually evaluate all generated candidates in parallel; produce MxM scores for M ensemble models), followed by an online learning mechanism to adjust the weight for better-performing models over time dynamically. Next, the top-ranked segment is selected as the best output and forwarded to all models for subsequent rounds of drafting-verification.\nThe model’s evaluation score is the average of the logits produced by the model over the tokens in the segment. The final score for each candidate is a weighted sum of the scores from all verifier models, where scaler weights are dynamically updated by the proposed online feedback mechanism based on a simple reward (counting the number of times a model candidate is scored higher than another candidate by a third model).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper is clearly written, and the proposed method is positioned well wrt prior art in ensemble-while-generation. The online feedback mechanism seems to be novel in this context.\n\nThe weakest conceptual point is that this method is not a fusion in any form (as opposed to what has been called in the paper) but simply a selection (the highest-scoring candidate is selected as the best output). Hence, the title, abstract, and intro are misleading and should be adjusted accordingly. \n\nFor technical weakness, see questions.\n\nQUESTIONS:\nQ1) Focusing on widely known benchmarks of MMLU and GSM8k in Table 2, the proposed method by ensembling two or three LLMs (including Gemma-2) shows marginal score improvement (max 1.5%) compared to the best LLM, which is Gemma-2. This raises the concern whether simpler methods, such as self-consistency with majority voting on Gemma-2, would reach similar scores without necessarily using different LLMs?\n\nQ2) Please elaborate on this statement: \n> Despite using only several 7B–9B LLMs, SpecFuse performs comparably to 70B-scale single models while remaining more parameter efficient. \n\nIt seems that it doesn’t quite hold, because 70B-scale LLMs are consistently outperformed in the first place by smaller base LLMs in Table 1 and Table 3 (except on BARTScore on English Scenario in Table 1), and there are no large LLMs for Table 2.\n\nQ3) It would be great to see the impact of the method on more recent benchmarks such as GSM-hard, or MATH-500\n\nLIMITATIONS:\nNo limitation is discussed. One possibility is to open up the limitation of he method compared to larger LLMs, or smaller LLMs when using a simple test-time scaling approach, on the studied benchmarks.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SpecFuse, a novel, training-free framework for ensembling large language models (LLMs) at inference time. Unlike previous ensemble methods, SpecFuse enables segment-level collaboration among multiple LLMs by iteratively performing two main stages: drafting (where each model generates a candidate segment) and verification (where all models evaluate each other's segments). A central innovation is the online feedback mechanism, which dynamically updates each model’s influence via multiplicative weight updates, rewarding models whose candidates are more often preferred by peers. The authors show strong empirical results across six benchmarks and five model families (from 7B to 72B parameters), demonstrating that SpecFuse consistently outperforms both single LLMs and state-of-the-art ensemble baselines in instruction-following, commonsense, and reasoning tasks. The method is entirely plug-and-play, requiring no additional training or fusion modules.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- **Originality:**  \n  SpecFuse innovates by combining speculative decoding concepts with model ensembling, creating an entirely training-free and adaptable approach for integrating multiple LLMs. The segment-level iterative collaboration is a significant departure from generate-then-ensemble and ensemble-while-generation paradigms.\n\n- **Quality:**  \n  The methodology is clearly described, with thorough ablations and analysis. The attention to technical details, such as efficient mutual verification (with verify-in-line attention masks and position id remapping), is impressive. The authors also provide a thoughtful investigation into hyperparameters and scalability.\n\n- **Significance:**  \n  Ensembling is a key path to robust and safe LLM deployment, and a training-free, plug-and-play solution with strong empirical gains is both timely and impactful. The consistent improvements across diverse benchmarks and model scales underscore broad utility.\n\n- **Clarity:**  \n  The paper is clearly written, well-organized, and includes helpful figures to clarify the workflow (e.g., the iterative drafting-verification-feedback loop). The empirical validation of the link between generation and verification ability is a valuable addition.\n\n- **Efficiency:**  \n  Compared to previous methods, SpecFuse achieves lower first-token latency and faster inference, making it more suitable for real-time or interactive applications.\n\n### Weaknesses\n\n- **Potential for Dominance by a Single Model:**  \n  The multiplicative weighting scheme, while adaptive, risks allowing the \"strongest\" model to dominate over time, particularly in settings where one model is consistently preferred. This could diminish the diversity benefits of ensembling, especially in heterogeneous task environments where different models have specialized strengths. The authors mention and partially investigate this, but further analysis and possible mitigation would strengthen the work.\n\n- **Assumption of Generation-Verification Alignment:**  \n  The core assumption is that models strong at generation are also reliable as verifiers. While empirically supported, there may be edge cases (e.g., adversarial or highly specialized tasks) where this alignment breaks down. More discussion on potential failures of this assumption would be valuable.\n\n- **Implicit Assumption of Calibration in Logit-Based Scoring:**  \n  The segment verification process relies on the average logit (or normalized average logit) for scoring candidates. This approach implicitly assumes that all models are at least somewhat well-calibrated: that is, their token-level logits meaningfully reflect output quality and are comparable across models. However, if a model is systematically overconfident or underconfident, or if models are miscalibrated in different ways, the segment selection mechanism might be biased or unreliable—even with normalization. The paper does not discuss the sensitivity of the method to calibration errors, nor does it compare to alternative scoring functions (e.g., mean log-probability, entropy, or reference-based metrics). Deeper analysis or empirical study of this point would help clarify whether average logit is the best possible choice for robust ensembling.\n\n- **Limited Discussion of Negative Societal Impacts:**  \n  The paper focuses primarily on technical contributions and empirical results. Broader discussion on societal implications, such as ensembling models with conflicting value systems or potential misuse in amplifying harmful outputs, is not addressed in depth.\n\nQUESTIONS:\n1. **On Weighting Dynamics and Diversity:**  \n   Given the online feedback mechanism’s tendency to upweight high-performing models, is there a risk of converging to a single-model regime, thus losing the core ensembling advantage? Have you considered or tested approaches that explicitly preserve some minimum influence for weaker or specialist models, such as entropy regularization or soft lower bounds on weights? Under what conditions do you observe diversity collapse in practice?\n\n2. **Failure Modes of Generation-Verification Alignment:**  \n   Can you elaborate on cases where a model may be strong at generation but unreliable at verification, or vice versa? For example, might there be scenarios where a model is verbose and persuasive but inaccurate, or vice versa? How robust is SpecFuse to such misalignments, and could alternative reward/weighting schemes improve resilience?\n\n3. **Calibration and Scoring Metric Choice:**  \n   Your method uses average logit (normalized across models) as the primary segment scoring metric in the verification stage. This appears to implicitly assume that all models are reasonably calibrated, such that a higher logit truly reflects better output quality. If a model is systematically miscalibrated (overconfident or underconfident), could this bias the verification and ensemble process? Have you considered or compared alternative scoring functions (e.g., mean log-probability, entropy, or learned reward models), and do you have empirical evidence that average logit is the most robust choice? A deeper discussion or ablation here would be valuable.\n\n4. **Long-Range Dependencies and Semantic Consistency:**  \n   Does segment-level iterative generation ever result in local coherence but global inconsistency or contradictions? Have you observed failure cases where context is not preserved well across rounds, and if so, are there modifications that can mitigate this?\n\nLIMITATIONS:\nYes, but I think the paper would still benefit from an expanded discussion of both technical and societal limitations. In particular, a more detailed analysis of when and why the weighting mechanism may reduce diversity, deeper consideration of calibration and scoring metric choice, and a clearer articulation of possible negative societal impacts and mitigation strategies would strengthen the work. I think it's worth adding some discussions on the risks of value amplification or emergent behaviors when ensembling diverse LLMs. For example, could combining LLMs with divergent or poorly-aligned objectives lead to unpredictable or harmful outputs? What safeguards would you recommend? Not saying this should be a focus of the paper or be a basis for accept/reject decisions, but such discussions could be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a training-free, plug-and-play LLM ensemble method, SpecFuse, that integrates multiple models' outputs by iteratively coordinating drafting and verification with an online weights feedback mechanism to adjust the influence of each model based on their contributions. By interleaving drafting and verification at the segment level, it keeps first-token latency near that of a single model while adding only ~20 % runtime overhead. Tested on six instruction-following and reasoning benchmarks with 7-72 B models, SpecFuse outperforms base models and state-of-the-art ensemble baselines.\n\nSTRENGTHS AND WEAKNESSES:\n- Quality\n  - Overall solid work with comprehensive experiments on performance and efficiency, comparison with prior methods and ablation studies, clearly outlining the advantages of SpecFuse\n  - Although the authors claim that the online feedback mechanism ensures stronger models exert greater influence in the ensemble, this claim is only supported by final performance metrics, without any analysis or visualization of how model weights actually evolve during generation.\n  - The learning rate is designed to ensure that the updates become more stable over time (k). However, I am not sure if this would handle situations where there's code switching (First English, then later Chinese; or first English then code at some point). The concern is different models are good at different aspects and the design of weights might not allow it to adapt to changes of tasks during generation.\n- Clarity\n  - The paper is generally well written, logically organized, and easy to follow.\n  - The motivation behind multiplicative weight update is a bit unclear\n    - Why are small initial weights a problem at all? Line 175\n    - Why is the learning rate designed this way? Equation (7)\n  - There are a few typos, formatting or notation issues (see questions/suggestions).\n- Significance\n  - Interesting idea of combining models at the segment-level with unsupervised online weight adjustment, achieving strong performance with relatively low computational overhead compared to previous ensemble methods.\n  - Although the improvement on FuseEval is notable, the SpecFuse (All) for other benchmarks like MMLU, GSM8K (Table 2) is much smaller and can even hurt the model performance (IFEval). It's good that the authors touch on this in the limitation section. However, this slightly contradicts the discussion in line 288-290 \"The improvements are more pronounced when stronger models are introduced, **while even weaker models still contribute positively**.\" Given the fact that one has to host multiple models and sacrifice some inference overhead, it casts some doubt on the practical usefulness of the method.\n- Originality\n  - Fresh blend of segment-level speculative drafting and verification with online multiplicative-weights voting.\n\nQUESTIONS:\n1. Can you explain why to mitigate scoring bias caused by some models producing systematically higher or lower logits the scores are normalized across the **model dimension** (I think this might be a typo)? \n   1. Combining quation 3 and 4: $y^k_j = \\sum_i^N w_i^k \\frac{s^k_{i,j}}{\\sum_i^N s^k_{i,j}}$, so at the beginning when you have all weights are equal, $y_j^k = \\sum_{i=1}^{N} \\frac{1}{N} \\frac{s^k_{i,j}}{\\sum_{i=1}^{N} s^k_{i,j}}=\\frac{1}{N}$, meaning that all candidates will have the same score ignoring completely the scores. Is this expected?\n   2. If model i produce very high logits, i.e. $s_{i,j} \\rightarrow \\infty, \\forall j$, by equation 3, $s_{i,j} \\approx 1$ for i and other r, $s_{r,j}\\approx 0$ for all j, then $y_j$ would be dominated by the model i's judgement\n2. Why in table 3 SpecFuse improve Chinese FuseEval so much? A deep dive into this might help illustrate why SpecFuse is effective.\n3. Slight abuse of notation in equation (3), consider using hat: $\\hat{s}^{(k)}_{i,j} \\leftarrow ...$\n4. Duplicated reference line 378 380\n5. Other typos: end of line 4, beginning of line 230, 303\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SpecFuse, a training-free, plug-and-play LLM ensemble framework designed to address several key limitations in current large language model (LLM) ensemble techniques. Rather than relying on token-level aggregation or post-generation fusion, SpecFuse employs a segment-level iterative process that includes drafting, verification and online feedback three stages. It is evaluated on six benchmarks (including FuseEval, MMLU, ARC-C, GSM8K, IFEval, AlpacaEval 2.0) and five model families (7B to 72B), consistently outperforming individual models and strong ensemble baselines (MOA, UniTE, MBR, PairRank, etc.). The framework offers both accuracy gains and latency improvements.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. SpecFuse is training-free and plug-and-play. It can directly incorporate diverse LLMs without any task-specific fine-tuning, making it highly accessible for practical use.\n\n2. Strong empirical validation: The method is tested on five major LLM families and six benchmarks, spanning open-domain instruction following, knowledge and reasoning (MMLU, ARC-C, GSM8K, IFEval), and multilingual contexts (English/Chinese). These results show consistent and sometimes large gains over both strong individual models and state-of-the-art ensemble approaches.\n\nWeaknesses:\n\n1. The verify-in-line mechanism requires direct control over attention masks and position IDs, which may not be feasible when using closed-model APIs (e.g., GPT-4, Claude).\n\n2. Most evaluation is conducted via automated metrics, with no human inspection of qualitative failure cases. It remains unclear how SpecFuse handles incoherent, contradictory, or hallucinated candidates, or whether it produces undesirable blends of inconsistent partial segments.\n\n3. As described in Figure 5, the choice of segment length L=10 is reported as optimal based on development set results. However, it remains unclear whether this setting was tuned primarily on short or mid-length inputs. In long-form generation tasks that require extended reasoning or discourse coherence, such a short segment length may lead to fragmented context understanding or premature truncation.\n\nQUESTIONS:\nSee weaknesses.\n\n1. Equation 3 normalizes scores across models to mitigate bias (e.g., due to logit scale differences), but this assumes linear comparability. Is this sufficient when combining models with heterogeneous architectures, tokenizers, or calibration characteristics (e.g., mixing a 7B and 70B model)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}