When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensembling
Authors: Heecheol Yun
Kwangmin Ki
Jung Hyun Lee
Eunho Yang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Ensemble
probability-level ensemble
speculative decoding
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a method that identifies the optimal points for applying ensembling, thereby improving both accuracy and efficiency in LLM collaboration.

Abstract: Ensembling Large Language Models (LLMs) has gained attention as a promising approach to surpass the performance of individual models by leveraging their complementary strengths. In particular, aggregating models’ next-token probability distributions to select the next token has been shown to be effective in various tasks. However, while successful for short-form answers, its application to long-form generation remains underexplored. In this paper, we show that using existing ensemble methods in long-form generation requires a careful choice of ensembling positions, since the standard practice of ensembling at every token often degrades performance. We identify two key factors for determining the ensembling positions: tokenization mismatch across models and consensus in their next-token probability distributions. Based on this, we propose $\textbf{SAFE}$, ($\textbf{S}$table $\textbf{A}$nd $\textbf{F}$ast LLM $\textbf{E}$nsembling), a framework that selectively ensembles by jointly considering these factors.  To further improve stability, we apply a probability sharpening strategy when the ensemble distribution becomes overly smooth, enabling the selection of more confident tokens during ensembling. Our experiments on diverse benchmarks, including MATH500 and BBH, demonstrate that SAFE outperforms existing methods in both accuracy and efficiency, with gains achieved even when ensembling fewer than 1\% of tokens.

Published as a conference paper at ICLR 2026
WHEN TO ENSEMBLE: IDENTIFYING TOKEN-LEVEL POINTS FOR STABLE AND FAST LLM ENSEMBLING
Heecheol Yun', Kwangmin Junghyun Lee', Eunho Yangl...
KAIST, "Korea University, 'AITRICS {yoon6503, eunhoy]@kaist.co
ABSTRACT
Ensembling Large Language Models (LLMs) has gained attention as a promis- ing approach to surpass the performance of individual models by leveraging their complementary strengths. In particular, aggregating models' next-token proba- bility distributions to select the next token has been shown to be effective various tasks. However, while successful for short-form answers, its application long-form generation remains underexplored. In this paper, we show that us- ing existing ensemble methods in long-form generation requires a careful choice of ensembling positions, since the standard practice of ensembling at every to- ken often degrades performance. We identify two key factors for determining the ensembling positions: tokenization mismatch across models and consensus in their next-token probability distributions. Based on this, propose SAFE, (Stable And Fast LLM Ensembling). a framework that selectively ensembles by jointly considering these factors. To further improve stability, we apply proba- bility sharpening strategy when the ensemble distribution becomes overly smooth, enabling the selection of more confident tokens during ensembling. Our experi- ments on diverse benchmarks, including MATH500 and BBH, demonstrate that SAFE outperforms existing methods in both accuracy and efficiency, with gains achieved even when ensembling fewer than 1% of tokens. The code is available at https: / / thub. com yyoon 6503/500055 /SFFF.
INTRODUCTION
Recently, Large Language Models (LLMs) have achieved remarkable performance across diverse domains, including mathematics (Yang et al., 2024b), coding (Guo al., 2024) and reasoning (Yang et al., 2025; OpenAI, 2024). Despite this progress, each LLM possesses unique strengths shaped by its training recipe, and no single model dominates across all domains. complementary strengths of multiple models at inference time has emerged to surpass the performance of any individual model (Wang et al., 2025a; Yao et al., 2025; Chen et al., 2025a). Compared to training new model that jointly integrates all such capabilities, these collaborative approaches provide more practical and efficient pathway to superior performance.
a result, combining the
promising way
Among various training-free collaboration methods, probability-eeve ensemble, which aggregates the next-token probability distributions of multiple LLMs to select the most confident token, has emerged one of the most effective ways (Yao et al., 2025; Yu et al., 2024; Huang et al., 2024; Xu et al., 2024). It enables collaboration across diverse model architectures and effectively leverages the knowledge of multiple models embedded in their probability distributions. Consequently, has out- performed individual models, particularly when directly answering multiple-choce or short-answer questions without reasoning.
A natural question then arises: are probability-leee ensemble methods equally effective for long- form generation? We find that, in long-form generation, the effectiveness of ensembling critically depends on deciding when to ensemble. Our analysis reveals that accuracy and efficiency improve when ensembling occurs at appropriate token positions, guided by two key factors: tokenization mismatch across models and their consensus in next-token probability distributions.
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Tokembbrr
So fia
"fia"
"À"
"So"
fia
"fia"
LLM
OOV-like token for LLM 2
Toweter'
Sofa
X
"So"+"?"
"A"
"A-
selected
So
LLM
after token alignment
Figure I: Illustration the OOV-like token problem. When So is fed into LLM2, which tokenizes Sofia single token, the probability distribution of the next token becomes corrupted.
Problem
Terry eats yogurts day. They are currently sale yogurts for $5.00. How much does he spend yogurt over 30 days?
Sofie and Tess will each randommy choose one of the 10 integers from to 10 What the probabiltty nelther integer chosen will be the square of the other?
determine how Terry spends yogurt over days,
To determine the probability that neither integer chosen by Saf and Tess will be the square of the other, need to follow these steps 2. Determine the total number of possible outcomes: Each of SotÄašaÄAA
down the problem into several steps:
1. Calculate the daily yogurt consumption in torms of the sale unit: Terry eats 2 Terry consumes
yogurts day.
yogurts
(2/4 = 0.5) of a sale unit per day.
2. Determine the daiy cost based on the sale price:- Since yogurts cost $5.00, (which
Unnatural Token Repetition X
yogurt costs (5/4 1.25). Therefore,
yogurts) costs (0.5
0.625), - So, Terry spends $0.625 per day on yogurt.
Answer
3. Calculate the total cost over 30 days: Terry sponds $0.625 per day. Over 30
"*Option Incorret because coronary arteries However, the galibleddle r does not se crete insul in... nourisment feeding, which involves digesive system
days, total
(0.625 30= 18.75).
Therefore. the total amount Terry spands on yogurt over days \boxee((8.75)
Wrong Information
Typo
Figure 2: (Left) Failure cases of existing ensemble methods in long sequence generation. (Right) Feeding OOV-like tokens into model often causes to produce wrong tokens.
The first factor, tokenization mismatch across models, is crucial for stability, especially in long- form generation where such mismatches occur more frequently. A mismatch arises when an ensem- ble selects a token that conflicts with the tokenization scheme of a participating model. We refer to these tokens as OOV-like tokens because while not truly out-of-vocabulae model to predict the next token in an out-of-distribution state, often resulting in the generation of erroneous tokens, as illustrated in Figure 1. Suppose the ensemble process first generates the token when constructing the word Sofia. However, since LLM2 tokenizes So as single token, So acts next-token probability distribution, leading to an erroneous output (i.e., Å). Such errors accumulate in long-sequ sequeeccence generation, degrading output quality. For example, in the case of unnatural token repetition shown in Figure 2, an initial error in generating the word Sofia propagates, causing the model to repeatedly output corrupted tokens like "À" on the subsequent generation. Consequently, existing ensemble method (Yao et al., 2025) that performs ensembling at every token, suffers sub- stantial performance degradation in Chain-of-Thougee in Table 1. Therefore, accounting for tokenization mismatch across models is essential to prevent the introduction of OOV-like tokens and ensure stable ensembling under COT.
(OOV), they force
OOV-like token for LLM2, Conditioning on this unnatural prefix corrupts LLM2's 's
(CoT) reasoning (Wei et al., 2022), as shown
The second factor, consensus in models' next-token probability distributions, relates to efficiency. Given the next-token probability distributions from multiple models, an ensemble operation is per- formed to aggregate these probability distributions. However, this introduces inefficiency when gen- erating long sequences because the number of ensemble operations grows with sequence length. The primary expense of the ensemble operation arises from aligning next-token probability distributions defined over different vocabularies into a shared vocabulary space, a process that requires mapping across large vocabulary sets. However, when individual models' next-token probability distributions exhibit sufficient consensus, the most confident token from the aggregated distribution can be identi- fied without explicitly aligning distributions from multiple models. Leveraging this property, can determine the most confident token directly from the models' next-token probability distributions, thereby improving efficiency by skipping alignment operations.
2

---

Published as a conference paper at ICLR 2026
MMI.U-redux
ARC-C
MATH500
Method
No CoT
CoT
No CoT
CoT
CoT
Qwen2.5-7B
68.86
74.88
87.37
88.74
72.4
Interni3-8BB
67.52
76.89
88.57
90.27
74.8
UniTE
69.36 (+0.50)
73.39 (-3.50)
88,40 (-0.17)
87.97
(-2.30)
59.6 (-15.2)
UniTE + Ours
69.36 (+0.50)
77.92 (+1.03)
88.40 (-0.17)
90.78 (+0.51)
77.6 (+2.8)
Table I: Performance of the baseline ensemble method (UniTE) degrades under CoT prompting. In contrast, it matches or outperforms individual models when directly answering multiple-choice questions, since tokenizer mismatches do not arise. All models are instruction-tuned.
To this end, we propose SAFE (Stable And Fast LLM Ensembling), which identifies the opportune moments for ensembling in long-sequence generation by considcring the two key factors above. SAFE adopts a speculative strategy in which one model, the drafter, generates lookahead quence of tokens, while the remaining models, the verifiers, identify token-level ensemble points within that sequence. Similar to speculative decoding (Leviathan et al., 2023), this role separation reduces computational cost by limiting autoregressive generation to the drafter, whereas conven- tional ensemble methods require every model do so. Specifically, SAFE iterates a three-step cycle: Generate-Veen Verify-Enseetne (Verify) Next, the verifiers examine drafter's 's tokens in a single forward pass to determine whether ensembling at each token is both stable and necessary. Ensembling is triggered among the drafter's tokens only when the following two conditions are satisfied: (i) OOV-like token and (ii) the verifiers exhibit insufficient agreement on the token. (Ensemble) Finally, ensembling is applied only at the tokens validated the Verify step, replacing them with the ensembled tokens. At these points, if the ensemble distribution is overly smooth, we apply probability sharpening strat- egy that concentrates the probability mass onto the most plausible token for precise token selection.
(Generate) First, the drafter produces a lookahead sequence of tokens.
introduced
Overall, find that probability level ensembling should occur appropriate token positions, espe- cially when generating long sequences with models that use different tokenizers. We then propose SAFE, method that determines these positions by jointly considering the two key factors. Conse- quently, our method offers the following key advantages.
Efficiency: SAFE significantly reduces computational cost in two ways. First, its specula- tive strategy restricts costly autoregressive generation to single drafter. Second, its selec- tive ensembling reduces the number of ensemble operations. Therefore, SAFE can achieve inference speed comparable to individual models, even on long sequences.
Stability: SAFE ensures that tokens are generated from an uncorrupted ensemble distribu- tion by preventing OOV-like tokens from being fed into models. As a result, SAFE enables stable text generation and outperforms existing ensemble methods in CoT settings.
Plug-and-Play: SAFE can be seamlessly integrated with existing ensemble methods by simply adding the generate-verify logic. SAFE consistently improves recent ensembling approaches across diverse model combinations.
2 RELATED WORK
2.1 LLM ENSEMBLE
LLM ensemble methods can be broadly categorized according whether ensembling occurs after inference or during inference (Chen et al., 2025b). Research in both directions has progressed in parallel, each of which detailed below.
2.1.1 ENSEMBLE AFTER INFERENCE
These approaches aggregate complete responses from multiple LLMs. While early work relied on debate-style interactions to reach consensus (Du et al., 2023; Chen et al., 2024a; Liang et al., 2024), recent focus has shifted toward stacking L.LMs in cascade, parallel, or hybrid structures.
3

---

Published as a conference paper at ICLR 2026
Cascade structure. Models are invoked sequentially to balance cost and performance. Frugal- GPT (Chen et al., 2024b) orders models by cost, invoking more expensive models only when cheaper ones produce unreliable responses. Similarly, AutoMix (Aggarwal et al., 2024) uses self- velffieriiiittion to guide routing decisions, while Gupta et al. (2024) employs token-level uncertainty.
Parallel structure. In contrast to cascading, parallel ensembling runs multiple models inddpen- dently and then selects the best response among them. MORE (Si et al., 2023) trains a classifier select the optimal response by considering model expertise, confidence, and agreement across responses. LLM-Blender (Jiang et al., 2023) employs a pairwise ranker to score responses and then fuses the top-k candidates into single answer.
Hybrid structure. Recent work has also explored combining the advantages of cascade and parallel structures. MoA (Wang al., 2025a) proposes a framework that iteratively feeds responses from multiple models into an aggregator LLM, which consolidates them into a single answer. Within this framework, Self-MoA (Li al., 2025) shows that using a single best-performing model can outperform using multiple distinct models in some cases, while Symbolic-MoB (Chen et al., 2025a) introduces query-adaptive model routing. Nevertheless, such frameworks require numerous LLM calls, and consolidating multiple responses remains challenging (Wang et al., 2025b).
2.1.2 ENSEMBLE DURING INFERENCE
In this setting, ensembling occurs during response generation, most commonly at the token level. Co-LLM (Shen et al., 2024) adopts a routing method which dynamically selects which model to use for generating each token. CoSD (Wang et al., 2025c) improves efficiency by introducing lightweight router and integrating speculative decoding. These approaches primarily target models with identical tokenizers and rely on routing rather than aggregating probability across models.
To better exploit the collective intelligence of multiple models, another line of work explores probability-level ensemble methods. These methods average the next-token probability distributions of different models to select the most confident token. Since probability distributions are defined over heterogeneous vocabularies, prior work has focused on constructing the ensemble distribution by aligning different vocabularies across models. GaC (Yu et al., 2024) integrates probabilities by taking the union of all model vocabularies and then mapping each model's vocabulary to this union. DEEPEN (Huang et al., 2024) projects each model's 's vocabulary into shared embedding space, merges distributions there, and maps them back to the individual vocabulary spaces. UniTE (Yao et al., 2025) demonstrates that aligning only the top-k tokens from each model is effective both in performance and efficiency. While these methods achieve strong performance in directly generating answer tokens by selecting the most confident token, they face challenges in long-sequence genera- tion that involves reasoning. In such cases, an increase in OOV-like tokens destabilizes the ensemble, and repeated autoregressive generation across multiple models, along with the need align their vo- cabulary spaces, makes such approaches inefficient. Therefore, we aim to simultaneously improve the stability and efficiency of probability-Ievel ensembling by introducing a verification algorithm that determines when to ensemble.
2.2 SPECULATIVE DECODING
Speculative decoding (Leviathan et al., 2023) is a widely used technique for reducing the cost of autoregressive generation in LLMs. To alleviate the repeated forward passes required by a large model during token generation, speculative decoding replaces this process with small drafter that speculates sequence of candidate tokens. The large target model then performs a single forward pass to determine how many of the drafter's proposed tokens to accept. This allows multiple tokens to be generated in a single forward pass of the target model, thereby reducing computational cost.
Recently, speculative decoding has been explored as way to accelerate probability- LLM en- sembling. However, existing approaches (Fu et al., 2025) are limited to settings in which all models share an identical tokenizer, and cases where the drafter and target models use different tokeniz- ers remain underexplored. In such scenarios, the drafter's 's tokens cannot be properly evaluated by other participating models due to tokenization misalignment and also exhibit OOV-like issues. To address this, we extend speculative decoding to ensembles composed of models with heterogeneous tokenizers by proposing appropriate acceptance criterion.
4

---

Published as a conference paper at ICLR 2026
Generate
Verify
Drafter
En
semb
pasition
Inc
oei
oct
>>
Tokesberi
Tekenbzer 2
0.8
position
posiiiion
Incorrect
Verifier
Verifler 2
"Ensemble position Correc"'
Resume generation from
Compusie probesiiity of ouch tokmn
Ensemble with Probability sharpening
ambble
position
drrect
Sombie
position
Incorect
oooa
"Ine"
Varity the drofter's toilnn
Varlerr
Model
sermb
position
Inc
0.8
required)
postion
eeseptid bolan beceuure the procoatig Ioleoru Ubs
Figure 3: Overview of SAFE. The drafter generates sequence of tokens, which the verifiers re- tokenize with own tokenization schemes and identify the necessary ensembling point. At this posi- tion, ensembling is performed with probability sharpening applied selectively to enhance precision.
SAFE: TOWARD STABLE AND FAST LLM ENSEMBLING
We aim to solve the problem of instability and inefficiency that arises when ensembling next-token probability distributions across LLMs with heterogeneous tokenizers, particularly long-sequence generation. To this end, we propose SAFE, an algorithm that preemptively determines optimal points for ensembling by jointly considering tokenization mismatch and consensus in next-token probabil- ity distributions. Notably, SAFE can be seamlessly integrated with existing ensemble methods.
Given & different LLMs, our method begins by dividing the models into two roles: a drafter Märafts which generates lookahead sequence of tokens, and verifiers Mver, which identify the ensem- ble points among the drafter's tokens. We select the best-performing model as Maraft. while the remaining models serve Mver. We then iterate the Generate (Section 3. 3.1)- -e-eiy (Section 3.2)- Ensemble (Section 3.3) cycle. In each iteration, Marah. first generates a sequence of tokens, which Mver then examine to find a token that requires ensembling. At such points, ensembling is performed to replace the token with the most confident token from the averaged distributions of all models, after which Maruft resumes generation from the ensembled token. Figure 3 shows the overview of SAFE.
3.1 GENERATE
The drafter Maraft generates a predefined number of tokens (4:n.).;(4nn1)) Producing multiple tokens rather than a single token allows SAFE to account for the different tokenization schemes of the various models. For example, consider the word Incorrect. Suppose Maraft generates it as three tokens (Inc, orr, ect), while other models generate as single token (Incorrect). If Maran: were to generate only the first token Inc, would fail to capture the tokenization schemes of the other models. Therefore, at this stage, Maratt produces a sequence of tokens to ensure com- patibility with diverse tokenization schemes. The choice of n is discussed in Section 4.4.
3.2 VERIFY
In this step, the verifiers Myer collaboratively examine the drafter's tokens (f,, ,, 14+n-1) to iden- tify which tokens require ensembling. Ensembling is triggered at the earliest token t, that satisfies two checks: (i) OOV-like token verification, requiring that the immediately preceding token fj-1 not 00V-like token, and (ii) ensemble distribution verification, requiring that łj is not the most confident token in the ensemble distribution Pens- This selective process addresses instability by preventing the introduction of OOV-like tokens and improves efficiency by skipping unneces- sary ensembling. Importantly, this entire verification process is efficient, as the drafter's tokens are processed by the verifiers in single forward pass rather than autoregressively.
5

---

Published as a conference paper at ICLR 2026
(i) OOV-like Token Verification. This check requires that the preceding token is not OOV-like token to prevent OOV-like tokens from corrupting the model's next-token probability distribution. To determine whether drafter token f, is OOV-like token, up to 1 aligns with the tokenization boundaries of the verifiers, ensuring that each verifier can be conditioned valid prefix tokens. For example, in the word Incorrect, tokens such Ine or orr are OOV-like, but ect is not. This is because the tokenization boundaries up to Inc or orr inconsistent with the other model's s tokenization boundary Incorrect, foreing that model to be conditioned on an invalid prefix such as Inc or Incorr. The detailed verification process are as follows. First, each verifier model LLM, E Myer tokenizes the drafter's sequence t<i+n into its own tokenization, t" <er_. Then, the drafter's token is defined an OOV-like token for LLM, the tokenization boundary of b<j++ does not match any boundary in LLM,'s 's tokenization. This condition is formally stated in Equation (1):
examine whether the token boundary
l is OOV-like LLM, < Vx [0, Ui+n - 1J, Decode(t<sta) + Decode(t" <.+1), (1)
where Decode(-) means merging tokens back into text. If is identified as OOV-like by any verifier, ensembling is not triggered the subsequent token fJ+1- Therefore, in the word Incorrect, ensembling is skipped at orr and ect, but can be triggered at the token following ect.
(ii) Ensemble Distribution Verification. For tokens that pass OOV-like token verification, method further checks whether the token is the most probable prediction in the ensemble distri- bution. To avoid the cost of repeatedly constructing the ensemble distribution, whether token t, is the most probable token by examining each model's 's own distribution. Specifi- cally, given LLM,'s 's tokenization t <ey of the drafter's tokens tzj. the drafter's token ł, regarded as the most confident and ensembling is therefore skipped, if either of the following holds:
instead verify
1. (Unanimous consensus among verifiers) If f., = arg max; (t t²=x,) for all LLM, E Mveers we skip ensembling, where P. is the probability distribution of LLM
2. (Average probability above one half)
If MycaUMMaaaal LLLM, UMarna Po(ty, [tt=2" <v,) we skip ensembling at ty.
Intuitively, the first condition checks whether ty is the most probable one across all verifiers, while the second checks whether its average probability across all models greater than 0.5. Adopting these criteria does not compromise accuracy compared to using the exact ensemble distribution, which is proved in Appendix C.
Algorithm SAFE: Generate- Generate-Verify- -erifyy- algorithm
Require: Marant Myers p. prompt, n: drafter's sequence length
1:
2: to < BOS token
Sentence do 4: 4. t.... 1++n-1 + Maratt (p. tes)
3: while not End-of-Sentence
D 1. Generate
5:
<octa t TOKENIZE, (t)):t), t²
Vo E M U Marart
6:
for j = i + i + n do if 1--1 * OOV-like token and Lj passes Ensemble Distribution Verification then
b 2. Verify
7:
8:
Pens + AVERAGEDIST (: p,t" ",,t",,) <ey))ve (e.))veMonnn MoneUMartte
> >. Ensemble
9:
t, arg max, Pens( (t p, pptx))
10:
i 6 j +1
else
12:
i it-t i-itn
13:
end
14:
end for
15:
end while
3.3
ENSEMBLE: SHARPENING ENSEMBLE DISTRIBUTION
In the Ensemble step, any token that passes both verifications in the Verify step is replaced with the most probable token from Pens. which is constructed as the average of all models' probabil- ity distributions using existing ensemble methods. However, different tokenization schemes across

---

Published as a conference paper at ICLR 2026
models can scatter probability mass for the same word across multiple sub-word tokens, resulting in an overly smooth ensemble distribution (i.e., max Peus < 0.5) that hinders confident token selection. To address this, apply probability sharpening strategy to consolidate the diffused probability mass. We explore two different sharpening strategies. The first adopts a heuristic approach that con- solidates the diffused probability by reallocating the probability mass from variant subword tokens to their common prefix token. To prevent inflating probabilities of low-quality tokens, reallocation applied only to the drafter's tokens with initial probability greater than a threshold X. Formally, the entire sharpening process is defined as:
Pens (1z) +- Pons(L) +
Σ
Pons(ti) where Pans(l)) >
# 4..atartsswwttg th(ts)
The second strategy replaces the arithmetic mean with the geometric mean when aggregating the models' probability distributions. Since the geometric mean strongly penalizes tokens that receive low probability by any individual model, it effectively concentrates probability mass on tokens that are consistently supported across models. These two strategies are compared in section 4.4.
After selecting the most confident token from the (potentially sharpened) ensemble distribution, the drafter resumes generation from this token.
4 EXPERIMENTS
This section evaluates SAFE across various benchmarks and model combinations. We begin by outlining the experimental setup, followed by the implementation details, including our KV caching strategy to further improve efficiency. We then provide an analysis of the results.
4.1 EXPERIMENTAL SETTINGS
Models. We select three widely used LLMs with similar ca- pability but heterogeneous tokenization schemes: Internim3-89 -BB Instruct (Cai al., 2024), Qwen2.5-7B-Imm Innstructtttttttteeeeeeeeee (Qwen et al., 2025). and M..ON--3.5-7.8B-Inntttee
Suoeiiat en Stediar by dufond5cce
Inamminaa
100.0
53.06
46.8
(An et al., 2024). Figure 4 presents
the tokenization similarity across model pairs on Oxford 5000 words (Oxford, 2018 8), which consists of commonly used English words. As illustrated, only a small portion of the words are tok- enized identically across models, with agreement rates ranging from 40% to 60%. To also evaluate ensembling on models with nearly identical tokenizations, we include two widely used LLMs with high agreement rates: Qwwnn-7B--nstrucc (Yang et al., 2024a) and 1-0ma-3.--B-Inntrreeee (Dubey et al., 2024). For these models, more than 99% of Oxford 5000 words are tokenized identically. For further study, 32B-scale models are experimented in Appendix G.
46.8
63.51
100.0
0we923
Figure 4: Tokenization agree- ment rates between each model pair Oxford 5000 words.
Benchmarks. To evaluate performance across diverse domains, we use five benchmarks. For general knowledge, we adopt MMLU-redux (Gema et al., 2025), refined subset of MMLU (Hendrycks et al., 2021) that covers 30 subjects with numan-annotaan we use MATH500 (Lightman al., 2024) and GSM8K (Cobbe et al., 2021). For general reasoning, we employ ARC-Challenge (Clark et al., 2018) and BBH (Suzgun et al., 2023). All benchmarks are evaluated under a zero-shot CoT setting, except for BBH, which uses 3-shot COT. For BBH, choose 15 subjects where the models exhibit comparable performance. Further details, including prompt templates and selected BBH subjects, are provided in Appendix A.
corrections. For mathematical reasoning,
Baselines. We apply our method to two recent SOTA probability- ensemble methods: GaC (Yu et al., 2024) and UniTE (Yao et al., 2025). GaC performs ensembling only when the main LLM's next-token probability falls below 0.5. In contrast, UniTE represents the SOTA among methods that ensemble at every generation step. In our setting, "X SAFE" means that the ensemble method X is applied only the token positions that SAFE identifies requiring ensembling.
7

---

Published as a conference paper at ICLR 2026
Method
MMLU-aela
MATHE500
GSMIK
ARC-C
Avg.
Aocuracy
B/T
Aucuracy
E/T
Accuracy
Ascuracy
Accuracy
E/T
Aucuracy
Internlm33-8 888
76.89
74.3
90.14
82.26
90.27
$2.87
Qwen2.5-77 5-78
74.
72.4
9131
79.15
88.
$1.40
EXXONE3.5-77
73:25
728
90145
78.75
90.44
81.14
Two-modee enzombling (Intermlm3 Qwen22.5)
G.C
(-0001)
8.43
74.2 (-0.6)
1.01
(-0.53)
0.82
82.34 (+0.08)
5.69
90.61 (+0.34)
10.22
83.09 (-0-22)
GaC + SAFE
77.11 (-0.22)
5:23
76.0 (+1.3)
0.71
91.36 (-0.45)
0.67
82,34 (+0.06)
3.73
91.13 (-0.56)
6.22
83.59 (-0.72)
UNITE
(3.59
100
(-
100
75.06 (-16.75)
100
(-266)
100
(-2.0)
100
(-777)
UNITE + SAFE
(-0.92)
12.59
77.4 (-2.6)
382
92,04 (-0.23)
5.16
82,97 (40.71)
10.35
90.78 (-0.51)
14.47
84.20 (-1.33)
Too-model ensembling (Qwen2.5+
EXAONEA.5)
GaC
76.01 (+1.33)
13:42
(+2.6)
2:31
92.65 (-0.24)
2.00
79.61 (+0.46)
8.15
90.27 (-0.17)
14.06
82.79 (-1.29)
GaC+ SAFE
76.79 (-199)
7.52
76.4 (+3,6)
1.09
92.57
(-0.76)
1.26
79.66
(+0.51)
4.51
90.78 (+0.34)
8.31
83:24 (+1.84
UNITE
(-2751-
100
43.4 (-294)
100
77.03
(-14 78)
100
67.45 70)
100
72.61 (-17.33)
100
62.85 (-18 55)
UNITE SAFE
(-.66))
17.21
76.4 (+3.6)
4.69
92.72
(-0.91)
5.60
81,69 (+2.51)
14.03
90.76 (+0.34)
19.24
83,63 (-2.23)
Proomodel ousembling (Jutembat EXAONEE.S)
GaC
7-.36(-0.53)
8.71
75.8 (-1.0)
1.14
90.75 (-0.30)
0.58
81.57 (-0.69)
6.32
90.78 (+0.34)
10.07
83:05 (-0.18)
GaC SAFE
77,21 (10.32)
5.94
77.2 (x2.4)
0.84
90.07 (-0.22)
0.72
81.54 (-0.72)
4.38
91.72 (+
6.92
83.67 (-0.80)
UNITE
(-.38)
100
(-12)
100
(-1.4)
100
78.04 (-4.20) 22)
100
(-2.21)
100
(-2.33)
UNITE SAFE
(6001)
15.84
(-2.22
(-0.30)
(40.89)
90.27 (-0.17)
17.89
83.09
Three-model ensembling (Interalm) Qwen2.5 + EXAONER.5)
UNITE
(-299))))9)
100
76.0 (+12)
100
91.28 (-0.53)
100
(-4.7(-4.79)
100
(-3.244
100
$1.17
UNITE SAFE
77,60 (-0.71)
16.18
79,0
(-4
4.12
92,04 (-0.23)
82.77 (+0.51)
12.74
91.55 (+1.11)
18.60
84.59 (-1.72)
Table 2: Ensembling results of models with substantially different tokenizations using COT. E/T (%) represents the percentage of ensembling during generation, computed as Ensemble parentheses denote the performance gap relative to the best-performing individual model.
(%). Numbers in
Token
4.2
IMPLEMENTATION
KV Cache Implementation. KV caching is essential for efficient generation in LLMs. However, unlike standard generation settings where previously generated tokens remain fixed, ensemble gen- eration may replace tokens during the ensembling process, leading to inconsistencies between the cache and the actual input sequence. Consequently, prior approaches have typically avoided imple- menting KV cache management, leaving it as future work. In contrast, our method updates each model's KV cache at the end of every ensemble step to align with the ensembled output, and uses this updated cache in the next step, thereby ensuring cache consistency. We apply our KV cache management to all baselines in our experiments. Please refer Appendix D for details.
Hardware and Hyperparameters. We configure our method as follows. Following the approach of UniTE (Yao al., 2025) for selecting primary model, we select the model with the best average performance as Maran- For probability sharpening, we apply the heuristic strategy in our main results with the threshold À set to 0.1. The drafter generates tokens in chunks of 5, and all models use greedy decoding with a maximum output length of 2048. For ensembling, each model is loaded onto a separate GPU, with all experiments conducted on RTX 3090 GPUs with FP16 precision and F (Dao, 2023) enabled.
4.3
MAIN ANALYSIS
Results of ensembling models with substantially different tokenization schemes are shown in Ta- ble 2. We also report E/T, the percentage of tokens that undergo ensembling over the entire sequence.
SAFE improves performance with less ensembling. Overall, SAFE generally outperforms indi- vidual models, making existing ensemble methods practical even under COT. As shown in Table 2, the baseline UniTE struggles significantly under a CoT setting, consistently underperforming indi- vidual models across all experiments. This is because it ensembles at every generation step, which increases the frequency of OOV-like tokens and consequently corrupts probability distributions. In contrast, applying SAFE enables UniTE to achieve the best performance in many cases (9/15) while reducing the ensemble frequency (E/T) to fewer than 20% of tokens. This highlights the importance of determining when to ensemble, particularly when generating long sequences with models that use heterogeneous tokenizers. GaC, on the other hand, is more robust, since it performs ensembling only when the main LLM's probability falls below 0.5, yielding an unintended but beneficial ef- fect of preventing the introduction of OOV-like tokens. Nevertheless, SAFE further improves GaC's performance while reducing the number of ensemble operations.
8

---

Published as a conference paper at ICLR 2026
Bac 8-M0
Dugr
BECA
Latencyy compariion UniTE with and win SAFF
(b) Latency comparison of CaC with and wio SAFF
(c) Latency of Cia wio KV exchinn
Figure 5: Latency comparison on MATH500. Our method shows similar latency compared to indi- vidual models, even when generating long sequences. w/ KV indicates that our KV caching strategy is applied. Note that the time-axis scale in (c) differs from (a) and (b).
One interesting finding is that much less ensembling is required in math datasets. When SAFE is applied with UniTE, ensembling is triggered for only 4.85% of tokens on average in math datasets, whereas it rises to 15.24% general-domaie datasets, which is nearly three times higher. We tribute this to the nature of math responses, which often contain equations or structured expressions with limited variation, leading higher agreement among verifier models. contrast, responses in -omann datasets allow for greater linguistic variability, which reduces agreement across models and thus requires more frequent ensembling.
Increasing the number of models is not always optimal. As shown in Table 2, ensembling three models does not consistently outperform two-model ensembling, whereas ensembling the top-2 best performing models typically yields the strongest results. This suggests that when model rankings are known, restricting ensembling to the top-2 models is both effective and efficient. On the other hand, when rankings unknown, ensembling multiple comparable models provides stable, though not necessarily optimal, performance.
SAFE can be as fast individual models. A key challenge in LLM ensembling is achieving in- ference speed comparable to running a single model. As shown in Figure 5, SAFE closely matches the latency of individual models when generating hundreds of tokens, regardless of the underlying ensemble method. Moreover, under the same computational resources, it significantly improves effi- ciency over existing ensemble methods when generating long responses. This efficiency stems from three properties of our approach. First, only Maraft is responsible for autoregressive generation, Sec- ond, SAFE substantially reduces the number of ensembling. Third, our KV caching strategy further improves efficiency, illustrated in Figure 5(c). We provide more comparisons in Appendix E.
SAFE further improves performance when ensembling models with similar tokenization. We further evaluate SAFE on models with highly similar tokenizations, where more than 99% of Oxford 5000 words are tokenized identically. As shown in Table 3, the performance drop of existing methods less severe than in Table 2. This is because highly aligned tokenization schemes greatly reduce the occurrence of OOV-like tokens, Ieading to more stable ensembling. Nevertheless, applying SAFE existing methods consistently improves performance, yielding over 9% gain MATH500 compared to the best-performing individual model.
4.4 ABLATION STUDY
We conduct ablation studies on probability sharpening strategy and the drafter's sequence length.
Method
U-edux
MATH500
GSMSK
Method
MMLU-redux
MATH500
GSMBK
Internlm3-58
76.89
Qwen2.7P
69.25
49.8
85.90
Qwen2.5.7B
74.88
72.4
91.81
Llama3.1-88
68.51
82.56
Two-sodde ensombiing (Interaluu3 Qween.5)
Two-model ensembling (Qwen2 Llama3.1)
GaC
77.00
74.2
91.28
GaC
69.50 (+0.25)
52.4 (+2.6)
85.37 (-0.53)
GaC SAFE (wlo sharpen.)
77.11
75.2
91.36
GaC + SAFE
69.99 (+0.74)
59.4 (+9.6)
86.66 (-0.76)
GaC SAFE
77.11
76.0
91.36
UniTE
68.9 (-0.35)
54.0 (+4.2)
79.98 (-5.92)
73.39
59.6
UniTE SAFE
69.71 (+0.46)
55.6 (+5.8)
84.08 (-1.82)
UnITE SAFE (wle sharpen.)
UnITE + SAFE
77.81
77.53
77.4
76.6
91.66
92.04
Table 3: Ensembling results of models with sim- ilar tokenization.
Table 4: Ablation on probability sharpening strategy.

---

Published as a conference paper at ICLR 2026
Sharpening Method
MMI.U-redux
MATH500
GSM8K
Avg.
UniTE + SAFE (no sharpening)
77.53
76.6
91.66
81.93
UniTE + SAFE (A 0.1)
77.81
77.4
92.04
82.42
UniTE + SAFE (A = 0.2)
77.28
77.6
92.04
82.31
UniTE + SAFE (A 0.3)
77.21
76.6
91.89
81.90
UniTE + SAFE (geometric mean)
78.31
77.6
92.27
82.73
Table 5: Performance across different probability sharpening methods. UniTE ensemble two models, Intemlm3-8B-Instin and Qwen2.5-7B-Instruct.
SAFE is used to
Method
MMLU-redux
GSM8K
ARC-C
InternIm3-8B
76.89
90.1
90.27
Qwen2.5-7B
74.88
91.81
88.74
Two-model ensembling (Internim3 + Qwen2.5)
UniTE SAFE;
77.67
91.66
90.19
UniTE + SAFE,
77.81
92.04
90.78
UniTE SAFEs
78.31
92.04
90.78
Table 6: Ablation on drafter sequence length. SAFE,, denotes generation of n-token sequences.
Probability sharpening. As shown in Table 4, incorporating probability sharpening consistently improves performance across benchmarks and ensemble methods. This result highlights that sharp- ening is beneficial for choosing accurate token when the ensemble distribution becomes overly smooth. To further examine whether different sharpening strategies provide similar benefits, we conduct additional experiments by varying the threshold in our heuristic sharpening method and by applying the geometric mean. As demonstrated Table 5, multiple sharpening strategies offer consistent gains. The geometric mean generally yields strong performance, which we attribute to its ability to gather the probability mass dispersed across multiple tokens by individual models and concentrate on the token with the highest consensus among the models. However, considering that the arithmetic mean is widely used and often required, our heuristic strategy remains useful in such settings. Therefore, we leave the specific choice of sharpening method as flexible de- sign decision, allowing each ensemble method to adopt the sharpening strategy most suitable for its characteristics. The core takeaway is the importance of mitigating an overly smooth ensemble distribution. Regarding the threshold À used in our heuristic sharpening strategy, the performance does not change drastically across different values, However, setting the threshold too high reduces the number of tokens subject to sharpening, which in turn diminishes its effectiveness.
Drafter's sequence length. Table 6 presents the ablation on drafter sequence length. Generating short sequences may fail to capture differences in tokenization across models, causing slight perfor- mance drops. Conversely, generating overly long sequences does not harm accuracy but may reduce efficiency, as shown in Figure 10. This is because longer sequences force the drafter regener- ate tokens more often from the ensembled token. length of 5 provides the best balance between accuracy and efficiency.
CONCLUSION
In this paper, we examined probability-Ieve ensemble methods for long-form generation and showed that deciding when to ensemble critical for both accuracy and efficiency. To this end, we pro- posed SAFE, a mnmeeratevveiff-enneeete framework that triggers ensembling only when safe and necessary, guided by tokenization mismatch and consensus in models' next-token probability distri- butions. SAFE further improved accuracy via probability sharpening to mitigate smooth ensemble distribution, and our KV cache implementation enabled much faster ensembling in long-for gener- ation. Experiments demonstrated that SAFE outperforms existing methods with only few ensem- ble operations across widely used 7C-seale model combinations. We believe SAFE offers a practical step toward making L.MM ensembling both robust and deployable in real-world applications.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was partly supported by Institute for Information & communications Technology Plan- ning Evaluation (IITP) grants (No. RS-2019-I1190075, Artificial Intelligence Graduate School Program (KAIST), No. RS-2024-00457882, AI Research Hub Project, No.2022-0-00713, Meta- learning applicable to real-world problems, No. 2022-0-00984, Development of Artificial Intel- ligence Technology for Personalized Plug-and-Play Explanation and Verification of Explanation) funded by the Korea government (MSIT).
REFERENCES
Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang. Shyam Upadhyay, Manaal Faruqui, and Mausam Automix: Automatically mixing language models. In The Thirty- eighth Annual Conference Neural Information Processing Systems, 2024. URL https:// openreview n forrmmii-=s www.wwessWwwvx.
Soyoung An, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Seokhee Hong, Junwon Hwang, Hyojin Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, et al. Exaone 3.5: Series of large language models for real-world use cases. arXiv e-prints, pp. arXiv 2412, 2024.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang. Tao Jiang. Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong. Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang. Guoteng Wang. Jiaqi Wang. Jiayu Wang. Rui Wang, Yudong Wang. Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu. Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024.
Kris Cao and Laura Rimell. You should evaluate your language model on marginal likelihood tokenisations. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 2104-2114, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/11/2221.com cmmnlp..mmmppppee 161. URL https: pettps//accles http:////aaaaaaaaaaeaceee
org/ /ttgg/22211022 emnlp-ma in. 161/.
Ivi Chatzi, Nina Corvelo Benz, Stratis Tsittsis, and Manuel Gomez-Rodriguez, Canonical autore- gressive generation. arXiv preprint arXiv:2506.066 250.0066666 2025.
Justin Chen, Swarnadeep Saha, and Mohit Bansal. ReConcile: Round-table conference improves reasoning via consensus among diverse LLMs. Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7066-7085, Bangkok, Thailand, August 2024a. As- sociation for Computational Linguisties. doi: 10.18653/v1/200 1,1553/w1/2224aa-oooogg.....1 URL https: //aclanthology wrg. a01-1aancnonng 381/.
Justin Chih-Yao Chen, Sukwon Yun, Elias Stengel-Eskin, Tianlong Chen, and Mohit Bansal. Symbolic mixture-of-experrs Adaptive skill-based routing for heterogeneous reasoning. arXiv preprint arXiv:2503.05641, 2025a.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. Transactions Machine Learning Research, 2024b.
11

---

Published as a conference paper at ICLR 2026
Zhijun Chen, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Dingqi Yang, Hailong Sun, and Philip S Yu. Harnessing multiple large language models: A survey IIm ensemble. arXiv preprint arXiv:2502.18033 2025b.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, htss/8803.0555., 2018. URL http: arxiv. org/ org/abs////eeeoogggg 1803. 05457.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168. 2021. URI.
https::/arxiv..orm wws. 2212011110 0 14168.
Tri Dao. Flashattentionn-:: Faster attention with better parallelism and work partitioning, 2023, URL https://arxix: org/ abs 2207 08691.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch, Improving fac- tuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Jiale Fu, Yuchu Jiang, Junkai Chen, Jiaming Fan, Xin Geng, and Xu Yang. Fast large language model collaborative decoding via speculation. In Forty-two International Conference on Machine Learning. 2025.
Renato Geh, Honghua Zhang, Kareem Ahmed, Benjie Wang, and Guy Van den Broeck. Where is the signal in tokenization space? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3966-3979, 2024.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong. Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile Van Krieken, and Pasquale Minervini. Are we done with MMLU? In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associa- tion for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp. 5069-5096, Albuquerque, New Mexico, April 2025, Association for Computational Lin- guisties. ISBN 979-8-89176-189-6 //aclanthology wttp:/y.c.r//222// www.saaaaaaannana.ea
doi: 10.18653/v 1ttp://wv/222...e/ 1/200/0.m00m0mmmoogggeg nancl-long.2222
URL https:
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming- the rise of code intelligence. arXiv preprint arXiv:2401. 14196. 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Neha Gupta, Hari Narasimhan, Ankit Singh Rawat, Wittawat Jitkrittum, Aditya Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In International Confer- ence on Learning Representations, 2024. URL https: /ttp:////pee/eeeeeeee net faat/forumiin forum?id- KgaBScZ4VI.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Ting Liu, and Bing Qin. Ensemble learning for heterogeneous large language models with deep parallel collaboration. Advances in Neural Information Processing Systems, 37:1 19838-1 19860, 2024.
12

---

Published as a conference paper at ICLR 2026
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-blender: Ensembling large language mod- els with pairwise ranking and generative fusion. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), pp. 14165-14178, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. 188653//112022.acllooo //aclanthology org/2023. aaw-aclloooooo long. 792//
URL https:
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMIR, 2023.
Wenzhe Li, Yong Lin, Mengzhou Xia, and Chi Jin. Rethinking mixture-of-agents:: Is mixing different large language models beneficial? arXiv preprint arXiv:2502.0000) 2025.
Tian Liang. Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang. Rui Wang, Yuju Yang, Shum- ing Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp- 17889-17904, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10. wee.222..mmimmmaiii main.992. URL ht https: www.anchallooooooo org/2024. o-nnlpmmale in. 992/.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lec, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL https:///////t... /tttt:::///ppeeeeeewe net/forumt 1d=v810pN 6E01.
Cong Liu, Xiaojun Quan, Yan Pan, Weigang Wu, Xu Chen, and Liang Lin. Cool-fusion: Fuse large language models without training. In Proceedings the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 10617-10627, 2025.
OpenAl.
Gpt-o4
mini.
https: http://pooaaa.cco//eeeeeeee cww/indem///////////co
Introducing-o3 -0w---33a------mimmmm 2024..
Oxford. The oxford 5000. https:/ /www. htttps:/www.oofoolllleett.... www..oofocllarreeeeaccttooono...
2018.
Qwen, :, An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin 7hu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https:
/ /ttpivaivvvv//////////.a./...av. org/ alos/ / 15115.
Zejiang Shen, Hunter Lang, Bailin Wang, Yoon Kim, and David Sontag. Learning to decode col- laboratively with multiple language models. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pp. 12974 12990, Bangkok, Thailand, August 2024. Asso- ciation for Computational Linguistics. doi: 10. 11111////2224.211l- //aclanthology wwt.ooggg///... acl- 7011
URL https:
Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd-Graber. Getting MoRE out of mixture of language model reasoning experts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8234-8249, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 00.18653/11/003, findings-emnlp.. e52 URL https: /awaa/aanaoooyyoorrgo/ooo org/2023 2023.2023.22 wwndings-ennnnneeeee em 552/.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of thouuught can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL
13

---

Published as a conference paper at ICLR 2026
2023, pp. 13003-13051, Toronto, Canada, July 2023. Association for Computational Linguis- org/2023.
tics. doi: 10.18653/v1/203...o 1t0.://////22..nddinn------ findings-acl.824, acc.824. URL findings-aci, 824/.
Tim Vieira, Tianyu Liu, Clemente Pasti, Yahya Emara, Brian Dusell, Benjamin Lebrun, Mario Giulianelli, Juan Luis Gastaldi, Timothy J. O'Donnell, and Ryan Cotterell. Language mod- els over canonical byte-pair encodings. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Pro- ceedings of the 42nd International Conference Machine Learning, volume 267 of Proceed- ings of Machine Learning Research, pp. 11443,31443, PMLR, 13-19 Jul 2025. / //proceedings ww..m.....ss... v227 wttv/viiiiieeee www.iiirrr2..... html.
URL https:
Junlin Wang. Jue WANG, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities. In The Thirteenth International Conference Learming Representations, 2025a. URL https:/ http::://ppeeeeieeee net ftr.mmirrmmmi/// hOZEDIzj7T.
Junlin Wang, Shang Zhu, Jon Saad-Falcon, Ben Athiwaratkun, Qingyang Wu, Jue Wang, Shuai- wen Leon Song, Ce Zhang, Bhuwan Dhingra, and James Zou. Think deep, think fast: Investigat- ing efficiency verifier-free inference-time--calinng methods. arXiv preprint arXiv:2504. 14047, 2025b.
Ziyao Wang. Muneeza Azmat, Ang Li, Raya Horesh, and Mikhail Yurochkin. Speculate, then col- laborate: Fusing knowledge of language models during decoding. In Forty-second International Conference on Machine Learning, 2025c. URL https: : /wppnroveeeieeeeeee ew. net forummmmmm d= XCBYTFu9Fs:
Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Yangyifan Xu, Jinliang Lu, and Jiajun Zhang. Ilm ensemble. In North American Chapter of the Association for Computational Linguistics, 2024. URL https: / api semant comanticsschnn wcw.hhlar.cc.....c.oooccoa.a...e org/ CorpusID::69149900
Bridging the gap between different vocabularies for
Yangyifan Xu, Jianghao Chen, Junhong Wu, and Jiajun Zhang. Hit the sweet spot! span-level en- semble for large language models. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st Interna- fional Conference on Computational Linguistics, pp. 8314-8325, Abu Dhabi, UAE, January 2025. Association for Computational Linguisties. URL https: aall anthol ogy org//022 2025.
coling ma main.. 555/.
An Yang. Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang. Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu. Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024a. URL https: /twpvv..vvvv///////////.///..av. org/ abss 22477400222 10671.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. 12122, 2024b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.0940 093888 2025.
14

---

Published as a conference paper at ICLR 2026
Yuxuan Yao, Han Wu, Mingyang LIU, Sichun Luo, Xiongwei Han, Jie Liu, Zhijiang Guo, and Linqi Song. Determine- then ensemble: Necessity of top-k union for large language model ensembling. In The Thirteenth International Conference Learning Representations, 2025. URL hotps: / //oenrevieww.. net. forum? dFFDn
thenminn-the--ensmmlleee
Yao-Ching Yu, Chun Chih Kuo, Ye Ziqi. Chang Yucheng, and Yueh-Se Li. Breaking the ceiling of the LLM community by treating token generation as classification for ensembling. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Com-
putational Linguistics: EMNLP 2024, pp. 1826-1839, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. (0.18633/11/0244 0665:///224.fcciiggeem....... findings-emnlp.99, URL https://aclanthol.ee org/2022 findings - emnlp. 99/.
15

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL DETAILS
In this section, we provide additional experimental details. We first present the exact prompts and BBH subjects used in experiments, followed by an explanation of baseline selection.
Prompts For multiple-choice questions, we follow the template in Figure 6. For math datasets, we the template in Figure 7.
Question: [question]
A.
B.
C.
D.
Provide your step-by-step reasoning first, and then print "The answer is (X)" where X is the answer choice (one capital letter), at the end of your response.
Figure 6: Prompt template used for multiple-choice questions.
[question]
Please reason step by step, and put your final answer within \boxed
Figure 7: Prompt template used for math problems.
BBH subjects In our main experiments (Table 2), we filter BBH subjects include only sub- jects where models exhibit comparable performance. This choice is based the observation of UniTE (Yao et al., 2025) that ensemble is meaningful when the base models exhibit similar perfor- mance levels. The chosen 15 BBH subjects are: boolean expressions, causal judgement, date under- standing, disambiguation qa, formal fallacies, logical deduction three objects, movie recommenda- tion, navigate, penguins in table, reasoning about colored objects, ruin names, salient translation error detection, snarks, temporal sequences, and tracking shuffled objects three objects.
Baseline selection As purpose is enhance the stability and efficiency of probability- ensemble methods, we focus on recent, state-of-the-ar probability-Ievel ensemble methods as base- lines. Specifically, we consider formed. The first is GaC, which ensembles only when the main LLM's probability falls below 0.5. To the best of our knowledge, this the only work that does not ensemble every token generation. The second is UniTE, which achieves state-of-the- performance among methods that ensemble every generation step. By applying SAFE both methods, we demonstrate its versatility and its effectiveness in determining when to ensemble.
representative methods that differ in when ensembling is per-
B LIMITATIONS AND FUTURE WORK
SAFE does not always guarantee superior performance compared to the best-performing individual model. As shown in Table 2, in a few cases SAFE performs slightly worse than the best pest-performing individual models. We believe this is because the most confident token the ensemble distribution not necessarily the optimal choice, poorly performing models distort the ensemble distribution by clevating an incorrect token as the most confident. Nevertheless, SAFE generally outperforms individual models by ensuring stable ensemble distribution when applied to ensembles of models with similar capability.
Additionally, experiments limited non-reasoning models. Extending SAFE to reasoning models (Yang et al., 2025: Guo et al., 2025) would be a promising direction, as reasoning models have recently gained significant attention.
16

---

Published as a conference paper at ICLR 2026
C CORRECTNESS OF ENSEMBLE DISTRIBUTION VERIFICATION
Theorem guarantees that applying the ensemble distribution verification criteria does not compro- mise accuracy.
Theorem 1. Let tzj denote the drafter's prefix. For each verifier model LLMo2 E Mvers let te, be its next token, aligned such that t. <v; is the LLM,'s 's tokenization of V2.j. Let Po be the next- token probability distribution of LLM,, Suppose (i) all verifier models unanimously agree on the next token Egg. where Ei arg max, Po(t t² <g) for all or (it their average probability in to, exceeds 4:
E
Polly, t <w;) 2
Myer U Maraft
VEMuorUMara
Then 1 is the token selected by the ensemble, i.e., 1/ arg max, Pens(t tzj), where Pens is the ensemble distribution.
Proof. Define the ensemble distribution average of all models' distributions which is aligned with Maran's tokenization
Pens (l) tzj)
Σ
Po Pr(Ees t" t²<co))
Mverl
+1
w VE UMarat
(i) Unanimous consensus across verifiers.
If for all v Mver we have ty arg max, (t
ty <uy), then for any token u,
(o t <ey) 2 Po(u t" <x,) for all v.
Since by is aligned with tj for all v, averaging over preserves the inequality:
(ens(t |tz)== |Mvrrl + E t <n)) 2 M + Po(u t" <xy))= = =Pons(u tej)-
Hence, ty arg max, Peus (t ttz).
(ii) Average probability above one half.
Assume
Pens (ty tzr) [Mverl
Σ
Po t <"<vv) 2.
UMaaa
Because Pens(: tz2) is a probability distribution, Penatt tz2) = Thus, no other token tj can have (u t<i) > Pens(ty tzj), since two distinct tokens cannot both exceed 1/2. Therefore, 1j = arg max, Pens (t tzz))
In either case (i) or (ii), the ensemble selects lg, proving the claim.
KV cache
Drafter
2
The
answer
is
Inc
orr
ect
The
answer
is
Inc
orr
ect
Prune cache
LLM Ensemble with verifier models
Drafter
After Ensemble
answer
is
Correct
The
answer
Correct
Figure 8: Our KV cache management. The cache pruned to ensure alignment with the ensembled output.
17

---

Published as a conference paper at ICLR 2026
D DETAILS OF OUR KV CACHE MANAGEMENT
While KV caching is essential for generating long responses, it introduces a challenge in ensem- ble settings because the token selected by the ensemble may differ from the tokens generated by individual models. As illustrated in Figure 8, the drafter initially generates The answer is Incorrect, but after ensembling the output becomes The answer is Correct. In this case, the drafter's s KV cache, which contains states for the discarded token Incorrect, must be updated. To resolve this inconsistency, we prune each participating model's KV cache by fixed buffer the end of every ensemble step. This pruning ensures that each model's KV cache is consistent with the actual input sequences before producing the next token.
E LATENCY COMPARISON
We present additional lateney comparisons general-domain datasets, where the proportion of ensembled tokens is higher than in math datasets. Figure shows the latency of SAFE on MMLU- redux dataset. As illustrated, SAFE substantially reduces latency compared existing ensemble methods and achieves speeds comparable to individual models when generating hundreds of tokens, in general-dome tasks. These results highlight the practical applicability of SAFE, demon- strating that it enables efficient ensembling across diverse domains.
fast Sefi
SAVA
Uwer
ERACAE
SAFE
Vaic
Latency comperison
(6) Lateney comparison GaC
(6) Latenoy caching
Figure 9: Latency comparison on MMLU-redux. SAFE significantly improves efficiency on general- domain tasks. Note that the time-axis scale in (c) differs from (a) and (b).
F ABLATION ON THE DRAFTER'S SEQUENCE LENGTH
SAFE,
UnITE SAFE: UNITE SAFE,
SAFE,
Time
Number
Tokonss
Figure 10: Lateney comparison depending the drafter's sequence length, where denotes the drafter's sequence length SAFE,, Generating a longer sequence reduces efficiency. We use MMLU-redux for the comparison.
G ENSEMBLING LARGER MODELS
Table 7 presents the results for ensembling 32B-scale models: Dwwen..5-32B-Inee (Qwen et al., 2025) and EXAONE-3.5-32 S.ONN--3--32B-nssructt (An et al., 2024), evaluated on MMLU-redux and MATH500. For MMLU-redux, redux* includes only subjects, excluding subjects where Qwen2.5- 3---332BIInsstr largely out- performs AXAONE-335-3215 Instrrutt by more than 10%, making ensembling less meaningful. As
report two variants: MMLU-redux* and the full MMLU-redux. MMLU-
18

---

Published as a conference paper at ICLR 2026
shown in the table, applying SAFE to existing ensemble methods consistently outperforms the base- lines, demonstrating its effectiveness on larger-scale models. The 9 subjects excluded in MMLU- redux* are: college chemistry, college mathematics, college physics, formal logic, electrical engi- neering, high school chemistry, professional accounting, clinical knowledge, and econometrics.
MMLU-reduxx
MMLU-redux
MATH500
Method
Accuracy
Accuracy
Accuracy
Qwen2.5-32B
85.06
84.54
80.8
EXAONE3.5-32B
82.34
79.26
77.2
Two-model ensembling (Qwen2.5 + EXAONE3.5)
GaC
84.70
82.73 83.79
80.4
GaC + SAFE
85.11
81.6
Table 7: Results of ensembling 32B-scale models.
H RELAXED ACCEPTANCE THRESHOLD
the Ensemble Distribution Verification step, the acceptance threshold can be relaxed to support various scenarios, such as sampling or weighted combinations of probability distributions, rather than simple averaging. First, if the ensemble distribution is defined as a weighted sum the mod- els' next-token probability distributions, the second condition of Ensemble Distribution Verification becomes:
(Average probability above half) If LLEM, P.Potty t ("<vy)= >
1.4 we skip ensembling at ty,
where w is the weight assigned to LLM, and Σ, wi 1.
If we wish to use sampling rather than greedy decoding, we can incorporate speculative sam- pling (Leviathan et al., 2023) into our framework. In this case, we do not use the first condition (Unanimous consensus among verifiers), and the second condition relaxed as follows:
(Average probability above one half) - (Speculative sampling) We skip ensembling at t, with probability min(1, (t,
(tne(t,(ter))
Similar to greedy decoding setup, P Pena(ty (te))z))) is derived as:
Pens (tg)
Σ
Po( tt <xg))
Myver Marat.
LLM EMMeeUMaaaMMaaa where ty is the token corresponding to iú under LLM,'s 's tokenization.
I FURTHER DISCUSSION OF OOV-LIKE TOKENS
In this section, we provide a more detailed explanation about OOV-like tokens, which we use to de- termine whether tokenization mismatch occurs. 0OV-like tokens are defined as tokens that can cor- rupt the next-token probability distribution of a participating model due to differences in tokeniza- tion across models. As described in Section 3.2, whether a drafter token is classified as OOV-like determined by checking whether the token boundary up to that token aligns with the tokenization boundaries of the verifiers.
One similar concept to OOV-like token is non-canonical tokenization, which refers to any tokeniza- tion other than the canonical tokenization for given model. Several studies (Cao Rimell, 2021; Geh et al., 2024; Chatzi al., 2025; Vieira al., 2025) have explored the of non-canonical tokenizations in generation within a single LLM. However, we highlight two important distinc- tions between OOV-like tokens and non-canonical tokenizations. First, OOV-like tokens are defined
19

---

Published as a conference paper at ICLR 2026
strictly the token level. For example, consider the sequence "Hello, world". If (Hello, world) the canonical tokenization, then alternative tokenizations such as (Hell, o, wwolld) or (Hello, do not treat entire
w, orld) would be considered non-canonical tokenizations. However, tokenization like (Hell, 1, o, world) as an OOV-like case. Instead, we identify only the specific mismatched tokens within such non-canonical tokenizations, such Hell kens. Second, OOV-like tokens consider only the tokenizations used by the participating models, whereas non-canonical tokenization tions of sequence. For example, if the participating models tokenize "Hello" in only one way, (Hello), then no OOV-like token exists. In contrast, non-canomical tokenization includes any tok- enization that can compose Hello, such as (H, 1, a) or (He, 11, o), regardless of whether it is produced by the models. These distinctions directly relate to how we handle tokenization mismatch. We only assess mismatch at the token level, using only tokenizations of participating models. These two distinctions clearly represent differences from prior work on non-canonical tokenization.
_w, OOV-like
broader concept that encompasses all possible tokeniza-
J COMPARISON WITH METHODS THAT ENSEMBLING AT A SPAN-LEVEL
We discuss how our method differs from methods that ensemble at a engger-gaanularity. such as span-level ensembling, in addressing tokenization mismatch. Recently, several approaches (Xu et al., 2025; Liu et al., 2025) have proposed ensembling at a larger granularity, such as a set of words, to bypass the tokenization mismatch problem when ensembling LLMs that use heterogeneous tokeniz- ers. These methods avoid tokenization mismatch by not performing ensembling at the token level and instead ensemble only the span level. As result, ensembling cannot occur inside span, thereby sacrificing token-level probability information. In contrast, our method addresses tokeniza- tion mismatch within token-level ensembling. Our method resolves OOV-like issues by identifying token-level positions in the drafter's tokens where no mismatch occurs, and only these positions are considered valid candidates for ensembling. This allows ensembling to take place within a span while avoiding OOV-like issues, providing greater flexibility in where ensembling can occur.
K QUALITATIVE RESULTS
In this section, we present qualitative results of SAFE. Tables 8 to 11 compare responses from existing ensemble methods with those obtained when SAFE applied. As shown Table SAFE can successfully yield the correct solution even when individual models fail to produce the correct answer or reasoning. Tables and 11 further illustrate how existing ensemble methods suffer from repeated generation of erroncous tokens due to the introduction of OOV-like tokens. Finally, Tables and 10 highlight cases where existing ensemble methods generate incorrect reasoning that leads to wrong answers.
L LLM USAGE
We used the LLM solely for the purpose of polishing the writing.
20

---

Published as a conference paper at ICLR 2026
Question
Sofia and Tess will each randomly choose one of the 10 integers from I to 10. What is the probability that neither integer chosen will be the square of the other?
(A) 0.64 (B) 0.72 (C) 0.81 0.95
Answer
(D) 0.95
Internim3-
To determine the probability that neither integer chosen by Sofia and Tess will be the square of the other, we can follow these steps: 1. Ideetiffy the squares of the integers from to 10: The squares of the integers from I to 10 are: [12 = 1, 1,2² 4,3² = 9] So, the squares are 1, 4, and 9, Determine the total number of possible outcomes: Each of Sofia and Tess can choose any of the 10 integers, so the total number of possible outcomes is: [10 x 10 = 100] 3. Count the number of outcomes where one integer is the square of the other: We need to count the pairs ((a, b)) where (a) is the square of (b) or (b) is the square of (a). The pairs are: [(1, 1), (1, 4), (1, 9). (4, 1), (4, 4), (4, 9), (9, 1). (9, 4), (9. 9)] There are 9 such pairs. 4. Calculate the number of outcomes where neither integer is the square of 91] 5. Calculate the probability: 100 0.91] However, none of the provided answer choices match 0.91. The answer is C.
8B-Instruct
the other: [100-
Qwen2.5- 7B-Instruct
To solve this problem, we need to determine the probability that neither of the two chosen integers is the square of the other. First, let's identify the squares of integers from I to 10: (12 = (2 4), (32 9) So, the only pairs of numbers in the range from I to 10 where one number is the square of the other are: 1), (4, 2), (4, 4), (9, 3), (9, 9) There are 5 such pairs. Since each person can independently choose any of the 10 integers, there are a total of (10 x 10 = 100) possible outcomes when both Sofia and Tess make their choices. Next, we calculate the number of outcomes where at least one of the chosen integers is the square of the other. This includes the pairs listed above and their reverses (since if (4, 2) is a pair, so is (2, 4)). However, since (1, 1), (4, 4), and (9, 9) are symmetric, they are already counted once. Therefore, we have:(1, 1) I way, (4, 2) and (2, 4) ways, (4, 4) way, (9, 3) and (3, 9) 2 ways, (9, 9)- way. This gives us a total of (1 + 1 + 2 1 = 7) pairs where one number is the square of the other. Thus, the number of outcomes where neither integer is the square of the other is (100 7 = 93). The probability that neither integer chosen will be the square of the other is therefore: 93 0.93] However, this does not match any of the provided options exactly.... Given the options, the closest match to our calculated probability of 0.93 is 0.95. The answer is (D).
100
UniTE
To determine the probability that neither integer chosen by SofĀa and Tess will be the square of the other, we need to follow these steps: 1. Identify the squares of the integers from to 10: The squares of the integers from 1 to 10 are: [12 = 1, 4, = 9] So, the squares are 1, 4, and 9. 2. Determine the total number of possible outcomes: Each of SofÄÄÄÄÄ.
UniTE SAFE
To determine the probability that neither integer chosen by Sofia and Tess will be the square of the other, we can follow these steps: 1. Identify the squares of the integers from I 10: The squares of the integers from I to 10 are: [12 = 1, 4,3² = 9] So, the squares are I, 4, and 9. 2. Determine the total number of possible outcomes: Each of Sofia and Tess can choose any of the 10 integers. Therefore, the total number of possible outcomes is: [10 x 10 100] 3. Count the number of outcomes where one integer the square of the other: We need to count the pairs ((a, b)) where (a) is the square of (b) or (b) is the square of (a). The pairs are:
[(1, (4,2), (2, 4), (9, 3), (3, 9)] There are 5 such pairs. 4. Calculate the number of outcomes where neither integer is the square of the other: [100- 5 95]
5. Calculate the probability: [ -f 0.95] Therefore, the probability that neither integer chosen will be the square of the other is D.
Table 8: Response comparisons of UniTE and UniTE + SAFE. Some verbose outputs are omitted due to space constraints.
21

---

Published as a conference paper at ICLR 2026
Question
Terry eats 2 yogurts a day. They are currently on sale at 4 yogurts for $5.00. How much does he spend yogurt over 30 days?
Answer
75
Qwen2.5-
Let's break down the problem step by step to find out how much Terry spends on yogurt over 30 days. 1. Determine the number of yogurts Terry eats in 30 days: Terry eats yogurts per day. Over days, he will eat (2 x 30 = 60) yogurts. 2. Calculate how many sets of yogurts are in 60 yogurts: Since the yogurts are sold in sets for $5.00, we need to find out how many such sets are
7B-Instruct
yogurts. The number of sets is ( 4 = 15). 3. Calculate the total cost: Each set of 4 yogurts costs $5.00. For 15 sets, the total cost is (15 x 5 75) dollars. Therefore, the amount Terry spends on yogurt over 30 days is
60
75
EXAONE- 3.5-7.8B- Instruct
To determine how much Terry spends yogurt over 30 days, let's break down the problem step by step: 1. Calculate Daily Yogurt Consumption: Terry eats yogurts per day. 2. Determine Total Yogurt Consumption Over 30 Days: Over 30 days, Terry will consume (2 yogurts/day x 30 days = 60 yogurts). 3. Understand the Sale Price: The yogurts are sold rate of 4 vogurts for $5.00. 4. Calculate the Cost Per Yogurt: The cost per yogurt is dollars = 1.25 dollars/yogurt). yogurts
5. Calculate Total Cost for 60 Yogurts: If each yogurt costs $1.25, then for 60 yogurts, the total cost will be (60 yogurts x 1.25 dollars/yogurt = 75 dollars).
Therefore, the total amount Terry spends on yogurt over 30 days is 75
UniTE
break down the problem into several steps: 1. Calculate the daily yogurt consumption in terms of the sale unit: Terry eats yogurts per day. The sale is yogurts for $5.00. Therefore, Terry consumes ( = 0.5) of a sale unit per day. 2. Determine the daily cost based on the sale price: Since 4 yogurts cost $5.00, yogurt costs (5.00 1.25). Therefore, 0.5 a sale unit (which equivalent 1.25 0.625). So, Terry spends $0.625 per day on yogurt. 3. Calculate the total cost over days: Terry spends $0.625 per day. Over 30 days, the total cost (0.625 30 18.75). Therefore, the total amount Terry spends on yogurt over 30 days is 78.75
To determine how much Terry spends yogurt over 30 days,
yogurts) costs (0.5
UniTE SAFE
To determine how much Terry spends on yogurt over 30 days, we can break down the problem into several steps: 1. Calculate the daily yogurt consumption cost: Terry eats 2 yogurts per day. The sale price is yogurts for $5.00. To find the cost per per yogurt, we divide the total cost by the number of yogurts: Cost per yogurt -4 = 1.25 dollars per yogurt. Since Terry eats yogurts per day, the daily cost is: Daily cost 2 x 1.25 2.50 dollars 2. Calculate the total cost over 30 days: Terry spends $2.50 per day on yogurt. Over 30 days, the total cost is: Total cost = 2.50 x 30 75.00 dollars Therefore, the total amount Terry spends on yogurt
over 30 days is 75.00
Table 9: Response comparisons of UniTE and UniTE + SAFE.
22

---

Published as a conference paper at ICLR 2026
Question
Compute 992 + 99 + 1 in your head.
Answer
9901
Internlm3-
To compute 992 99 + in my head, I can use algebraic manipulation to simplify the expression. Let's start by rewriting the expression in more convenient form. First, let's denote x 99. Then the expression becomes: x² + x + 1. Notice that this expression can be rewritten using the identity for the square of binomial: x²²
8B-Instruct
back into the expression, we get: 99a + 99 1 (99 + 1)² 99. Simplifying inside the parentheses,
(x: + 1)² z. Substituting x
have: 99 + 100. So the expression becomes: 100 99. Now, we know that
100 10000, the expression simplifies to: 10000 99 9901. Therefore,
the value of 9g2 99 is 9901
Qwen2.5- 7B-Instruct
To compute 992 + 99 + my head, will break down into simpler steps. First, let's calculate 99 [992 (100-1)² 100 1--100-1+111 1 10000-200++ 9801.] Next, we add 99 to the result: 9801 + 99 = 9900. Finally, we add to the
result: 9900 + 1 = 9901. So, the final answer is 9901
GaC
To compute 9g + 99 + in my head, I can use algebraic manipulation simplify the expression. Let's start by rewriting the expression in more convenient form. First, let's denote need to compute x² + x: + 1 for x = 99. Notice that x² + + 1 can be rewritten using the identity for the sum of cubes: x² + x +
99. Then the expression becomes: x² +
Now, we
(xr + 1) x. However, simpler approach is to recognize that x22 x can be expressed as: x²a + (x+1)² 2 + = (x + 1)2. So, we have: 992 + +99 (99+1) 1)2 = 1002. Now, we jusu need to compute 1 100 10000. Therefore, the value of 992 99 +1
is 10000
GaC SAFE
To compute 992 + 99 + in my head, I can use algebraic manipulation to simplify convenient form. can use the identity for the square of binomial to rewrite x:2
the expression. Let's start by rewriting the expression in First, let's denote x = 99. Then the expression becomes: x:2 + x + 1. Now, we
Notice that: x² + + = (x + 1)2 x. Substituting æ 99 back into the expression, we get:9g²
+ (99 1)a 99. Simplifying inside the parentheses, we have: 99 + 1 = 100. So the expression becomes: 1002 99. Next, we compute 1002: 100 10000. Now, we subtract 99 from 10000: 10000 99 9901. Therefore,
the value of 992 99 1 is 9901
Table 10: Response comparisons of GaC and GaC + SAFE.
23

---

Published as a conference paper at ICLR 2026
Question
At 25"C, the maximum amount of Pbl₂ that can be dissolved in 1.00 L of pure water is 1.0 mmol. Assuming complete dissociation, the solubility product. Ksp, for lead iodide at 2 is (A) 1.0 x 10-3, (B) 1.0 x 10-6, (C) 1.0 x 10-9, (D)
4.0 x 10-9
Answer
(D)
Qwen2.5-
To solve this problem, we need to understand the relationship between the sol- ubility of compound and its solubility product constant K Ksp- 1. Dissociation of Lead Iodide Pbl2: Pbl2(s) = -bb* (ag) 21 (ag) 2. Solubility in Water: Given that 1.0 mmol (or 1.0 x 10-3)) of Pbl2 dissolves in 1.00 L of pure wa- ter, we can determine the concentrations of the ions: The concentration of Pb is 1.0 10-3M. The concentration of I is 2 x 1.0 10 = 2.0 x 10-3M. 3. Solubility Product Constant Kap: The expression for the solubility product constant Kep for Pbly is: Ksp [ Substituting the concentrations:
7B-Instruct
Ksp = (1.0 x (0.0 x 10- = (1.0 x 10 x 10-6) = 4.0 x 10-9
Therefore, the solubility product, Ksp. for lead iodide at 25 4.0 x 10-9, The answer D.
EXAONE- 3.5-7.8B- Instruct
To determine the solubility product constant Ksp for lead iodide Phl2, need to follow these steps: 1. Understand the Dissociation Reaction: Lead iodide dis- sociates in water as follows: PbI2 - Pb - 21 2. Determine the Concentra- tions: Given that the maximum amount of Pbl2 that can dissolve in 1.00 L of water is 1.0 mmol, we can find the molar concentrations of Pb2+ and I ions produced. Since I mmol of Pbl2 dissociates into mmol of Pb2+ and 2 mmol of I-: [Pb2-] 2.0 mmol/L. = 2.0 x 10-3 M 3. Write the Expression for Ksp: The solubility
1. mmol 1.0 mmol/L 1.0 10-3 M, [ 1.0.0 mmol
1.0L
1.0L
product constant expression for Pbl2 is: Kap 2pb21 [ 4. Substitute the Concentrations into the Kyp Expression: Kap (1.0 x 10 3) (2.0 10-
(1.0 x 10-3). (4.0 10 4.0 x 10- Therefore, the solubility product con- stant Ksp for lead iodide Pbl, at 25 is 4.0 x 10 9, The answer is (D).
GaC
To solve this problem, we need to understand the relationship between the sol- ubility product constant Kap and the solubility of a compound. 1. Identifing the Dissociation Equation: Lead(II) iodide Pbl, dissociates in water as follows: PbI2 = 2b (ag) + 21 (aq) 2. Expressing Solubility in Terms of Concentra- tions: Let be the molar solubility of Phl2- This means that when Phl2 dissolves, it produces moles of ag per liter of solution and 2s moles of Iaaaaa...
GaC SAFE
To solve this problem, need to understand the relationship between the sol- ubility product constant Kap and the solubility of a compound. For lead iodide Pbl,, the dissociation in water can be represented as: Pbl9 = pb - 21 Given that the maximum amount of Phl2 that can be dissolved in 1.00 L. of pure water is 1.0 mmol (which is 1.0 x 10 mol, we can determine the concentrations of the ions in solution. Determine the concentration of 2b ions: Since 1.0 mmol of Pbl, dissociates into 1.0 mmol of pb²- ions, the concentration of pb ions is: 3 10-3 2. Determine the concentration of I- ions: Each mole of Pbl, dissociates into 2 moles of I ions. Therefore, the concentration of I 3. Calculate the solubility product constant Kap The expression for Kap for Pbly is: Kap = Pb I Substituting the concentrations we found: Kap = (1.0 x 10-0) 10 10-3y2 (1.0 10 (4.0 10 B) = 4.0 x 10 Therefore, the solubility product, Ksp- for lead iodide at 25cC is 4.0 x 10-9. The answer is D.
[pb2+] = 1.0 x 10
ions is: [[] = 2 x 1.0 X 10 10-3M = 2.0 x 10 -
Table 11: Response comparisons of GaC and GaC + SAFE. Some verbose outputs are omitted due to space constraints.
24


Reviews:
[{"review_id": "p4hHTZTwLQ", "reviewer": "Reviewer_4xbi", "summary": "This paper proposes a LLM Ensemble method, specifically categorized as an ensemble-during-inference method.   \nThe proposed method uses a main model to generate fragment-level information, and then utilizes several other models as validators to check the \"potential error-prone areas\", and finally integrates the outputs of multiple models in these areas.", "questions": "Please refer to the above \"Weaknesses and Suggestions\" and the following:  \n1) The 5.72% performance improvement mentioned in the introduction—where is it reflected in the tables? The explanation of this in the experiment section seems unclear.  \n2) What aspect or meaning does the term \"stable\" in the paper's title primarily refer to? It would be helpful to explain and address this question from both the methodological and experimental result perspectives. It is recommended that the authors consider this issue in the final version of the paper. \n3) If the authors intend to emphasize efficiency, it is recommended that they provide a comparison of the overall runtime of the proposed method and the baselines on these datasets.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed generate-verify-ensemble method, in terms of its overall approach and framework design (i.e., from a high-level perspective), is reasonable and intuitive.", "weaknesses": "**Weaknesses and Suggestions.**\n\n\n1) The presentation of the method and experiments is not clear enough. This version of the paper requires significant revisions to make the presentation clearer and more accessible for readers.  \nSpecifically: The presentation in the method section (Section 3) is not clear enough. Regarding the experiment section, for example: (i) In Section 4.2, when introducing the set of LLMs considered, mentioning the prior work proposed by Yao et al. is unnecessary and may cause confusion, as the LLM set in this paper differs from that in the prior work. (ii) The meanings of \"GaC + SAFTE\" and \"UniTE + SAFE\" are not clearly explained in the Section 4.2.\n\n2) For the related work section, it is recommended to include an introduction to some classic works on speculative decoding, as the research in this paper is closely related to speculative decoding.\n\n3) The main experimental results (Table 2) presented in this paper seem to show marginal performance improvement. It is recommended to include an overall average result in Table 2 to show the average performance improvement across all cases. Further experiment tuning and the design of new tricks could help improve the performance of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TFfvzdblyr", "reviewer": "Reviewer_gPZv", "summary": "The paper argues that LLM ensembling only needs to occur for a subset of tokens determined by: (1) the mismatch in tokenization across different models and (2) the level of consensus in their next-token probability distributions. The authors introduce SAFE (Stable And Fast LLM Ensembling), a framework designed to find the best tokens for ensembling during long-sequence generation. SAFE uses a speculative strategy where one \"drafter\" model generates a lookahead sequence of tokens, and the other \"verifier\" models identifies the specific token-level points within that sequence that require ensembling.", "questions": "1. To confirm my understanding, is it true that if you apply SAFE ensembling between a model and itself, it will give the same result as just sampling directly from the model?  I want to confirm that ensembling does not induce temperature-scaling-like effects that boost performance.\n\n2. The authors repeatedly claim that vocabulary alignment is a primary expense, e.g., \"the number of costly ensemble operations grows with sequence length. This expense arises primarily from vocabulary alignment...\" and \"high computational cost of repeated vocabulary alignment makes such approaches inefficient.\" Can you elaborate on this? My understanding is that the primary bottleneck for ensembling is the cost of forward passes through multiple models, not vocabulary alignment. Why is alignment expensive in this setup?\n\n3. The latency is shown to be lower, but what is the maximum memory usage (peak memory) of SAFE compared to the baselines?\n\n4. What is the computational overhead of repeatedly calling the verifier models' tokenizers during the verify step?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-motivated and clearly written.\n\n* The experiments are fairly rigorous. The method achieves better quality and efficiency over baseline ensembling methods.\n\n* A notable systems contribution is the implementation of KV caching for ensembling, which the authors also apply to baselines.\n\n* It is an interesting finding that ensembling at a small fraction of tokens (less than 20%) can significantly improve the overall generation quality.", "weaknesses": "* The paper introduces the term \"OOV-like,\" but this appears to be identical to the existing concept of \"non-canonical\" tokenization found in prior work [1, 2, 3, 4]. The paper fails to cite or discuss this relevant literature on tokenization.\n\n* The probability sharpening strategy seems arbitrary. The choice of the 0.1 threshold is not well-justified. There are other standard ways to sharpen distributions (e.g., using a geometric mean instead of an arithmetic mean) that are not explored or compared.\n\n* The experiments relating to efficiency could be more complete to show where exactly the speedups are coming from (see questions below).\n\n[1] Cao, Kris, and Laura Rimell. \"You should evaluate your language model on marginal likelihood over tokenisations.\" arXiv preprint arXiv:2109.02550 (2021).\n\n[2] Geh, Renato Lui, et al. \"Where is the signal in tokenization space?.\" arXiv preprint arXiv:2408.08541 (2024).\n\n[3] Vieira, Tim, et al. \"Language Models over Canonical Byte-Pair Encodings.\" arXiv preprint arXiv:2506.07956 (2025).\n\n[4] Chatzi, Ivi, et al. \"Canonical Autoregressive Generation.\" arXiv preprint arXiv:2506.06446 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vVPw30OKGR", "reviewer": "Reviewer_ux3H", "summary": "This paper introduces a new framework, SAFE, to determine when to ensemble models' next-token generation probability. The framework allows ensemble happen only when (1) no tokenization mismatch occurs across models and (2) the models’ next-token distributions show low consensus. SAFE increase model inference efficiency and accuracy on long-form generation under ensemble.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Disclaimer: reviewer is not an expert on the topic.\n\n1. The paper is well-presented.\n2. I think the paper's idea of using token mismatching and token confidence to decide whether to ensemble is intuitive. SAFE can be applied to any existing next-token ensemble method.\n3. The paper experiments clearly show gains for efficiency and accuracy for using SAFE.", "weaknesses": "Disclaimer: reviewer is not an expert on the topic.\n\nI'd like to see an ablation study for how each of the two criteria in SAFE impact ensemble method performance.\n\nI will check other reviewers' comment on the weakness part of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D312rE8Ybw", "reviewer": "Reviewer_vm3X", "summary": "This paper studies ensemble methods in long-form generation tasks, demonstrating that existing ensemble methods fail to perform well in these tasks because of tokenization mismatch across models and divergence in their next-token probability distributions. To address this problem, they propose SAFE, a novel ensemble method. Experiments show that SAFE achieves better performance and efficiency.", "questions": "1. What is the main contribution compared to existing approaches?\n2. What would be the result if greedy decoding is replaced with common sampling techniques?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The focus on long generation/CoT tasks is a novel perspective for model ensemble research.\n2. The proposed approach leads to better performance across different benchmarks and ensemble strategies. It also address the huge inference overhead.\n3. The writing is clear and easy to follow.", "weaknesses": "1. Some related works should be discussed and compared, which could impact the novelty and relative performance:\n    a) Speculative decoding is already utilized to accelerate model ensemble in CoS [1].\n\n    a) The problem of OOV can be addressed by ensembling at a larger granularity (e.g., word of span), as shown in SweetSpan [2]. I think the same idea is used in SAFE.\n\n    b) EVA [3] (which is the first token-level ensembling approach) shows that the divergence between models could be addressed by dynamically adjusting the weight of different models according to their confidence. I would suggest that the authors try these approaches.\n\n2. For the verification step, the proposed token is only accepted when it is the most possible one for all models. However, modern LLMs typically use random sampling with topk/topp renorm for diversity. This is crucial for generating long sequences without rotting and further test-time scaling. I think expanding the acceptance threshold would be a more reasonable approach.\n\n[1] Fu, Jiale, et al. \"Fast Large Language Model Collaborative Decoding via Speculation.\" \n\n[2] Xu, Yangyifan, et al. \"Hit the sweet spot! span-level ensemble for large language models.\"\n\n[3] Xu, Yangyifan, et al. \"Bridging the Gap between Different Vocabularies for LLM Ensemble.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)