Diverse Text Decoding via Iterative Reweighting
Authors: Ruiqi Shi
Sinno Jialin Pan
Venue: ICLR 2026 Poster (2026)
Keywords: Natural Language Processing
Large Language Models
Text Generation
Decoding Method
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose OverRIDE, a dynamic decoding method that improves text generation diversity.

Abstract: Recent advances in large language models (LLMs) have led to impressive results in text generation. However, current decoding methods still lack diversity when combined with popular sampling techniques. We propose a Reweighting-based Iterative DEcoding (OverRIDE) approach that dynamically adjusts the decoding process with history responses. Our method fine-tunes auxiliary output heads iteratively on previously generated sequences to capture and suppress semantic patterns that appear in the history responses. This inference-time training process only incurs minimal loss of efficiency. We conduct extensive experiments on various tasks, including code generation, mathematical reasoning and story generation, demonstrating that OverRIDE increases output diversity while maintaining quality. We implement OverRIDE on LLM serving systems like vLLM, achieving a 6.4% throughput loss for 72B models under parallel decoding. The code is available at https://github.com/shi-rq/OverRIDE.

Published as a conference paper at ICLR 2026
DIVERSE TEXT REWEIGHTING
DECODING
VIA
ITERATIVE
Ruiqi Shi
Sinno Jialin Pan
The Chinese University of Hong Kong rqshi@link cuhk. hk, sinnopan@cuhk. edu. hk
ABSTRACT
Recent advances in large language models (LLMs) have led to impressive results in text generation. However, current decoding methods still lack diversity when combined with popular sampling techniques. We propose a Reweighting-bass Iterative DEcoding (OverRIDE) approach that dynamically adjusts the decoding process with history responses. Our method fine-tuner tin-ttnes auxiliary output heads iter- atively on previously generated sequences to capture and suppress semantic pat- terns that appear in the history responses. This inference-time training process only incurs minimal loss of efficiency. We conduct extensive experiments on var- ious tasks, including code generation, mathematical reasoning and story genera- tion, demonstrating that OverRIDE increases output diversity while maintaining quality. We implement OverRIDE ing a 6.4% throughput loss for 72B models under parallel decoding. The code is available ptthttss////t httpggtthhhhtcee com shi shi-rq/OverRIDE
LLM serving systems like vLLM, achiev-
INTRODUCTION
Recent advances in Large Language Models (LLMs) have brought significant improvements to ap- plications in natural language processing, such as creative writing (Fan et al. 2018 Brown et al.,
2020
Stiennon et al.) 2020 Rafailov et al. 2023), mathematical reasoning Cobbe et al.) 2021 : Yu
al.
2024| Luo al. [025] Rorder-Paarddes al. 2024), and code generation (Li et al. 2022]
Rozière et al. 2023 Li et al. 2023a] El-Kishky et al. 2025). To enhance the quality and diversity of
LLM generations, various sampling techniques have been proposed. For instance, top-k sampling
(aan et al. 2018) and top-p sampling (Holtzman al. 2020)) work by filtering out low-confidence tokens, which helps ensure the overall quality the generated text; Temperature sampling (Ackley
al.
1985) controls the randomness of the sampling process, allowing for more creative outputs.
Despite these advances in sampling techniques, current generation strategies still struggle to pro- duce adequately diverse outputs. This limitation comes from the autoregressive nature of mod- emn language models. The standard decoding process for autoregressive language models involves sequentially predicting each token conditioned all previously generated tokens, creating path- dependent generation process. In practice, this often leads to responses that share similar beginnings, and only diverge toward the end, as illustrated Figure Often, the generated samples are struc- turally similar, with only minor variations in local details. For example, in code generation tasks, models may produce different variable names or formatting styles, but the underlying logic remains the same, which leads to the same mistake across multiple generations.
The issue lies in the decoding strategy: given the same context, the same next-token probability distribution is applied at every round of response, regardless of what has been generated in previous rounds. Although various sampling methods adjust this distribution by, for example, truncating the tail or rescaling the distribution, this process is done only at the token level independently. While these sampling-best based approaches enable stochastic decoding, they fail to learn from the history of generated samples and lead redundant exploitation of high-probabilityy regions in the generation space. This often causes degeneration problems like repetitions or unfavorable outputs. (Holtzman et al. 2020 Welleck et al. 2020 Su et al. 2022)
To address the aforementioned limitation, we introduce OverRIDE Reweighting-base Iterative DEcoding), decoding method that dynamically adjusts token probabilities across different de-

---

Published as a conference paper at ICLR 2026
Next-Token Distribution
Standard Decoding
To determine how many possible numbers Rick and Steve could be thinking of
Round
To solve this problem, we first need Identify the positive factors
Reweighting
Risk
Eait
Pasz
AWo
Stin
To solve this problem, need determine the common factors of 14 and 42
Round
OverRIDE
To determine how many possible numbers Rick and Steve could be thinking
Round
Rick is thinking of a positive factor $14$, and Steve is thinking of a posilive
$14$ has factors $1, 7, 14$ and $425 has factors $1, 2. 3, 7, 14, 21, 42$
Stave
Figure 1: Standard autoregressive decoding often leads to similar outputs. OveRIIDE dynamically reweights token probabilities by learning from the patterns of previous generations.
coding rounds based on previously generated responses. OverRIDE iteratively fine-tunes guide model on previously generated samples to identify common semantic patterns. This guide model then used to reweight the next-token distribution of the original model by suppressing the prob- ability of tokens that lead to previously seen patterns. In such way, encourage exploration of lesv-traveled paths in the generation space. By learning from previous generations, OverRIDE can effectively suppress repetitive patterns, leading to more diverse outputs, as illustrated in Figure I Although the training of the auxiliary model is performed at inference time, it is relatively efficient compared to the cost of autoregressively generating entire response. Additionally, OverRIDE can be combined with existing sampling methods to further improve the generation process.
Furthermore, design parallel version of OverRIDE, which allows seamless integration with LLM serving systems like vLLM (Kwon et al. 2023) and SGLang (Zheng et al. 2024), and benefits from the efficiency of parallel decoding. We restrict the trainable parameters to tiny amount in the output head, allowing for minimal efficiency loss. We also synchronize the sampling and fine-tuning processes, which makes it compatible with the decoding mechanism L.M serving systems.
We evaluate OverRIDE on a variety of tasks, including code generation (HumanEval (Chen et al. 202T)), mathematical reasoning (MATH500 (Hendrycks al., (02I], GSM8K (Cobbe et al.) 2021) and story generation (CCNews (Common Crawl] (007)). Our experiments demonstrate that Over- RIDE improves the response diversity while maintaining or improving quality. Additionally, Over- RIDE prevents models from generating over-confident responses. We further analyze the decoding dynamics of OverRIDE, showing how it explores high- and low-probable probbabiiilyy regions in the genera- tion space achieve better diversity. We implement OverRIDE on vLLM (Kwon et al.) 2023), and achieve a 6.4% throughput loss for 72B models under parallel decoding. We test on models ranging from 3B to 72B, and verify that OverRIDE is effective on models of different sizes.
2
METHODOL.OGY
2.1 OVERVIEW
Let Mp be a pretrained LLM, and C C= [cib.N, be a set of contexts. For each context c; our goal generate T diverse responses. As illustrated in Figure 2 each round E 21,,,..,,, we: 1) Generate new response for each context; 2) Fine-tune a guide model Mg. on previously generated responses; 3) Use Mg. to reweight the next-token distribution of Mp. producing the reweighted distribution Pt; 4) Use Pt to generate new responses in the next round. The core idea is that the reweighted distribution should be close enough to the next-token distribution the original model Mp. while avoiding the distribution that would likely result in previously generated responses. In this way, we encourage the model to explore less-traveled paths at every decoding round.
In the following sections, explain each component of OverRIDE: Section 2.2 describes how to capture common patterns from history responses by fine-tuning the guide model Mg Section
2.3 describes the reweighting mechanism that adjusts token probabilities. Section 2.4 describes
a parallel implementation of OverRIDE, which allows for efficient parallel decoding and can be directly integrated with existing L.LM serving systems.
2

---

Published as a conference paper at ICLR 2026
Round t
$ 2.3 Reweight
Rick and Btave are sach dhinking of posliig factor of Siu8 and Sap respetivvly If and Shrve ame thinking they thinking
Mp
pla [
exme mumber, how many
Context
Prit
Wi Serve
Pa sx)
Rick thinidng postte thinking lid the protliou factons $u49:
PACK
FWST
Skoe
numbers
and Fins,
1,2, 14
Nest, lets list the positive factioe of Sea8:
thinking they thinking
Responiee
Fine-tune 2.2
Rick thinking of positive Tharefie,, the final snswer is
Response Set D
Figure 2: The overall framework of OverRIDE, implemented sequentially.
2.2 CAPTURE HISTORY PATTERNS
standard autoregressive language model Mp predicts the next token with probability p(a; xe(), where 2; is the i-th token in the sequence, and Tei is the sequence of tokens before z;. In the initial round, the model generates response r; for each context c; by sampling each token sequentially from the original distribution p(a; (=e)). The generated responses and corresponding contexts stored in response set D {(c,, M= Since no prior generations exist at the initial round, we decode responses with the original model explore its default behavior and high probabillbb paths.
Starting from the second round, we fine-tune guide model My. from the original model Mp. using the previously generated responses in D:
C..
Σ (C.r)ED
log c).
(1)
D
This guide model Mg. learns to predict the common semantic patterns that appeared in previous samples. The key insight here is that we cannot directly use previously generated samples to guide new generations, since the sampling process is non-deterministic, and responses vary each round. By fine-tuming guide model on previous responses, we enable it to learn and generalize the common patterns that have emerged across samples. This allows capture and suppress these patterns for more diverse responses in subsequent rounds.
2.3 REWEIGHTING THE NEXT-TOKEN DISTRIBUTION
After fine-tuning the guide model Mn., we use it to reweight the original model's next-token dis- tribution. We construct reweighted probability distribution pe(æ;; stes)) that leverages both the original model M, and the guide model Mq.:
p(1; x<s) (x x Z ge x<s))
p( see)),
where p(a; xre)) is the next-token distribution from the original model, qe (x( (22)) is the next- token distribution from the guide model, is hyperparameter that controls the strength of reweight- ing, and Z is the normalization term.
The intuition behind this reweighting mechanism that since the guide model Mg. predicts pat- terns that appear in previous samples, if it assigns higher probability to token, the weighting ratio p(s,(ee)) becomes smaller, reducing the probability of selecting this token in a new round. ((2]] Conversely, tokens that were less frequently chosen in previous rounds receive relatively higher probability, encouraging exploration of altermative responses. The hyperparameter À controls the diversity- suuarsity-quality trade-off. This reweighting process performed at each decoding round, allowing the model to adaptively adjust its generation strategy based the history of generated responses.

---

Published as a conference paper at ICLR 2026
2.4 PARALLEL IMPLEMENTATIO FOR LLM SERVING SYSTEMS
The described OverRIDE method operates sequentially. requiring all responses to be collected and the guide model to be fine-tuned before proceeding the next round. However, modern LLM serv- ing systems like vLLM (Kwon et al. 2023) and SGLang (Zheng et al. 2024) have implemented parallel decoding that significantly reduces sampling time and computational overhead. These effi- ciency gains primarily come from the reuse of the KV cache. For example, when generating multiple responses for the same context, the KV cache for the context only needs to be computed once, and subsequent sampled tokens or generation paths can also be reused if they are identical.
We now describe our architectural and procedural improvements for parallelizing OverRIDE. Our design is directly compatible with existing LLM serving systems, as demonstrated by our implementation, which maintains tigh-performance parallel de- coding without compromising the method's core objective of generating diverse responses.
p p(z; x m>s))
U [z;
Softmax
logits
Output head adapters To minimize the efficiency loss from fine-tuning, we restrict the trainable parameters to the output heads. As shown in Figure 3 we set low-rank adapter for each decoding round, with a structure and training approach similar LoRA (Hu et al. 2022). This allows us to compute both Pt and de simultaneously in a single forward pass given hidden states h:
Output Head
W C R
W E 12 hxv
W Rax
hidden states
logit, Wh, and logit,, Wh + W
(3)
Transformer Layers
Intuitively, the adapter captures the difference between the model's original distribution and the categorical distribution of the actual sampled token. In the next decoding round, this dif- ference is suppressed by a factor of to avoid previous patterns. Since the guide model M,, only drifts slightly from the original distribution, this tiny amount of trainable parameters is sufficient capture the difference.
Frozen during inference
Update during inference
Architecture of sin- gle output head. is the dimen- sion of the hidden states, is the rank of the adapter, and v is the dimension of the vocabulary.
Figure 3:
Additionally, the use of adapters comes with a desirable bene- fit: the model size grows larger, the efficiency loss gets even smaller, since the adapter takes up smaller proportion of the to- tal parameters. We conduct experiments on different model sizes and show that this method is still effective.
Pe(ut
| #
Sampler
Synchronized fine-tuning The original OverRIDE method requires collecting all the sampled responses before fine-tuning, which is not acceptable in the case of parallel decoding. To solve this problem, we propose to perform fine-tuning right after the sampling process. As shown Figure 4 at the (-th round, we head compute the reweighted distribution Pt and sam- ple the next token. Then, we update the adapter weights in head t ++ using cross entropy loss with respect to the sampled token, which is the same as Equation I but at the token level.
Rewnight
Sample
p(#: (xa)
(e a
Next
Token
Fine-tune
Head 1
Head
Head
Head t+1
Head
This design aligns with the implementation of LLM serving sys- tems. To perform parallel decoding, the same context is dupli- cated into identical requests, and sent to the model for exe- cution. These requests are consecutively processed in order to rouse the KV cache the memory. While this original imple- mentation doesn't require execution order, we sort these re- quests by the order of their round number. In this way, after the t-th request is processed by head head t + 1 is updated subse- quently, and is immediately used to perform the + 1-th round of decoding. The parallel version of OverRIDE is described in Algorithm
hidden states
Transformer Layers
Decoding process at round t. Head 1 is the original head. Head t consists of the orig- inal head and the adapter. Here, we simplify them as a whole.
Figure 4:
4

---

Published as a conference paper at ICLR 2026
Algorithm OverRIDE: Reweighting Iterative Decoding
Input: model Mp: context number of rounds T'; reweighting parameter X.
1: for f- to T do
2: if't 1 then
3: r1 ~ p(-
> Sample response with the original model
4: else
5:
Pr( xee)) normalize( p(mr) (x xec)
p(a; (re)))
Reweighting
6:
r 1e- cc
> Sample response with the reweighted distribution
7:
end if
8:
if' T then
9:
Ma + train(M fay {(c, re)f) & Fine-tune head +++ with the sampled response
10:
end if
11:
end for
12: Output: Diverse responses (rI, r2...... pT).
3 EXPERIMENTS AND ANALYSIS
3.1 DATASETS AND METRICS
Datasets We evaluate OverRIDE's effectiveness across different domains, including code genera- tion (HumanEval (Chen et al. 2021)), mathematical reasoning (MATH500 (Hendrycks et al. 2021),
GSM8K (Cobbe al. 2021) and story generation (CCNews (Common Crawl 2007)).
Evaluation metries To evaluate the quality and diversity of multiple generated responses, employ the following metrics: (1) PASS@k: the pass rate of problem if allowed to sample times; (2) Cosine similarity: the average pairwise cosine similarity between the embeddings of generated responses; (3) CodeBLEU (Ren et al., 2020): the average pairwise CodeBLEU score between all generated code snippets, which indicates the degree of matching between different code; (4) MAUVE (Pillutla et al. [021]: metric to determine the similarity between model generations
and human answers. Details of the datasets and metrics can be found in Appendix A
3.2 MAIN RESULTS
We evaluate OverRIDE using Qween2255-7BIInstruce Yang et al. 2024) and Mistral-7B-Instruct v0.3 (Jiang al. 2023) our base models. For each model, we implement OverRIDE with the
following sampling methods: (1) Greedy decoding; (2) Top-p sampling (Holtzman et al. 2020)
with different temperature settings (r 0.6, 1.0); (3) Top-k sampling (Fan al. 2018): (4) Min-p
sampling (Minh et al. 2025). The implementation details can be found in Appendix
OverRIDE balances quality and diversity. Table T shows the results on (hhen et al., 2021] and MATH500 (Hendrycks
2021). Results on GSM8K (Cobbe al.) (2021) are pre- sented in Appendix B We report PASS @k accuracy, pairwise CodeBLEU score, and pairwise cosine similarity. While PASS @k accuracy mainly indicates generation quality, it also reflects generation diversity. Lower CodeBLEU score and cosine similarity indicate higher diversity between generated responses. For PASS@5 accuracy, PASS @ 10 accuracy, and the similarity score, we present paired results where the left value represents the baseline sampling method, and the right value represents the same method integrated with OverRIDE.
OverRIDE increases PASS@5 and PASS @ 10 accuracy while reducing response similarity across sampling methods with both models. Additionally, OverRIDE's effectiveness persists across differ- ent temperature settings. This derives from OverRIDE's ability to dynamically explore both high and low probability responses, avoiding the typical trade-off between diversity and quality. This in- sight suggests that when implementing OverRIDE, it is advantageous to base it on high-probability decoding methods. This allows OverRIDE to explore high- hiobbbrobybility paths in the initial rounds, and lower-proyability alternatives in later rounds, thus maximizing OverRIDE's effectiveness.
5

---

Published as a conference paper at ICLR 2026
Table I: Results of different sampling methods on HumanEval and MATH500.
HumanEival
MATHI
Model
Method
PASS@1
P
PASS@ 10
CodeBLEU
P
PASS@5
PASS 10
Similarity
Greedy
64.9
64.//11.2
64.9/86.0
1.000/ 0.622
72.5
72.5/ 84.3
7 87.0
1.000/
0.938
Owen 2,5-7B
Top-p. 0.6
63.7
78.4 81,6
81.4/86,4
0.754 / 00610
72.1
84.1 84,8
86.8 88.3
0.950/ 0,936
Top-i. Top-k Min-p
1.0
60.3
82.1 83.6
85.5/88.5
0.666 0.578
71.6
84.8
87.2/ 87.6
0.940 0.929
62.5
80.0 83.2
84.4/87.9
0.715 0.603
72.2
84.2/ /8.77
86.7 /8..5
0.948/
0,937
63,2
79.7 83.7
83.2/ 88.1
0.736 0.610
71.8
84.3/
8 87.6
0.946 0.932
Greedy
31.3
31.3 45,9
52.4
1.000 / 0,537
13.1
13.1
13.1 36.6
1.000/
0.874
Mistral 7B
Top-p, 7 0.6
29.9
47.41 49.0
54.9/ 56.9
0.563 0.474
11.3
27.7
28.4
36.47 37.4
0.871
0.862
Top-p.
1.0
29.6
/9.3/51.5
58.1 / 58.5
0.448 0,400
10.9
27.1 28.0
36.2/377.2
0.854 0,850
Top-k
29.7
48.9 49.8
56.0/ 57.6
0.521 0.450
11.8
27.1 28.5
36.4/37.22
0.870/ 0.863
Min-p
30,0
/48.6
57.1 / 57,6
0.547 / 0.469
11.5
27.0
28,5
36.4/ 37,3
0,867 0.860
OverRIDE generates human-like responses. Table shows the results on CCNews (Common Crawl 2007). Models are provided with the first 32 tokens of each news text sample, and required to generate the subsequent 256 tokens. We use MAUVE (Pillutla al. 202I) to evaluate how all 10 rounds model generations match the original news. Cosine similarity is still measured pairwise between generations. We present paired results where the left value represents the baseline sampling method, and the right value represents the same method integrated with OverRIDE.
The results demonstrate that OverRIDE can improve human-like generation quality while enhancing generation diversity. OverRIDE achieves consistent improvements in MAUVE scores and reduction in cosine similarity scores across all settings, indicating better alignment with human writing patterns while increasing di- versity between generated responses. OverRIDE alleviates the typical I.IM generation issues of repetition (Holtzman et al. 2020). For repetitive patterns generated by the original model in the first round, OverRIDE avoids generating similar patterns in subsequent rounds through reweight- ing, thereby improving generation quality.
Table 2: Results of methods on CCNews.
CCNews
Model
Method
MAUVE
Similarity J
Greedy
0.803 / 0. 0.753
1.000/ 0.606
Top-p.
0.6
0.901 / 0.938
0,726 0.678
Qwen 2.5-7B
Top-p, 7 = 1.0
0.974 / 0.977
0.651 / 00.641
Top-k
0,953 / 0.960
0.715 0.679
Min-p
0.946 0.946
0.714 0.668
Greedy
0.883 / 0.891
1.000 0.675
Top-p. 7 0.6
0.937 0,952
0.75% 0.703
Mistral 7B
Top-p.
1.0
0.942 / 0.947
0.697 0,685
Top-k
0,904/ 0.937
0.744 / 0,713
Min-p
0.903 / 0.934
0.759 0,707
OverRIDE can scale up. To validate whether OverRIDE remains effective across different model sizes, we test models of varying scales. Table shows the results on HumanEval and MATH500 across the Qwen-2.5 model series, ranging from 3B to 72B parameters. OverRIDE demonstrates consistent improvements across all model sizes, enhancing both quality and diversity.
Table 3: Results of different sized models on HumanEval and MATH500.
HumanEyal
MATH
Model
PASS@1
PASS@5
PASS@ 10
CodeBLEU
PASS@)
PASSes
PASS@ 10
Similarity J
Qwen-2.5-33
58.2
78.7 79.0
82.9/83,2
0.629/ 0.547
63.3
78.3/79.2
83,5
0.944/ 0.937
Qwen-2.5-7B
63.7
78.4/81.6
81.4/ 86.4
0.754/ 0.610
72.1
84.1 84.1/84.8
86.8 87.3
0.950/ / 0.936
Qwen-2,5.148
40.0
63.4 70.5
77.1/7722
0.698 / 0.588
75.8
85.1 / /85.7
87.7 / 88.0
0.941 / 0.929
Qwen-2.5-322
67.8
78.8 80.5
81.2/84.2
0.695 0.579
78.5
/6.4/87.2
88.6 89.3
0.945 0.933
Qwen-2.5-72B
73.5
82.1 /3.88
83.4/87.0
0.803 0.674
78.9
87.0 88.0
88.8 90.5
0.955 0.941
Furthermore, compared to smaller models (3B), larger models (72B) achieve greater performance gains through OverRIDE, despite the trainable adapter parameters constituting a smaller proportion of the total parameters. We attribute this the enhanced capacity of larger models, which provides more expressive hidden states that enable adapters to easily capture the differences between the original next-token distribution and the distribution of the sampled tokens.

---

Published as a conference paper at ICLR 2026
3.3 DECODING DYNAMICS
To better understand OverRIDE's behavior and its impact generation diversity, analyze the entropy of token distributions and the log-likelihood of generated sequences across decoding rounds. For the following experiments, implement OverR IDE with greedy decoding using the Qwen-2.5- 7B model, and evaluate on the MATH500 dataset.
Entropy over Rounds
Likei load Rounds
1.4
A-0.2
-20
A-0.4
122
-30
A-0.8
Aw100
Log Litelhoood
-40
Enioopy
0.6
- OveRIIDE
Top.p, r-0.3
0.4
-70-
Top-p.
topp. t=0.9
0.2
-80
Top t-1.2
Top-p.
10
Round
Figure 5: Next-token distribution entropy across decoding rounds for different values. Higher entropy indicates greater diversity.
Figure 6: Log-likelihood of responses across decoding rounds. under OverRIDE and top-p sampling with different temperatures.
Figure [ shows how the reweighting parameter À controls the diversity of token selection during the decoding process. We observe a similar pattern across all values: In round 1, the average token entropy starts at low level because of greedy decoding. In round 2, a significant spike in entropy occurs as OverRIDE begins actively suppressing high high-probability patterns. In later rounds, the entropy gradually decreases and eventually stabilizes. Higher À values lead to higher entropies, with 1.0 reaching a peak of nearly 10 times its initial value. This demonstrates that stronger reweighting leads to more aggressive exploration and higher diversity in the generated responses, particularly in early rounds where the model actively diverges from previous generation patterns.
Figure 6 shows the average log-likelihood of generated responses across decoding rounds. Over- RIDE initially experiences a sharp decline log-likelihood in early rounds as it actively suppresses previously seen patterns, forcing exploration into low-prbbability regions of the original distribu- tion. The likeliood partially recovers in later rounds and eventually stabilizes at an intermediate level. This reveals OverRIDE's decoding dynamics the interplay between OverRIDE's 's suppres- sion mechanism and the model's original distribution. As the response set grows, it contains both high and low likelihood generations. When suppressing patterns common all previous responses, the model discovers alternative paths that, while distinct from the initially preferred responses, still represent valid solutions with higher likelihood.
The comparison with top-p sampling at various temperatures reveals the differences in how these methods approach diversity. While sampling with higher temperatures consistently produces low log- lieeeikeeiood across all rounds, OverRIDE's E's strategy is more dynamic: exploring between low- and progabiibbbbiitty responses. This indicates that OverRIDE's approach to increasing diversity fundamentally different from temperature scaling, which is only flattening the distribution.
3.4 EFFICIENCY ANALYSIS
To assess the efficiency impact of our parallel implementation, apply the parallel version of OverRIDE on vLLM (won et al. 2023). We conduct experiments using the Qwen-2.5 model series, with size ranging from 3B 72B. All models are deployed on single node, and distributed on single or multiple GPUs. Specifically, for 3B, 7B, 14B, 32B, 72B models, we use 1, 1, 2, 4, 8 GPUs, respectively. More details about the setup can be found in Appendix B Models are evaluated on the MATH500 dataset with top-p sampling (r = 0.6). For each context, we sample 10 rounds with parallel decoding. Throughput is measured as output tokens per second.

---

Published as a conference paper at ICLR 2026
Table shows the throughput comparison across different model sizes. As the model size increases from 3B to 72B parameters, the throughput loss gradually decreases from 8.2% 6.4%. This validates our hypoth- esis in Section [.4 that, since the adapter parameters constitute smaller proportion of total parameters as the model scales up- OverRIDE becomes more efficient for larger models. This provides assurance for deploy- ing OverRIDE with even larger models in production environments.
Table 4: Comparison of throughput across different
model sizes under parallel decoding.
Throughput (token/s)
Model
Drop
Baseline
OverRIDE
Qwen-2.5-3B
5439.9
4992.2
-8.2%
Qwcn-2.5-7B
3709.6
3435,8
-7.4%
Qwen-2.5-14B
2798.0
2600.3
-7.1%
Qwen-2.5-32B
2091.3
1947.4
-6.9%
Qwen-2.5-72B
1463.0
1369.9
-6.4%
3.5 OUTPUT HEAD CONFIGURATION
To determine the optimal rank for the output head adapters our parallel implementation, we con- duct experiments with different rank settings. Experiments conducted with Qwen-2.5-7B on the MATH500 dataset with top-p sampling (r = 0.6). The rank r refers to the rank of the adapter ma- trices WA and W 1 in the output heads. We compare against "Full" setting that uses independent output head parameters W for each decoding round to compute de without adapters.
Table 5 reveals trade-off between model ca- pacity and computational efficiency. When the rank too small (r 4), the adapter lacks sufficient parameters to accurately model the distribution q. leading to suboptimal perfor- mance. When the rank becomes too large (r = 256), we observe severe efficiency degradation r = 4
Table 5: Comparison of performance and effi- ciency across different rank settings.
Rank
PASS@ 10
Throughput
Drop
Baseline
86.8
3709.6
86.9
3453.1
-6.9%
with 30.6% throughput drop, while perfor- mance gains are minimal compared to smaller ranks. We eventually set = 16, which pro- vides sufficient modeling capacity while main- taining reasonable computational overhead.
r = 16
87.5
3435.8
-7.4%
r = 64
87.3
3301.0
-11.0%
256
87.1
2569.2
-30.7%
Full
87.3
1984.1
-46.5%
3.6 SENSITIVITY ANALYSIS
To understand how pattern suppression in OverRIDE affects model performance, we conduct a sensitivity analysis on the hyperparameter X, which controls the intensity of the reweighting ef- fect. Higher values of À lead to more aggressive suppression of patterns in previously generated responses. Figure 7 shows the performance of OverRIDE with varying values of X. Results evaluated on the MATH500 dataset with top-p sampling (T 0.6).
The results show that the reweighting parameter À is model-specific. For Qwen-2.5-7B, we observe PASS @5 and PASS @ 10 reaching their highest at À = 0.8. In contrast, Mistral-7B shows an optimal performance with lower As, with PASS@5 and PASS @ 10 reaching their highest 0.4. Beyond
Qwen-2.5.78
Mistral-7B
0.56
36
0.87
0.95
86.5
0.86
Accrrscy (50
Accurasy 150)
85.0
RASS-295
0.85
PASS@10
-22
PASS-910
85.5
Simisrit
Simlarity
0.84
30
0.92
28
84.0
0.5
1.0
0.2
0.4
0.6
0.8
5.0
Figure 7: Effect of reweighting parameter À on PASS@k performance (%) and similarity. The left figure shows results for Qwen-2.5-7B, while the right figure shows results for Mistral-7B.
8

---

Published as a conference paper at ICLR 2026
this range, performance gradually declines as X increases. This shows that Mistral-7B is more sensitive to the reweighting mechanism. For both models, similarity scores consistently decrease with higher À values, confirming that stronger reweighting leads to more diverse responses.
Such sensitivity to À can be attributed to the characteristics of output diversity for different models. From main results in Tables T we observe that Mistral-7B consistently produces more diverse outputs than Qwen-2.5-7B with different sampling methods, as evidenced by their lower Code and cosine similarity scores across all settings. This suggests that models with already sufficiently diverse responses benefit from lower À values, since excessive pattern suppression becomes coun- terproductive and may cause degradation in performance.
4 RELATED WORK
Stochastic Decoding Deterministic decoding approaches like greedy search and beam search (Freitag and Al-Onaizan 2017) tend to select tokens that maximize model confidence, resulting in outputs that lack diversity, and often suffer from issues such as repetition or dull content. Stochastic decoding methods address these limitations by introducing randomness in the generation process. Temperature sampling (Ackley et al. (985) flattens or sharpens the probability distribution to control randomness. Top-p sampling (Holtzman et al. 2020) and top-k sampling (Fan et al. 2018) truncate the distribution tail to exclude low-confidence tokens. More adaptive approaches (Basu et al. [021] Hewitt et al., [2222] Minh et al. (2025) dynamically adjust sampling parameters based on distribution entropy model confidence generate creative and coherent responses. However, these methods operate independently at the token level without considering previous responses, which systemati- cally limits their ability produce diverse outputs across multiple generations.
Decoding with Guidance Recent works have explored guiding the decoding process with addi- tional models or in-context information. A series contrastive methods enhance generation faith- fulness and quality by contrasting between different models (Li et al. 2023b), different layers within the same model (Gera et al. 2023 CChang et al. 2024 Das et al.) 2025), or model outputs with dif- ferent contexts (Shi et al.) 2024). Speculative decoding approaches (Xia al. 2023 Leviathan al.] 2023 Chen et al. 2023) use lightweight draft models to predict multiple tokens in parallel, so as to improve decoding efficiency. Multi-token prediction methods (Fu et al. 2024 Gloeckle al. 2024 Guo
Cai et al. 2024| Li et al. 2024] Guo et al., 2025) introduce additional decoding heads or branches to accelerate inference while maintaining output quality. Guided search methods (Yao et al. 2023 Xic cet al. 2023 Zhu et al. 2024) employ verifiers evaluators to navigate complex reasoning. While unlikelihood training Welleekk et al. 2020) and SimCTG (Su et al. 2022) also attempt to encourage diversity by addressing degeneration, they require extensive training large corpora. In contrast, our method dynamically guides generation based on the specific patterns observed in previous re- sponses. This requires only lightweight fine-tuning during inference without depending on external knowledge or models. Additionally, our method enables context-specific diversity that adaptively evolves with each decoding round.
5 CONCLUSION
In this paper, we propose OverRIDE, an iterative decoding method that enhances the diversity of LLM outputs. OverRIDE works by reweighting the next-token distribution of autoregressive mod- els based on previously generated responses. By fine-tuning guide model on previously generated responses, OverRIDE effectively suppresses patterns in history responses and encourages explo- ration of diverse alternatives across decoding rounds. To extend OverRIDE to parallel decoding for higher efficiency, we propose improvements in architecture and procedural design, and further implement OverRIDE existing LLM serving systems.
Our experimental results demonstrate that OverRIDE improves generation diversity while maintain- ing or improving quality across various models and sampling methods. We also provide insights into OverRIDE's s decoding dynamics, revealing how balances exploration and exploitation during the generation process. Experiments with different-sized models show that OverRIDE can scale up to larger models, while maintaining performance and efficiency with parallel decoding. Our work suggests that OverRIDE is promising approach for enhancing the diversity of L.MM outputs.

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
OverRIDE dynamically modifies the next-token distribution LLMs. By suppressing previous pat- terns and encouraging exploration of low-probability regions, our method could potentially damage the safety alignment of the original model. This might lead to the generation of harmful or biased contents. The primary negative effect we observe is repetitive content generation at high reweighting parameter values A. However, there exist cases where OverRIDE can produce unexpected outputs, see Table 12 for an example. We recommend using moderate reweighting parameters and imple- menting additional safety filters when applying OverRIDE in production environments.
REPRODUCIBILITY STATEMENT
We provide the following materials for reproducing the experiments in paper: datasets and met- rics information in Appendix A implementation details Appendix B prompts used for generation in Appendix [ Our code is available at htttps://gittie
com shi - shii-r//vverDDD including imple- mentation for vLLM (Kwon et al. 2023) v0.11.1 and v0.8.5.post) We also provide the OverRIDE implementation for EvalPlus (Liu al. 2023) and Livecodebench (Jain al. 2025).
ACKNOWLEDGMENTS
The research work described in this paper was conducted the JC STEM Lab of Machine Learning and Symbolic Reasoning, funded by The Hong Kong Jockey Club Charities Trust.
REFERENCES
David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for Boltz- mann machines. Cognitive Science, 1985.
Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varsh- ney. Mirostat: neural decoding algorithm that directly controls perplexity. In International Conference on Learning Representations, 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning, 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating Large Language Model decoding with speculative sampling. arXiv preprint arXiv:2302.013//8 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating Large Language Models trained on code. arXiv preprint arXiv:2107.03337. 2021.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. DoLa: Decoding by contrasting layers improves factuality in large language models. In International Conference on Learning Representations, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano. Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1rXiv:2110.14166 2021.
Common Crawl. Common Crawl: Open repository of web crawl data. htttps:/ccomoneee org/ 2007.
10

---

Published as a conference paper at ICLR 2026
Souvik Das, Lifeng Jin, Linfeng Song, Haitao Mi, Baolin Peng, and Dong Yu. Entropy guided ex- trapolative decoding to improve factuality in large language models. In International Conference Computational Linguistics, 2025.
Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman. Ignasi Clavera Gilaberte, Jakub Pachocki, et al. Competitive programming with large reasoning models. arXiv preprint arXiv:2502.0680 068077 2025.
Angela Fan, Mike Lewis, and Yann N. Dauphin. Hierarchical neural story generation. In Annual Meeting of the Association for Computational Linguistics, 2018.
Markus Freitag and Yaser Al-Onaizan. Beam search strategies for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, 2017.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang- Break the sequential dependency of LLM inference using lookahead decoding. In International Conference on Machine Learning, 2024.
Ariel Gera, Roni Friedman, Ofir Arviv, R. Chulaka Gunasekara, Benjamin Sznajder, Noam Slonim, and Hyal Shnarch. The benefits of bad advice: Autocontrastive decoding across model layers. In Annual Meeting of the Association for Computational Linguistics, 2023.
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster Large Language Models via multi-token prediction. In International Conference on Machine Learning. 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.2178: 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, al. DeepSeek-RI: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems, 2021.
John Hewitt, Christopher D. Manning, and Percy Liang. Truncation sampling as language model desmoothing. Findings of the Association for Computational Linguistics: EMNLP 2022, 2022.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. International Conference on Learning Representations, 2020.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of Large Language Models. In International Conference on Learning Representations, 2022.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of Large Language Models for code. International Conference on Learning Repre- sentations, 2025.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for Large Language Model serving with PagedAttenttonn. In ACM Symposium Operating Systems Principles, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from Transformers via specula- tive decoding. In International Conference on Machine Learning, 2023.
11

---

Published as a conference paper at ICLR 2026
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. StarCoder: may the source be with you! Transactions on Machine Learning Research, 2023a.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Perey Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Annual Meeting of the Association for Computational Linguistics, 2023b.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires
rethinking feature uncertainty. In International Conference on Machine Learning, 2024.
Yujia Li, David H. Choi, Junyoung Chung. Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et. Competition-level code generation with AlphaCode. Science, 2022.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of Large Language Models for code generation. In Advances in Neural Information Processing Systems, 2023.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-Guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for Large Language Models via reinforced evol-instruct. In International Conference on Learning Representations, 2025.
Nguyen Nhat Minh, Andrew Baker, Clement Neo, Allen G. Roush, Andreas Kirsch, and Ravid Shwartz-Ziv. Turning up the heat: Min-p sampling for creative and coherent LLM outputs. In International Conference on Learning Representations, 2025.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaïd Harchaoui. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Advances Neural Information Processing Systems, 2023.
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. CodeBL.EU: a method for automatic evaluation of code synthe- sis. arXiv preprint arXiv:2009. 10297, 2020.
Bernardino Parede- Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with Large Language Models. Nature, 2024.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Llama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. Trusting your evidence: Hallucinate less with context-aware decoding. In North American Chap- of the Association for Computational Linguistics, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, 2020.
Yixuan Su. Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. In Advances in Neural Information Processing Systems, 2022.
Sean Welleck. Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In International Conference on Learning Rep- resentations, 2020.
12

---

Published as a conference paper at ICLR 2026
Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decod- ing: Exploiting speculative execution for accelerating Seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023.
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Zhao, Min- Yen Kan, Junxian He, and Michael Qizhe Xie. Self-evaluation guided beam search for reasoning. In Advances in Neural Information Processing Systems, 2023.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu. Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint 2IXi::4122.1115.. 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate problem solving with Large Language Models. In Advances in Neural Information Processing Systems, 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. MetaMath: Bootstrap your own mathematical questions for Large Language Models. In International Conference on Learning Representations, 2024.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems, 2024.
Tinghui Zhu, Kai Zhang, Jian Xie, and Yu Su. Deductive Beam Search: Decoding deducible ratio- nale for thain-of-thouuht reasoning. arXiv preprint arXiv:4401.16666 2024.
A DATASETS AND METRICS
Datasets consider four datasets to evaluate OverRIDE's effectiveness across different do- mains, including code generation, mathematical reasoning. and story generation:
HumanEval (Chen et al.
2021 code generation dataset consisting of 164 programming problems that require to generate Python functions based on function descriptions;
MATH500 (Hendrycks et al. 2021): y a standard 500-sample subset of the original MATH
dataset with 12,500 competition mathematics problems;
GSM8K (Cobbe et al. 2021): dataset of 8,500 grade school math word problems;
CCNews: subset of the Common Crawl (Common Crawl 2007) corpus that contains news articles from news sites all over the world.
Evaluation metries To evaluate the quality and diversity of multiple generated responses, we employ the following metrics:
PASS@k:
Following previous work (Chen et PASS@K accuracy to evaluate the correctness of multiple generated solutions. PASS @k estimates the pass rate of problem if allowed to sample times.
2021 Grattafiori al. 2024 we the
CodeBLEU (Ren et al., 2020 For code generation tasks, use CodeBLEU evaluate the similarity of generated code snippets. CodeBLEU is a weighted combination of BLEU, BLEU-weighted, AST match and data-flow match scores. We calculate the pairwise Code- BLEU scores between all generated code snippets for each context:
CodeBLEU; =
CodeBLEU(T.s Fi,)),
(4)
j=1 A=
where riy is the j-th code snippet for the i-th context, and is the number of code snippets.
13

---

Published as a conference paper at ICLR 2026
Cosine similarity: To measure output diversity, we compute pairwise cosine similarities between the embeddings of generated responses. For each context c, we obtain embed-
dings (fr,,) ) for each response F, using the OpenAl reb---mbedddngg--smmm embeddddnn model, and calculate the average pairwise similarity:
( j=lk=j+1 lo(rr,))lllll(r...
Similarity,
Σ
(5)
where Fi,j is the j-th response for the i-th context, and n is the number of responses.
MAUVE (Pillutla et al., 2021]: use MAUVE to measure the similarity between model generations and human answers. MAUVE is obtained by computing the KL divergence
between the two distributions in a quantized embedding space of a foundation model.
B IMPLEMENTATION DETAILS
OverRIDE Configuration We conduct all experiments on 8 NVIDIA L40S GPUs. For the reweighting parameter X, use X = 0,8 for Qwen-2.5-7B (Yang et al. 2024 and X 0.4 for Mistral-7B (Jiang et al., 2023), based on our sensitivity analysis in Section 3.6 When fine-tuning the output head adapters, N (0,, 0.02), and Wn a is initialized with zeros.
learning rate of 1e-3. WA is initialized with normal distribution
vLLM Configuration We use the default settings of vLLM (wwon et al. 2023 except for the following: gpu memory utilization is set to 0.8 for all the experiments; we use tensor parallel size of for 3B and 7B models, 2 for 14B models, for 32B models, and for 72B models.
Sampling Methods We implement sampling methods with the following parameters all the datasets: (1) Top-p sampling (Holtzman et al. (2020) with top-p = 0.9; (2) Top-k sampling (Fan al. 2018 with k 20 and temperature 0.6; (3) Min-p sampling (Minh et al., 2025) with
min-p 0.05 and temperature 0.6.
C PROMPTS
Table/6]shove the prompts used in our experiments. The prompt for HumanEval (Chen et al.) 2021 adjusted from EvalPlus (Liu al. 2023).
Table 6: Prompts used in our experiments HumanEval, MATH, GSM8K, and CCNews.
Dataset
Prompt
HumanEval
Please provide self-contained Python script that solves
the following problem in a markdown code block: python
Problem
MATH
( Problem
Let's think step by step and output the final answer within {boxed{].
GSM8K
Problem
Let's think step by step and output the final answer within \boxed(].
CCNews
First 32 tokens of the news)
14

---

Published as a conference paper at ICLR 2026
D THEORETICAL DERIVATION OF THE REWEIGHTING FORMULA
In this section, we provide a theoretical derivation of our reweighting formula in Equation 2 We formulate the problem of determining the optimal next-token distribution r(x) as constrained optimization problem that balances exploration and exploitation. Let p(z) denote the next-token distribution from the current model and ((x) denote the distribution of previous patterns. We seek target distribution r(z) that satisfies the following properties:
Exploration: The distribution r(z) should assign lower probability to tokens favored by
the previous pattern distribution ((x) compared the current model p(x).
Exploitation:
The distribution r(x) should stay close to the current model's prediction p(z). This is achieved by minimizing the KL divergence DKL p).
Formally, we define the objective as minimizing the divergence from p(z) while maximizing the expected log-likelihood ratio between p(r) and g(z). The optimization problem is formulated as:
p(x) log
min rEA
= p) - AB
g(z))
s.t. Er(z==1, =1 r(x) 20
where À > is hyperparameter controlling the strength the exploration term, and A represents the probability simplex.
To solve this constrained optimization problem, we introduce a Lagrange multiplier for the nor- malization constraint. The Lagrangian function L(r, a) is given by:
r(x) p(x)
p(x) (r()))-)) ((x)
C(r,a) = Σ r(x) ((x) log
Σ r(x) log
Taking the derivative of L with respect to r(z) and setting it to zero yields the optimality condition:
OC
log r(x)
+ - Alog p(x)
+a = 0.
0r(x))
p(x)
(xx)
Rearranging the terms to solve for log r(x): logr(x) = (1 + >) logp(z) Alog (()) (1 +
(9)
Exponentiating both sides, obtain the unnormalized form of the optimal distribution:
p(x)
r(x) p(x))+)g(x) = p(x)
(10)
((x))
The non-negativity constraint r(x) 2 0 is automatically satisfied since the exponential function is strictly positive.
Considering the normalization constraint Σx r(x) 1, get the closed-form solution:
1
p(x)
r" (x)
Z p(x))
(11)
g(x)
where 3= ((=)))))))))tt))( the partition function. This derivation demonstrates that our reweighting formula is the theoretically optimal solution for balancing fidelity to the current model (cxploitation) with the suppression previous patterns (cxploration).
15

---

Published as a conference paper at ICLR 2026
E RESULTS ON GSM8K
Table shows the results on GSM8K (Cobbe et al. 2021 We report PASS @k accuracy and pair- wise cosine similarity. For PASS@5 accuracy, PASS@ @ 10 accuracy, and the similarity score, present paired results where the left value represents the baseline sampling method, and the right value represents the same method integrated with OverRIDE.
Table 7: Results of different sampling methods GSM8K.
GSM8K
Model
Method
PASS@1
PASS@5
PASS@ @ 10
Similarity 4
Greedy
89.8
89.8 94.7
89.8 95.6
1.000 0.951
Top-p, 0.6
89.6
94.8/ 95.2
95.8 96.3
0.960 / 0.947
Qwen 2.5-7B
Top-p. 1.0
88.8
95.1/ /5.1
96.0 / 96.5
0.947 0.940
Top-k
89.6
94.9 /
96.0 / 96.3
0.955 0.945
Min-p
89.7
95.0 95.0
95.8 96.3
0.957 0.947
Greedy
39.5
39.5 / 65.6
39.5 / 74,2
1,000
0.938
Top-p. 0.6
39.2
68.5 / 69.4
77.4 78.2
0.927
0.920
Mistral 7B
Top-p, 1.0
36,9
69.6 / 69.9
79.6 / 80.2
0,910
0.907
Top-k
37.9
69.1 69.6
78.0 80.0
0.921 0.915
Min-p
38.5
68.9 / 69.6
77.8 / 79.1
0.924 0.918
The results GSMSK further validate our main findings. OverRIDE consistently improves both quality (PASS@5 and PASS@ 10) and diversity (lower similarity scores) across all sampling meth- ods and temperature settings for both models. These results confirm OverRIDE's effectiveness in mathematical reasoning tasks, suggesting that the dynamic reweighting mechanism prevents over- confidence and encourages exploration of diverse yet accurate responses.
COMPARISON WITH OTHER DIVERSE DECODING METHODS
We compare OverRIDE with additional diversity-enhancemmen decoding baselines: Contrastive
Decoding (CD) Li et al. 2023b) and SimCTG (Su et al. 2022). For Contrastive Decoding, follow-
ing the original paper, we conduct experiments on XL. We set the amateur model GPT2, amateur model temperature Tamateur 0.5, and plausibility threshold a 0.1. We set À = 0.4
OverRIDE. For SimCTG, we set degeneration penalty a 0.6, and top-k prediction A: =
Table 8 shows the comparison results with Contrastive Decoding GPT2-XL on CCNews. Table 9 shows the comparison results with SimCTG on 2.5en-2.5-B-Innsteee and OverRIDE achieves better diversity with better quality compared to the basclines.
Table 8: Comparison with Contrastive Decoding on GPT2-XL.
Model
Method
MAUVE f
Similarity 1
0.752
0.676
GPT2-XL
OverRIDE
0.846
0.664
Table 9: Comparison with SimCTG on Qwen-2.5-7B and Mistral-7B
HumanEval
MATH500
CCNews
Model
Method
PASS@10 10
CodeBLEU
PASS@ 10
Similarity
MAUVE
Similarity
Qwen 2.
SimCTG OverRIDE
77.2%
0.747
86.8%
0.954
0.976
0.756
86.4%
0.610
87.3%
0.936
0.938
0.678
Mistral 7B
SimCTG
56.1%
0,528
36.8%
0.868
0.915
0.758
OverRIDE
56.9%
0.474
37.4%
0.862
0.952
0.703
16

---

Published as a conference paper at ICLR 2026
G TUNING AND SAFEGUARD IMPLEMENTATION FOR À
We recommend to tune on a small validation set, starting from and gradually increasing it until the desired diversity is achieved. Although not presented in our experiments, in extreme cases when the optimal À is too high, the reweighting ratio (p(a; x<zi//a (x; x<x))² may explode. Thus, we additionally provide safeguard prevent the reweighting ratio from exploding. Specifically, we can clip the reweighting ratio with a clipping threshold 3:
(1tttt/ I (((( xees) '3 p(a; x<i),
1 clip
1
Pe(x; xx<))
(12)
where p(a; xice) is the distribution from the original model, (a (<e)) the distribution from the guide model at round t, À is the reweighting factor, and Z is the normalization term.
H CASE STUDY
To intuitively illustrate how OverRIDE works, we provide qualitative analysis on the token prob- ability changes across decoding rounds in distinct scenarios in Table [O] These examples high- light how our method balances between improving diversity and preserving quality.
Table 10: Next-token probability (%) comparison across decoding rounds for OverRIDE.
(a) Case 1: Diversity
(b) Case Quality
Prompt: To calculate the area of triangle,
Prompt:
The Fibonacci sequence can be de-
the first step is to
fined recursively: SF(n) F(n-2) F(n- J$
Token
Round
Round 2
Round 10
Token
Round
Round
Round 10
measure
21.3
17.2
18.7
100.0
100.0
100.0
determine
18.0
19.2
19.4
0.0
0.0
0.0
calculate
17.0
11.8
15.4
(space)
0.0
0.0
0.0
find
15.8
19.7
14.6
0.0
0.0
0.0
identify
14.7
11.7
15.2
)
0.0
0.0
0.0
know
1.6
2.6
1.5
0.0
0.0
0.0
Case illustrates OverRIDE's effectiveness in encouraging diversity when multiple valid continu- ations exist. For the same mathematical problem with multiple valid choices as the first step. the model's token preferences evolve the rounds. This dynamic reweighting illustrates how Over- RIDE continuously adapts based on previously generated responses, encouraging exploration of equally valid but less-traveled paths the generation space.
Case 2 illustrates OverRIDE's ability to preserve quality when there is only a single correct answer. For the definition the Fibonacci sequence, F(n) Fnn-2)+ (nn-1) is the only mathematically correct completion in this case. OverRIDE maintains 100% probability for the token "1" across all rounds, demonstrating that our method does not force diversity at the expense correctness.
These cases highlight OverkIDE's core strength: the ability to distinguish between scenarios where diversity should be encouraged and where precision should be maintained. This ability derives from the reweighting mechanism. When a token originally assigned with high probability, reweighting does not dramatically increase the probability of other tokens. When multiple tokens are assigned with similar probabilities, reweighting encourages the model to explore less likely options.
17

---

Published as a conference paper at ICLR 2026
I EXAMPLES OF GENERATED OUTPUTS
Table 11: Examples of Qwen-2.5-7B generated responses on MATH500.
Method
Top-2, 0.3
Top-p. 0.3 + OverRIDE
Context
Determine the graph of the equation below parabola, circle, ellipse, hyperbola, point, line, lines, or empty.
( --33 3)
x²
Response
The given equation is ( + = 10. To determine the type contc section
The given equation is ( - 3) + = 10. This equation is in the standard form of circle, which is (at-A) A)2 (y k)2 center of circle and r is the radios. To this more clearly. we can rewrite the given equation ( 33 + (v u)² 10, Here, we can identify h 6 (since &
represents, we can compare to the standard forms of conic sections. The general form for circle is (ar 1)) + [yy k)) = r², where (h, k) is the center and is the radius. We can rewrite the given equation match this form: (# +-² 10. Notice that ($ 3) can be rewritten
y where (h, k)
= k and y2 10. Therefore, the center circle is (6, 0) equation matches the standard form of cirele, the graph the given equation is circle. Thus, the answer circle
2 So the equation becomes: { + 10. This is in the form (2 - 11)2 [y k)) r where h = 6, = O, and ,² = 10. Therefore. r = VIO. Since the equation matches the standard
radius VIO. Since
circle, the graph
equation
is circle. Thus, the answer is circle x
Response
The given equattion is (5 3) + y²² 10. To determine the type conic section this represents, standard forms of conic sections. First, let's rewrite the equation
To determine the type of conic section represented by the equa- tion ( - 3) + "²" = 10, we can compare it to the stan- dard forms of conic sections. First, let's rewrite the equation in more recognizable form by expunding and simplifying it:
compare to the
in a more recognizable form: ( - 3) + y² = 10. This equation is in the form (X_ h) + (Y kj² r, which the standard form of a circle. Here, X = & and Y = a. and the center of the circle is (h, k) (6. 0) because
1. The given equation is ( - -3) + v² = 10. 2. Ex-
pand (3 3) (3 3) (5) - 2 3 3
3 3 and thus 2 2(X + 3), so the center in terms of x and y is (6, (). The radius r vIU. Therefore. the given equation represents circle. The answer circle
z² 3x + 9. 3. Subatitute this back into the original equa-
X implies X
4
tion: 4 32 + + V² = 10. Simplity by moving
0. 5. To bet- ter undersiund the form, we can complete the square for the i
10 to the left side: =² 32 + v2
terms: a 32 # 12x). Completing the square inside the parentheses: x² 12x
(2 0)2 36. Thus,
1/ 2= = 1 (xx 6)2 2---------30 -36)
2= 9. Substitute
this back into the equation: (a 2)
9 + y²
0. Sim-
plity the constants: (a-0)²
+ 0²
10. Divide by 10 to get
equation in standard form: m stundard form an ellipse,
to 1. This the
+ <v ky-
1. where
(ft, k) is the center of the ellipse, a²
= 10, and 62 10.
Therefore, the graph of the equation 5 a)2 y² 10 is an ellipse. The final answer is: ellipse
Response
The given equation is (5 3) +y² 10. To determine the type of conic section this ropresents. we can compare it to the standard forms of conic sections. First, let's rewrile the equation
The given equation is ( - = 10. determine the type of conic section this equation represents, we can sewrite it recognizablle form. Let's staatt by simplifying and com- paring it to the standard forms of conic sections. 1, "Rewiitee the equation:** ** (# 3)²+,y = 10. 2. **Isalate the squared form that resembles the terms:* ** This equation is already
in
recognizable form: (5 3) + y² 10. This equation is in the form (X- h)3 the standard form of circle. Here, X = 1. h = 3, Y
39 + (Y x>2 2 which
V10 So, the equation (5 - 3) 2 = 10 represents a circle with center (6. 0)
= 0, and 22 10. Therefore,
(a-h)² + (y=a)² 1. but standard form of an ellipse. _2
need to manipulate to match this Form. 3. *DDivide both sides
and radius <TO. TO. Thus, the answer is circle
by 10:0* (5 3)
Yo
4. **Simplity the denomina-
tors: (5-3)
(x-B)² So the equation
becomne:: ox
S. *ooomgrree standard
of ellipse: The equation (x-6)²
10 matches
standard form an ellipse [=- (w-k)² where
2 = 40 and 6, 10. Since the equation mutches the stan- dard form of an ellipse, the graph of the given equation is ellipse. ellipso /
Similarity 0.983
0.795
18

---

Published as a conference paper at ICLR 2026
Table 12: Examples of Qwen-2.5-7B generated responses on GSM8K.
Method
Top-k
Top-k OverRIDE
Context
How much does cost you lunch today Subway you pay for foot-long fish sub cold cut combo sub?
much for six-inch
Response
To determine the total cost of lunch loday Subway, we need add the cost of the foot long fish suh to the cost of the six inch cold-cut combo sub. First, we know that the cost the foot-long fish sub is $40. Next, we need to calculate the cost of the six- inch cold-cut combo three times the cost of the foot-long fish sub. So, we multiply the cost of the Toot-long fish sub by 3: 40 x the cost of the foot long fish sub the cost of the six inch cold- cut combo suh to find the total cost of lunch: 40 + 120 160 Therefore, the total cost for lunch today at Subway is 160
To determine the total cost for lunch today Subway, need to calculate the cost of the items mentioned and then sum them up. The cost of the six inch cold cut cambo sub is three times the cost calculate
the foot-long fish sub given $40. The
According to the problem,
the foot-long six-inch cold-cut combo sub as follows: 3 x 10 120 Now. we add the cost of the foot-long fish sub and the six-inch cold- combo sub to find the lotal cost: 40 120 160 Thus,
suh. Therefore,
cost
120 Now, we add
cost lunch today
160
Response
To determine total cost lunch for the day. we need culate the cost of hoth the foot long fish suh and the six ixch cold-cut combo sub.
your question. As 1 don't It consume food or have a cost for meals. The information you' 've provided about the prices of the foot-long fish sub and the six inch cold cut combo suh Subway is unrelated functioning or costs. These prices are for illustrative purposes and not related any actual expenses for me.
appeeurs there's 's misundrrsaanding
First, know the cost of the foot long fish suh is $40.
Next, we are told that the six-inch cold-cut combo sub costs three
foot-long fish sub, Therefore, we calculate the cost of the six-inch cold-cut cut combo sub as follows:
much
If you're 're asking about the cost of meal for an Al. it would be zero hecause entitiss da not consume for functionality. If you have any other questions or need further assistance with calculations or information related to Subway's ss menu prices, feel free to ask? x
or require payment
Now; we add the cost of the foot-long fish suh and the six-inch cold-cut combo sub to find the total cost lunch: + 120 160
Thus, the tocal cost for lunch today is 160
Response
To determine the total crsst of the lunch, need calculate the cost of euch item and then sum them up. First, we know the cost $40. Next, we
To calculate the total cost of the lunch, üilying the cost of each item and then summing them up. given as
by iden-
the cost of the six-inch cold-cur combo suh. The problem states that the six-inch cold-cut combo sub costs three times as much as the foot-tong fish sub. Therefore, we calculate: 3 x 40 120. So, the six inch cold-eut combo sub costs $120. Now, we add the cost of the foot long fish suh and the six-inch cold-cut combo sub to get the total cost of the lunch: 40 + 120 = 160. Thus, the total cost of the lunch is 160
long
foot, long The cost of the six-inch cold-cur combo suh is three times cost of the foot-long fish sub. Therefore, we calculate: Cost of six-inch cold-cut combo sub = 3 x $40
The cast
$120 3. Adding the two costs logether gives the total cost of the lunchr Total cost $40 $120 8160 Sn, the total cost for the
lunch today at Subway is 160
Response
To determine the total cost of lunch for Subway, we need to cal.- culate the cost of both the foot-long fish sub and the six-inch cold-cut combo sub. 1. The cost of the foot-long fish sub is given 540.
It seems there might he bit misunderstanding As an arti- ficial intelligence, I don't consume food require meals, the concept of paying for lunch docsn't apply to me. The scenario you' 've presented involves oosts for food. but since physical entity. don't 't have personal expenses the noed to pur- chase lunch. Therefore, the question about how much it costs me for lunch today at Subway is not applicable. Not Applicable
The cost of the six-inch cold-cut combo suh three times the cost of the foot-long fish sub. Therefore, we calculate: 3 x 40 120 So, the six-inch cold-cut combo sub costs $120. 3. To find the lotal cost of lunch, add the of the foot-long fish sub and the six- cold-aut combo sub: 40 + 120
Thus, the total cost lunch today Sub-
way is
160
Similarity
0.927
0.626
19


Reviews:
[{"review_id": "m2FFqRS6LJ", "reviewer": "Reviewer_om5c", "summary": "This paper proposes a novel decoding method called OverRIDE, increasing semantic diversity in the generation of text by LLMs. It dynamically adjusts the probability distribution of the next word in the inference stage, encouraging the model to explore new and low-probability generation paths. The experiment proved that this method can enhance semantic diversity and achieve better results in multiple sampling processes.", "questions": "1. Can you add evaluations on the time to first token metric and the overall memory cost compared with normal sampling?\n2. Does OverRIDE also have considerable superiority on more challenging benchmarks? Can robustness research be conducted on more difficult data such as AIME and LiveCodeBench?\n3. The recent contrastive decoding method mentioned by the author should be compared with OverRIDE [1]. \n4. Can you provide fine-grained sensitivity analysis on the lambda performance of more model families on more challenging benchmarks?\n\n[1] Su, Yixuan, et al. \"A contrastive framework for neural text generation.\" Advances in Neural Information Processing Systems 35 (2022): 21548-21561.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The homogenization of the content generated by LLMs is a widespread problem that needs to be addressed.\n2. The authors designed and implemented a parallel algorithm based on high-performance inference frameworks, which is beneficial for the use of the open-source community.", "weaknesses": "My primary concerns relate to the comprehensiveness of the experimental evaluation. To be confident in the paper's claims, the following points need to be addressed:\n1.  Incomplete Efficiency Evaluation. The paper claims \"minimal loss of efficiency\" based solely on throughput. For a method that introduces training at inference time, this is insufficient. A more thorough analysis of computational overhead is critical. Please report Time To First Token metric for user-perceived latency, and the overall memory cost compared with normal sampling.\n2.  Limited and Potentially Dated Datasets. The experiments are conducted on relatively simple tasks. Several studies have demonstrated the crucial role of the \"critical token\" in ensuring the correctness of the model's responses. Unselectively suppressing the tokens generated in the first round may affect the model's accuracy. The benchmarks evaluated by the authors, such as MATH-500 and GSM8k, are not challenging for Qwen2.5 and might overlook this issue. It is recommended to add more cutting-edge data, at least including AIME.\n3.  Missing Comparison with key baselines. The paper compares OverRIDE with popular sampling techniques but omits a comparison with other advanced decoding methods that also aim to improve generation quality and diversity. A direct comparison with Contrastive Search is essential. Both methods modify the decoding objective to balance model likelihood with a penalty term. A head-to-head comparison on both quality and diversity metrics is necessary to properly contextualize the contributions of OverRIDE.\n4.  Concern of Hyperparameter Sensitivity Analysis. The proposed algorithm likely depends on a key hyperparameter $\\lambda$. Although the author conducted the sensitivity experiment, \\lambda may still be a challenging parameter. Users will not have fine-grained prior knowledge during the usage process. Robustness and usability of the method are supposed to be verified through a broader model family and more difficult data (as mentioned in the second concern)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zm3EcFWoaF", "reviewer": "Reviewer_wTcW", "summary": "The paper introduces OverRIDE, an iterative reweighting decoding method designed to increase output diversity while preserving task performance. OverRIDE introduces test-time trainable output-head adapters that iteratively reweight next-token logits to suppress tokens leading to previously seen patterns, steering decoding away from already-explored distributions. The adapters are confined to the output head, updated after each round, and synchronized with sampling, which makes the method compatible with parallel inference in serving systems. Combined with standard sampling, this history-aware reweighting increases diversity while preserving quality across code, math, and open-ended generation tasks.", "questions": "1. There is a discrepancy between your Table 1 and the Qwen-2.5 technical report: on HumanEval, the official PASS@1 is 52.9 for the 7B base model (Table 4 [1]) and 84.8 for the 7B Instruct model (Table 8 [1]), whereas your Table 1 lists 64.9 under “greedy.” After reviewing your released code, I see you evaluate Qwen-2.5-X-B-Instruct rather than the base Qwen-2.5-X-B. Please verify why your evaluation diverges from the official numbers, and standardize the model naming in the paper (or re-run the baselines with the base model) to avoid confusion. \n2. Additionally, I would like to know the exact evaluation setup for CCNews reported in Table 2. The relevant code does not appear to be included in your supplementary materials.\n3. In Table 3, row 315 shows Qwen-2.5-14B with a notably lower HumanEval PASS@1  (greedy)  than other sizes of the same family. Is this a typographical error?\n\n\n[1] Qwen Team. (2025). *Qwen2.5 Technical Report*. arXiv. arXiv:2412.15115.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The experimental coverage is broad, including two LLM families (Qwen2.5 and Mistral), five parameter scales from 3B to 72B, and four standard benchmarks.\n2. OverRIDE shows consistent diversity improvements over common decoding baselines in nearly all settings, with pass@k accuracy gains, while incurring only a modest slowdown.", "weaknesses": "1. For the CCNews open-ended long-form generation in Table 2, Distinct-n and Self-BLEU may be more appropriate diversity or similarity metrics than paragraph-level embedding cosine similarity.\n2. Comparisons focus on basic sampling schemes and min-p. How does your method perform compared with other classic baselines such as beam search and group beam search?\n3. Training and implementation details are under-specified. The core Output head adapters lack information on parameter counts, learning rates, and other settings. Section 3.4 reports only throughput in tokens per second without the hardware configuration.\n4. When restricted to a single iteration, OverRIDE has no effect and collapses to the baseline, which limits scenarios that demand single-pass decoding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5nSPfkyGW5", "reviewer": "Reviewer_Drqn", "summary": "The paper proposes OverRIDE, a history-aware decoding scheme to improve sample diversity without retraining the base LLM. The method keeps the original head to produce distribution, and adds a lightweight guide head (LoRA-style adapter on the output layer) that is updated online on previously generated samples to produce $q_t$. At each round, the next-token distribution used for sampling is reweighted by suppressing tokens that the guide head deems “common in prior rounds.” The paper provides an implementation aligned with parallel decoding (e.g., vLLM/SGLang-style KV reuse) and reports gains on code, math reasoning, and open-ended generation with small throughput drops that shrink for larger models.", "questions": "- **Equal-diversity controls:** Can you report results with *matched entropy* (or matched average NLL) against temperature/top-p, with significance testing?  .  \n- **Scheduling bias:** Does randomizing candidate update order or using async updates change results? Provide curves vs. batch size/KV hit rate.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Clear, pragmatic idea:** A *history-aware* decoding objective that can be layered onto existing stacks with minimal changes (only the output head; small adapter; online updates).\n- **Engineering relevance:** Thoughtful integration with **KV-cache reuse** and round-synchronous updates; throughput overhead appears modest and decreases with model size.\n- **Quality–diversity trade-off:** Reported increases in diversity (lower candidate similarity) with **PASS@k/MAUVE** not degrading and sometimes improving.\n- **Simple hyperparameters:** Two main knobs—$\\lambda$ (suppression strength) and adapter rank $r$—make the method operationally accessible.", "weaknesses": "1. **Temperature/top-p parity not rigorously controlled**  \n   The central claim is that OverRIDE provides more than generic flattening (i.e., increasing temperature). However, most comparisons are at author-chosen settings rather than **equal-diversity (equal entropy/KL)** conditions. Without controlled comparisons, it is hard to isolate the benefit beyond “being hotter.”\n\n2. **Stability and guardrails of ratio reweighting**  \n   The method hinges on $p_t \\propto (p/q_t)^\\lambda p\\$. If $q_t$ is small or miscalibrated on some tokens, the ratio can explode and induce overly aggressive exploration. The paper does not fully quantify **failure modes** (invalid/format-broken outputs) under large \\(\\lambda\\), nor detail **safeguards** (logit clipping, $\\epsilon$ -floors, temperature stacking).\n\n3. **Guide-head “slight drift” is asserted but under-measured**  \n   The text claims the guide model “only drifts slightly,” but lacks systematic analysis of $\\(\\mathrm{KL}(q_t\\|p)\\)$ over rounds, **adapter-norm growth**, or how drift scales with $r$ and $\\lambda$. This matters for safety and to understand when the guide becomes effectively an *anti-LM*.\n\n4. **Parallel scheduling bias**  \n   The implementation updates head $t{+}1$ immediately after sampling round $t$ and enforces a **round-ordered schedule** for KV reuse. This could introduce **order bias** across candidates. There is no study of randomized/async updates or batch-size sensitivity to rule out engineering confounds.\n\n5. **Ablations don’t fully decompose gains**  \n   Improvements in PASS@k could come from (a) genuine coverage of new solution modes vs. (b) more surface-different but semantically near misses. The paper lacks **semantic de-dup** analysis, **first-hit round distributions**, and **cost-normalized** comparisons (same steps/candidates).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4iWEL4zCdM", "reviewer": "Reviewer_BHNo", "summary": "This paper introduces OverRIDE (Reweighting-based Iterative DEcoding), a novel decoding method that enhances diversity in LLM text generation by learning from previously generated responses. The key innovation is fine-tuning a guide model on past generations and using it to reweight the next-token distribution, thereby suppressing repetitive patterns. The authors implement a parallel version compatible with vLLM, achieving minimal throughput loss. Experiments on code generation, mathematical reasoning, and story generation demonstrate improved diversity while maintaining or improving quality.", "questions": "1. How does OverRIDE compare to other diversity methods like SimCTG or contrastive decoding? A direct comparison would strengthen the claims.\n2. Can you provide theoretical analysis or intuition for why the specific reweighting formula works better than alternatives (e.g., additive vs. multiplicative)?\n3. How do you recommend selecting λ for a new model or task without extensive hyperparameter search?\n4. What is the computational breakdown between inference and adapter fine-tuning? How many gradient steps per round?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Learning from generation history is an elegant solution to the diversity problem that goes beyond token-level adjustments.\n2. Strong experimental validation across multiple domains (code, math, story), model sizes (3B-72B), and sampling methods.\n3. The parallel version with vLLM integration shows the authors considered real-world deployment.\n4. The decoding dynamics analysis (Section 3.3) provides valuable insights into how the method works, showing the interplay between exploration and quality.", "weaknesses": "1. No comparison with other diversity-enhancement methods like SimCTG, contrastive decoding variants, or other inference-time techniques. The paper only compares against standard sampling.\n2. Why is the reweighting formula in Equation 2 the right choice? No analysis of convergence properties or theoretical guarantees. Connection to exploration-exploitation tradeoffs not formalized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)