Semantic-guided Diverse Decoding for Large Language Model
Authors: Weijie Shi
Yue Cui
Yaguang Wu
Jingzhi Fang
Shibo Zhang
Mengze Li
Sirui Han
Jia Zhu
Jiajie Xu
Xiaofang Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Diverse Decoding
Sampling Strategy
Large Language Model
Primary area: deep_learning
TL;DR: SemDiD generates semantically diverse LLM outputs by guiding decoding in embedding space through orthogonal direction vectors and inter-group repulsion, outperforming existing methods in Best-of-N evaluations and accelerating RLHF training.

Abstract: Diverse decoding of large language models is crucial for applications requiring multiple semantically distinct responses, yet existing methods primarily achieve lexical rather than semantic diversity. This limitation significantly constrains Best-of-N strategies, group-based reinforcement learning, and data synthesis. While temperature sampling and diverse beam search modify token distributions or apply n-gram penalties, they fail to ensure meaningful semantic differentiation. We introduce Semantic-guided Diverse Decoding (SemDiD), operating directly in embedding space that balances quality with diversity through three complementary mechanisms: orthogonal directional guidance, dynamic inter-group repulsion, and position-debiased probability assessment. SemDiD harmonizes these competing objectives using adaptive gain functions and constraint optimization, ensuring both quality thresholds and maximal semantic differentiation. Experiments show SemDiD consistently outperforms existing methods, improving Best-of-N coverage by 1.4-5.2% across diverse tasks and accelerating RLHF training convergence by 15% while increasing accuracy by up to 2.1%.

Semantic-guided Diverse Decoding for Large Language Model
Weijie Shi'; Yue Cui Yaguang Wu?, Ju'gJinnzhh Fang', Shibo Zhang", Mengze Li', Sirui Han Jia Zhu", Jiajie Xu², Xiaofang Zhou
The Hong Kong University of Science and Technology. 2MetaX. Allibbaa Group, *Zhejiang Normal University, SSoochow University
Abstract
Diverse decoding of large language models is crucial for applications requiring multiple semantically distinct responses, yet existing methods primarily achieve lexical rather than semantic diversity. This limitation significantly constrains Best- of-N strategies, group-based reinforcement learning, and data synthesis. While temperature sampling and diverse beam search modify token distributions or apply n-gram penalties, they fail to ensure meaningful semantic differentiation. We introduce Semantic-guided Diverse Decoding (SemDiD), operating directly in embedding space that balances quality with diversity through three complementary mechanisms: orthogonal directional guidance. dynamic inter-group repulsion, and position-debiasse probability assessment. SemDiD harmonizes these competing objectives using adaptive gain functions and constraint optimization, ensuring both quality thresholds and maximal semantic differentiation. Experiments show SemDiD consistently outperforms existing methods, improving Best-of-N coverage by 1.4-5.2% across diverse tasks and accelerating RLHF training convergence by 15% while increasing accuracy by up to 2.1%.
Introduction
Diverse decoding of large language models (LLMs) aims to produce multiple semantically diverse responses for the same query. High-quality, creative, and non-repetitive multiple responses play crucial role in numerous fields. In best-of-N strategies, multiple-answer candidates coupled with verification mechanism (like self colf-consistency enable smaller models to surpass larger ones under equivalent computational budgets [1, 2, 3]. In group-based reinforcement learning from human feedback (RLHF) algorithms like Iterative-RL. facilitates celf-improvement through reward-based evaluation of answer groups. Additionally, in seed-based data synthesis, diverse decoding serves as a fundamental component for generating varied training data, thereby enhancing the robustness and generalization capabilities [8, 9, 10, 11].
[4, 5], RLOO [6]. and GRPO [7], diverse decoding
Current methods like temperature sampling [12], nucleus sampling [13], diverse beam search [14, 15), and code-book [16] create lexical variations, but they often produce semantically similar responses. To ensure semantic diversity among multiple-answe candidates, an ideal approach is to generate nu- merous candidates, embedding and clustering them semantically, and then selecting nigh-probability representatives from different clusters. Unfortunately, this is computationally prohibitive in practice, and calls for resource-efficient methods achieve semantic diversity during the decoding process.
"wshi ah@connect. ust... hk t Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We propose Semantic-gur gunddd Diverse Decoding (SemDiD), decoding algorithm that generates multiple (k) semantically diverse and high-quality answers through three key mechanisms: directional guidance, inter- inter-group repulsion, and probability preference. Directional guidance steers each decod- ing trajectory toward distinct semantic regions, inter-group repulsion maintains semantic distances between groups, and probability preference prioritizes tokens with higher likelihood to ensure quality, SemDiD runs groups of beam searches simultaneously, with the first group employing greedy decoding to establish a quality baseline. Despite the conceptual simplicity, effective implementation presents several challenges:
Diminishing Returns in Probability- Quality Assessment: While probability offers a compu- tationally efficient quality metric, has sigmificant limitations. Extremely low probabilities correlate with poor outputs, but quality improvements plateau as probabilities increase beyond moderate levels. Rather than aggressively maximizing probability, SemDiD treats probability as a lower bound using a harmonic gain function that adaptively prioritizes quality or diversity based on their relative performance. This ensures both metries remain above acceptable thresholds while optimizing for overall effectiveness.
Position and Length Bias in Probability Estimation: As sequences progress, tokens appearing later typically receive higher probabilities due to increasing contextual certainty, resulting in distorted quality assessment. To address this. SemDiD employs position- based regularization, applying diminishing weights to tokens in later sequence positions while imposing a threshold to limit the influence of excessively high-confidence tokens.
Balancing Competing Forces: To simultaneously maintain quality and maximize diversity, SemDil) integrates harmonic mean theory with c-constraint optimization guarantee that each group's quality score remains above the greedy baseline threshold while pursuing semantic diversity. Within the diversity dimension, SemDiD implements stage-aware transition between directional guidance and inter-group repulsion. During early exploration stages, partial answer embeddings are closely in semantic space, so directional guidance predominates establish initial diversity. As decoding progresses, weighting gradually shifts toward inter- group repulsion, recognizing that predetermined directional guidance may become suboptimal and potentially limit model precision.
We evaluate SemDiD on multiple benchmarks to test its effectiveness. For Best-of-N, our ap- proach enhances performance across various tasks, with improvements ranging from 1.4% on ARC-Challenge to 5.2% on MMLU-Pro+ with 25 samples. In the RLHF framework, SemDiD enables more efficient exploration of the solution space through the IterativeRRLHF, GRPO, and RLOO algorithms, reaching target performance levels more quickly during training and achiev- ing 1.8-2.1% % higher accuracy compared the best baseline methods. The code is available at https: / //ggttttttuutuuo/gg wtw.hhho.oooooeecocoooocccceeeee shiwe wwwwwwwwwweeeee eieeer///// SemDiD.
2 Related Work
Decoding algorithms determine how tokens are selected during language model generation. Diverse decoding methods extend this concept by producing multiple distinct outputs for a single prompt, and can be categorized into parallel sampling approaches and diverse beam search variants.
Parallel samplings generate tokens independently across different decoding paths. Methods such [17], nucleus sampling [13], and typical decoding [18] modify the conditional token distribution to increase randomness, but provide no guarantees against duplicate outputs. Arithmetic sampling [16] uses an implicit arithmetic code-book to ensure character diversity. To prevent sampling tokens with excessively low probabilities that might compromise quality, truncation sampling [19] and epsilon sampling [20] introduce cutoff mechanisms based on probability thresholds and distribution shape.
as temperature sampling, top-k sampling
Diverse beam search variants [15, 14] explicitly model interactions between candidate sequences to prevent redundancy, introducing diversity penalties between beam groups like Hamming, Cumulative, and n-gram penalties. Determinantal beam search [21] formulates decoding as a series of subdetermi- nant maximization problems based on determinantal point processes (DPPs), inherently encoding intra-set interactions to promote n-gram coverage. Stochastic beam search with Gumbel-TTpp- trick [22] and Conditional Poisson [23] enable both exploration and principled sampling without replace- ment from sequence models. While these interaction- methods improve lexical diversity, they

---

Empirically Optimum But Intractable
Semantically- Diverse Decoding
Embeedding
Breedy
Question
Search
Clustering
Question Embedding
LLM Samples
Stepa
Sampled Answer,
Step2
Step3
where size proportional to probability
Quality Score
Clustered
+
Beam Group, where grey means pruned candidate, colored means selected candidate
Diverse Scare
Mnmm Angle
Deraciuuaa Drashance
Inter Group Repuiston
Figure I: Comparison between ideal semantic decoding and SemDiD. While generating numerous samples followed by embedding and clustering would optimally identify diverse representatives. this is computationally intractable in practice. Our proposed SemDiD achieves similar diversity more efficiently by directly guiding decoding semantic space through orthogonal directions and inter-group repulsion, while maintaining quality with debiased probability assessment.
primarily operate the token n-gram level rather than semantic space. Our proposed SemDiD advances beyond these approaches by explicitly guiding the decoding process in embedding based semantic space. Furthermore, ensure the case of deployment, our design principle aims to be usable out-of-the-box without any training.
Additionally, recent works have explored semantic diversity in language generation from domain views. [24] use gradient-based attribution scores to identify semantically critical tokens for uncertainty estimation, employing importance sampling with NLI-guided token substitution.
[25] propose
controlled embedding perturbation at the first token position combined with Bayesian optimization for reasoning in latent space. While these works share our motivation for semantic diversity, SemDiD differs in several key aspects: (1) we operate during beam search decoding rather than post-hoc token substitution or first-token perturbation, (2) our approach balances quality and diversity through harmonic optimization rather than focusing solely on uncertainty or correctness, and (3) we target Best-of-N and RLHF applications rather than uncertainty estimation reasoning tasks.
Semantic-guided Diverse Decoding
Let g denote query input and M pretrained language model with conditional distribution PM pM(v)e) (vlq) over possible responses y E V. Our Semantic- guidddd Diverse Decoding (SemDiD) aims to generate k responses {yı, 32.-.., . y/k) that exhibit both high quality and semantic diversity.
SemDiD Architecture. As shown in Figure SemDiD extends the group beam search framework using k groups and selecting one representative output from each group. Candidates are evaluated on both quality using debiased sequence probability as a lower bound and diversity by assessing embedding space representations through exploration direction and inter-group distance. The first group uses greedy decoding to establish a quality bascline and reference direction.
3.1 Efficient Quality Assessment with Debiased Probability
Quality evaluation is crucial during generation. Although dialogue-based LLMs could be used, applying such evaluations to each candidate branch during generation would incur substantial computational costs. An efficient alternative is using the language model's own token likelihood scores, which reflect its confidence level. When model exhibits uncertainty, tends to produce lower quality outputs [26].
Probability as Quality Lower Bound. Unfortunately, our experiments reveal that blindly pursuing high probability does not necessarily yield higher quality, as shown in Figure 2. Log
Detailed experimental setup is available in Appendix A and
3

---

Probubility
0.2
Log Probubility
Accuracy
Logprob 1 Threshold
-0.5
0.2
LLM-Judge (108)
LLM-Judge
Moan
0.4
LLM Judge (3B)
Mean
Gredyy Decoding
-0.6
Mean Dev
100
200
300
400
-3
Token Position
20
40
Distance to Punctuation
Logprob Threshold
Figure 4: Mean log probability by distance to previous punctu- ation.
Figure 2: Accuracy in evaluating an- Figure 3: Mean log probabil- swer quality.
ity by token position with stan- dard deviation bands.
probability thresholds up to -2.5 demonstrate rapid improvements in evaluation accuracy, but this accuracy generally plateaus thereafter. When the log probability threshold exceeds -1, the accuracy of quality assessment significantly decreases, indicating that many correct responses are mistakenly classified incorrect. This suggests probability-ban thresholding is better suited as lower bounds to avoid extremely low-probability sequences and ensure basic quality, rather than as maximization targets for optimal performance.
Position and Length Biases in Token Probabilities. Figures and 4 reveal systematic variations in token probabilities that compromise quality assessment. Tokens appearing later in sequences receive artificially inflated probabilities due to richer contextual information. Similarly, probabilities increase sentences progress after punctuation marks. These position-dependent biases create unfair advantages for longer responses and certain sentence structures. Without correction, probability- boaaee quality assessment becomes unreliable for comparing candidate sequences of different lengths.
We propose a position-debiased probability that applies diminishing weights to token probabilities based their positions in sequence and sentence:
Pdnbinssed (4i) = p(4 (k<s) exp(-3seg" - exp(-Bsent ds)
where is the absolute position of token t; the sequence, is the distance (in tokens) from token t; to the most recent centence-ending punctuation, and Bseg and Beent are hyperparameters controlling the respective decay rates. Additionally, apply a saturation threshold to prevent excessively high-confidence tokens from dominating quality assessment:
Squality (y) ((in(ppaainseed((4le))))
(2)
3.2 Semantic Diversity Assessment
0.96
Correct Answer
Incomeet Answer
Best-of-N Coverage
0.94
0.92
0.9
Hlighett probability
Cluster
0.88
N-gram
0.86
Semantic clastering (0.58)
0.84
Semantic clustering (1.58)
7
10
13
16
20
25
Top- diverse samples retained (from samples per problem)
Figure Visualization of 100 answers for the same question, where circle sizes represent probability magnitudes.
Comparison of Best-of-N coverage across highest probability selection, character di- versity, and semantic diversity methods.
Figure 6:
While probability-based quality assessment provides an efficient mechanism for filtering low-quality candidates, optimizing solely for probability often yields semantically redundant solutions. As

---

illustrated in Figure 5, correct answers (circles) are scattered across distinct semantic clusters rather than concentrated in single region. This distribution pattern suggests multiple valid reasoning paths exist in the embedding space.
Furthermore, Figure 6 demonstrates that semantically-drivv selection strategies consistently outper- form both probability-basee ranking and character-level approaches in Best-of-N evaluations. When selecting top-k diverse candidates from 100 samples per problem, semantic clustering achieves up to 95% coverage at k=25, compared to 92% for regular probability-based selection. Surprisingly, we found that embedding model size has minimal impact on performance effectiveness, allowing us to employ smaller, computationally efficient models without sacrificing diversity quality or incurring excessive computational overhead. We employ a small sentence embedding model that maps partial or complete sequences to semantic vector space. Afterward, two key mechanisms drive our semantic diversity:
Directional Guidance for Initial Exploration: In the early stages of decoding, embeddings of partial sequences tend to cluster closely in semantic space. This proximity occurs because initial tokens have limited context to differentiate their semantic trajectories substantially. Without explicit guidance, these trajectories would likely converge toward similar semantic regions despite different token selections, resulting in lexical variations with minimal semantic diversity.
To efficiently explore different semantic regions, each group 9 (except the first greedy group) is assigned distinct directional vector d, in embedding space. These directional vectors serve as semantic "targets" that guide the decoding process toward different regions of the embedding space. The first group follows the highest probability path using greedy decoding, establishing a reference direction:
E(v)) E(g) [E( - E(4)
dj
where E(v!) is the embedding of the partial sequence generated by the first group at time step {() is the embedding of the query. For the remaining groups, we apply the Gram-Schmidt orthogonalization process to generate a set of orthogonal directions:
9
89
J = rg
d.
d,
=
1
where Fy is a randomly initialized vector, 8. is the intermediate vector orthogonalized against all previous directions, and d, is the normalized directional vector for group g. The orthogonality ensures each group explores fundamentally different semantic trajectories through the directional score:
Sair (vr)) = cos((E(ff) (((((()))))))))()) - E(g). d.))
Inter-Group Repulsion for Dynamic Divergence: While directional guidance establishes initial trajectory divergence, predetermined directions may become suboptimal generation progresses. Forcing candidates to follow fixed directions regardless of the language model's I's natural tendencies can lead to suboptimal outputs in terms of quality and coherence, To this end, we introduce an inter- group repulsion that dynamically maintains semantic distance between groups without constraining them to rigid paths:
S Srep (vi) max 9'#9
(6)
where (--)) denotes the dot product operation. This score becomes negative as candidate approuches other groups' semantic regions in the embedding space, thereby encouraging continued divergence throughout the generation process.
The final diversity score for a candidate sequence vi is computed as weighted combination of directional guidance and inter-group repulsion:
Sase (vi) (1 - a.) Sur + au Srep (v?)
(7)
where ae is time-dependent weighting factor calculated as:
5

---

0t minl
1,
Ttrans
Here, 1 represents the current decoding step, and Ttrans hyperparameter controlling the transition point. During early steps Tirans), the weight gradually shifts from directional guidance toward repulsion, with repulsion completely dominating after step Ttras- This dynamic weighting recognizes that directional guidance is more valuable during early exploration stages, while repulsion becomes increasingly important as sequences develop their semantic identity.
3.3
Balancing Quality and Diversity
Having assessed quality and diversity, it is critical optimize these competing objectives.
3.3.1
e-Constraint Quality Guarantee
To ensure the lower bound of quality, we adopt the e-constraint method from multi-objective opti- mization:
max Sdiv subject to Squality (v) 2
(9)
yEy
where e represents our quality threshold based on the greedy decoding baseline:
min Squatity (y²)
(10)
[E[1.t]
Here, y is the partial sequence generated by the first (greedy) group at time step i, and (0, 1] serves relaxation parameter balancing quality requirements with optimization flexibility.
Theorem (Quality Guarantee) With quality threshold mechanism, the quality difference between any group g and the first greedy group is bounded by:
E Squality((") Squality (v S 6(1-7) -)
(11)
where 5 represents the maximum quality dispersion the response space.
The first group follows greedy decoding, selecting maximum probability tokens each step. Through e-constraint, any candidate sequence y with (quality)) < e is eliminated regardless of diversity score. Since e = min,gel.e) Squatity (vt) 7, and each group must maintain quality scores at least y fraction of the greedy baseline every step, the cumulative quality difference cannot exceed (1 - 7) multiplied by the maximum possible quality dispersion d.
Proof:
3.3.2 Harmonic Gain-Basse Balancing
While the e-constraint method provides quality guarantees, its binary acceptancefrejection mechanism can be too rigid in practice. In practical applications, we neither want candidates with high probability but low diversity, nor those with low probability but high diversity. To allow more nuanced trade-offs, we employ a harmonic gain function that prioritizes improving the weakest aspect of each candidate:
OC, X-S (u'') Saun
if Squatsty (vi) Squaticy (M)) Sat otherwise
S Scombined (vi)
(12)
where Squality (vi) max(0, Squatity (V²) e) represents the quality surplus above the threshold and À a hyperparameter that controls the strength of harmonic mean.
Proposition 2 The harmonic gain function exhibits diminishing returns for improvements in either quality diversity when the other metric is significantly lower.
Proof: The partial derivatives of the harmonic gain with respect to quality and diversity are:
OScombined
X S2 LEo
as @Soominedd OSai
A (Squaisty) )
(13)
aS Squatity
(S (quaatity + S )2
(S (Squatity + S )2

---

As Squality > Saie. we have 0Seamiined R O and 0s aaammnaned 8S
2 S-Suuiitty X, indicating that
8 @Squality
Squatrev
improvements in quality yield minimal gains while diversity improvements Conversely, when Sdiv > Squality. improvements in diversity yield minimal gains while quality improvements highly rewarded. This adaptive balancing ensures that neither metric is neglected during optimization.
highly rewarded.
Theorem (Diversity Guarantee) The expected minimum pairwise semantic distance between any two responses generated by SemDiD satisfies:
cos( kk)
Emmin dsem (yrs Uy)) 2 a.
(14)
2
where a is the semantic space variance and is the number of groups.
The directional guidance vectors {dl, di...., J are constructed to be orthogonal in the semantic embedding space. The minimum angle between any two vectors is at least T/ /k radians.
Proof:
Since the cosine distance between two unit vectors with angle 0 L-cos(#) and the expected 2 magnitude of semantic differences in the embedded space is proportional to the minimum expected
pairwise distance bounded by --os(s//)))
All scoring components undergo group-based normalization to ensure comparability across different assessments. Detailed algorithm procedures are provided in Appendix C.
4 Experiments
We evaluate SemDiD on two categories of tasks: (1) Best-of-N evaluation across variety of bench- marks assess diversity and quality of generated responses, and (2) the impact on reinforcement learning from human feedback (RLHF) algorithms measure training efficiency and performance improvements. We evaluate SemDiD against baselines including Temperature Sampling [12], Arith- metic Sampling [16], Diverse Beam Search [14], and Determinantal Beam Search [21] on diverse generation tasks.
4.1 Best-of-N Evaluation
4.1.1 Experimental Setup
Datasets. We evaluate the effectiveness of SemDiD in Best-of-N settings across diverse tasks: Reasoning tasks include ARC-Challenge [27], Big Bench Hard (BBH) [28]. GSM8K [29], and Minerva Math [30]. Question answering tasks include CoQA [31], PubMedQA [32], and MMLU- Pro+ [33]. Machine translation tasks include WMT16 [34] (English-German German- -nngish)
Evaluation Metrics. For each query, we generate N responses using different decoding strategies. We measure performance using two key metrics. Coverage represents the percentage of test examples with at least one correct answer among the N generated responses. Accuracy indicates the percentage of test examples where the selected answer (via majority voting or LLM judge) is correct.
4.1.2 Best-of-N Results
Figure 7 demonstrates that SemDiD consistently outperforms other decoding methods across diverse benchmarks. On reasoning tasks, SemDiD shows superior coverage even with fewer samples, achieving 82.4% on ARC-Challenge, 85.6% on BBH, 98.1 % on GSM8K, and 86. % on Minerva Math with just 25 samples. For question answering tasks, SemDiD maintains its advantage with 46.7% on CoQA, 82.6% on PubMedQA, and 82.63% on MMLU-Pro+ at 25 samples. The translation tasks show similar trends with SemDiD reaching 37.2% and 44.7% coverage on WMT16 English- German and German-English respectively at 50 samples. Group Beam Search generally performs well as the second-best method in many tasks, particularly in translation tasks where it closely trails SemDiD, indicating that structured diversity approaches outperform independent sampling methods. Temperature sampling shows inconsistent performance across tasks, with T=1.5 performing relatively well on BBH 77.46% at 25 samples) but underperforming Minerva Math (71.2% at 25
7

---

ARC-Challnge
Bench
GSMBKK
100
80
Covergge (S)
82
90
80
90
10
20
50
70
100
3
10
20
50
75
100
3
25
50
100
Simples
Samples
Samples
Minersa Math
CoQA
PuhMeeJJA
50
80
Cosergg: (%)
40
80
30
60
3.10
25
50
75
100
3
10
25
50
75
100
3.10
25
50
100
Samples
Samples
Samples
MMLU Pro+
Faglish-Gemmnn
Germaan-- Farglsn-Fiiissh
50
80
Covnnge (3)
30
20
200
10
25
50
75
100
25
50
100
3
3.10
25
50
100
Sumples
Samples
Samples
SanDDD (Ours)
Tomp-0.5 Divaree Borm
Temp-10 1.
Temy=1.5
Arith, Sampling
Determinental Berm
Greedy
Figure 7: Coverage comparison for Best-of-N using Qwen-2.5-3B.
samples). Notably, all diverse decoding strategies substantially outperform greedy decoding (shown by the horizontal dashed line), with coverage gaps of up to 40 percentage points on certain tasks, underscoring the critical importance of diverse sampling for complex reasoning and generation tasks.
4.2 Impact on RLHF Training
4.2.1 Experimental Setup
We evaluate SemDiD's impact on reinforcement learning from human feedback (RLHF) by:
Iterativ-RLHHF [5]: Progressively enhances policy models through bootstrapped reward signals in an online learning fashion, utilizing generated responses as training examples.
DeepSeek-GRRE [7]: A group-based RLHF algorithm that improves mathematical reasoning by selecting optimal solutions from answer groups
RI.00 [6]: simplified REIy optimization that outperforms both PPO and "RI-freee" methods with lower computational costs while preserving alignment performance.

---

Diontiie-RR (Qwms-2.5-8)-
GIRP (Qwen-2.-7R)
(Qwen-2.5.7))))
70
70
Was Kate
60
60
60
20
10
40
Rollont Number
Rallout Number
Rolkout Number
Ieerative RLHF
GSMBK
GRPO (Qwen 2.5-78)..-7) GSMKK
(Qwen (wwen-2.55-7- GSMIK
81
Accuraty
80
78
76
Rollout Number
Rollout Number
Rollout Number
(Ous)
Temp=0.5
Temp=10
Temp=1.5
Antth, Sampling
Beam
Decermiinnnal Boam
Figure 8: Performance comparison across RLHF algorithms with varying rollout numbers using Qwen-2.5-7B.
For our experiments, we employ Qwen-2.5-7B and Pythia- base models, training them on the mathematical dataset GSM8K* and the summarization dataset TLDR*. We evaluate performance using accuracy for GSM8K and win rate for TLDR, where the win rate is assessed by GPT-ol-mini against the untrained baseline. For training feedback, we utilize rule-based reward for GSM8K and a reward model? for TLDR.
4.2.2 RLHF Results
Figure 8 reveals that SemDiD consistently enhances RLHF training across all evaluated algorithms and tasks. For TLDR summarization, SemDiD achieves the highest win rates of 72. %, 73.4%, and 71.2% with Iterative-RI.HF GRPO, and RL.O respectively at 60 rollouts. On GSM8K, SemDiD reaches impressive accuracy scores of 85.5% with ItcrativeRLLHF, 88.2% with GRPO, and 82.4% with RLOO. The performance gap between SemDiD and other methods widens rollout numbers increase, with minimal differences at rollouts but substantial advantages at 50 rollouts, suggesting that semantic diversity becomes increasingly valuable with extended exploration. Importantly, SemDiD's s ability to generate semantically diverse sequences results in more varied reward signals (neither all Os nor all 1s), creating larger advantage estimates that prevent policy collapse and drive more effective learning. Group-based diversity strategies (Diverse Beam Search and Determinantal Beam Search) consistently outperform independent sampling methods (Temperature Sampling and Arithmetic Sampling). indicating that explicit inter-sequence interactions are crucial for effective exploration in RLHF, with GRPO combined with SemDiD yielding the strongest overall performance.
5 Conclusion
We introduced Semantic-guided Diverse Decoding (SemDiD), addressing the critical limitation of semantic diversity in I.LM outputs. By operating directly embedding space through orthogonal
hhups:/huggingffce hung://hungiigheee.oovvaattttttp co/lattseetsoooeniiignnn..
Tttps:/tuuggingfaae..odddat co/dattaeetsttlllibbiiie
Verify
'bttss//huggingffcc... co/trl lib/pythia- de-deeuped--dd--nn
co/tt

---

directional guidance, dynamic inter- group repulsion. and debiased probability assessment, SemDiD significantly outperforms existing methods. Our approach balances quality and diversity using an e-constraint mechanism and harmonic gain function that adaptively prioritizes the weaker aspect of each candidate. With stage-aware transitions between exploration mechanisms, SemDiD maintains semantic differentiation throughout the generation process. Our extensive evaluations across Best-of- and RLHF frameworks demonstrate SemDiD's effectiveness improving response diversity and quality simultaneously without additional training.
Acknowledgments
We would like to specially thank the support from the B2 project of the HKUST & MetaX Joint Laboratory. The research work described in this paper was supported by Hong Kong Research Grants Council (grant# 16202722, 16210625, T43-513/23-N, T22-607/24N). It was partially conducted in JC STEM Lab Data Science Foundations funded by The Hong Kong Jockey Club Charities Trust. We acknowledge the support of the National Natural Science Foundation of China (Grant No.62272334, 6257073827, 62577050) and the Natural Science Foundation of Zhejiang Province (Grant No.LY23F020010).
References
[1] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177 2022.
[2] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03311 2024.
[3] Furui Cheng, Vilém Zouhar, Simran Arora, Mrinmaya Sachan, Hendrik Strobelt, and Menna- tallah El-Assady. Relic: Investigating large language model responses using self-consistency. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages
1-18, 2024.
[4] Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprini arXiv:2405. 07700783. 2024.
[5] Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37:116617-116666 2024.
[6] Arash Ahmadian, Chris Cremer. Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint
2024.
[7] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.035 30000 2024.
[8] Weijie Shi, Jipeng Zhang, Yaguang Wu, Jingzhi Fang, Ruiyuan Zhang, Jiajie Xu, Jia Zhu, Hao Chen, Yao Zhao, Sirui Han, et al. Dids: Domain impact-aware data sampling for large language model training. arXiv preprint 2xXv:2504.13227, 2025.
[9] Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing, arXiv preprint arXiv:2-466088464 2024.
[10] Yitian Luo, Yu Liu, Lu Zhang, Feng Gao, and Jinguang Gu. A survey quality evaluation of instruction fine-tuning datasets for large language models. Data Intelligence, 7(3):527-566, 2025.
10

---

[11] Jan Sawicki, Maria Ganzha, and Marcin Paprzycki. The state of the art of natural language pro- cessing a systematic automated review of nlp literature using nlp techniques. Data Intelligence. 5(3).707-749, 2023.
[12] Matthew Renze. The effect of sampling temperature on problem solving in large language models. In Findings of the Association for Computational Linguisties: EMNLP 2024, pages 7346-7356, 2024.
[13] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:/944.0970 2019.
[14] Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes. In Proceedings of the AAAI Conference on Artificial Intelligence, number 1, 2018.
[15] Jiwei Li, Will Monroe, and Dan Jurafsky. simple, fast diverse decoding algorithm for neural generation. arXiv preprint 1eXiv:611108560 2016.
[16] Luke Vilnis, Yury Zemlyanskiy, Patrick Murray, Alexandre Tachard Passos, and Sumit Sanghai. Arithmetic sampling: parallel diverse decoding for large language models. In International Conference on Machine Learning, pages 35120-35136. PMLR, 2023.
[17] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv:8805.00 04833, 2018.
[18] Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Locally typical sampling. Transactions of the Association for Computational Linguistics, 11:102-121, 2023.
[19] John Hewitt, Christopher D Manning, and Percy Liang. Truncation sampling as language model desmoothing. arXiv preprint arXiv:2210.15119 2022.
[20] Markus Freitag, Behrooz Ghorbani, and Patrick Fernandes. Epsilon sampling rocks: Investi- gating sampling strategies for minimum bayes risk decoding for machine translation. arXiv preprint arXiv:2305. 2305.09860, 2023.
[21] Clara Meister, Martina Forster, and Ryan Cotterell. Determinantal beam search. arXiv preprint arXiv:2106.07400, 2021.
[22] Wouter Kool, Herke Van Hoof, and Max Welling, Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In International Conference on Machine Learning, pages 3499-3508. PMLR, 2019.
[23] Clara Isabel Meister, Afra Amini, Tim Vieira, and Ryan Cotterell. Conditional poisson stochastic beams. Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 664 681. Association for Computational Linguistics, 2021.
[24] Lukas Aichherger, Kajetan Schweighofer, Mykyta lelanskyi, and Sepp Hochreiter. Improving uncertainty estimation through semantically diverse language generation. In The Thirteenth International Conference on Learning Representations, 2025.
[25] Qinglin Zhu, Runcong Zhao, Hanqi Yan, Yulan He, Yudong Chen, and Lin Gui. Soft reasoning: Navigating solution spaces in large language models through controlled embedding exploration. arXiv preprint arXiv:2505.24668. 2025.
[26] Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. Llm-check: Investigating detection of hallucinations in large language models. Advances in Neural Information Processing Systems, 37:34188-34216 2024.
[27] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint
2018.

---

[28] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi. Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:22 10.09261, 2022.
[29] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 2021.
[30] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
[31] Siva Reddy, Danqi Chen, and Christopher Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266,
2019.
[32] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567-2577, 2019.
[33] Saeid Asgari Taghanaki, Aliasgahr Khani, and Amir Khasahmadi. Mmlu-pro+: Evaluating higher-order reasoning and shortcut learning in 1lms. arXiv preprint arXiv:2409.02257, 2024.
[34] Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, al. Findings of the 2016 conference on machine translation (wmt16), In First conference on machine translation, pages 131-198. Association for Computational Linguistics, 2016.
[35] Juyeon Heo, Miao Xiong, Christina Heinze-Deml, and Jaya Narain. Do Iims estimate uncertainty well in instruction-follovinng arXiv preprint arXiv:2410. 145822 2024.
[36] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation natural language generation. arXiv preprint arXiv:2302.099: 09660,944 2023.
[37] Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfi-embedding-:: Advanced text embedding with multi-stage training. 2024.
[38] Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. Jasper and stella: distillation of sota embedding models, 2025.
[39] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv:2309.03883 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and introduction clearly state the main contributions: (1) identifying limitations of existing decoding methods, (2) introducing SemDiD with its three components, and (3) demonstrating performance improvements that match the experimental results in Sections 4.1 and 4.2.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Section 3.1 (probability- -bsed quality assessment), with detailed analysis in Appendix C.3 noting the 25-35% additional time overhead.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Complete proof' are provided for all theoretical claims, including Quality Guarantee Theorem, Diversity Guarantee Theorem, and harmonic gain function properties, with additional analysis in Section 3.3.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 4 and Appendix specify all datasets, evaluation metries, model archi-
tectures, and hyperparameters. Algorithm provides detailed pseudocode for implementing SemDiD.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
14

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code available the repository URL. All experiments use publicly available benchmarks with citations, and implementation details are thoroughly documented in the
Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines fbttps://nippe cc/
public/ gewies//gee/////eee CodeSubmi ss pissoooolcyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/nips. cc/ ccc/publiie public//ee c//guu dee /C/deeudubbn ss onPol apPolicyy ccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Sections 4.1.1 and 4.2.1 detail the experimental setup, with hyperparameters listed in Table 2. RLHF experiments specify base models, datasets, and reward mechanisms used.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
15

---

Results are reported across 9 benchmarks for Best-of-N evaluations, and RLHF experiments show performance across multiple algorithms and tasks, providing robust evidence of consistent improvements. We also provide the reward growth log for RLHF in Figure 14.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computational requirements are detailed in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tt/rirusppp..e// cc/ htt.b//i//thicssueeeeeee..
Answer:
[Yes]
Justification: The research uses publicly available benchmarks and models, doesn't involve human subjects, doesn't encourage harmful applications, and properly cites all prior work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
16

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper doesn't include a dedicated discussion societal impacts, focusing primarily on technical contributions without addressing potential benefits risks society.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper introduces a decoding algorithm rather than high-risk models or datasets, safeguards beyond those in the base models aren't 't necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets, models and embedding models properly cited with references.
17

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papersw wit thcode. com/ dataaaaaassss tasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release code with well-documented instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicablle as well as details about compensation (if any)?
Answer: [NA]
Justification; The research doesn't involve human subjects or crowdsourcing; all evaluations use automated metrics on benchmark datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
18

---

Answer:
[NA]
Justification:
No human subjects were involved in this research.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer;[NA]
Justification:
We just use LLMs to refine the grammar.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: //t.urieisppp/pp//e ee cc//oofereeccss /2025/ LLM) for what should or should not described.
19

---

A Analysis of Probability-Basee Quality Assessment
A.1 Experimental Setup
To investigate the relationship between token probabilities and answer quality, we conducted series of experiments using mathematical reasoning and common-sense question answering tasks. We sampled from three datasets: GSM8K [29], -.CC-hhalengge [27]. and BBH [28], selecting 500 problems from each. For each problem, generated 100 candidate answers using nucleus sampling
with p = 0.95 and temperature = 1.0 by Qwen-2.5-3B. Each generated answer was evaluated using two methods:
LLM-as-Judge: large language model evaluated each answer for correctness (binary classification).
Probability Thresholding: Answers were classified as correct/incorrect based on various log probability thresholds.
For LLM-as-Judge, we used a stronger model than the one generating the answers to ensure reliable evaluation, including Qwen-2.5-3B, 7B, and 70B.
A.2 Probability-Quall Relationship
Our analysis revealed non-linear relationship between sequence probability and answer quality. As shown in Figure 2, there exists a critical threshold region where quality assessment accuracy changes dramatically.
For log probability thresholds below -255, accuracy of quality assessment increases rapidly with higher threshold values. This suggests that extremely low-probability sequences strongly correlate with incorrect answers. However, accuracy plateaus in the region between 225 and -1.0, indicating diminishing returns for higher probability thresholds. When log probability thresholds exceed 1..0, we observed a significant decrease in assessment accuracy. This unexpected decline occurs because many correct answers were misclassified as incorrect these higher thresholds. The finding suggests that probability is better suited as lower bound to filter out low-quality outputs rather than as maximization target.
Additionally, we observed that model size significantly impacts the effectiveness the LLM-as-judge approach. The 70B model achieved accuracy of 0.8185. While smaller 7B and 3B models reached only 0.7685 and 0.7417 respectively, comparable to probability thresholding. However, larger models inevitably introduce substantial inference costs, making them impractical for real-time evaluation during the decoding process.
A.3 Position and Length Bias Analysis
Further analysis revealed systematic biases in token probabilities that compromise fair quality assessment:
A.3.1 Position Bias
We computed average log probabilities for tokens at different positions across all generated sequences. As shown Figure 3, clear upward trend emerges: tokens appearing later in sequences receive consistently higher probability scores. This effect is particularly pronounced after position 20, where average log probabilities increase by approximately 0.15 per 10 tokens.
This position bias creates an unfair advantage for longer responses in probability-based quality assessment, their aggregate scores benefit from the artificially inflated probabilities of later tokens.
Initial Token Bias. We also observed significantly higher confidence (i.e., higher probability) for initial tokens across generated responses. This phenomenon occurs because language models are trained on common opening phrases such as "To solve this problem...", "According to the given information..... "Let's approach this step by step...".
20

---

A.3.2 Sentence Progress Bias
Similar patterns appear within sentence structures. We analyzed the relationship between token's distance from the most recent sentence-ending punctuation and its assigned probability. Figure illustrates that tokens appearing further into a sentence receive progressively higher probabilities, with an average increase of 0.08 per 5 tokens after punctuation. This sentence-level bias favors certain syntactic structures and further distorts quality assessment when using raw probability scores.
A.4 Debiasing Methodology Evaluation
To address these biases, implemented the position-debiased probability described in Section 3.1. We conducted an ablation study to evaluate the effectiveness of our debiasing approach on SemDiD, which used the Qwen-2.5-3B model with 10 samples per question. The combined approach with saturation threshold achieved the highest accuracy, validating our debiasing methodology.
Table Performance comparison of different debiasing approaches under 10 samples per question.
Method
Avg. Coverage
Avg. Accuracy by LLM-as-Judge
Raw Probability
67.85%
61.02%
Position Debiasing Only
68.41% %
61.90%
Sentence Debiasing Only
67.95%
61 11.23%
Combined Debiasing
68.64 %
62.36%
+ Saturation Threshold
68.76 %
62.47%
B Analysis of Semantic Diversity Assessment
B.1 Experimental Setup
To investigate the value of semantic diversity response generation, we conducted experiments comparing different selection strategies for Best-of-N evaluation. Using GSM8K and BBH reasoning problems as our testbed, we generated 100 candidate responses for each problem using nucleus sampling with p 0.95 and temperature T 1.0 with Qwen-2.5-3B. We then applied various selection strategies to identify the most representative diverse subsets.
B.2 Selection Strategies
We compared three distinct selection methods:
Highest Probability: The standard approach of selecting answers based solely on their log probability scores, taking the top-k candidates with highest probabilities.
N-gram Clustering: Candidates were clustered based character-level n-gram similarity. For each cluster, we selected the representative with the highest probability score. This approach ensures lexical diversity but may not capture deeper semantic differences.
Semantic Clustering: We embedded all 100 candidates using 0.5B and 1.5B parameter sentence embedding models, performed K-means clustering in this semantic space, and selected the highest- probability candidate from each cluster.
B.3 Semantic Space Visualization
Figure 5 provides t-SNE visualization of the semantic embedding space for specific GSM8K problem, where correct answers (circles) and incorrect answers (crosses) are scattered across distinct semantic clusters. The problem analyzed is:
21

---

GSM8K Problem Example
rectangular band formation is formation with band members in each of rows, where m and r are integers. A particular band has less than 100 band members. The director arranges them in rectangular formation and finds that he has two members left over. If he increases the number of members in each row by 1 and reduces the number of rows by 2, there exactly enough places in the new formation for each band member. What is the largest number of members the band could have?
For this problem, 100 responses were generated, of which 22 were correct answers. These responses clustered into 10 semantic groups. Notably, 9 of the 10 clusters contained at least correct answer, and in 5 clusters, the highest-probabillty response was correct. Several key observations emerge:
Multiple Valid Solution Paths: Correct answers appear in multiple distinct clusters rather than concentrated in single region, confirming that multiple valid reasoning paths exist for solving mathematical problems.
Cobbability-Correchme Correlation: Circle sizes represent probability magnitudes, showing that while higher probability generally correlates with correctness within clusters, this relationship not uniform across the entire semantic space.
Cluster Purity: Some semantic clusters contain predominantly correct answers (e.g., purple, blue, and orange clusters), while others show a mix of correct and incorrect solutions (c.g., red and yellow clusters). This suggests that certain reasoning approaches are inherently more reliable than others.
Distribution Patterns: Incorrect answers also form clusters, indicating systematic error patterns that could potentially be useful for model improvement and error analysis.
Caltect Alceer
Cornect Toue!
rage
Figure 9: t-SNE visualization of semantic embedding space for GSM8K problems, where correct answers (circles) and incorrect answers (crosses) are scattered across distinct semantic clusters. Circle sizes represent probability magnitudes.
More examples are provided in Figure 9. This visualization highlights the importance of exploring different semantic regions during decoding rather than simply maximizing token probabilities, as valuable correct answers may exist across diverse semantic clusters rather than being concentrated in a single figh-probabilit region.
B.4 Results and Analysis
As shown in Figure 6, semantic clustering consistently outperforms both probability- ranking and character-level approaches across all sample sizes. When selecting top-k diverse candidates from 100 samples per problem, semantic clustering with even the smaller 0.5B model achieves up to 95% coverage at k=25, compared to 92% for regular probability- particularly pronounced at lower sample counts (k=3,5,7), where resource efficiency is most critical. At k=3, semantic clustering provides 3-4% absolute improvement in coverage compared to highest probability selection.
selection. The performance gap is
Embedding Model Size Impact: Surprisingly, we found minimal difference between the 0.5B and 1.5B embedding models in terms clustering effectiveness. The 0.5B model achieved 91.7% coverage at k=10, only 0.1% below the 1.5B model's 91.8%. This finding suggests that even lightweight embeddiin models can effectively capture the semantic distinctions necessary for diversity assessment, allowing for computational efficiency without sacrificing diversity quality.
22

---

Implications for Diverse Decoding, Our analysis demonstrates that semantic diversity provides significant advantages over lexical diversity or probability- baseed selection in Best-of-N scenarios. The findings support the design decisions in SemDiD, particularly the use of directional guidance to explore distinct semantic regions rather than relying solely on token-level diversity, the importance of balancing quality (via probability) with semantic diversity, and the viability of using lightweight embedding models for semantic assessment during the decoding process. Furthermore, these re- sults validate the efficacy of semantic clustering candidate responses are available. For practical applications where generating large number of candidates is feasible, semantic clustering provides an effective mechanism for identifying diverse and representative subset.
post-hoc selection strategy when multiple
C Algorithm and Optimization Details
C.1 SemDiD Algorithm
Algorithm I provides a detailed overview of the Semantic-guided Diverse Decoding (SemDiD) procedure. The algorithm simultaneously manages k groups of beam searches, with the first group following greedy decoding to establish a quality baseline.
C.2 Efficient Implementation
A naive implementation of the SemDiD algorithm would incur substantial computational overhead due to duplicated embedding calculations. To address this, we implemented several optimization techniques:
C.2.1 Dynamic Multi-step Lookahead
Unlike traditional Diverse Beam Search methods that rely on lightweight heuristic rules for token selection, operating in the semantic space requires embedding computations that could become prohibitively expensive. Traditional approaches evaluate all tokens in the vocabulary (typically around 100K tokens) against beam scoring functions, which is computationally feasible for simple n-gram penalties.
With SemDiD, evaluating the semantic impact of each vocabulary token through embedding calcula- tions impractical. Moreover, the semantic change from single token is often subtle for reliable diversity assessment. To address this, we implement dynamic multi-step lookahead that explores E, sequences per beam, each extending forward by I,, steps. It allows evaluate substantial semantic deviations rather than token-level variations.
The exploration width E, is dynamically adjusted based on sentence position [35,66];
Ebase
Pps position & follows punctuation
E,
(15)
Ebase Pas otherwise
where (tp and 1hn are hyperparameters, smaller values helps reduce the computational cost of SemDiD. Alternatively, a linear function of sentence position can also be used adjust these parameters dynamically.
1.5 and 0.7 respectively in implementation. Setring
We set maximum lookahead depth of Lmazz but terminate the exploration early if a punctuation mark is encountered:
ExploreDepth(y) min( StepeUnnilleeccecuucuuaoooo////
(16)
This adaptive approach ensures that we explore complete semantic units (sentences clauses)
without wasting computational resources on unnecessarily deep lookahead. Our analysis shows that approximately 26% of decoding steps qualify for increased exploration width. While we considered using entropy-based dynamic adjustments, token-level entropy exhibits high variance and can introduce noise, opted for the simpler punctuation-base approach.
23

---

Algorithm Semantic-guided Diverse Decoding (SemDiD)
Require: Query q. language model M, embedding model &, number of groups k, beam size b, quality relaxation ", transition step Trrass: harmonic strength X
Ensure: k semantically diverse responses {y1, ................................
1: Initialize Yo -- {g}
Greedy group has single beam Diverse groups start with query
2: Initialize Y0 < {q} for groups g [2, k]
3: Initialize , f-
(enddomunitVector( for all groups E [1,
> Random vectors
4: Initialize group directions dy null for all groups g [1,
5: for = 1, 1....... Tmax do
6: 04 min(1, Terams
Update transition weight Process each group in parallel D Candidates for group at step Update directions periodically
7:
for 9 ......... do
8:
C <-
9:
if mod T Tupdate
then
10:
if g = 1 then
Let y² be the sequence in group I
d1 +- 8(y))-8(g)
12:
[(((r))-((n))
b First group direction
13:
else
B gg i=
Fg d
d4
Cramm-Schmidt
||
15:
d
gg
B. Normalize direction
end if
17:
end if
if g
1 then
> Greedy group with single beam
19:
Let y be the sequence in Y,
y' GreedyLookahead y. L,)
b Greedy lookahead
21:
Sqality (1/) +- DebiasedProb(ppM (w'lv), wv'l lul. dpanct)
22:
Y, f {3/}
Maintain single beam Update threshold Diverse groups with multiple beams For each beam in the group Dynamic exploration width b Generate E, lookahead sequences Generate one path
23:
Update < m-minc,, Squalit (v/) 7)
24:
else
for
y E Y do E, DettemineExplooationiiiii for e = 1, 1. E, do
26:
27:
28:
y - GenerateeLokaaaeedd L,)
29:
Squality ( ) + DebiasedProb(pM (v/ (v/lv), 1b'1 - 11yl, dpwnet)
30:
(y'r + cos
- E(g),
Directional score
31:
Srep +- max. ( (Best(Y
> Repulsion
32:
C7 + CJ U (1// Squatyy (v/), ( (z/))
33:
end for end for Use percentile- based normalization for Squality - Squality Sair Saie. Srop Step in O do
34:
35:
36:
for each candidate (3/, (y)()') + (1 az) ) Sar(y') + ar (v))
37:
if Squality (V)) then Soumbined (1/) t 00
39:
>> Below quality threshold
else
41:
S qualtiy (V') & max(0, Squalil (v') -c) A- (D/) )a
D Quality surplus
42:
Soumbined (5/)
> Harmonic
8 Syan (an
43:
end if end for Y < SclcctTopK(Cy.))
45:
>> Keep top-b candidates per group
end if
47:
end for
if all groups have complete responses or reached max length then
49: break
50: end
51: end for
52: return (Y77 (Y)) Best(Y (Y²)
> Return best from each diverse group
24

---

Each group functions as independent beam search representing distinct search direction. The groups explore asynchronously reduce computational bottlenecks. After every T Tupdate steps of greedy decoding, we update the reference direction and guidance vectors for each group.
C.2.2 KV-Cache Utilization for Embedding Efficiency
To further reduce computational overhead, we leverage the key-value (KV) cache mechanism when computing embeddings. We recommend using autoregressive embedding models such as BeapeeeSSSR-EEeedennnn------
[37] or NovaSearch/stella, per), they allow for efficient reuse of computation in sequential token processing. These models can share the same KV cache architecture used in language model decoding.
1.3B_ v5 [38] (we chose in this pa-
Alternatively, small BERT-style models like the 30M all-MiniL.M-L--22 can be used without introducing excessive computational costs. Our experiments indicate that embedding model size has minimal impact on diversity quality, making lightweight models practical choice for production deployments.
C.3 Computational Complexity Analysis
Theorem 4 (Computational Complexity) The time complexity of SemDiD is:
O b- E. (I+ CLAS + CE))
(17)
where k is the number of groups, b is the beam size, T is the total decoding steps, Eg is the exploration width per beam, L is the lookahead depth, CLM the cost language model forward pass, and Cg is the cost of an embedding model forward pass.
Proof: SemDiD maintains k parallel groups executing beam search until sequence completion, requiring T' total decoding steps. The first group performs greedy decoding with single beam (b = 1), while groups 2 through & maintain beams each.
At each decoding step f, the greedy group generates lookahead sequence extending Lg tokens from its current state. This requires Le sequential forward passes through the language model, resulting in L, CLM computational cost per step.
For diverse groups (2 through k:), each beam explores E, different continuation paths at every step. Each continuation generates lookahead sequence of length L4, requiring L,, language model forward passes. After generating each complete lookahcad sequence, the algorithm computes its embedding for semantic diversity assessment. Therefore, each beam in diverse groups incurs Eg (L, CLM +CB) cost per step.
The total computational cost across all T steps is:
Cost = T [Greedy Group + Diverse Groups]
(18)
T [L. CI.M (k 1) b. E. (L, CI.M + CF)]
(19)
Since the greedy group can be viewed as having E, = 1 and b = 1, and noting that it doesn't 't require embedding computation, this simplifies to the stated complexity O(k b T E (Le CLM Cg)).
KV-cache optimization significantly reduces the effective cost of forward passes by reusing previously computed key-value states. Early termination at punctuation marks bounds the actual lookahead depth below Liman, while dynamic adjustment of E, based on sentence position further optimizes resource utilization.
Traditional Diverse Beam Search operates with complexity O(k b. T. (T., CLM + V Cscore) )) where V is the vocabulary size and Cecore is the cost of evaluating diversity penalties. At each step, DBS performs forward pass through the language model obtain token probabilities, then evaluates diversity penalties across all V vocabulary tokens. While Cscore < C'LM, scoring 100K tokens creates small overhead. The key distinction is that DBS applies shallow, token-level diversity penalties, whereas SemDiD invests computation in deeper semantic exploration through multi-token lookahead.
25

---

Temperature sampling achieves O(k-T -..-TLL. Lt CLM) complexity per sequence, appearing more efficient. However, achieving comparable semantic diversity requires generating more independent samples where ktemp >>
DiseDDDD b, resulting in total complexity that may exceed SemDiD'ss cost.
In practice, despite introducing threefold computational requirements, the actual end-to-end lateney of SemDiD increases by only 25-35% compared to group beam search (huggingface official imple- mentation) with typical parameters (A: 25, for real-wer weal-woldd applications. This modest increase in time cost delivers substantial improvements in semantic diversity and downstream performance, creating favorable trade-ofi between resource consumption and quality gains for applications where diverse, high-qualit, outputs are critical. Under conditions of unlimited computational resources or for high-value applications such as RLHF train- ing, the significant performance improvements that SemDiD delivers greatly outweigh its marginal computational costs.
Le 10), making a practical solution
D Hyperparameter Settings and Analysis
D.1 Default Hyperparameter Settings
Table 2 presents the default hyperparameters used in our experiments. These values were determined through extensive grid search optimization on a held-out validation set. All experiments were conducted on cluster of 8 NVIDIA H800 GPUs. The Best-of-N evaluations across all benchmarks required approximately 70 hours, while each RLHF training took approximately 40 hours for Qwen- 2.5-3B.
Table 2: Default hyperparameter settings for SemDiD
Parameter
Description
Default Value
N
Number of groups
k or k/2
b
Beam size per group
5/3
Etase
Base exploration width Maximum lookahead depth
4 4/3
Lmaz
20 10 5
T Ttrane
Transition step for a Quality relaxation parameter Harmonic gain strength
10
X
Y
0.25
2.0
3xeg
Sequence position decay rate Sentence position decay rate
0.001
Ssent
0.005
T
Probability saturation threshold
-0.8
Temperature
Sampling temperature
1.0
Top-p
Top-p (nucleus sampling) threshold 0.95
The number of groups N is adaptively set to either the target number of samples or k//2, depending on the desired trade-off between computational cost and diversity quality. When N = k, each group returns one candidate, maximizing inter-group diversity. When k/2, each group returns two candidates, reducing computational overhead while maintaining reasonable diversity.
D.2 Exploration Width and Beam Size Analysis
We conducted experiments to analyze the relationship between exploration width (E,) and beam size (b) on the GSM8K dataset. Figure 10 shows the coverage achieved with different parameter combinations.
Results show that increasing the exploration width from to yiclds significant improvements in coverage. For instance, with 6 3, coverage improves from 87.3% to 93.8%, representing 6.5 percentage point gain. However, further increases to E, = 8 and E, = 16 provide diminishing returns, with coverage plateauing around 94.7%. This saturation effect can be attributed to the limited semantic variance available most problems, where exploring beyond a certain width captures only redundant variations.
Similarly, beam size shows positive correlation with coverage up to = 3, after which the gains become marginal. For example, E, 4, increasing beam size from b 1 to = 3
26

---

96
94
Coverage (%)
b=
92
b=2
90
b=3
88
5
86
10
84
8
16
Exploration Width (E,)
Figure 10: Impact of exploration width and beam size on coverage. Results are averaged over 500 GSM8K problems with 10 groups.
improves coverage from 89.8% to 93.8%, but further increasing to b = 10 only yields an additional 0.5 percentage points. The computational cost, however, increases linearly with both E, and suggesting that E, = 4 and b = 3 offer the optimal trade-off between performance and efficiency.
D.3 Group Structure Analysis
We investigated alternative group configuration where each group returns multiple candidates rather than a single representative. Table presents the coverage results when varying the number of outputs per group while maintaining constant total of 10 outputs.
Table Coverage comparison with different group structures on GSM8K. Total outputs fixed at 10.
Outputs per Group
Number of Groups
Coverage (%)
1
94.8
2
5
94.5
3
3 (with group returning 4)
92.2
5
2
87.9
10
84.3
The results demonstrate that the single-output- pwr--erorr configuration achieves optimal coverage. When groups return multiple candidates, coverage remains relatively stable for 2-3 outputs per group but degrades significantly beyond that. This pattern suggests that inter-group diversity mechanisms are more effective than intra-group selection for maintaining semantic separation.
The decline in coverage with fewer groups can be explained by the reduced effectiveness of orthogonal direction guidance. With fewer groups, the semantic space cannot be partitioned as efficiently, leading to increased overlap between exploration regions. Additionally, the inter-group repulsion mechanism becomes less effective with fewer groups, there fewer distinct trajectories maintain separation between.
The choice of using multiple groups with single outputs as the default configuration, while allowing flexibility to use N k/2 when computational resources are constrained. The modest decrease in coverage (0.3%) when using two outputs per group makes this attractive option for resource-limited settings.
D.4 Lookahead Depth Analysis
Unlike character-level diversity methods that can evaluate each vocabulary token individually with minimal computational cost, semantic diversity assessment requires embedding model forward passes, making boken-by-toker evaluation prohibitively expensive. Additionally, single-token semantic changes are often too subtle for reliable diversity measurement. Therefore, SemDID introduces lookahead depth parameter Lynar to control how many tokens ahead we explore when evaluating semantic diversity, allowing assessment of more substantial semantic deviations while managing computational overhead.
27

---

We conducted experiments using Qwen-2.5-3B across different Lmaz values to demonstrate its sensitivity and identify optimal settings. Table shows the performance and computational overhead analysis across different lookahead depths.
Table 4: Performance and computational overhead analysis across Imaz values using Qwen-2.5-3B.
max
GSM8K Coverage (N=25)
ARC Coverage (N=25)
Computational Overhead
5
95.3%
80.1%
+15%
10
98.1%
82.4%
+25%
15
98.2%
82.6%
+35%
20
98.6%
82.7%
+45%
Performance saturates around Tymax 10, with diminishing returns for deeper lookahead while computational overhead continues to increase substantially. This suggests that 10 tokens provide sufficient context for reliable semantic diversity assessment without excessive computational cost.
D.5 Quality-Diversity Balancing Analysis
The quality relaxation parameter 7 and harmonic strength control the trade-off between maintaining quality thresholds and pursuing semantic diversity. We conducted systematic sensitivity analysis by varying each parameter independently understand their effects on both coverage and aceuracy.
Effect y (with 2.0 fixed): The quality relaxation parameter determines how much the quality threshold can be relaxed to promote diversity. Lower values maintain stricter quality requirements, while higher values allow more quality sacrifice for diversity gains.
Table 5: Sensitivity analysis of quality relaxation parameter y with 2.0 fixed.
Task
? = 0.15
?
0.20
Y y= 0.25
?
0.30
? 0.35
GSM8K Coverage (N=25)
96.6%
97.1%
98.1%
97.8%
97.4%
GSMSK Accuracy (N=25)
75.9%
76.6%
77.5%
77.2%
77.2%
WMT16 Coverage (N=25)
36.7%
36.8%
37.2%
36.9%
36.7%
WMT16 Accuracy (N=25)
20.2%
20.4%
20.7%
20.5%
20.3%
Effect of A (with 0.25 fixed): The harmonic strength parameter À controls the intensity of the harmonic gain mechanism that balances quality and diversity objectives. Higher values strengthen the diversity promotion effect, while lower values prioritize quality preservation.
Table 6: Sensitivity analysis of harmonic strength parameter À with y 0.25 fixed.
Task
== 1.0
À = 1.5
À = 2.0
À = 2.5
À = 3.0
GSMBK Coverage (N=25)
96.8%
97.4%
98.1%
97.9%
97.6%
GSM8K Accuracy (N=25)
77.3%
77.1%
77.5%
77.3%
76.9%
WMT16 Coverage (N=25)
36.8%
36.5%
37.2%
36.9%
36.7%
WMT16 Accuracy (N=25)
20.3%
20.6%
20.7%
20.7%
20.6%
The results demonstrate that default settings (y 0.25, 2.0) consistently achieve near-
optimal performance across different tasks, with performance remaining stable within reasonable parameter ranges. The sensitivity analysis shows that SemDiD robust parameter variations, with performance degrading gracefully rather than sharply when moving away from optimal values.
D.6 Transition Weight Analysis
The parameter Tirans controls the transition point from directional guidance to inter- groue- repulsion. We conducted experiments varying Ttrans across multiple datasets:
The results show clear optimal performance at Terans 10. When transition occurs too early (T = 5), groups haven't established sufficient semantic differentiation before repulsion dom- inates, leading to suboptimal exploration. When transition delayed (Ttrans 20), groups may converge to similar semantic regions before inter group repulsion becomes effective.
28

---

Table 7: Transition weight parameter Tirans analysis across datasets, where Cov. denotes coverage
and Acc. denotes accuracy.
T Trrans
GSM8K Cov.
GSM8K Acc.
ARC Cov.
ARC Acc.
MMLU-Pro+ Cov.
5
97.2%
76.8%
81.2%
79.1%
81,4%
(default)
98. 1%
77.5%
82.4%
82.0%
82.6%
15
97.9%
77.2%
82.1%
81.7%
82.3%
20
97.5%
76.9%
81.7%
81.2%
81.8%
25
97.2% 76.8%
81.8%
80.9%
81.6%
Intuitive Explanation: Tirans = 10 corresponds to the typical number of tokens needed to establish meaningful semantic context. Most mathematical reasoning problems require 8-12 tokens to establish the early thought process for problem-solving before semantic trajectories become distinguishable.
D.7 Guidance for Hyperparameter Settings
Due to SemDiD's involvement of numerous hyperparametters provide the following guidelines for setting them based on the comprehensive analysis presented above.
Firstly, temperature, Top-p. N (number of groups), and b (beam size) are inherited from standard Group Beam Search parameters, not SemDID additions. The SemDID-speccfic parameters serve dis- tinct purposes across three categories: semantic diversity assessment (E,, Lmar), quality assessment (Bsog Ssent, 7), and quality-diversity balancing (Itrans. 7, A).
Automatically Derivable Parameters: Several key parameters can be systematically determined rather than manually tuned. The position bias parameters Beeq and Beent be automatically fitted using a scipy. curve__li from patterns remain consistent across tasks. The saturation threshold is derived from probability-quall analysis as Figure 2 and set to -0.8 for most tasks based on the empirical study. As demonstrated in Section D.6, the transition point T Transs 10 corresponds the typical number of tokens needed to establish meaningful semantic context before inter-group repulsion becomes effective.
position curves shown in Figures 3 and 4, as these
Resource-Dependen Parameters: The exploration parameters E., b. and N balance exploration breadth versus computational cost. Our analysis in Sections D.2 and D.3 shows diminishing returns beyond E, and 3, establishing clear "sweet spots" without extensive tuning requirements. The lookahead depth analysis in Section D.4 demonstrates that Lmax = 10 provides optimal performance-cost balance.
Quality-Diversit Balance: The quality-diversiy balancing analysis Section D.5 reveals that our default settings (y 0.25, À = 2.0) achieve robust performance across tasks, with graceful degradation when parameters deviate from optimal values. The transition weight analysis in Section D.6 confirms that Trrons 10 provides optimal timing for the shift from directional guidance to inter-group repulsion.
For practitioners seeking to deploy SemDiD, we recommend starting with our provided default parameters for initial implementation, adjusting E,, b, and N based on available computational budget, and fine-tuning A. and Tirans only for highly specialized applications where task-specific optimization critical.
E
Accuracy Evaluation in Best-of-N
To provide comprehensive evaluation of our SemDiD approach, we conducted experiments measur- ing the accuracy of the selected answers under Best-of-N settings. Unlike coverage, which measures whether at least one correct answer exists among the N candidates, accuracy evaluates whether we can effectively identify the best answer from the generated candidates. For this purpose, employed L.MM-leende PairRM as our evaluation model.
For each query, we generated N responses using 2.5n-B.5-3B. We then used the LLM-Blender PairRM model to evaluate and select the most likely correct answer from among the candidates. This
29

---

ARC-Challagge
Rig Hand (BRH)
GSMBK
52
Accuncy (8)
48
78
3
10
25
50
75
100
3.
10
22
50
75
100
10
25
50
75
100
Sumples
Sumples
Samples
Minerva Marh
CotjA
PubMeddQA
Accuraty (8)
62
20
3
10
25
50
75
100
3
3.10
25
50
75
100
10
25
50
75
100
Sumples (N)
Samples (N)
Samples (N)
MMLU Pro+
Faglish-Germson
German Faglish
222
355
16
3 3.10
25
50
75
100
3 1.0
25
75
100
3 10
25
50
75
100
Samples (N)
Samples (N)
Samples (N)
SemDiD (Curs)
Tempr0.5
Temp-1.0
Tamp-1.5
Arith Sampling
Diverse Beam
Determnronal Rram
Grondy
Figure 11: Accuracy comparison for Best-of-N using Qwen-2.5-3B with evaluation model.
PairRM as
process simulates real-world scenario where a verification mechanism must determine the optimal answer from of alternatives.
Figure 11 presents accuracy results across all benchmarks. The results demonstrate that SemDiD consistently outperforms baseline methods across all tasks, though by modest margins. This indi- cates that while various decoding strategies can increase the likelihood generating least one correct answer (coverage), the ability to identify the best answer (accuracy) remains challenging. Interestingly, we observe that performance differences between methods become more pronounced the number of samples increases, suggesting that semantic diversity becomes increasingly valuable with larger candidate pools. Mathematical reasoning tasks (GSM8K and Minerva Math) show the greatest improvements with SemDiD, likely because these tasks benefit most from exploring diverse solution paths. Translation tasks exhibit the smallest performance gaps, potentially due to their more constrained solution spaces compared to open-ended reasoning tasks.
Cross-Model Generalization Analysis
In addition our primary experiments with Qwen-2.5-3B for Best-of-N and Qwen-2.5-7B for RLHF, extended our analysis to different model architectures and sizes to assess how SemDiD's
30

---

ARC-Clallagge
Rig Ronch Hand
GSMBK
100
Covenge (8)
3
1.
225
50
75
100
3
10
25
50
75
100
3
10
23
50
75
200
Samples (N)
Samples
Sumples (N)
Mineroi Mach
CodJA
PabMeddA
90
600
Corcaage (%)
70
10
60
3
3.10
25
50
75
100
10
25
50
75
100
10
25
50
75
100
Sumples (N)
Samples (N)
Samples (N)
MMLU Pro+
Faglish-Grmmoon
WMT16 German-Faglish
100
Covenge (S)
40
30
20
3
25
50
75
100
10
25
50
75
100
10
25
75
100
Samples (N)
Samples (N)
Samples (N)
((uo))
Temgr0.5 0.5
Temp-1.0
Temp-1 1 Grrody
Arihh Sampling
Divens Ream
Determrrnnal Rrom
Figure 12: Coverage comparison for Best-of-N using Llama-3 1.mma-33.11-0 3.B.
effectiveness generalizes across different language models. Specifically, we evaluated SemDiD using Llama-3.1-8B for Best-of-N tasks and Pythia-1B for RLHF training.
F1 Llama-3.1-8B in Best-of-N
For the Best-of-N experiments with Llama-3. 1-8B, followed the same experimental setup with Qwen-2.5-3B, evaluating performance across all nine henchmarks. As shown in Figure 12, SemDiD consistently outperforms baseline methods across all tasks with Llama-3.1-8B demonstrating even more pronounced improvements compared to Qwen-2.5-3B. The performance gains are particularly striking on reasoning tasks, with SemDiD achieving 99.1% coverage on GSMSK and 97.4% coverage on BBH at just 25 samples improvements of 0.5-1.8% over other methods. For language under- standing tasks, SemDiD shows exceptional performance on MMLU-Pro+ with 89.4% coverage 25 samples versus 86.9% for Temperature sampling (T=0.5). On translation tasks, while Diverse Beam Search remains competitive, SemDiD maintains a clear advantage with 43.1 and 51.3% coverage on WMT16 English-German and German-Fnglish respectively at 25 samples. These results suggest that SimDiD's semantic diversity mechanisms are particularly effective when combined with larger, more capable models, further validating the architecture-agnosti nature of our approach.
31

---

Deemiive-RL.. (Pythia-
(Pythin- TLDRR
60
60
60
Wia Rate
45
45
45
10
20
40
10
20
40
10
20
40
60
Rollont Number
Rallout Number
Rollout Number
Dierative RLHE (Pythia-1B) GSMHK
GRPU (Pythio GSMMKK
RLOO (Pythin GSM8K
Accunay (8)
66
6.5
65
55
55
55
Rallout Number
Rollout Number
Rollout Number
SimDID (Ours
Temp=0.5
Temp=1
Tempe1.5
Anith, Sumpling
Diverse Beam
Desermimental Beam
Figure 13: Performance comparison across RLHF algorithms with varying rollout numbers using PBt
F22 Pythia-1B in RLHF
To investigate SemDiD's effectiveness with smaller models in RLHF settings, conducted experi- ments using Pythia- IB on the same GSM8K and TLDR datasets. As illustrated Figure 13, despite the significant reduction in model size, SemDiD continues outperform baseline methods across all RLHF algorithms, though with lower absolute performance values compared to Qwen-2.5-7B.
For TLDR summarization with Pythia-IB, SemDiD achieves win rates of 61.55 61.5%, 62. %6 and 60.0% with Iterative-RAL RLHFF GRPO, and RLOO respectively at 60 rollouts, which are approximately 10.6%, GSM8K, SemDiD
11.3%, and 11.2% lower than those achieved with Qwen-2.5-7B. Similarly, with Pythia- 1B reaches accuracy scores of 71.7% with ItrHa, 72.9% with GRPO, and 68.6% with RLOO, representing gaps of 13.8% %, 15.3%, and 13.8% compared to Qwen-2.5-7B.
Interestingly, the performance gap between SemDiD and baseline methods is proportionally larger with Pythia- IB than with Qwen-2.5-7B. This suggests that semantic diversity becomes even more critical for smaller models with limited representational capacity, it enables more efficient explo- ration of the solution space during RLHF training. The results demonstrate that SemDiD's ' approach diverse decoding is particularly valuable eesuurce-constraine scenarios where maximizing the utility of smaller models is essential.
G RLHF Reward Growth Analysis
Figure 14 illustrates the reward growth curves during RLHF training across different decoding methods, model sizes, and RLHF algorithms. Our experiments evaluate performance on both the TLDR summarization dataset and the GSM8K mathematical reasoning dataset using Qwen-2.5-7B and the smaller Pythia-T IB models. The results consistently demonstrate that SemDiD accelerates reward convergence compared to other decoding strategies. For Q.55-2..-7B on TLDR, SemDiD (blue line) achieves faster initial reward growth within the first 750 training steps across RLHF algorithms, reaching plateau performance approximately 15-20% earlier than Temperature sampling (red line). This acceleration more pronounced with Pythia- 1B, where SemDiD maintains consistent reward advantage throughout training, particularly evident in the RLOO implementation.
32

---

Denniivs-R HF (Qwen-2.5-7B)- TL.DR
(Q=a1-2.5-78)
KLHE Rowand
-1
-1.5
1.5
-1.5
-2.5
2.5
22.5
500
1,000
1,500
2,000
2,500
500
1,000
1,500
2,000
2,500
0
500
1,000
1,500
2,000
2,500
Iterative RLTIT (Pythia-IB)
(Pythia
(Pythia (0)hia-10)-aa
RL.H Reward
-1
-1.5
1.5
1.5
-2
2.5
2.5
2.55
500
1,000
1,500
2,000
2,500
500
1,000
1,500
2,000
2,500
500
1,000
1,500
2,000
2,500
(Qwen-2.5.7B)
(Qwmn 5-78)
(Qum-225.78)
2.4
2.4
2.4
RLHF Reward
1.8
1.6
1.6
1.2
0.8
0.8
0.8
500
1,000
1,500
2,000
2,500
500
1,000
1,500
2,000
2,500
500
1,000
1,500
2,000
2,500
Beraaiie-RLHH (Pythis-8))-
(Pythia-1R) 18)-
RLOD (Pythia-1R) (R)
RLAF Rowand
1.6
1.6
1.6
1.2
1.2
1.2
0.8
0.8
0.8
0.4
0.4
500
1,000
1,500
2,000
2,500
500
1,000
1,500
2,000
2,500
500
1,000
1,500
2,000
2,500
Training Steps
Training Steps
Training Steps
SemDDD
Temp-1.0
Deserminnntal Beam
Arits. Sampling
Figure 14: Reward growth comparison of different decoding methods on RLHF algorithms.
On the more complex GSM8K dataset, SemDiD's ' advantage becomes even more significant, with reward curves showing both steeper initial growth and higher final convergence values. This improved learning efficiency can be attributed to SemDiD's 's ability to generate semantically diverse response groups that provide more informative training signals, allowing the model explore broader solution space and identify optimal policies more efficiently during RLHF training.
33

---

H
Ablation Studies
To understand the contribution of cach component in SemDiD, we conducted systematic ablation experiments across both Best-of-N coverage (averaged across all tasks) and RLHF settings (using GRPO GSM8K). Table 8 presents the results when removing individual components from the full SemDiD framework. For Best-of-N evaluations, we used Qwen-2.5-3B with 25 samples per query, while RLHF experiments employed Qwen-2.5-7B with 25 rollouts during training.
Table 8: Ablation study of SemDiD components.
Method
Avg. Coverage with 25 samples (%) GRPO-GSM8K Accuracy (%)
Full SemDiD
74.2
81.6 79.9 (-1.7)
Directional Guidance
72.1 (-2.1)
Inter-Group Repulsion
71.5 (-2.7)
79.6 (-2.0)
Debiased Probability
73.3 (-0.9)
80.5 (-1.1)
Harmonic Gain
71.2 (-3.0)
79.3 (-2.3)
Only Probability (Greedy)
57.2 (-17.0)
71.8 (-9.8)
Our ablation results demonstrate the essential contribution of each SemDiD component, with har- monic gain showing the largest impact by reducing coverage 3.0% when removed, highlighting its effectiveness in balancing quality-diversity trade-offs. Inter-group repulsion exerts significant influence, decreasing coverage 2.7% and RL.FF accuracy 2.0% when ablated, confirming the im- portance of maintaining semantic distances between candidates. Directional guidance particularly benefits RLHF scenarios with an accuracy reduction of 1.7% when removed, demonstrating its value in enabling diverse exploration for policy improvement. Debiased probability, though having the smallest individual impact with coverage decreasing 0.9% upon removal, ensures fair quality assess- ment across varied sequence structures. Most notably, greedy decoding alone produces dramatically worse results with coverage dropping 17.0% and accuracy declining 9.8%, highlighting probability maximization's 's inability to explore diverse semantic regions. These findings validate SemDiD's integrated approach to semantic diversity, substantially outperforming simple token-level diversity methods.
I
Comparison to Contrastive Decoding and Scalability Analysis
We conducted additional experiments comparing SemDiD against contrastive decoding methods and embedding-bass based post-hoc clustering across multiple model sizes.
I.1 Contrastive Decoding and Clustering Baselines
We evaluated SemDiD against DoLa (Decoding by Contrasting Layers) [39] and embedding based post-hoc clustering using Qwen-2.5-3B across all benchmarks. The clustering approach generates 100 independent samples, clusters them semantically, and selects representatives from different clusters.
Table 9: Comprchensive bascline comparison with computational overhead analysis using Qwen-2.5- 3B
Method (N=25)
ARC
BBH
GSM8K Minerva CoQA
Pub MMLU+
WMT16 Compute Latency
SemDiD (Ours)
82.4%
85.6%
98.1%
86.1%
46.7%
82.6%
82.6%
40.95%
+200%
+27%
DoLa Contrastive
80.3%
81.7%
95.2%
82.8%
43.2%
78.4%
77.9%
37.855
-66%
-66%
Clustering (100 >25)
80.8% 80.8%
83.9%
96.4%
84.3%
45.7%
80.0%
80.7%
39.35%
+33%
+32%
Temperature (T=1.0)
78.9%
82.1%
94.8%
81.7%
44.1%
77.8%
76.3%
36.2%
-66%
-66%
Diverse Beam Search
79.8%
83.2%
95.5%
83.1%
45.3%
79.2%
78.9%
38.1%
+0%
+0%
DoLa consistently underperforms SemDiD because it lacks explicit diversity mechanisms and often converges toward similar high-confidence solutions. This limitation is particularly evident tasks requiring diverse reasoning strategies, where DoLa's 's conservative boken-by-toker contrasting favors safe, conventional solutions rather than exploring semantic diversity.
The clustering approach shows competitive performance but becomes increasingly inefficient as N increases. While it sometimes matches SemDiD's performance at smaller sample sizes, it requires
34

---

generating 100 independent sequences without coordination, making KV-cache reuse impossible and substantially increasing computational overhead.
I.2 Large Model Scalability Analysis
We evaluated SemDiD's effectiveness and computational efficiency on Qwen-2.5-70B to assess scalability to larger models:
Table 10: Performance and computational analysis on Qwen-2.5-70B
Method (N=25)
ARC
BBH
GSM8K Minerva
CoQA
Pub MMLU+
WMT16
Compute
Latency
SemDiD (Ours)
84.7%
89.2%
99.8%
93.4%
52.1%
89.3%
91.2%
47.05%
+180%
+ +18%
DoLa a Contrastive
82.1%
86.4%
98.9%
90.7%
48.6%
86.2%
87.9%
43.6%
-66%
-66%
Clustering (100-225)
83.9%
88.1%
99.0%
91.8%
51.4%
88.6%
89.8%
46.05%
+33%
+32%
Temperature (T=1.0)
82.8%
86.9%
99.1%
90.2%
49.3%
86.8%
88.1%
44.45%
-66%
-66%
Diverse Beam Search
83.1%
87.2%
98.6%
91.1%
50.7%
87.5%
88.9%
45.4%
+0%
+0%
The 70B model results reveal several important patterns:
Improved Performance Scaling: SemDilD shows consistent improvements across all tasks, with particularly strong gains on complex reasoning tasks like BBH (+2.0%), Minerva Math (+2.3%), and MMLU-Pro+ (+2.3%) compared to the best baselines.
Reduced Computational Overhead: SemDiD's latency increase drops to only + +18% with 70B models, significantly lower than the +27% observed with 3B models. This occurs because embedding computation becomes negligible compared to large model forward passes, while K KV-cache reuse within groups remains highly effective.
Cache Efficiency: SemDiD's coordinated beam exploration allows extensive KV cache reuse within groups since beams share common prefixes. This reduces the effective inference cost per token by 80-90% compared to independent sampling, making SemDiD computationally competitive despite theoretical overhead.
These results demonstrate that SemDiD's advantages become more pronounced with larger, more capable models, while computational overhead decreases relative to model size, making it particularly suitable for deployment with state-of-the-art aht large language models.
I.3 Equivalent Computational Budget Analysis
We conducted experiments using equivalent latency budgets rather than equivalent sample counts. For parallel decoding methods, we generated larger candidate pools and selected N outputs using embedding- clustering. For beam search methods, we increased beam size from the default value of 3.
Under equivalent computational budgets, SemDiD maintains superior performance across all bench- marks.
Diminishing Returns for Independent Sampling: Generating larger candidate pools for parallel methods (88 samples select 25) leads to rapidly increasing costs with minimal diversity gains due to inability reuse KV-cache across independent sequences.
Limited Beam Size Benefits: Simply increasing beam size in diverse beam search methods yields negligible improvements while substantially increasing computational overhead, as evidenced by the modest gains from beam=3 beam=4.
Efficient KV-Cache Utilization: SemDiD's coordinated beam exploration within groups allows extensive KV-cache reuse since beams share common prefixes, resulting in 80-90% reduction in effective inference cost per token despite theoretical overhead.
Systematic Semantic Exploration: SemDiD benefits from directional guidance and inter-group re- pulsion that systematically explore semantic space, efficiently utilizing every computational allocation to maximize coverage rather than generating redundant variations.
35

---

Table 11: Performance comparison under equivalent computational budgets using Qwen-2.5-70B
Method
ARC
BBH
GSM8K Minerva CoQA PubMed MMLU+ WMT16 Compute Latency
N-10
SemDiD (Ours)
83.4%
16
96.8%
82.3%
47.2%
82.1%
76.8%
40.25%
+180%
+18%
(3 Arith. Sampling (35-10) 10)
82.4%
73.9%
38.5%)
82.6%
84.7%
95.6%
81.5%
46.2%
81.1%
75.0%
39.1%
+18%
+18%
Diverse Beam (beam=4)
82.4%
84.9%
95.3%
80.5%
46.7%
81.4%
74.3%
39.0%
+33%
+33%
Determinantal Beam (bcam=s)
82.7%
84.1%
94.9%
80.4%
45.8%
81.1%
74.7%
38.8%
+33%
N=25
SemDiD (Ours)
84.7%
89.2%
99.8%
93.4%
52.1%
89.3%
91.2%
47.05%
+180%
+18%
Temp=1.0 (88-225)
83.6%
87.5%
19.1%%
91.2%
49.9%
87.4%
88.8%
45.15%
+18%
+18%
Arith. Sampling (88-255)
83.9%
87.8%
91.9%
51.2%
89.4%
45.85%
+18%
Diverse Beam (beam=4)
83.2%
87.2%
98.9%
91.0%
50.8%
87.5%
88,9%
45.3%
+33%
+33%
Determinintal Beam (beum=4)
83.0%
86.8%
98,
90.5%
50.2%
87.3%
88. 4%%
45.15% 15%
+33%
+33%
N=50
SemDiD (Ours)
85.3%
1
95.2%
53.8%
91.7%
48.95%
Temp=1.0 (177-+50)
84.6%
88.4%
99.2%
94.3%
51.7%
89.9%
92.8%
47.7%
+18%
+18%
Arith. Sampling (177-150)
84.4%
89.3 3%
99. 6%
94.1%
51.9%
91.0%
92.3%
47 65%
+18%
+18%
Diverse Beam (beam=4)
84.3%
88.2%
99.2%
93.2%
51.7%
89.1%
91.4%
47.45%
+33%
+33%
Determinantal Beam (beum=4) 83.8%
88.6%
99.0%
92.8%
51.5%
88.64
91.0%
46. 85%
4.3%%
+33%
J Comparative Analysis of SemDiD Response Patterns
We provide detailed analysis of the response patterns generated by SemDiD compared to baseline methods. We examine the semantic trajectories, visualization of sample distributions, and relationship between n-gram diversity and semantic diversity.
J.1 Semantic Trajectory Perspective
Reflecting on mean-pont pooled embedding models during autoregressive generation, each candidate answer be conceptualized trajectory in semantic space. The trajectory begins at the question representation and evolves with each generated token, creating path through the embedding space. From this perspective, diverse decoding can be reformulated generating multiple high-probabillty trajectories that maintain significant distances from one another.
The directional guidance in SemDiD establishes initial trajectory divergence, while inter- stte-group repulsion maintains semantic distance as generation progresses. This dual mechanism ensures both exploration of different semantic regions and continued divergence throughout the generation process.
J.2 Comparison of N-gram and Semantic Diversity
To quantify responses on lexical diversity and semantic diversity, we conducted a systematic compari- son across different decoding methods. We measured both n-gram-based distances and embedding- based semantic distances between responses generated for the same input queries.
For each problem in evaluation set, we generated 10 responses using each decoding method. We then computed pairwise distances between these responses using two primary metrics:
We calculated the Jaccard similarity of character- n-grams (n=4) between each pair responses, then converted distance (1 similarity).
N-gram Overlap:
Semantic Distance: We measured the cosine distance between sentence embeddings of each response pair.
Table 12 presents the average pairwise distances across all problems in our evaluation set. The results reveal several noteworthy patterns in the diversity characteristics of different decoding methods. SemDiD achieves the highest semantic distance 0.371 while maintaining moderate n-gram diversity of 0.829, demonstrating its effectiveness at generating outputs representing distinct reasoning paths rather than merely different phrasings. Arithmetic Sampling shows the highest n-gram diversity at 0.882 but falls significantly behind in semantic distance at 0.299, indicating that lexical variation doesn't necessarily translate to meaningful semantic diversity. Temperature sampling demonstrates clear correlation between temperature values and both diversity metrics, with T=0.5 showing
36

---

Table 12: Average pairwise distances between responses using different diversity metrics
Method
4-gram Distance Semantic Distance
Temperature (T=0.5)
0.627
0.183
Temperature (T=1.0)
0.764
0.241
Temperature (T=1.5)
0.832
0,295
Determinantal Beam Search
0.857
0.336
Arithmetic Sampling
0.882
0,299
SemDiD (Ours)
0.829
0.371
the lowest diversity scores at 0.627 and 0.183 for n-gram and semantic distances respectively. Determinantal Beam Search maintains strong performance across both metries with scores of 0.857 and 0.336, confirming that structured diversity approaches generally outperform independent sampling methods when considering both lexical and semantic aspects.
These results underscore the importance of explicitly modeling semantic diversity during the decoding process, rather than relying n-gram diversity proxy. They also explain why SemDiD demon- strates superior performance in Best-of-N scenarios, where having truly diverse solution candidates improves the likelihood of including least one correct answer.
J.3 Visualization of Sampling Distributions and Case Analysis
To better understand the distributional characteristics of different decoding methods, we sampled responses for three representative problems from GSM8K with varying difficulty levels and visualized their distribution in semantic space using t-SNE dimensionality reduction. Figure 15 presents these visualizations.
Lortect Ataee
Comnt
Figure 15: 1-SNE visualization of semantic embedding spaces for GSM8K problems using three different decoding methods: temperature sampling (left column, T==10),, determinantal beam search (middle column), and SemDiD (right column). Each row represents a different problem. Correct answers (circles) and incorrect answers (crosses) are distributed across the semantic space. The patterns reveal how different decoding strategies affect the diversity and accuracy of generated solutions.
The visualizations reveal distinct patterns in how different decoding strategies explore the solution space:
Temperature Sampling (T=1.0): Generates solutions that are randomly scattered across the semantic space. While this approach produces some diversity, lacks structured exploration, often leading to redundant solutions missing promising areas entirely.
37

---

Determinantal Beam Search: Shows more structured patterns but tends to cluster solutions in limited regions, creating densely populated "islands" of similar reasoning approaches with large unexplored gaps between them.
SemDiD: Demonstrates comprehensive coverage of the semantic space with solutions more uniformly distributed. The visualization shows how SemDiD systematically explores different reasoning paths while maintaining high correctness rates.
The case studies below analyze three problems of increasing difficulty, examining how different decoding methods approach each problem and illustrating the effectiveness of semantically-get guided diverse decoding.
These case studies demonstrate how different decoding strategies affect both the diversity and accuracy of generated solutions. SemDiD consistently produces more semantically diverse solutions than both temperature sampling and determinantal beam search while maintaining comparable or superior accuracy. The character-level diversity focus of traditional methods often leads syntactically diverse but semantically redundant or flawed solutions, particularly evident in the more complex problems (rows 2 and 3). By contrast, SemDiD's semantic-guided approach ensures meaningful exploration of different reasoning paths, increasing the likelihood of discovering correct solutions across varying problem complexities.
Case 1: Simple Arithmetic (Row 1)
Janet's ducks lay 16 eggs day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day the farmers' market?
Problem:
Correct Answer: Janet sells 16 at the farmer's market.
duck eggs a day. She makes 9 x 2 = $18 every day
Analysis: This simple arithmetic problem has straightforward solution path. All decoding methods achieved perfect or near-pent pear-perfect accuracy (100%, 97%, and 100% for temperature sampling, determinantal beam search, and SemDiD respectively). However, the visualization reveals that temperature sampling and determinantal beam search produce solutions clustered in similar semantic regions, while SemDiD's solutions the embedding space. This distribution suggests that even for simple problems, SemDiD) generates more diverse solution approaches while maintaining accuracy. Temperature Sampling (T=1) Results: Sample 1: (Correct)
evenly distributed across
First, we need to calculate how many eggs Janet has left to sell after eating and using some for baking.
She starts with 16 eggs per day. She eggs for breakfast, that leaves her with: 16 - 3 = 13 eggs
Then she uses 4 eggs to bake muffins: 13 - 4 = 9 eggs
Janet sells the remaining eggs at the farmers' market. Each egg sells for $2 the amount she makes per day is: x dollars Therefore, Janet makes dollars every day farmers' market.
Sample 2: (Correct)
First, need to determine how many eggs are sold at the farmers' market every day.
1. Janet starts with 16 eggs laid per day. 2. She eats 3 eggs for breakfast. 3. She uses 4 eggs bake muffins.
To find out how many eggs are left for sale: 16 eggs 3 (caten) (baked)
= eggs
Janet sells the remaining 9 eggs the farmers' market, and she earns $2 per egg-
Therefore, her daily earnings from selling the eggs are: 9 eggs 2 dol- lars/egg = 18 dollars
So. Janet makes 18 dollars every day at the farmers' market.
38

---

Sample 3: (Correct) First, let's determine how many eggs Janet has left to sell after eating and baking.
Janet starts with 16 eggs per day. She eats eggs for breakfast and bakes 4 eggs into muffins. So, the number of eggs left for sale is: 16 - 3
Janet sells each fresh duck egg for $2 the farmers' market. Therefore, the amount she makes each day is: 9 eggs $2 per egg $18 Hence, Janet makes $18 every day at the farmers' market.
Sample 4: (Correct)
First, let's determine how many eggs are left after Janet's consumption and baking:
Total eggs laid per day: 16 eggs Eggs eaten for breakfast: eggs Eggs baked into muffins: 4 eggs
Eggs left after these activities: 16 - 4 eggs
Janet sells the remaining eggs the farmers' market for $2 per egg. There- fore, her daily earnings from selling the eggs are: 9 eggs x $2 per egg = $18
So, Janet makes $18 every day at the farmers' market.
Sample 5: (Correct)
First, let's determine the total number of eggs Janet's ducks lay per day:
Total eggs laid per day 16
Next, we need find how many eggs Janet uses:
She eats eggs for breakfast. She uses eggs to bake muffins.
So, the total number of eggs used per day is: Eggs used per day 3 + 4 4
Now, subtract the number of eggs used from the total number of eggs laid to find out how many eggs she sells:
Eggs sold day = 16 - 7
Finally, since she sells each egg for $2 at the farmers' market, calculate her daily earnings by multiplying the number of eggs sold by the price per egg:
Daily earnings 9 18
So, Janet makes $18 every day the farmers' market.
Determinantal Beam Search Results:
Sample 1: (Correct)
First, we need to determine how many eggs Janet has left after eating and using some for baking.
Janet starts with 16 eggs per day. She eats 3 eggs for breakfast and 4 eggs to make muffins. So, the number of eggs she has left each day is:
16 3 4 9
Janet sells the remaining eggs at the farmers' market for $2 per egg. There- fore, the amount she makes each day is:
9 eggs x dollars/egg = 18 dollars
So, Janet makes 18 dollars every day at the farmers' market.
Sample 2: (Correct)
First, we need to determine how many eggs Janet has left after eating and using some for baking.
Janet starts with 16 eggs per day. She 3 eggs for breakfast and 4 eggs to make muffins. So, the number of eggs she has left each day is:
39

---

16 3 9
Janet sells the remaining eggs at the farmers' market for $2 per egg. There- fore, the amount she makes each day is: 9 eggs x dollars/egg = 18 dollars Thus, Janet makes 18 dollars every day the farmers' market.
Sample 3: (Correct)
First, we need to determine how many eggs Janet has left after eating and using some for baking.
Janet starts with 16 eggs per day. She eats 3 eggs for breakfast and uses 4 eggs to make muffins. So, the number of eggs she has left each day is: 16 3
9
Janet sells the remaining eggs the farmers' market for $2 per egg- There- fore, the amount she makes each day is:
9 eggs x dollars/egg = 18 dollars
So. Janet makes $18 every day the farmers' market.
Sample 4: (Correct)
Janet starts with eggs per day. She eats 3 eggs for breakfast and uses 4 eggs to make muffins. So, she uses total of 3 + 4= 7 eggs.
This leaves her with 16 7 eggs that she sell the farmers' market.
Since she sells each fresh duck egg for $2, she makes 9 $2 18 dollars every day the farmers' market.
Therefore, Janet makes $18 every day at the farmers' market.
Sample 5: (Correct)
Janet starts with 16 eggs per day. She eats eggs for breakfast and uses 4 eggs make muffins. So, the number of eggs she sells the farmers' market is:
16 (total eggs) (eaten for breakfast) (used for muffins) 16 3 9 eggs.
Since she sells each fresh egg for $2, her daily earnings at the farmers' market are:
eggs * $2 per egg = $18.
So. Janet makes $18 every day at the farmers' market.
SemDiD Results:
Sample 1: (Correct)
First, let's determine how many eggs Janet keeps out of those laid: Janet starts with 16 eggs per day.
She eats 3 eggs for herself: 16 eggs 3 eggs 13 eggs Then, she makes muffins with additional 4 eggs: 13 eggs 4 eggs eggs
These 9 eggs remain as leftovers which she can sell at the farmer's 's market.
Now we know she sells 9 eggs per day at S2 per egg: 9 eggs $2/egg $18
Therefore, Janet makes $18 every day at the farmers' market.
Sample 2: (Correct)
To determine how much money Janet makes the farmers' market every day, we need to follow these steps:
1. Calculate the total number of eggs laid per day. 2. Subtract the number eggs eaten by Janet for breakfast. 3. Subtract the number of eggs used to bake muffins. 4. Multiply the resulting number of eggs sold by the price per egg.
40

---

Let's break it down step-by-step:
1. Total number of eggs laid per day: 16 eggs
2. Eggs eaten by Janet for breakfast: 3 eggs
3. Eggs baked into muffins: eggs
Now, calculate the remaining eggs available for sale: 16 (total) 3 (eaten)
4 (baked) eggs
Janet sells these remaining eggs at per egg: eggs $2/egg $18 Therefore, Janet makes $18 dollars every day at the farmers' market.
Sample 3: (Correct)
First, let's calculate the total number of eggs laid by the ducks per day:
Total Eggs Laid 16
Janet consumes eggs for breakfast:
Eggs Used For Breakfast =
She bakes muffims with an additional 4 eggs:
Eggs Used For Muffins 4
Next, subtract the total number of eggs used (for breakfast and baking)
from the total number of eggs laid to find out how many eggs remain:
Remaining Eggs Eggs Used For Muffins) Remaining Eggs Remaining Eggs 16 7 Remaining Eggs = 9
Total Eggs Laid (Eggs Used For Breakfast +
16
(3
+
Now, Janet sells the remaining eggs at S2 per egg:
Daily Revenue
Number of Remaining Eggs
Price Per Egg
Daily Revenue = 9 x 2 Daily Revenue = 18
Therefore, Janet makes $18 every day at the farmers' market.
Sample 4: (Correct)
First, we need to determine the total number of eggs laid per day: Janet's ducks lay 16 eggs per day.
Next, subtract the number of eggs eaten for breakfast: 16 3 = 13
Then, subtract the number of eggs used to bake muffin:: 13 4 Therefore, Janet has fresh eggs available for sale each day.
She sells each fresh egg for $2: 9
18
So. Janet makes $18 every day at the farmers' market.
Sample 5: (Correct)
To determine how much Janet makes every day selling the remaining eggs:
First, calculate the total number of eggs laid per day: Total eggs per day 16
Next, subtract the eggs caten for breakfast: Eggs used for breakfast 3 Remaining eggs after breakfast = 16 13
Then, subtract the eggs baked into muffins: Eggs baked into muffins = 4 Remaining eggs sold 13
Finally, calculate the revenue generated from selling the remaining eggs: Price per egg $2 Revenue Therefore, Janet makes $18 every day at the farmers' market.
x 2 $18
One key observation from these samples is that the determinantal beam search solutions show remarkably similar wording across samples, particularly in the first three examples, indicating limited lexical diversity despite the method's design. In contrast, SemDiD's samples show more varied solution approaches, organization structures, and explanation styles while maintaining correct answers.
41

---

Case 2: Intermediate Problem (Row 2)
Problem: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?
Correct Answer: It takes 2/2=1 bolt of white fiber. So the total amount of fabric is 2+1=3 bolts of fabric.
Analysis: This moderately complex problem involves fraction calculation followed by addition. The visualization shows that SemDiD (98% correct) significantly outperforms determinantal beam search (57% correct) and even improves upon temperature sampling (97% % conrect). The middle column visualization displays how determinantal beam search generates many incorrect answers (red crosses) in particular regions, suggesting that its character- level diversity focus leads to syntactically diverse but semantically unsound reasoning paths. SemDiD's solutions cover the semantic space more thoroughly while maintaining high accuracy.
Temperature Sampling (T=1) Results:
Sample I; (Correct)
First, we need to determine the amount of white fiber required for one robe.
Since the robe takes half much white fiber as blue fiber:
White fiber needed Blue fiber needed / 2 White fiber needed 2 bolts White fiber needed = bolt
Now, know that one robe requires bolts of blue fiber and bolt of white fiber.
Therefore, the total number of bolts required for one robe is: Total bolts = Blue fiber White fiber Total bolts bolts bolt Total bolts bolts So. it takes a total of 3 bolts to make one robe.
Sample 2: (Correct)
A robe requires 2 bolts of blue fiber. Since it takes half much white fiber as blue fiber, it needs 2 2 1 bolt of white fiber.
To find the total number of bolts needed for the robe, we add the bolts of blue fiber and white fiber together:
2 (blue fiber) 1 (white fiber) = bolts Therefore, it takes a total bolts to make the robe.
Sample 3: (Correct)
A robe takes bolts of blue fiber. Since it takes half as much white fiber blue fiber, it uses:
2 2 1 bolt of white fiber.
To find the total number of bolts needed, we add the blue and white fibers
together:
2 (blue) + (white) 33 bolts in total.
So. it takes total of 3 bolts.
Sample 4: (Correct)
A robe takes 2 holts of blue fiber. It takes half much white fiber as blue fiber, so for the white fiber, it would take 2 1 bolt.
In total, it takes bolts (blue) + 1 bolt (white) 3 bolts. Therefore, it takes 3 bolts in total.
Sample 5: (Correct)
First, determine the amount white fiber needed for one robe. The problem states that the robe takes half much white fiber as blue fiber.
Since 2 bolts of blue fiber are used: The amount of white fiber needed is
2 = bolt.
42

---

Now, add the blue and white fiber amounts together to find the total number bolts required: Total bolts Blue fiber bolts + White fiber bolts = 2 + = bolts.
Therefore, it takes 3 bolts in total to make one robe.
Determinantal Beam Search Results:
Sample 1; (Correct)
A robe takes bolts of blue fiber. Since it takes half as much white fiber blue fiber, it takes 2/ bolt of white fiber. Therefore, the total number of bolts needed is 2 (blue) + 1 (white) = 3 bolts.
So, it takes a total of 3 bolts.
Sample 2: (Correct)
A robe takes bolts of blue fiber. Since it takes half as much white fiber blue fiber, takes 2 2/2 bolt white fiber. Therefore, the total number of bolts needed (blue) + (white) = bolts.
So, it takes 3 bolts in total.
Sample 3: (Correct)
A robe takes bolts of blue fiber. Since it takes half as much white fiber as blue fiber, takes 2 bolt of white fiber. Therefore, the total number of bolts needed (blue) + (white) bolts.
So, it takes total of 3 bolts to make the robe.
Sample 4: (Incorrect)
To find out the total number of bolts fiber needed for the robe, let's break it down:
The robe takes 2 bolts of blue fiber. It takes half much white fiber blue fiber, so takes / = bolt of white fiber. Therefore, the total number of bolts of fiber needed 2 (blue) + [stopped by model] [truncated]
Sample 5: (Incorrect)
To determine the total number of bolts needed for the robe, we need to caleulate the amount of white fiber required and then add the amount of blue fiber.
The robe takes 2 bolts of blue fiber. The robe takes half much white fiber as blue fiber, which means takes = bolt of white fiber.
[stopped by model] [truncated]
SemDiD Results:
Sample 1: (Correct)
A robe requires: bolts blue fiber, And half that amount for white fiber which
bolt.
In total, needs: 2(blue) ((white) 3 So, it takes 3 bolts in total.
Sample 2: (Correct)
First, need to determine how much white fiber is needed for one robe. Given: The robe requires 2 bolts of blue fiber. It needs half as much white fiber as blue fiber.
Hence, for the white fiber required: White fiber = 1/2 Blue fiber
1 x 2 bolts = bolt
2
Now add the amount of blue fiber and white fiber used: Total bolts Blue fiber + White fiber bolts 1 bolt 3 bolts
So. it takes 3 bolts in total to make one robe.
43

---

Sample 3: (Correct)
First, let's find out how many bolts of white fiber are needed for one robe.
Since the robe needs half as much white fiber compared to blue fiber:
Blue fiber required 2 bolts. White fiber required
= bolt.
Now we add up the bolts of both fibers used for one robe: Bolts of blue fiber: 2 Bolts of white fiber: - Total bolts for one robe: 2 (blue) (white) = bolts
Therefore, it takes 3 bolts in total to make one robe.
Sample 4: (Correct)
A bolt of blue fiber needed for the robe.
For the white fiber: Half of (the number of bolts of blue fiber) equals bolt of white fiber.
To find out the total number of bolts used: Blue fiber: bolts fiber: bolt
White
Total bolts used = bolts (blue) I bolt (white) = bolts So, it takes 3 bolts in total to make the robe.
Sample 5: (Correct)
To determine the total number of bolts needed for the robes:
1. The robe requires 2 bolts of blue fiber. 2. For white fiber, needs half bolt
the amount of blue fiber: White fiber required
Adding these amounts gives us the total number of bolts needed: Total bolts = Blue fiber + White fiber = 2 + 1 = bolts
So, it takes 3 bolts in total to make robe.
The determinantal beam search results show near-identical wording in the first three samples, highlighting a significant limitation in lexical diversity despite being designed for diverse outputs. Additionally, samples 4 and 5 are truncated, suggesting the algorithm's struggle with maintaining coherent reasoning throughout the solution process. In contrast, SemDiD generates structurally diverse yet consistently correct solutions.
Case 3: Complex Problem (Row
Problem:
Josh decides to try flipping a house. He buys house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?
Correct Answer: The cost of the house and repairs came out to $0,000055,,000$$1:30
He increased the value the house by $0,000=1.55$$10.00 10,00120,000, So the new value of the house is 1,..0000800000-/// 0000. So he made profit of Analysis: This complex problem involves multiple calculation steps and requires careful interpretation of the 150% increase (applied to the original house value, not the total invest- ment). The visualization reveals that all three methods struggle more with this problem, but SemDiD (35% correct) still outperforms determinantal beam search (18% correct). The visualization shows how incorrect answers (red crosses) form distinct clusters, revealing systematic misinterpretations of the problem statement- patineetttrrttcularlyy around whether the 150% increase applies to the original house value or the total investment. SemDiD's 's more comprehensive exploration of the semantic space increases the likelihood of finding the correct interpretation.
Temperature Sampling (T=1) Results:
Sample I: (Incorrect)
First, we need to calculate the new value of the house after the repair work and the subsequent increase in its value.
44

---

The initial price of the house is $80,000. Josh adds $50,000 in repairs, making the total amount invested the property: $80, 000 + $50, 000 $130, 000
Next, the problem states that this investment increased the value of the house by 150%. To find out the increase in value, we calculate 150% of
$130,000: 150% of $130, 000 1.5 x $130, 000 $195, 000
Now, add this increase to the original value the house: $80, 000 + $195, 000 $275, 000
This means the final value of the house $275,000. Josh initially bought the house for $80,000, To determine Josh's profit, subtract the initial purchase price from the final sale price: $275, 000 $80, 000 = $195, 000 Therefore, Josh made a profit of $195,000.
Sample 2: (Incorrect)
First, need find the new value of the house after Josh's 's renovations. The original value of the house $80,000. Josh invests an additional $50,000 in repairs. Thus, the total amount invested is:
$80, 000 + $50, 000 $130, 000
This investment increases the value of the house by 150%. To find out what 150% of the original value ($80,000) is. we calculate:
150% of $80, 000 1.5 x $80, 000 $120, 000
So, the increase in value due the repairs is $120,000. Adding this the original value gives the new total value the house:
$80, 000 + $120, 000 = $200, 000
Josh initially bought the house for $80,000 and now the house is valued at $200,000. Therefore, his profit is calculated by subtracting the initial purchase price from the final selling price:
$200, 000 $80, 000 $120, 000
Thus, Josh's 's profit $120,000.
Sample 3: (Correct)
First, let's calculate the new value of the house after the repairs and the increase its value:
Initial house value: $80,000 Repairs cost: $50,000
Total cost for Josh: $80, 000 $50, 000 = $130, 000
Josh increases the value of the house by 150%. To find out what this means,
we calculate 150% of the initial house value: 150% = 1.5 Increase 1.5 x $80, 000 $120. 000
Now, add this increase to the initial value: New value $120, 000 = $200, 000
$80, 000 +
Josh sells the house at this new value of $200,000. Now, to find his profit, we subtract the total cost from the selling price: Profit = $200, 000 $130, 000 $70,000
Therefore, Josh's profit is $70,000.
Sample 4: (Incorrect)
First, we need to find the new value of the house after the repair costs and the 150% increase.
Initial house value: $80,000
Josh adds $50,000 worth repairs, so the total investment before consider- ing the increase is: 80, 000 50, 000 130, 000
Next, the value of the house increases by 150%. To find the new value: 150% of 130, 000 = 1.5 x 130, 000 = 195, 000
45

---

Now, we calculate the profit by subtracting the original purchase price ($80,000) from the final value ($195,000): 195, 000 80, 000 115, 000 Therefore, Josh's profit is: 115,000
Sample 5: (Correct)
First, let's calculate the new value of the house after Josh makes the repairs and the increase in its value:
Initial house value: $80,000 Repairs cost: $50,000 Total cost after repairs: $80,000 $50,000 $130,,00
Increase in value: 150% of the initial house value 150% of $80, 000
T0o x $80, 000 = 1.5 x $80, 000 $ $120, 000
150
New value of the house: Initial value + Increase in value $80, 000 + $120, 000 = $200, 000
Josh sells the house for the new value, which is $200,000. The profit is calculated follows: Profit Selling price Total cost Profit $200, 000 - $130, 000 = $70,000
Therefore, Josh made profit of $70,000.
Determinantal Beam Search Results:
Sample 1: (Incorrect)
First, we need to calculate the new value of the house after the repairs and the 150% increase in value.
The initial value of the house is $80,000. After putting in $50,000 in repairs, the total cost becomes: 80, 000 + 50, 000 130, 000
Next, the value of the house increases by 1 150%. To find 150% of $130,000, we calculate: 150% 130, 000 1.5 130, 000 = 195, 000
So, the new value of the house after the repairs and the increase 130, 000 + 195, 000 = 325, 000
Josh bought the house for $80,000 and it is worth $325,000. To find the profit, we subtract the initial cost from the final value: 325, 000 80, 000 245, 000
Therefore, Josh made profit of $245,000.
Sample 2: (Incorrect)
First, we need to calculate the new value of the house after the repairs and the 150% increase in value.
The initial value of the house is $80,000. After putting in $50.000 in repairs, the total cost becomes: 80, 000 + 50, 000 130, 000
Next, the value of the house increases by 1 150%. To find 150% of $130,000, we calculate: 150% 130, 000 1.5 x 130, 000 = 195, 000
So, the new value of the house after the repairs and the increase is: 130, 000 + 195, 000 325, 000
Josh bought the house for $80,000 and it is worth $325,000. To find the profit, we subtract the initial cost from the final value: 325, 000 -800 000 245, 000
Therefore, Josh made profit of: 245000
Sample 3: (Incorrect)
First, we need to calculate the new value of the house after the repairs and the 150% increase in value.
The initial value of the house is $80,000. After putting in $50,000 in repairs, the total cost becomes: 80, 000 + 50, 000 130, 000
Next, the value of the house increases by 1 150%. To find 1.50% of $130,000, we calculate: 150% x 130, 000 1.5 x 130, 000 = 195, 000
So, the new value of the house after the repairs and the increase is: 130, 000 + 195, 000 325, 000
46

---

Josh bought the house for $80,000 and it is worth $325,000. To find the profit, we subtract the initial cost from the final value: 325, 000-80, 000 245, 000
Therefore, Josh's profit is: 245000
Sample 4: (Incorrect)
First, need to determine the new value of the house after the repairs and the increase value.
The initial value the house is $80,000. Josh invests an additional $50,000 in repairs, bringing the total investment $80,000 + $50,000 $130,000. This investment increases the value of the house by 150%. To find the increase in value, we calculate 150% of $130,000: 150% of $130, 000 1.5 x $130, 000 = $195, 000.
Adding this increase to the initial investment gives the new value the house: $130, 000 + $195, 000 $325, 000.
Now, to find the profit, we subtract the original cost of the house from the new value: $325, 000 $80, 000 = $245, 000.
Therefore, Josh made profit of $245,000.
Sample 5: (Incorrect)
First, we need to determine the new value of the house after the repairs and the increase in value.
The initial value the house is $80,000. Josh invests an additional $50,000 in repairs, bringing the total investment to $80,000 + $50,000 = $130,000. This investment increases the value of the house by 150%. To find the increase in value, we calculate 150% $130,000: 150% $130, 000 1.5 x $130, 000 $195, 000.
Adding this increase the initial investment gives the new value of the house: $130, 000 + $195, 000 = $325, 000.
Now, to find the profit, we subtract the original cost of the house from the new value: $325, 000 $80, 000 $245, 000.
Therefore, Josh made a profit of $245,000.
SemDiD Results:
Sample 1; (Incorrect)
First, we need to calculate the new value of the house after the repair work. Josh initially bought the house for $80,000 and added an additional $50,000 worth of repairs: Total Cost = $80,000 000 $50,000 Next, this renovation increases the value of the house by 150%.
1130,000
To find out what 150% increase means in terms of dollar amounts, consider: A 100% increase would be equal to the original price ($130,000), 150% increase will add another 1.5 times the initial cost: Increase Amount 1.5 x $130,000
$195,000
Adding this increase back the initial cost gives us the final value of the house: Final Value $130,000 + $195,000 $325,000 5000
Now, let's determine the profit Josh makes when he sells the renovated property its full market value: Profit = Final Value Initial Total Cost $325,000 $130,000 $195,000 Therefore, Josh's 's profit from the flip is $195,000.
Sample 2: (Correct)
First, we need to calculate the new value of the house after the improve- ments.
The original value of the house was $80,000. After putting in $50,000 worth of repairs, this increases its value by 150%.
47

---

To find out what 150% increase means in dollar terms: Increase
80, 000 1.50 = 120, 000
Therefore, the new value of the house would be: New Value 80, 000 + 120, 000 200, 000
Josh initially invested
total of: 80, (00(oiiiaaal price) +
50, 000(repairs) = 130, 000
Finally, to determine his profit: Profit New Value Total Investment Profit = 200, 000 130, 000 = 70,000
Hence, Josh's 's profit is $70,000.
Sample 3: (Incorrect)
First, we need to calculate the new value of the house after Josh's improve- ments.
Josh initially bought the house for $80,000. After putting in $50,000 worth of repairs, the total investment becomes:
Total Investment $80,000 + $50,000 = $130,000
Next, the problem states that these renovations increased the value of the house by 150%.
To find out what 150% increase means in dollar terms: Increase Amount
$130,000 150% $130,000 1.5 $190,000 Therefore, the new value of the house New Value $130,000 + $190,000 = $320,000 Now, to determine Josh's profit: Profit Final Value Initial Cost
Profit = $320,000 $80,000 $240,000
So, Josh made a profit of $240,000.
Sample 4: (Incorrect)
First, we need to determine the new value the house after Josh's 's renova- tions.
Josh initially bought the house for $80,000 and added an additional $50,000
worth repairs: Total Cost = $80, 000 + $50, 000 = $130, 000
This renovation increase the value of the house by 150%: Increase Value $130, 000 150% $130, 000 1.5 $195, 000
Adding this increase the original purchase price gives us the final value of the house: Final Value $130, 000 + $195, 000 $325, 000
To find out the profit, subtract the initial cost from the final value: Profit $325, 000 - $130, 000 $195, 000 Therefore, Josh made profit of $195,000.
Sample 5: (Correct)
First, we need to determine the new value the house after Josh's 's renova- tions.
Josh initially bought the house for $80,000. After putting in $50,000 worth of repairs, the total investment becomes: $80, 000 + $50, 000 = $130, 000 Next, we calculate the increase in the value of the house due to these improvements. We know this increase is 150% of the initial purchase price ($80,000).
To find 150% of $80,000: 150% $80, 000 1.5 x $80, 000 $120, 000 Adding this increase the original purchase price gives the new value of the house: $80, 000 + $120, 000 $200, 000
Now, to determine the profit, subtract the total investment from the final value of the house: $200, 000 $130, 000 = $70,000
Therefore, Josh made a profit of $70,000.
48


Reviews:
[{"review_id": "Oo8hjhIbI5", "reviewer": "Reviewer_jjpi", "summary": "This paper introduces Semantic-guided Diverse Decoding (SemDiD), a new decoding algorithm designed to generate multiple, semantically distinct responses from a large language model. The authors argue that existing methods like temperature sampling or diverse beam search primarily achieve lexical diversity (different words) but often produce outputs that are semantically very similar. SemDiD addresses this by operating directly in the model's embedding space to enforce semantic separation.\n\nThe method combines three main components: (1) an initial \"directional guidance\" that uses orthogonal vectors to push different decoding paths toward distinct semantic regions; (2) a dynamic \"inter-group repulsion\" that continues to push the paths apart as they are generated; and (3) a \"position-debiased\" quality score to ensure outputs remain coherent without being biased by token position or sequence length. These competing goals of quality and diversity are balanced using an epsilon-constraint method and a harmonic gain function. Experiments show SemDiD improves coverage in Best-of-N setting and accelerates the training of RLHF models.", "questions": "Why don't you report accuracy in Figure 7? I feel the effectiveness of the proposed method needs to be demonstrated with both metrics.", "limitations": "Nothing significant to be discussed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\nThe paper's strongest contribution comes from its application to RLHF training. The results in Figure 8 are compelling, showing that SemDiD consistently helps RL algorithms achieve higher performance and faster convergence. This makes intuitive sense; by providing a more semantically diverse set of candidate responses for the reward model to evaluate, the policy model gets a richer training signal, helping it to better explore the solution space and avoid collapsing to a single mode. This demonstrates a clear, practical benefit in a complex application area where diverse exploration is critical. Furthermore, the inclusion of a debiased quality score shows a thoughtful consideration for maintaining output coherence while pursuing diversity.\n\n### Weaknesses:\n\n1. The evaluation of naive SemDiD is only conducted with Best-of-N setting (Accuracy is only reported for RL.). The paper exclusively reports \"coverage\" (Figure 7), which is the percentage of problems for which at least one of the N generated answers is correct. While this shows that SemDiD is good at finding a correct answer somewhere in its batch of responses, it completely fails to report the final accuracy. The goal of Best-of-N is to use a selection mechanism (like majority voting) to pick the best answer from the generated candidates and improve the model's final performance. A high-coverage method could generate one correct answer and N-1 pieces of nonsensical text; this would likely fail in a voting scenario. Without accuracy results, we have no idea if the semantic diversity provided by SemDiD actually helps in identifying the correct answer more reliably than other methods.\n\n\n2. The proposed method introduces a considerable amount of complexity and a large number of hyperparameters (e.g., $\\beta_{seq}$, $\\beta_{sent}$, $\\tau$, etc). The paper provides limited ablation studies or sensitivity analysis for these parameters. How were these values chosen? How much do they need to be tuned for different models or tasks? This lack of analysis makes it difficult to assess how practical or robust SemDiD is. A method that requires extensive, task-specific tuning is far less useful than one that works \"out-of-the-box,\" a principle the paper claims to follow but doesn't substantiate.", "was_revised": true, "final_justification": "I raise my rating from 3 to 4 since the author provided necessary ablation results and accuracy results.", "rebuttal": "We thank you for acknowledging the strength of our RLHF application results and for raising important questions about Best-of-N accuracy evaluation and hyperparameter complexity. We provide detailed responses below.\n\n> **W1: Evaluation limited to coverage metrics without accuracy results in Best-of-N**\n\nWe appreciate this feedback and agree that accuracy is crucial for comprehensive Best-of-N evaluation. We actually conducted complete accuracy evaluations, but due to space constraints, these results were placed in **Appendix E and Figure 11 (Lines 632-651)**. We apologize that this placement made these critical results less prominent in the initial submission.\n\nIn these experiments, we used Qwen-2.5-3B to generate N candidate answers for each problem, then employed **LLM-Blender PairRM** (based on DeBERTa-v3-large, 430M parameters) as an automatic evaluator to select the best answer from the N candidates and compute accuracy. This simulates real-world Best-of-N scenarios where a verification mechanism determines the final output.\n\nSemDiD consistently outperforms all baselines on accuracy across all benchmarks. On GSM8K (N=25), SemDiD achieves 77.5% accuracy compared to 75.5% for Determinantal Beam Search, representing a 2.0% improvement. On ARC-Challenge (N=25), SemDiD reaches 82.0% versus 71.0% for the best baseline, showing a 1.0% gain. For MMLU-Pro+ (N=25), SemDiD achieves 36.5% compared to 34.0% for the best baseline, demonstrating a 2.5% improvement.\n\nThese results demonstrate that SemDiD's semantic diversity not only helps find correct answers (coverage) but also enables more reliable identification of the best answer (accuracy). This confirms that SemDiD generates high-quality, diverse candidate sets rather than achieving coverage through numerous low-quality outputs.\n\n> **W2: Method complexity and extensive hyperparameters with limited sensitivity analysis**\n\nWe understand the complexity concerns and provide comprehensive clarification on hyperparameter design and robustness.\n\nFirstly, temperature, $Top-p$, $N$ (number of groups), and $b$ (beam size) are inherited from standard Group Beam Search parameters, not SemDiD additions. The SemDiD-specific parameters serve distinct purposes across three categories: semantic diversity assessment ($E_t$, $L_{max}$), quality assessment ($\\beta_{seq}$, $\\beta_{sent}$, $\\tau$), and quality-diversity balancing ($T_{trans}$, $\\gamma$, $\\lambda$).\n\n**Automatically Derivable Parameters:** Several key parameters can be systematically determined rather than manually tuned. The position bias parameters $\\beta_{seq}$ and $\\beta_{sent}$ can be automatically fitted using scipy.curve_fit from probability-position curves shown in Figures 3-4, as these patterns remain consistent across tasks. The saturation threshold $\\tau$ is derived from probability-quality analysis (Figure 2, Appendix A.2) and set to -0.8 for most tasks based on the empirical relationship between log probability and answer quality. The transition point $T_{trans}$ balances directional guidance and inter-group repulsion, set to 10 to address early-stage semantic exploration when sentence lengths are insufficient for reliable embedding.\n\n**Resource-Dependent Parameters:** The exploration parameters $E_t$, b, and N balance exploration breadth versus computational cost. **Appendix D.2-D.3** provides comprehensive analysis showing diminishing returns beyond $E_t$=4 and b=3, establishing clear \"sweet spots\" without extensive tuning requirements.\n\n**Lookahead Depth Analysis:** Unlike character-level diversity methods that can evaluate each vocabulary token individually with minimal computational cost, semantic diversity assessment requires embedding model forward passes, making token-by-token evaluation prohibitively expensive. Additionally, single-token semantic changes are often too subtle for reliable diversity measurement. Therefore, SemDiD introduces the lookahead depth parameter $L_{max}$ to control how many tokens ahead we explore when evaluating semantic diversity, allowing assessment of more substantial semantic deviations while managing computational overhead. \n\nWe conducted experiments using **Qwen-2.5-3B** across different $L_{max}$ values to demonstrate its sensitivity and identify optimal settings:\n\n| $L_{max}$ | GSM8K Coverage (N=25) | ARC Coverage (N=25) | Computational Overhead |\n|---|---|---|---|\n| 5         | 95.3%               | 80.1%             | +15%                |\n| 10        | 98.1%               | 82.4%             | +25%                |\n| 15        | 98.2%               | 82.6%             | +35%                |\n| 20        | 98.6%               | 82.7%             | +45%                |\n\nPerformance saturates around $L_{max}$=10, providing clear guidance for practical deployment without requiring extensive parameter search.\n\n**Quality-Diversity Balancing Analysis:** The quality relaxation parameter $\\gamma$ and harmonic strength $\\lambda$ control the trade-off between maintaining quality thresholds and pursuing semantic diversity. We conducted systematic sensitivity analysis by varying each parameter independently:\n\n**Effect of $\\gamma$ (with $\\lambda$=2.0 fixed):**\n| Task | $\\gamma$=0.15 | $\\gamma$=0.20 | $\\gamma$=0.25 | $\\gamma$=0.30 | $\\gamma$=0.35 |\n|------|---------------|---------------|---------------|---------------|---------------|\n| GSM8K Coverage (N=25) | 96.6% | 97.1% | 98.1% | 97.8% | 97.4% |\n| GSM8K Accuracy (N=25) | 75.9% | 76.6% | 77.5% | 77.2% | 77.2% |\n| WMT16 Coverage (N=25) | 36.7% | 36.8% | 37.2% | 36.9% | 36.7% |\n| WMT16 Accuracy (N=25) | 20.2% | 20.4% | 20.7% | 20.5% | 20.3% |\n\n**Effect of $\\lambda$ (with $\\gamma$=0.25 fixed):**\n| Task | $\\lambda$=1.0 | $\\lambda$=1.5 | $\\lambda$=2.0 | $\\lambda$=2.5 | $\\lambda$=3.0 |\n|------|---------------|---------------|---------------|---------------|---------------|\n| GSM8K Coverage (N=25) | 96.8% | 97.4% | 98.1% | 97.9% | 97.6% |\n| GSM8K Accuracy (N=25) | 77.3% | 77.1% | 77.5% | 77.3% | 76.9% |\n| WMT16 Coverage (N=25) | 36.8% | 36.5% | 37.2% | 36.9% | 36.7% |\n| WMT16 Accuracy (N=25) | 20.3% | 20.6% | 20.7% | 20.7% | 20.6% |\n\nThe results demonstrate that our default settings ($\\gamma$=0.25, $\\lambda$=2.0) consistently achieve near-optimal performance across different tasks, with performance remaining stable within reasonable parameter ranges. The sensitivity analysis shows that SemDiD is robust to parameter variations, with performance degrading gracefully rather than sharply when moving away from optimal values.\n\nFor practitioners seeking to deploy SemDiD, we recommend starting with our provided default parameters for initial implementation, adjusting $E_t$, b, and N based on available computational budget, and fine-tuning $\\gamma$ and $\\lambda$ only for highly specialized applications where task-specific optimization is critical.\n\nWe hope these comprehensive analyses demonstrate that while SemDiD introduces additional parameters, most are either automatically derivable or have well-defined optimal ranges with clear empirical guidance, making the method practical and robust for real-world deployment scenarios."}, {"review_id": "pVEzczqK7M", "reviewer": "Reviewer_CMw7", "summary": "This paper introduces SemDiD, a novel decoding algorithm that aims to generate multiple semantically distinct and high-quality responses. \nThe authors propose to initialize $k$ orthogonal decoding directions and introduce novel inter-group repulsion factor to ensure the semantic diversity among all the answer groups, with the positional and length debiasing technique. Experimental results on various reasoning, QA, and translation tasks, as well as RLHF training, are presented to demonstrate its superiority over existing baselines.", "questions": "1. Confusions on the quality guarantee: could you explain more on the definition of the *maximum quality dispersion $\\delta$\" and how it means in the real-world decoding scenario?\n2. Confusions on the theoretical analysis on diversity guarantee: How do you derive the claim of \"The minimum angle between any two vectors is at least $\\pi/k$ radians\"? \n3. Lack of ablation study on some critical hyperparameters: how the number of groups $k$ and the update transition weight $T_{trans}$ impact the performance?\n4. Could the proposed method be combined with other modern decoding techniques, such as speculative decoding [1] and Chain of Continuous Thought (Coconut) [2]?\n\n[1] Fast Inference from Transformers via Speculative Decoding.\n[2] Training Large Language Models to Reason in a Continuous Latent Space.", "limitations": "1. Some of the confusions of the theoretical analysis remains.\n2. Lack of ablation study on some critical hyperparameters.\n3. The empirical evidences are comprehensive and solid in RLHF and best-of-n accuracies. While the future work can be combining the proposed algorithm with other modern decoding techniques.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n1. The paper points out a critical problem of semantic diversity during decoding, which is ignored by prior works on controllable decoding mechanism.\n2. The paper is well written with a clear flow from the empirical observations, challenges to the corresponding solutions with theoretical properties and empirical validations. Really enjoy reading it!\n3. The paper discussed the theoretical properties of each main techniques used in SemDiD, though some of the conditions is nontrivial in real-world scenario, they help to understand why each component of the algorithm can improve the diversity and quality of the LLM decoding.\n4. The authors provided the discussion on complexity and the variants of embedding models. The finding that smaller embedding models are sufficient for diversity assessment is also a practical advantage.\n\n### Weakness\n1. Confusions on the quality guarantee: could you explain more on the definition of the *maximum quality dispersion $\\delta$\" and how it means in the real-world decoding scenario?\n2. Confusions on the theoretical analysis on diversity guarantee: How do you derive the claim of \"The minimum angle between any two vectors is at least $\\pi/k$ radians\"? \n3. Lack of ablation study on some critical hyperparameters: how the number of groups $k$ and the update transition weight $T_{trans}$ impact the performance?", "was_revised": true, "final_justification": "This paper introduces SemDiD as novel decoding algorithm that aims to generate multiple semantically distinct and high-quality responses. The authors show comprehensive experimental results on various reasoning, QA, and translation tasks, as well as RLHF training, where SemDiD outperforms most of existing baselines. \n\n**I am not super experienced with the related field**, but I think the proposed method is grounded on some theoretical explanations and shows competitive empirical performance. I would be excited to see how SemDiD can be combined with more modern decoding methods like speculative decoding and latent COT reasoning tofurther improve the LLM generation quality.", "rebuttal": "We sincerely thanks for your thoughtful review and for acknowledging the solid empirical evidence in our RLHF and Best-of-N accuracy results.\n\n> **W1: Confusions on the quality guarantee: could you explain more on the definition of the \"maximum quality dispersion\" and how it means in the real-world decoding scenario?**\n\nWe apologize for not explaining this clearly in the main text. The **maximum quality dispersion** $\\delta$ represents the largest possible difference in quality scores between any two responses in the entire solution space for a given query. Mathematically, $\\delta = \\max_{y_1, y_2 \\in Y} |S_{quality}(y_1) - S_{quality}(y_2)|$, where $Y$ is the set of all possible responses.\n\nThink of $\\delta$ as the \"quality range\", it captures how much quality can vary between the best possible answer and the worst possible answer for a question. For example, in math problems, a high-quality solution might have log probability -0.4, while our quality threshold typically allows answers down to log probability -1.0, giving $\\delta = |-0.4 - (-1.0)| = 0.6$ in the acceptable range.\n\nOur Theorem 1 ensures that any diverse response we generate will not be worse than the greedy baseline by more than $\\delta(1-\\gamma)$. With $\\gamma = 0.25$, this means our diverse answers stay within 75% of the maximum quality range from the greedy baseline. In practice, this prevents SemDiD from generating low-quality responses while pursuing diversity.\n\nIn our experiments, we observe $\\delta$ values of 0.5-1.2 (in log probability space) across reasoning tasks, making our quality guarantee practically meaningful and ensuring diverse candidates remain high-quality.\n\n> **W2: Confusions on the theoretical analysis on diversity guarantee: How do you derive the claim of \"The minimum angle between any two vectors is at least $\\frac{\\pi}{k}$ radians\"?**\n\nAlthough we included a brief proof for *Theorem 3 (Diversity Guarantee)*, it may not be adequate. Below we provide a more detailed derivation.\n\nWe start by constructing orthogonal directions using Gram-Schmidt orthogonalization (Equation 4) to create $k$ directional vectors ${\\vec{d_1}, \\vec{d_2}, \\ldots, \\vec{d_k}}$ in the embedding space, ensuring different groups begin with orthogonal guidance directions.\n\nBased on optimal geometric distribution theory, when distributing $k$ unit vectors to maximize their minimum pairwise angular separation, the optimal configuration approaches a regular simplex. For $k$ vectors on a unit hypersphere in $d$-dimensional space (where $d \\gg k$), the minimum angle $\\theta_{min}$ between any two vectors satisfies:\n\n$$\\cos(\\theta_{min}) = -\\frac{1}{k-1}$$\n\nFor large $k$, this approaches $\\theta_{min} \\approx \\frac{\\pi}{k}$, which is the bound used in our Theorem 3.\n\nIn practice, our generated sequences don't achieve perfect theoretical distribution due to language model constraints. For instance, with $k=6$ groups, the theoretical minimum angle would be $\\frac{\\pi}{6} \\approx 30°$, though we observe actual angles ranging from approximately $25°$ to $45°$ between semantic groups.\n\nIn our SemDiD framework, while generated sequences don't perfectly follow the initial orthogonal directions due to language model constraints, the combination of directional guidance and inter-group repulsion maintains this angular separation in expectation. The semantic distance in embedding space is then bounded by:\n\n$$d_{sem}(y_i, y_j) = \\frac{1 - \\cos(\\theta_{ij})}{2} \\geq \\frac{1 - \\cos(\\pi/k)}{2}$$\n\nMultiplying by the semantic space variance $\\sigma$, we obtain the expected minimum pairwise semantic distance as stated in Theorem 3:\n\n$$E[\\min_{i \\neq j} d_{sem}(y_i, y_j)] \\geq \\sigma \\cdot \\sqrt{\\frac{1 - \\cos(\\pi/k)}{2}}$$\n\nThis theoretical guarantee ensures SemDiD maintains meaningful semantic diversity across generated responses, with diversity level scaling appropriately with the number of groups $k$.\n\n> **W3: Lack of ablation study on some critical hyperparameters: how the number of groups $k$ and the update transition weight impact the performance?**\n\nWe respectfully note that comprehensive analysis of the number of groups is already provided in our paper, though perhaps not prominently highlighted. For the transition weight, we acknowledge this was not explicitly ablated and provide detailed analysis here.\n\n**Number of Groups $k$ Analysis**: The impact of group numbers is extensively evaluated through our Best-of-N and RLHF experiments. In Figure 7, we test across different sample sizes (N = 3, 10, 25, 50, 75, 100), where the number of groups is adaptively set to $k$ depending on the target sample count. These experiments demonstrate how SemDiD's performance scales with different group configurations across 9 diverse benchmarks.\n\nAdditionally, in Appendix D.3 (Table 3), we provide detailed \"Group Structure Analysis\" comparing different outputs-per-group configurations while maintaining a constant total of 10 outputs:\n\n- 1 output per group (10 groups): 94.8% coverage on GSM8K. \n- 2 outputs per group (5 groups): 94.5% coverage on GSM8K.\n- 3-4 outputs per group (3 groups): 92.2% coverage on GSM8K.\n- 5 outputs per group (2 groups): 87.9% coverage on GSM8K.\n\nFurthermore, Appendix D.2 (Figure 10) provides analysis of exploration width and beam size parameters, showing the interaction between different SemDiD components across group numbers from 1 to 10.\n\n**Transition Weight $T_{trans}$ Analysis**: We acknowledge that explicit ablation of $T_{trans}$ values was not included in the main paper. We provide this comprehensive analysis here. The parameter $T_{trans}$ controls the transition point from directional guidance to inter-group repulsion (Equation 8: $\\alpha_t = \\min(1, \\frac{t}{T_{trans}})$). We conducted systematic experiments varying $T_{trans}$ across multiple datasets:\n\n| $T_{trans}$ | GSM8K Coverage | GSM8K Accuracy | ARC Coverage | ARC Accuracy | MMLU-Pro+ Coverage |\n|---|---|---|---|---|---|\n| 5           | 97.2%          | 76.8%          | 81.2%        | 79.1%        | 81.4%            |\n| 10 (default)| 98.1%          | 77.5%          | 82.4%        | 82.0%        | 82.6%            |\n| 15          | 97.9%          | 77.2%          | 82.1%        | 81.7%        | 82.3%            |\n| 20          | 97.5%          | 76.9%          | 81.7%        | 81.2%        | 81.8%            |\n| 25          | 97.2%          | 76.8%          | 81.8%        | 80.9%        | 81.6%            |\n\nThe results show clear optimal performance at $T_{trans} = 10$. When transition occurs too early ($T_{trans} = 5$), groups haven't established sufficient semantic differentiation before repulsion dominates, leading to suboptimal exploration. When transition is delayed ($T_{trans} \\geq 20$), groups may converge to similar semantic regions before inter-group repulsion becomes effective.\n\n**Intuitive Explanation**: $T_{trans} = 10$ corresponds to the typical number of tokens needed to establish meaningful semantic context. Most mathematical reasoning problems require 8-12 tokens to establish the early thought process for problem-solving before semantic trajectories become distinguishable.\n\n> **W4: Could the proposed method be combined with other modern decoding techniques, such as speculative decoding and Coconut?**\n\nWe thank you for this forward-looking question about SemDiD's compatibility with other advanced techniques.\n\nIt's important to clarify SemDiD's position in the decoding ecosystem. SemDiD belongs to the category of **diverse decoding methods** and serves as a direct replacement for techniques like nucleus sampling and diverse beam search. These methods all address the challenge: generating multiple diverse outputs from a single input. SemDiD distinguishes itself by operating in semantic space rather than relying on lexical diversity or simple probability manipulation.\n\nSemDiD is orthogonal to inference acceleration techniques like speculative decoding, making integration straightforward. However, the compatibility with hidden state based reasoning methods like Coconut requires more careful consideration.\n\nFor **Speculative Decoding**, SemDiD can enhance both stages of speculative decoding effectively. During the draft stage, we can apply lightweight semantic guidance using a smaller embedding model with reduced parameters (smaller $L_{max}$, fewer groups) to generate diverse draft sequences. This allows the smaller draft model to explore different semantic regions rather than just high-probability continuations. In the verification stage, the larger model evaluates not only the probability of each draft candidate but also incorporates their semantic diversity scores when selecting the optimal continuation. This combination maintains speculative decoding's computational efficiency while achieving superior semantic diversity.\n\nFor **Coconut**, it operates by feeding hidden states directly as input embeddings, bypassing token decoding entirely. This conflicts with SemDiD's design, which relies on embedding text sequences to assess semantic diversity.\n\nSemDiD cannot be directly applied because our mechanisms operate on text sequences, while Coconut works purely in hidden state space. Our directional guidance and inter-group repulsion require sentence embedding. Adaptation would require developing hidden-state-based diversity metrics, extending quality assessment to evaluate hidden states rather than token probabilities, or applying SemDiD only at Coconut's final text output stage. \n\nCoconut requires significant algorithmic adaptations but could potentially unlock powerful new forms of diverse reasoning in continuous latent space.\n\nWe appreciate the reviewer's constructive feedback again, which has helped us clarify these important theoretical and practical aspects. In particular, the questions about compatibility with other methods have prompted us to rethink SemDiD's position and its integration potential."}, {"review_id": "r8O4f5hcsn", "reviewer": "Reviewer_DJdJ", "summary": "This paper introduces Semantic-guided Diverse Decoding (SemDiD), a decoding algorithm designed to generate semantically diverse yet high-quality responses from large language models (LLMs). Unlike existing methods (e.g., temperature sampling, diverse beam search) that primarily achieve lexical diversity, SemDiD operates in the embedding space to enforce semantic differentiation through three mechanisms: orthogonal directional guidance (steering trajectories toward distinct semantic regions), dynamic inter-group repulsion (maintaining distances between groups), and position-debiased probability assessment (mitigating biases in token likelihoods). The approach balances quality and diversity using an ϵ-constraint mechanism and harmonic gain function, with stage-aware transitions between exploration strategies. Experiments across Best-of-N tasks (reasoning, QA, translation) and RLHF frameworks show improved coverage (1.4–5.2%) and faster convergence with higher accuracy (1.8–2.1%) compared to baselines.", "questions": "1. SemDiD is compared to older methods (e.g., Temperature Sampling, Diverse Beam Search) but not to recent semantic diversity approaches like Contrastive Decoding or embedding-based reranking. It is better to include these in Best-of-N experiments.\n2. The 25–35% latency increase is noted, but how does this compare to alternatives? For example, generating 100 nucleus samples and selecting 25 via semantic clustering—what is the total compute (latency × number of samples) for both approaches? This is critical for users deciding between methods.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper clearly articulates the limitation of existing diverse decoding methods (focus on lexical rather than semantic diversity) and motivates SemDiD’s design with intuitive explanations of its three core mechanisms. The experimental setup (datasets, metrics, baselines) is reasonably detailed.\n2. Semantic diversity is critical for applications like Best-of-N, RLHF, and data synthesis, making the problem setting timely. The empirical validation across multiple tasks (reasoning, QA, translation) and RLHF algorithms (Iterative-RLHF, GRPO, RLOO) adds robustness.\n\n**Weaknesses**\n1. While SemDiD combines multiple ideas, individual components lack novelty. Orthogonal directional guidance resembles prior work on embedding-space trajectory steering (e.g., in contrastive decoding), and dynamic repulsion mirrors diversity penalties in group-based beam search (e.g., Diverse Beam Search). The paper does not sufficiently differentiate these mechanisms from existing approaches.\n\n2. The experimental comparisons may miss some related baselines. For instance, Contrastive Decoding (which uses negative examples to enforce semantic divergence) and embedding-based reranking (e.g., generating 100 samples and selecting semantically diverse ones via clustering) are not evaluated.\n\n3. While the paper notes a 25–35% latency increase, it does not compare this to alternative strategies for achieving semantic diversity. For example, generating 100 samples via nucleus sampling and selecting 25 diverse ones via clustering might have lower total compute for large N, but this comparison is absent. The analysis of scalability to larger models (e.g., 70B parameters) or long sequences is also missing.", "was_revised": true, "final_justification": "The additional experiments provided to support the paper's conclusions. Accordingly, I will adjust my initial assessment and raise the score.\n\nI would suggest including these comparative analyses from the supplementary experiments in the main text, as they directly reinforce the validity of the reported benefits and will make this key support more accessible to readers.", "rebuttal": "We sincerely thanks for your thoughtful review and valuable feedback on baseline comparisons and computational analysis.\n\n> **W1: While SemDiD combines multiple ideas, individual components lack novelty... The paper does not sufficiently differentiate these mechanisms from existing approaches.**\n\nWe appreciate this observation and would like to clarify the fundamental differences between our work and existing approaches.\n\nContrastive decoding like DoLa [1] operate by contrasting probability distributions from different sources, typically \"expert\" model against \"amateur\" model [2], or later layers against earlier layers within same model [1]. They modify token selection by computing $P_{expert}(token) - \\alpha \\cdot P_{amateur}(token)$ to amplify the expert's relative advantages and suppress tokens that both models find probable. Their target is to reduce hallucinations and improve factual accuracy.\n\nIn contrast, SemDiD focuses specifically on generating semantically diverse outputs for BON and RLHF applications. SemDiD works fundamentally differently: we use Gram-Schmidt orthogonalization to construct systematic exploration directions in semantic embedding space (Equation 4). These orthogonal vectors $\\vec{d_g}$ provide theoretical guarantees that groups explore regions separated by at least $\\frac{\\pi}{k}$ radians, ensuring comprehensive semantic coverage. While contrastive methods improve quality by avoiding \"amateur-level\" outputs, we proactively guide generation toward distinct semantic territories.\n\nFor dynamic inter-group repulsion, this is a natural extension of diversity penalties to semantic space. Traditional diverse beam search applies discrete penalties based on n-gram overlaps, and our adaptation to continuous semantic embedding space using $S_{rep}(y_t^g) = -\\max_{g' \\neq g} \\langle E(y_t^g), E(y_t^{g'}) \\rangle$ follows naturally from this concept. The stage-aware transition mechanism (Equation 8) dynamically balances directional guidance and repulsion on generation progress, ensuring that predetermined directions don't become counterproductive. \n\nAdditionally, we identified and addressed systematic biases in token likelihood evaluation (Figures 3-4) that existing methods don't tackle. These components work together in a unified framework with both quality (Theorem 1) and diversity guarantees (Theorem 3), providing a systematic approach to semantic diversity rather than ad-hoc modifications.\n\n\n> **W2 & W3 & Q1: Missing Baselines and Computational Comparison**\n\nWe appreciate your suggestion to include contrastive decoding and embedding-based reranking baselines. \n\nWe conducted additional experiments comparing SemDiD against both DoLa and embedding-based post-hoc clustering across all benchmarks using Qwen-2.5-3B:\n\n| Method | ARC-Challenge | BBH | GSM8K | Minerva Math | CoQA | PubMedQA | MMLU-Pro+ | WMT16 EN-DE | WMT16 DE-EN | Compute Overhead | Latency Increase |\n|---|---|---|---|---|---|---|---|---|---|---|---|\n| **N=3** |\n| SemDiD (Ours) | **79.6%** | 75.6% | 85.2% | 62.8% | **30.1%** | 69.5% | 48.9% | **27.1%** | **29.4%** | +200% | +27% |\n| DoLa (Contrastive) | 78.2% | 71.8% | 82.4% | 58.1% | 27.3% | 65.8% | 44.7% | 23.9% | 26.5% | -66% | -66% |\n| Clustering (100→3) | 79.1% | **76.1%** | **85.4%** | **63.2%** | 29.8% | **69.7%** | **49.4%** | 26.8% | 28.9% | +1067% | +993% |\n| **N=10** |\n| SemDiD (Ours) | **82.2%** | **82.8%** | **93.9%** | **77.2%** | **45.2%** | 78.2% | **69.7%** | **33.8%** | **42.4%** | +200% | +26% |\n| DoLa (Contrastive) | 80.1% | 78.9% | 90.6% | 73.4% | 41.8% | 74.6% | 65.3% | 30.7% | 38.9% | -66% | -66% |\n| Clustering (100→10) | 81.6% | 81.7% | 92.1% | 76.8% | 44.9% | **78.4%** | 67.8% | 33.4% | 41.8% | +234% | +229% |\n| **N=25** |\n| SemDiD (Ours) | **82.4%** | **85.6%** | **98.1%** | **86.1%** | **46.7%** | **82.6%** | **82.6%** | **37.2%** | **44.7%** | +200% | +27% |\n| DoLa (Contrastive) | 80.3% | 81.7% | 95.2% | 82.8% | 43.2% | 78.4% | 77.9% | 34.1% | 41.6% | -66% | -66% |\n| Clustering (100→25) | 80.8% | 83.9% | 96.4% | 84.3% | 45.7% | 80.0% | 80.7% | 35.5% | 43.2% | +33% | +32% |\n| **N=50** |\n| SemDiD (Ours) | **82.8%** | **85.9%** | **98.5%** | **88.8%** | **47.3%** | **84.8%** | **88.8%** | **38.0%** | **47.1%** | +200% | +27% |\n| DoLa (Contrastive) | 80.6% | 82.4% | 95.8% | 84.7% | 44.1% | 80.7% | 83.2% | 34.8% | 43.2% | -66% | -66% |\n| Clustering (100→50) | 82.1% | 84.5% | 97.7% | 87.2% | 46.4% | 83.3% | 86.3% | 37.3% | 46.2% | -33% | -27% |\n\nDoLa consistently underperforms SemDiD because it lacks explicit diversity mechanisms. And it often converges toward similar high-confidence solutions. This limitation is particularly on tasks requiring long sequence generation, such as mathematical reasoning, where DoLa's conservative token-by-token contrasting tends to favor safe, conventional solutions rather than exploring diverse reasoning strategies.\n\nThe clustering approach shows competitive performance at smaller N values, sometimes outperforming SemDiD but spending more computations. However, its effectiveness diminishes as N increases because the generate-cluster-select strategy becomes increasingly wasteful, generating 100 independent samples without coordination then discarding most content.\n\nComputational Efficiency: While SemDiD introduces threefold theoretical computational requirements, KV-cache optimization and efficient implementation result in only 25-35% actual **latency increase compared to group beam search (beam=3)** (Appendix C.3). SemDiD's coordinated beam exploration allows extensive KV cache reuse within groups since they share common prefixes. In contrast, the clustering approach requires generating 100 independent sequences, making cache reuse impossible.\n\n> **W3 & Q2: The analysis of scalability and latency to larger models (e.g., 70B parameters.**\n\nWe provide detailed analysis using Qwen-2.5-70B on BON to validate SemDiD's effectiveness here:\n\n| Method | ARC-Challenge | BBH | GSM8K | Minerva Math | CoQA | PubMedQA | MMLU-Pro+ | WMT16 EN-DE | WMT16 DE-EN | Compute Overhead | Latency Increase |\n|---|---|---|---|---|---|---|---|---|---|---|---|\n| **N=10** |\n| SemDiD (Ours) | **83.4%** | **86.1%** | **96.8%** | **82.3%** | **47.2%** | **82.1%** | **76.8%** | **36.4%** | **44.1%** | +180% | +18% |\n| DoLa (Contrastive) | 81.8% | 83.2% | 94.9% | 79.7% | 44.8% | 80.2% | 73.6% | 34.1% | 41.2% | -66% | -66% |\n| Clustering (100→10) | 82.9% | 85.4% | 95.2% | 81.6% | 46.7% | 81.8% | 75.9% | 35.8% | 43.5% | +233% | +217% |\n| Temp=1.0 | 81.8% | 84.1% | 94.7% | 79.8% | 45.3% | 80.6% | 73.2% | 34.7% | 41.4% | -66% | -66% |\n| Arith. Sampling | 82.1% | 83.8% | 95.3% | 80.2% | 45.7% | 80.9% | 73.8% | 35.1% | 42.0% | -66% | -66% |\n| Diverse Beam | 82.6% | 84.8% | 95.1% | 80.9% | 46.1% | 81.3% | 74.8% | 35.6% | 42.7% | +0% | +0% |\n| Determinantal Beam | 82.3% | 84.5% | 94.8% | 80.4% | 45.8% | 80.9% | 74.3% | 35.2% | 42.3% | +0% | +0% |\n| **N=25** |\n| SemDiD (Ours) | **84.7%** | **89.2%** | **99.8%** | **93.4%** | **52.1%** | **89.3%** | **91.2%** | **42.8%** | **51.3%** | +180% | +18% |\n| DoLa (Contrastive) | 82.1% | 86.4% | 98.9% | 90.7% | 48.6% | 86.2% | 87.9% | 39.4% | 47.8% | -197% | -195% |\n| Clustering (100→25) | 83.9% | 88.1% | 99.0% | 91.8% | 51.4% | 88.6% | 89.8% | 41.9% | 50.2% | +33% | +32% |\n| Temp=1.0 | 82.8% | 86.9% | 99.1% | 90.2% | 49.3% | 86.8% | 88.1% | 40.5% | 48.4% | -66% | -66% |\n| Arith. Sampling | 83.2% | 87.4% | 99.3% | 91.8% | 50.1% | 87.2% | 88.9% | 41.3% | 49.7% | -66% | -66% |\n| Diverse Beam | 83.1% | 87.2% | 98.6% | 91.1% | 50.7% | 87.5% | 88.9% | 41.2% | 49.6% | +0% | +0% |\n| Determinantal Beam | 82.9% | 86.8% | 98.4% | 90.8% | 50.2% | 87.1% | 88.4% | 40.8% | 49.1% | +0% | +0% |\n| **N=50** |\n| SemDiD (Ours) | **85.3%** | **90.1%** | **99.9%** | **95.2%** | **53.8%** | **91.7%** | **94.3%** | **44.2%** | **53.7%** | +180% | +18% |\n| DoLa (Contrastive) | 82.8% | 87.1% | 99.3% | 92.4% | 49.7% | 87.9% | 90.2% | 40.8% | 49.1% | -66% | -66% |\n| Clustering (100→50) | 84.5% | 88.3% | 99.2% | 94.1% | 51.9% | 90.4% | 92.7% | 43.0% | 52.2% | -33% | -30% |\n| Temp=1.0 | 83.6% | 87.8% | 99.4% | 92.1% | 50.4% | 88.2% | 90.8% | 41.7% | 50.2% | -66% | -66% |\n| Arith. Sampling | 84.1% | 88.6% | 99.5% | 93.7% | 51.8% | 89.4% | 92.1% | 42.7% | 51.9% | -66% | -66% |\n| Diverse Beam | 84.0% | 88.4% | 99.2% | 93.2% | 51.8% | 89.1% | 91.6% | 42.5% | 51.9% | +0% | +0% |\n| Determinantal Beam | 83.8% | 88.1% | 99.0% | 92.8% | 51.5% | 88.6% | 91.1% | 42.1% | 51.4% | +0% | +0% |\n\nThe 70B model results reveal several important patterns. SemDiD consistently outperforms all baselines, with particularly strong improvements on complex reasoning tasks like BBH (+1.1%), GSM8K (+0.5%), and Minerva Math (+1.6% over the best baseline at N=25). \n\nNotably, SemDiD shows only a +16% latency increase with 70B models, significantly lower than the +27% observed with 3B models, because embedding computation becomes negligible compared to 70B model forward passes. Although SemDiD appears to use more computation on the surface, the KV-reuse within groups reduces the inference cost per token by 80-90%, since these beam share the same prefix in exploration. Additionally, clustering-based methods often require more samples to match SemDiD's diversity, significantly increasing their actual computational cost. When accounting for cache efficiency and quality-adjusted sample requirements, SemDiD becomes computationally competitive while delivering superior semantic diversity\n\nWe hope these analyses could address your concerns about novelty, baseline comparisons, and scalability. We will incorporate these experiments into revised version.\n\n[1] Chuang et al. \"DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models.\" ICLR 2023.\n\n[2] Li et al. \"Contrastive Decoding: Open-ended Text Generation as Optimization.\" ACL 2023"}, {"review_id": "gEDAv61bag", "reviewer": "Reviewer_sicL", "summary": "This paper addresses the problem of LLMs generating multiple different diverse responses. Current\nmethods do more rephrasing than providing original different responses to the same idea. The\nsolution in the paper operates in meaning space instead of word space which allows the LLM to\nexplore more diverse embeddings, making sure the LLM provides different solutions to the same\nidea. The paper outperforms baseline methods by increasing accuracy, however the complexity and\ncomputational overhead is increased as well.", "questions": "1. The hyperparameter choices in Eq 1 (βseq, βsent) seem somewhat arbitrary, how sensitive is the\nmethod to different values of (βseq, βsent)?\n\n2. Since the baseline methods use much less computation than your method, could you run an\nexperiment where the baseline methods use equivalent computation budget? Like if you ran k\nindependent temperature sampling processes and then selected the most diverse subset post hoc.\nThis would prove that the improvements in performance are not just from brute force exploration but\nfrom your semantic guidance method.\n\n3. When you measure coverage in best of N evaluations, is the method trading off having several\ngood answers instead of having one good answer and several poor ones? It would be interesting to\nsee the quality distribution of all generated answers.\n\n4. Theorem 3 assumes orthogonal directions in embedding space translate to semantic diversity in\ngenerated text. Can you empirically validate this assumption? Maybe show examples where\northogonal starting directions led to genuinely different reasoning paths vs cases where they\nconverged to similar solutions despite different starting directions? Was there a case where your\nmethod produced semantically diverse but all wrong answers?\n\n5. Do you believe the benefits of this method will still hold with much larger models like 70B+? I\nunderstand due to limited resources this experiment is harder to run. But do larger models reduce\nbenefits since they might naturally generate more diverse outputs?", "limitations": "Addressed limitations (complexity overhead) but no mention of positive or negative societal impact.\nNot needed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nProblem is well defined with strong motivation. It is important to make a distinction between lexical\ndiversity vs semantic diversity. The implementation is mostly technically sound (concerns mentioned\nbelow). The theory is validated by experimental results on 9 benchmarks for Best of N and RLHF\nalgorithms. Personally I feel its application in biomedical queries could allow for better answers,\nwhere quality is preferred over speed of generation.\n\nPaper is well written with clear figures. Figure 1 effectively explains the approach. I like the case\nstudies provided in the appendix, they give good insight. The limitations of baseline methods are\naddressed.\nGood engineering by combining known methods like beam search and embedding based similarity\nfor semantic-guided decoding achieving better performance. The method is also post-hoc which\nallows it to be plugged into existing LLMs, which allows for good generalizability.\n\nWeakness:\n\nA clear weakness mentioned in the paper is the introduction of 35% latency compared to regular\nbeam search method which might limit practical use. There are also a lot of hyperparameters to tune\nlike exploration width, beam size, transition timing etc.\n\nThe semantic diversity relies on embedding cosine distance which might not capture complete\nsemantic difference. Maybe there is room to explore other semantic similarity metrics. There is a\nchance of this method failing when general embeddings do not capture domain relevant semantic\nrelationships accurately.\n\nThe theoretical guarantees in Theorem 3 rely on assumptions that orthogonal guidance directions in\nembedding space will necessarily lead to semantically diverse final outputs. I am not entirely\nconvinced of it.\n\nThe method relies on computing embeddings of partial sequences and using these to make token\nlevel decisions. But the sentence embedding models are trained on large chunks of text and not\npartial sequences. It's possible that the method might make token level choices based on unreliable\nsequence level embeddings. This is something to address.\n\nGood engineering work which solves a real problem but seems a little incremental with minimal\ntechnical novelty.", "was_revised": true, "final_justification": "After reading the other reviews and rebuttal, I reaffirm my original assessment of a weak accept. The paper makes a moderate contribution with reasonable execution of the idea.", "rebuttal": "Thank you for recognizing we address an important real-world problem. Below we provide detailed responses for your concerns.\n\n> **W1 & Q2 & Q5: Concern about computational latency and comparison under equivalent computational budget and scalability to larger models (70B+)**\n\nWe acknowledge the computational overhead concern and provide comparison using **equivalent latency budgets** rather than equivalent sample counts. Experiments use **Qwen-2.5-70B** to simultaneously address scalability concerns. For parallel decoding methods, we first generate a larger candidate pool, then select N outputs using embedding-based clustering, and for Diverse/Determinantal Beam Search, we increase beam size from the default 3.\n\n| Method | ARC-Challenge | BBH | GSM8K | Minerva Math | CoQA | PubMedQA | MMLU-Pro+ | WMT16 EN-DE | WMT16 DE-EN | Compute Overhead | Latency Increase |\n|---|---|---|---|---|---|---|---|---|---|---|---|\n| **N=10** |\n| SemDiD (Ours) | **83.4%** | **86.1%** | **96.8%** | **82.3%** | **47.2%** | **82.1%** | **76.8%** | **36.4%** | **44.1%** | +180% | +18% |\n| Temp=1.0 (35→10) | 82.4% | 85.0% | 94.9% | 80.6% | 46.1% | 81.0% | 73.9% | 35.2% | 41.8% | +18% | +18% |\n| Arith. Sampling (35→10) | 82.6% | 84.7% | 95.6% | 81.5% | 46.2% | 81.1% | 75.0% | 35.4% | 42.8% | +18% | +18% |\n| Diverse Beam (beam=4) | 82.4% | 84.9% | 95.3% | 80.5% | 46.7% | 81.4% | 74.3% | 35.2% | 42.8% | +33% | +33% |\n| Determinantal Beam (beam=4) | 82.7% | 84.1% | 94.9% | 80.4% | 45.8% | 81.1% | 74.7% | 35.3% | 42.3% | +33% | +33% |\n| **N=25** |\n| SemDiD (Ours) | **84.7%** | **89.2%** | **99.8%** | **93.4%** | **52.1%** | **89.3%** | **91.2%** | **42.8%** | **51.3%** | +180% | +18% |\n| Temp=1.0 (88→25) | 83.6% | 87.5% | 99.1% | 91.2% | 49.9% | 87.4% | 88.8% | 41.0% | 49.3% | +18% | +18% |\n| Arith. Sampling (88→25) | 83.9% | 87.8% | 99.4% | 91.9% | 51.2% | 88.0% | 89.4% | 41.7% | 50.0% | +18% | +18% |\n| Diverse Beam (beam=4) | 83.2% | 87.2% | 98.9% | 91.0% | 50.8% | 87.5% | 88.9% | 41.5% | 49.1% | +33% | +33% |\n| Determinantal Beam (beam=4) | 83.0% | 86.8% | 98.4% | 90.5% | 50.2% | 87.3% | 88.4% | 41.3% | 49.0% | +33% | +33% |\n| **N=50** |\n| SemDiD (Ours) | **85.3%** | **90.1%** | **99.9%** | **95.2%** | **53.8%** | **91.7%** | **94.3%** | **44.2%** | **53.7%** | +180% | +18% |\n| Temp=1.0 (177→50) | 84.6% | 88.4% | 99.2% | 94.3% | 51.7% | 89.9% | 92.8% | 43.1% | 52.3% | +18% | +18% |\n| Arith. Sampling (177→50) | 84.4% | 89.3% | 99.6% | 94.1% | 51.9% | 91.0% | 92.3% | 42.9% | 52.4% | +18% | +18% |\n| Diverse Beam (beam=4) | 84.3% | 88.2% | 99.2% | 93.2% | 51.7% | 89.1% | 91.4% | 42.5% | 51.9% | +33% | +33% |\n| Determinantal Beam (beam=4) | 83.8% | 88.6% | 99.0% | 92.8% | 51.5% | 88.6% | 91.0% | 42.1% | 51.6% | +33% | +33% |\n\nSemDiD demonstrates improved efficiency scaling with larger models. The actual latency increase drops from +27% (3B) to +18% (70B) because embedding computation becomes negligible relative to the 70B forward passes, while the semantic guidance benefits remain consistent.\n\nWhile SemDiD appears to use more computation on the surface, the majority of operations benefit from KV-cache reuse since beams within groups share common prefixes, resulting in minimal actual latency increase. For parallel decoding methods, diminishing returns become evident as generating larger candidate pools leads to rapidly increasing costs with minimal diversity gains due to inability to reuse KV-cache. For Diverse Beam Search, simply increasing beam size yields negligible improvements while substantially increasing overhead.\n\nSemDiD benefits from systematic exploration in semantic space through directional guidance and inter-group repulsion, efficiently utilizing every computational allocation to maximize semantic coverage rather than generating redundant variations.\n\n> **W1 & Q1: Over-complex parameter design and arbitrary parameter selection**\n\nWe understand the complexity concerns and provide clarification on hyperparameter design and robustness. \n\nThe parameters can be categorized as:\n\n- **Inherited Parameters**: Temperature, top-p, $N$ (groups), and $b$ (beam size) come from standard Group Beam Search, not SemDiD additions.\n\n- **Automatically Derivable**: Position bias parameters $\\beta_{seq}$ and $\\beta_{sent}$ can be fitted from probability-position curves (Figures 3-4) using scipy.curve_fit. Saturation threshold $\\tau$ is derived from probability-quality analysis (Figure 2) and set to -0.8 based on empirical log probability-accuracy relationships.\n\n- **Resource-Dependent**: Exploration width $E_t$ and beam size $b$ balance exploration versus computational cost, with clear \"sweet spots\" shown in Appendix D.2.\n\nThe hyperparameter choices in Eq 1 ($\\beta_{seq}$, $\\beta_{sent}$) are not arbitrary but can be systematically derived from probability-position analysis. We analyze parameter sensitivity on BON (N=25) by Qwen-2.5-3B:\n\n| $\\beta_{seq}$ \\ $\\beta_{sent}$ | 0.003 | 0.004 | **0.005** | 0.006 | 0.007 |\n|---|---|---|---|---|---|\n| **GSM8K Coverage (%)** |\n| 0.0006 | 98.1 | 98.0 | 98.1 | 98.0 | 97.9 |\n| 0.0008 | 98.0 | 98.1 | 98.2 | 98.1 | 98.0 |\n| 0.0010 | 98.1 | 98.2 | 98.1 | 98.2 | 98.1 |\n| **0.0012** | 98.2 | 98.2 | **98.2** | 98.2 | 98.1 |\n| 0.0014 | 98.2 | 98.2 | 98.2 | 98.1 | 98.0 |\n\n| $\\beta_{seq}$ \\ $\\beta_{sent}$ | 0.003 | 0.004 | **0.005** | 0.006 | 0.007 |\n|---|---|---|---|---|---|\n| **BBH Coverage (%)** |\n| 0.0006 | 85.6 | 85.5 | 85.6 | 85.5 | 85.4 |\n| **0.0008** | 85.5 | 85.6 | **85.7** | 85.6 | 85.5 |\n| 0.0010 | 85.6 | 85.7 | 85.6 | 85.6 | 85.6 |\n| 0.0012 | 85.6 | 85.7 | 85.6 | 85.6 | 85.5 |\n| 0.0014 | 85.5 | 85.6 | 85.5 | 85.7 | 85.4 |\n\nBold values indicate curve-fitted optimal parameters: GSM8K ($\\beta_{seq}=0.0012$, $\\beta_{sent}=0.005$) and BBH ($\\beta_{seq}=0.0008$, $\\beta_{sent}=0.005$). The results demonstrate that (1) optimal parameters remain reasonably consistent across datasets, (2) performance is robust within reasonable parameter ranges with <=0.3% variation, and (3) our default values (0.001, 0.005) lie within the optimal regions, eliminating the need for task-specific tuning.\n\n> **W2 & W4: Cosine distance cannot capture complete semantic differences, and partial sentence embeddings may not reliably guide token-level decisions**\n\nWe agree that cosine distance in embedding space does not necessarily equate to semantic differences or different solution approaches. Additionally, partial sentence embeddings may exacerbate this issue.\n\nFortunately, recent advances in LLM-based sentence models (like Qwen3 Embedding) have introduced new paradigms. They allow the same sentence to be combined with different task instructions to obtain corresponding representations in embedding space.\n\nFor example: \n\n\"Classify the mathematical approach used in this solution: Question: A rectangular garden has a perimeter of 40 meters and an area of 96 square meters. What are its dimensions? Answer: Let length = l and width = w. From perimeter: 2l + 2w = 40, so l + w = 20, which gives us...\"\n\nVs.\n\n\"Classify the mathematical approach used in this solution: Question: A rectangular garden has a perimeter of 40 meters and an area of 96 square meters. What are its dimensions? Answer: Since area = 96, I need two numbers that multiply to 96. The factor pairs are: (1,96), (2,48), (3,32), (4,24), (6,16), (8,12). Now checking which pair has...\"\n\nIn this way, embeddings leverage the foundational capabilities of LLM and can focus on solution difference.\n\nFurthermore, we prioritize LLM-based embedding models with support of mean pooling, like NovaSearch/stella_en_1.5B_v5 (as Appendix C.2.2). Mean pooling averages all token embeddings, providing more stable representations for partial sentences compared to special token-based approaches.\n\n> **W3 & Q3 & Q4: Theoretical assumptions regarding orthogonal guidance directions leading to semantic diversity. Whether method trades off multiple good answers for one good answer and several poor ones**\n\nAs mentioned in previous response, LLM-based embeddings can better capture solution approach differences through instructions to classify solutions, enabling more reliable semantic diversity assessment.\n\nWe examined 20 GSM8K and BBH examples under N=10 setting and found an average of 2.3 distinct solution approaches with embedding distance 0.388. Notably, the embedding distances between different approaches reach 0.435, significantly larger than the 0.231 distance within the same approach. Even character-level 4-gram distances demonstrate similar patterns, with inter-approach distance of 0.884 exceeding intra-approach distance of 0.637. \n\nIndeed, orthogonal directional guidance cannot guarantee different solutions, but the increased similarity distances during decoding help generate different reasoning approaches. When we remove such guidance, it produces an average of 2.1 distinct solutions with semantic distance 0.365.\n\nIn contrast, standard temperature decoding with N=10 produces only 1.4 distinct solutions with semantic distance 0.253.\n\nWhile most solutions still follow similar reasoning patterns, the improvement from 1.4 to 2.3 distinct approaches represents meaningful gains in exploring alternative paths.\n\nRegarding correctness, for more challenging datasets like Minerva Math, we observe cases where N=10 produces 2-3 different approaches but all results are incorrect across SemDiD and baselines. This reflects the foundation model's limited capability rather than a flaw in our diversity mechanism.\n\nWe also provide detailed output samples and correctness analysis in Appendix I.3 and Figure 15 (Lines 759-799) to examine the quality distribution. Additionally, Appendix E presents Accuracy Evaluation in Best-of-N, showing performance when using selection models to choose one response in practical scenarios. Results show SemDiD consistently outperforms baselines in both coverage and accuracy metrics, proving it generates multiple high-quality diverse answers rather than trading one good answer for several poor ones."}, {"review_id": "VTNYMLr6T9", "reviewer": "Reviewer_xF5D", "summary": "The paper proposes Semantic-guided Diverse Decoding, introducing a beam search variant with explicit semantic diversity - contrary to previous methods focussing on lexical diversity.\nBy utilizing different scores, the method optimizes beams for semantic diversity and quality, balancing those objectives through harmonic gain-based balancing.\nExtensive experiments both for Best-of-N experiments and RLHF training show the efficacy of the approach.", "questions": "See weaknesses.", "limitations": "The main limitations, namely the reliance on an external lightweight embedding model and the increase in runtime of decoding are not properly discussed in the main paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strenghts\n\n- The method is well motivated and the scores make sense intuitively. Also the harmonic balancing and restricting to a \"high enough\" probability region is reasonable. While there are a lot of components and hyperparameters that can be adjusted when applying SemDiD, this is not necessarily a bad thing given the often very diverse nature of tasks a trained base LLM is applied to.\n- Directional guidance is a strong idea to overcome the lack of diversity at the beginning of generation.\n- There are plenty of experimental investigations on a broad range of tasks that underpin the value of semantically diverse generations provided by SimDiD.\n\n### Weaknesses\n\n- Figure 5 is lacking a lot of details. Which embedding space is used? Which projection method used? How is clustering performed? How is the probability magnitude computed - does it take into account answer length?\n- Important aspects such as the percentile-based normalization of individual scores only discussed in appendix. I would move the theorems on quality and diversity guarantees and the proposition into the appendix with a smaller summary theorem in the main paper and rather move those critical aspects into the main paper. Those are interesting insights, but to me it appears the main selling point of the method is empirical performance not theoretical guarantees, thus I would adjust the presentation to provide sufficient details on the empirical implementation.\n- An ablation on the importance of the individual scores is lacking, the same with a sensitivity analysis of important hyperparameters such as T_trans, lambda and epsilon.\n\n### Remarks\n\nMany parts of the methodology remind me of the sampling strategy proposed in Aichberger et al. 2025. While they use the gradient signal and focus on uncertainty estimation, it could be worthwile to discuss in the related work section. Evaluating SemDiD for uncertainty estimation could be an interesting direction for future work.\n\nAichberger, L., Schweighofer, K., Ielanskyi, M., & Hochreiter, S. (2025). Improving uncertainty estimation through semantically diverse language generation.", "was_revised": true, "final_justification": "All the weaknesses I pointed out have been resolved sufficiently.\nAs also pointed out in the public comment, I see a lot of merit in the ideas proposed in this work also beyond mere accuracy gains and am thus in favor of acceptance.", "rebuttal": "We thanks for your valuable feedback and will response them in below.\n\n> **W1:  Figure 5 is lacking a lot of details. Which embedding space is used? Which projection method used? How is clustering performed? How is the probability magnitude computed - does it take into account answer length?**\n\nWe apologize for the insufficient details in the main text. Complete experimental settings are provided in Appendix B.3 Semantic Space Visualization (Lines 456-479). Specifically: we use NovaSearch/stella_en_1.5B_v5 embeddings (Line 540) with t-SNE projection, K-means clustering (K=10), and position-debiased probability (Equation 1) that accounts for answer length through sequence position decay rate $\\beta_{seq}$. The probability magnitude shown represents the average debiased probability across all tokens in each response.\n\n> **W2: Important aspects such as the percentile-based normalization of individual scores only discussed in appendix. I would move the theorems on quality and diversity guarantees and the proposition into the appendix with a smaller summary theorem in the main paper and rather move those critical aspects into the main paper.**\n\nWe appreciate your constructive suggestion regarding presentation balance. Due to space constraints, we prioritized theoretical analysis in the main text, but you are correct that critical implementation details deserve more prominence. We agree that SemDiD's primary value lies in its empirical effectiveness, and we will restructure to emphasize practical implementation aspects while moving detailed proofs to the appendix for better readability.\n\n> **W3: An ablation on the importance of the individual scores is lacking, the same with a sensitivity analysis of important hyperparameters such as T_trans, lambda and epsilon.**\n\nWe thank the reviewer for pointing out this. We have conducted comprehensive ablation studies and sensitivity analyses for existing components, and provide additional experiments for detailed hyperparameter analysis.\n\nRegarding individual scores, we provide systematic ablation experiments in Appendix H (Table 4, lines 702-719). Each system component corresponds to its scoring function: directional guidance ($S_{dir}$), inter-group repulsion ($S_{rep}$), and harmonic gain ($S_{combined}$). We also provide supplementary experiments analyzing complete removal of scoring categories:\n\n| Method | Avg. Coverage (25 samples) | GRPO-GSM8K Accuracy |\n|---|---|---|\n| Full SemDiD | 74.2% | 81.6% |\n| - Directional Guidance ($S_{dir}$) | 72.1% (-2.1%) | 79.9% (-1.7%) |\n| - Inter-Group Repulsion ($S_{rep}$) | 71.5% (-2.7%) | 79.6% (-2.0%) |\n| - Debiased Probability | 73.3% (-0.9%) | 80.5% (-1.1%) |\n| - Harmonic Gain ($S_{combined}$) | 71.2% (-3.0%) | 79.3% (-2.3%) |\n| **- Quality Scoring ($S_{quality}$)** | **65.8% (-8.4%)** | **77.1% (-4.5%)** |\n| **- Diversity Scoring ($S_{div}$)** | **57.9% (-16.3%)** | **71.9% (-9.7%)** |\n| Only Probability (Greedy) | 57.2% (-17.0%) | 71.8% (-9.8%) |\n\nThe results demonstrate that removing diversity scoring has the most dramatic impact, with coverage dropping 16.3%, confirming that semantic diversity is crucial for effective exploration.\n\n**Regarding hyperparameter sensitivity**, we analyze the parameters in four categories:\n\n### Category 1: Automatically Derivable Parameters\n\nFor the position bias parameters, the hyperparameter choices in Equation 1 ($\\beta_{seq}$, $\\beta_{sent}$) are not arbitrary but can be systematically derived from probability-position analysis shown in Figures 3-4 in Appendix A.3. These parameters can be automatically fitted using scipy.curve_fit from probability-position curves, as these patterns remain consistent across tasks. We provide additional sensitivity analysis on BON (N=25) using Qwen-2.5-3B:\n\n$\\beta_{seq}$ and $\\beta_{sent}$ Parameter Sensitivity Analysis (GSM8K Coverage):\n\n| $\\beta_{seq}$ \\ $\\beta_{sent}$ | 0.003 | 0.004 | **0.005** | 0.006 | 0.007 |\n|---|---|---|---|---|---|\n| 0.0006 | 98.1 | 98.0 | 98.1 | 98.0 | 97.9 |\n| 0.0008 | 98.0 | 98.1 | 98.2 | 98.1 | 98.0 |\n| 0.0010 | 98.1 | 98.2 | 98.1 | 98.2 | 98.1 |\n| **0.0012** | 98.2 | 98.2 | **98.2** | 98.2 | 98.1 |\n| 0.0014 | 98.2 | 98.2 | 98.2 | 98.1 | 98.0 |\n\n### Category 2: Resource-Dependent Parameters\n\nFor the lookahead depth parameter $L_{max}$, semantic diversity assessment requires embedding model forward passes, making token-by-token evaluation computationally expensive. We supplement our analysis in Appendix D.2-D.3 with experiments using Qwen-2.5-3B across different $L_{max}$ values:\n\n| $L_{max}$ | GSM8K Coverage (N=25) | ARC Coverage (N=25) | Computational Overhead |\n|---|---|---|---|\n| 5 | 95.3% | 80.1% | +15% |\n| **10** | **98.1%** | **82.4%** | +25% |\n| 15 | 98.2% | 82.6% | +35% |\n| 20 | 98.6% | 82.7% | +45% |\n\nPerformance saturates around $L_{max}=10$, providing clear guidance for practical deployment without requiring extensive parameter search.\n\n### Category 3: Stage-Aware Transition Parameters\n\nWe apologize that explicit ablation of $T_{trans}$ values was not included in the main paper and provide this comprehensive analysis here. Parameter $T_{trans}$ controls the transition point from directional guidance to inter-group repulsion (Equation 8: $\\alpha_t = \\min(1, \\frac{t}{T_{trans}})$). We conducted systematic experiments varying $T_{trans}$ across multiple datasets:\n\n| $T_{trans}$ | GSM8K Coverage | GSM8K Accuracy | ARC Coverage | ARC Accuracy | MMLU-Pro+ Coverage |\n|---|---|---|---|---|---|\n| 5 | 97.2% | 76.8% | 81.2% | 79.1% | 81.4% |\n| **10 (default)** | **98.1%** | **77.5%** | **82.4%** | **82.0%** | **82.6%** |\n| 15 | 97.9% | 77.2% | 82.1% | 81.7% | 82.3% |\n| 20 | 97.5% | 76.9% | 81.7% | 81.2% | 81.8% |\n| 25 | 97.2% | 76.8% | 81.8% | 80.9% | 81.6% |\n\nThe results show clear optimal performance at $T_{trans} = 10$, corresponding to the typical number of tokens needed to establish meaningful semantic context before semantic trajectories become distinguishable.\n\n### Category 4: Quality-Diversity Balancing Parameters\n\nThe quality relaxation parameter $\\gamma$ and harmonic strength $\\lambda$ control the trade-off between maintaining quality thresholds and pursuing semantic diversity. While these parameters are discussed conceptually in Section 3.3, we provide detailed sensitivity analysis here by varying each parameter independently:\n\nEffect of $\\gamma$ (with $\\lambda=2.0$ fixed):\n\n| Task | $\\gamma=0.15$ | $\\gamma=0.20$ | $\\gamma=0.25$ | $\\gamma=0.30$ | $\\gamma=0.35$ |\n|---|---|---|---|---|---|\n| GSM8K Coverage (N=25) | 96.6% | 97.1% | **98.1%** | 97.8% | 97.4% |\n| GSM8K Accuracy (N=25) | 75.9% | 76.6% | **77.5%** | 77.2% | 77.2% |\n| WMT16 Coverage (N=25) | 36.7% | 36.8% | **37.2%** | 36.9% | 36.7% |\n| WMT16 Accuracy (N=25) | 20.2% | 20.4% | **20.7%** | 20.5% | 20.3% |\n\nEffect of $\\lambda$ (with $\\gamma=0.25$ fixed):\n\n| Task | $\\lambda=1.0$ | $\\lambda=1.5$ | $\\lambda=2.0$ | $\\lambda=2.5$ | $\\lambda=3.0$ |\n|---|---|---|---|---|---|\n| GSM8K Coverage (N=25) | 96.8% | 97.4% | **98.1%** | 97.9% | 97.6% |\n| GSM8K Accuracy (N=25) | 77.3% | 77.1% | **77.5%** | 77.3% | 76.9% |\n| WMT16 Coverage (N=25) | 36.8% | 36.5% | **37.2%** | 36.9% | 36.7% |\n| WMT16 Accuracy (N=25) | 20.3% | 20.6% | **20.7%** | 20.7% | 20.6% |\n\nThe results demonstrate that our default settings ($\\gamma=0.25$, $\\lambda=2.0$) consistently achieve near-optimal performance across different tasks, with performance remaining stable within reasonable parameter ranges.\n\nFor practitioners seeking to deploy SemDiD, we recommend starting with our provided default parameters for initial implementation, adjusting $E_t$, $b$, and $N$ based on available computational budget, and fine-tuning $\\gamma$ and $\\lambda$ only for highly specialized applications where task-specific optimization is critical. We hope these comprehensive analyses demonstrate that while SemDiD introduces additional parameters, most are either automatically derivable or have well-defined optimal ranges with clear empirical guidance, making the method practical for real-world deployment.\n\n> **Remark: Many parts of the methodology remind me of the sampling strategy proposed in Aichberger et al. 2025. While they use the gradient signal and focus on uncertainty estimation, it could be worthwile to discuss in the related work section. Evaluating SemDiD for uncertainty estimation could be an interesting direction for future work.**\n\nThank you for highlighting these important concurrent works. The following two works correlate with SemDiD, both of which explore semantic diversity in language generation through different views:\n\n- **Aichberger et al. (2025)** [1] uses gradient-based attribution scores to identify semantically critical tokens for uncertainty estimation, employing importance sampling with NLI-guided token substitution.\n- **Zhu et al. (2025)** [2] proposes controlled embedding perturbation at the first token position combined with Bayesian optimization for reasoning in latent space.\n\nWhile these works share our motivation for semantic diversity, SemDiD differs in key aspects: (1) we operate during beam search decoding rather than post-hoc token substitution or first-token perturbation, (2) our approach balances quality and diversity through harmonic optimization rather than focusing solely on uncertainty or correctness, and (3) we target Best-of-N and RLHF applications rather than uncertainty estimation or reasoning tasks.\n\nWe will incorporate these works into our related work discussion and acknowledge the potential of combining our semantic guidance with their gradient-based attribution methods.\n\n[1] Aichberger, Lukas, et al. \"Improving uncertainty estimation through semantically diverse language generation.\" ICLR. 2025.\n\n[2] Zhu, Qinglin, et al. \"Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Exploration.\" arXiv:2505.24688 (2025)."}]

Author rebuttal:

Meta-review: This paper aims to *semantically* diversify LLMs’ generation rather than lexical diversity. The proposed method, SemDiD, operates in the embedding space to promote semantic diversity. The motivation is convincing, and the potential applications of such a semantically diverse decoding are clear.

The main concerns raised by reviewers included (i) the introduction of several hyperparameters without sufficient ablation studies, and (ii) the limited set of baseline comparisons. The authors responded thoroughly by providing additional experimental results that directly addressed these points. All reviewers indicated that they were satisfied with the rebuttal and subsequently improved their ratings. Importantly, the reviewers requested that the additional experimental results be incorporated into the final version of the paper.

Taking into account the reviewers’ evaluations, the constructive discussion, and the strengthened evidence provided by the authors, I recommend acceptance of this submission.
Decision: Accept (poster)