Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs
Authors: Sergey Troshin
Wafaa Mohammed
Yan Meng
Christof Monz
Antske Fokkens
Vlad Niculae
Venue: COLM 2025 (2025)
Keywords: Natural Language Processing
Large Language Models
Text Generation
Sampling Methods
Truncation Sampling
Stochastic Sampling
Min-p Sampling
Top-p Sampling
Temperature Sampling
Decoding Methods
LLMs reasoning
TL;DR: We propose selective sampling, a method that dynamically switches between greedy and high-temperature sampling based on a sampling risk metric.

Abstract: Diversity is essential for language models to generate creative outputs. Temperature-based sampling is a common strategy to increase diversity. However, for tasks that require high precision, e.g., mathematical reasoning, uncontrolled high temperature sampling, e.g., min-$p$ or top-$p$ lowers reasoning quality. 
We demonstrate that the loss of accuracy is caused by sampling incorrect continuations in sensitive positions when entropy is high. To address this, in this paper, we propose selective sampling, a method that dynamically switches between greedy and high-temperature sampling based on a sampling risk metric. This risk metric estimates the likelihood of output errors when applying high temperature sampling on the current token position. We train a lightweight classifier on a small subset of verifiable problems to predict sampling risk. The classifier can be integrated with the base language model with minimal latency overhead. Experiments on mathematical reasoning tasks show that selective sampling improves the quality-diversity trade-off, even under high-temperature settings.

Published as a conference paper at COLM 2025
Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs
Sergey Troshin ':1, Wafaa Mohammed Yan Meng"),
Christof Monz', Antske Fokkens', Vlad Niculae'
LLanguage Technology Lab, University of Amsterdam
2 Computational Linguistics and Text Mining Lab, Vrije Universiteit Amsterdam
Abstract
Diversity is an essential metric for evaluating the creativity of outputs generated by language models. Temperature-based sampling is a common strategy to increase diversity. However, for tasks that require high precision, egg, mathematical reasoning, uncontrolled high temperature sampling, e.g. min-p or top-p, degrades reasoning quality. We demonstrate that the loss of accuracy is caused by sampling incorrect continuations in sensitive decoding positions. To address this, in this paper, we propose selective sampling, method that dynamically switches between greedy and high- temperature sampling based on a sampling risk metric. This risk metric estimates the likelihood of output errors when applying high-temperature sampling on the current token position. To predict sampling risk, we train lightweight classifier a small subset of verifiable problems. The trained classifier can be integrated with the base language model with minimal latency overhead. Experiments on mathematical reasoning tasks demonstrate that selective sampling enhances the quality-diversity trade- off, even in high-temperature settings.
1 Introduction
Recently, large language models (LLMs) have demonstrated unprecedented capabilities in mathematical reasoning tasks by efficiently harnessing task-specific rewards (OpenAI et al., 2024; Dubey et al., 2024; Yang et al., 2024; DeepSeek-Al et al., 2025). To maintain high reasoning accuracy, prior work has primarily focused on deterministic decoding, e.g,, greedy decoding, which generates outputs with the highest probability. However, this approach often leads to pathological solutions by over-optimizing for single objective, i.e., accuracy, at the expense of other desirable properties (Hashimoto et al., 2019). In particular, determínistic decoding tends to reduce the diversity in generated outputs (Le Bronnec et al., 2024; Kirk et al., 2024). This is problematic because human preferences are not always aligned with those of greedy generations due lack of diversity and fluency (Zhang et al., 2021; Holtzman et al., 2020).
Diversity is important for controllability in large language models: by having access probabilistic sampler, user can first generate multiple outputs, and then select the best ones using a task-specific metric (Stiennon et al., 2020); a user can improve parallel exploration of potential solutions (Yao et al., 2023), or control the generation process from black- box sampler according to additional constraints (Mudgal et al., 2024; Deng Raffel, 2023; Troshin et al., 2024). Temperature sampling is a common strategy to increase creativity and diversity in many LLM inference frameworks such as vLLM (Kwon et al., 2023) or Hugging Face (Wolf et al., 2020). However, it often comes at the cost of lower task accuracy compared to deterministic decoding (Shi et al., 2024). This trade-off has motivated research on how to narrow the quality gap between deterministic and temperature sampling (Holtzman et al., 2020; Basu et al., 2021; Hewitt et al., 2022; Minh et al., 2025).
* Authors contributed correpooeor to serj. trossineertttonnn.e

---

Published as a conference paper at COLM 2025
Previous sampling methods, such as top-p and min-p sampling, truncate the next-token distribution based on model confidence in order to improve the quality-diversity trade-off in language generation. These sampling methods prioritize high-likelihood candidates to improve output precision (Meister et al., 2023). However, one limitation of the truncation sampling approaches is that they solely rely model confidence to sample potential candidate tokens. While the shape of the model output distribution over candidate tokens can represent uncertainty, is hard to distinguish between different types of uncertainty (Baan et al., 2023), namely between variability due to numerous plausible continuations (Giulianelli al., 2023) and uncertainty about what the correct answer is. For example, if model assigns high probability to two different answers, are both of these answers plausible, or the model not confident which one is correct in a given context? As we demonstrate in $3, sampling at certain decoding time steps can lead to incorrect continuations in cases where the greedy continuations are correct. This shows that it is important to adopt different decoding strategies across decoding positions to balance quality and diversity.
To balance both the diversity and quality of LLMs output, we propose selective sampling. method that dynamically switches between greedy and temperature sampling based on a sampling risk metric. This metric measures the likelihood of output errors for a given decoding timestep when applying tem temperature sampling to it. We train lightweight classifier on a small subset of verifiable problems to predict sampling risk. Our approach preserves the original model outputs and is easy to implement, with the classifier integrating into the base language model with minimal latency overhead. We empirically demonstrate that temperature sampling with our classifier results better quality- diversity trade-off compared to commonly used suncation-basee and entropy-based sampling methods, both under standard and high-temperature settings.
The structure of our paper is as follows: In $2 we review the fundamental theoretical concepts behind the sampling methods explored in our study. In $3, we highlight the gap we aim to fill by analyzing when existing sampling methods fail. In 84, we detail the intuition and implementation of the selective sampling method. In $5 we outline the details of our experimental setup, including models, tasks, and evaluation metrics. Then, in $6, we present and discuss our results and findings. We describe related work in 57. Finally, we summarize our conclusions in 58.
2 Preliminaries
In this section, we cover key concepts for sampling mechanisms, including temperature sampling and its variants.
2.1 Temperature Sampling
At each step of decoding, the base model observes an already generated prefix x, and predicts the logits 2 E IRII. Logits are then used to form a sampling distribution p(l|x), which may involve filtering, temperature, and other modifications (Minh et al., 2025; Holtzman et al., 2020; Basu et al., 2021; Hewitt et al., 2022). In particular, temperature sampling uses scalar parameter to rescale the probabilities to make the distribution more peaked (low temperature) or flatter (high temperature). The next token is then sampled from categorical distribution defined by p(o(x).
2.2 Temperature Sampling Variants
There are many heuristics for selecting a short-list of most likely token candidates, and they rely on different statistics of the model distribution. Top-k always selects top most probable tokens (Fan et al., 2018). Top-p (Holtzman et al., 2020) selects top tokens whose cumulative probability exceeds a hyperparameter p. Recently, Minh et al. (2025) proposed min-p that uses the discounted probability of the top-1 token to define the probability threshold. e-sampling (Hewitt et al., 2022) allows any token with a probability greater than Some of the methods rely on the Shannon entropy of the model distribution (Shannon,
2

---

Published as a conference paper at COLM 2025
Question: qualified for spot on soccer team ... and a water bottle for V3. How much spend athletic gear? Answer: 115
CoT Continuation: Yara bought a jeneyy für Y24, The leggings cost 113 more than the jeneey... So the cleats cost
Probebiitt
Greedy .... So the cleats cost 20. The headbands cust Y11 and she got a discount ... The
final answer is 115
05
Non-Greedy X:
cleats cost 20 headbands cost Y11 and got #1
Vocalaalary
discount... ...lhe final answer is 105
Figure 1: High sampling risk example for the current decoding position. CoT Continuation is generated by greedy decoding. Greedy continuation 20 results in correct answer while non-greedy continuation 30 leads to an incorrect answer. The full example is shown in Appendix G, Figure 15.
1948), among which 7-sampling truncates words below an entropy-dependent probability threshold (Hewitt et al., 2022). Zhang et al. (2024b) propose EDT (Entropy-based Dynamic Temperature) sampling in which they dynamically adjust the temperature based on entropy. These methods generally lead to better results than sampling directly from the unmodified p(v| (()]x)), and improve diversity and accuracy. In $6, we find that even the best among them still exhibit quality degradation when increasing the temperature. We provide a more detailed description of the aforementioned methods in Appendix C.
3 When Do the Sampling Methods Fail?
In this section, we analyze the limitations of widely used sampling methods introduced in $2 by focusing their impact specific decoding time steps. We aim to identify sensitive positions where sampling significantly increases the risk of generating incorrect outputs compared greedy decoding. To quantify this, we propose a metric sampling risk that measures the likelihood of output errors when replacing greedy tokens with sampled ones. Our findings highlight the need for adaptive decoding strategies that switch selectively between greedy and sampling to maintain output quality.
3.1 Sampling Risk
Motivated by the concept of regret in reinforcement learning (Bubeck & Cesa-Bianchi, 2012), we want to know the risk of choosing sampling method the current state compared to greedy decoding. We define sampling risk of a current prefix x as follows:
s-risk(x) = R(x)- - Ev-p [R (Ix, v])],
(1)
where [x, vj denotes the concatenation of the current prefix with a sampled next token, and R(x) is the reward obtained by continuing from x with only greedy tokens until the stopping criteria is met. In our case, we use accuracy as reward." A higher sampling risk value a indicates a greater likelihood of generating incorrect outputs when applying temperature sampling for this decoding time step.
3.2 Case Studies of High Sampling Risk
To validate the definition of the sampling risk, we first conduct a case study to investigate whether we can identify certain decoding time steps with high sampling risk. For this study, we focus on an arithmetic reasoning task. We hypothesize that when model produces an arithmetic calculation, there are certain decoding steps with high sampling risk.
AAccrrcyy measured by matching between predicted and gold final answers.
3

---

Published as a conference paper at COLM 2025
Entrpy
Entopy
65-
Sampling
(a) Greedy
(b) min-p Sampling (p 0.1)
(c) top-p Sampling (p 0.9)
Figure 2: Entropy US Sampling Risk. Red squares: Incorrect final answers; Blue dots:
Correct final answers. Overall, we show that introducing sampling methods for high-risk token positions will lead to erroneous outputs.
3.2.1 Setup
We focus on an arithmetic reasoning task with the chain-of-thought GSM-Symbolic dataset (Mirzadeh et al., 2025), and use the instruction-tuned LLaMa-3.1 model (Grattafiori et al., 2024) as our language model. For this analysis, we subsampled 100 correct greedy outputs from the CoT GSM-Symbolic dataset to ensure our observations were not limited to few cases, Within these outputs, we identified potentially risky token positions considering the positions where the model's top-1 token is an integer number. These positions are critical for math tasks, as integer numbers often serve as intermediate results for arriving at the final answer.
3.2.2 Findings
Figure illustrates one high sampling risk example from the GSM-Symbolic dataset. It shows that selecting non-greedy token (e.g., 30) for the current greedy CoT continuation leads to an incorrect answer. This suggests that sampling under the model distribution at the selected integer positions tends to generate errors, as only a narrow set of candidates can yield correct outputs.
To further examine the impact of sampling at the selected positions, we compare min-p and top-p sampling with greedy decoding. Figure 2 shows the correlation between sampling risk and entropy for different decoding settings. We focus full CoT continuations that produce correct answers with greedy decoding and compute sampling risk and entropy at integer positions within these continuations. Overall, we show that there are certain decod- ing time steps when sampling tends to result in incorrect outputs, which also correlate with high model entropy. These results underscore the importance of the sampling risk metric in guiding decoding methods to improve the quality-diversity trade-off.
Methodology: Selective Sampling
Motivated by the observation that high sampling risk leads incorrect predictions ($3), in this section, we propose the selective sampling strategy by training a classifier so(x) E R+ to discriminate between safe and high-risk prefixes x.
4.1 Estimating Sampling Risk
We automatically label sampling risks defined Equation (1) for a small training set to train classifier for selective sampling. For the training set, either part of the training splits of the considered problems (GSM8k, Minerva Prealgebra tasks), or we split the test set into train and test (GSM-Symbolic task) in a proportion 60 40. For the GSM-Symbolic task,
4

---

Published as a conference paper at COLM 2025
Question: Find the sum of all positive integers $n0 such that $1 $1.22-4.4<5.25
Answer: 28
S-risk labels: We have that $1.2n-4.4<5.2 28, $1 21.2n<65.2 2+4 2-4.4-9.65. and thus $n<Virac(9.611 2)-85 Therefore, the positive
integers $n$ satisfying the inequsiity are $1 ,,,,,,,,,6,77, The sum of these integers is 21122-/445566777booee///// Answer: The answeris $28$. hope is correct.
Our classifier: We have that $1.2n-4.4=5,22 2$, so $1.2n=5,22 +4, 4-.66. and thus $n 1.. Therefore, the positive integers $n$ satisfying the inequality are $1, 5,,,,, 7s. The sum of these integers is 5111--3-4---56-7-ooo0o00888
Answer: The maalansweris is $28$. hope is correct.
Figure 3: S-risk labels: obtained sampling risk labels for the Minerva task validation
set; Our classifier: corresponding predictions from our classifier. Orange marks the high sampling risk positions where the greedy action would be chosen.
we split the data using the original problem IDs, such that versions of the same problem only appear in one subset.
For a given task, we assume that training example contains a source prompt u, and we can estimate sampling risk for a given decoding time step. We do not use any correct chain- of-thoughts from the data, and only use the model outputs. We only use correct greedy outputs for training. We mark prefix to be either risky or safe to sample from as follows:
y(x) = =(0
f1,
if s-ris((xx) < -1-6,
otherwise
where we set 0.05 in our experiments.
In practice, we observe that for the reasoning tasks we considered, greedy sampling often produces high-quality outputs. Given a set of training prompts U = {u] i= we obtain the subset of prompts and correct greedy continuations U, = (u, и E (() = 1]. Then, for every (u, x) E U+, we estimate sampling risks up to a maximal token position M (we use at most M = 300). To estimate the sampling risk at position for the current X.,, we create a short-list of the top most probable next tokens. We force decode each of the k next token candidates {u,}}_1 and then we finish each candidate using greedy decoding to estimate ([xx,1))) Here, we treat greedy continuation as low-cost approximation to the upperbound on the quality given selected next token. Different ways to estimate this property may be possible and may depend on the task or the domain.
4.2 Selective Sampling
Our goal is to train a classifier to predict the sampling risk based on a representation of the model's context. LLMs' hidden representations are known to be rich source of information for feature extraction (Duan et al., 2024; Mahaut et al., 2024). To obtain high- quality context features for the classifier, we simply use the hidden representations of the last token position. Namely, for a prefix x = [x), x,], we use hidden states of the base model from the residual stream, including the last input embedding, for the last position t: f(x) = [h,,... ,hr,), where L is the number of layers in the base model. Then, the classifier is applied on top of the frozen hidden states f(x). We parametrize simple linear binary classifier sa (x)) = af E' ht), where ayy) = 1/ e Our classifier is simple linear model applied
top of the hidden states, introducing minimal computational overhead. Most inference frameworks support access to the last hidden states, and we implement it directly inside the vLLM implementation of the LLaMa model (Kwon et al., 2023). During inference, we use our selective sampling classifier together with truncation sampling (we use min-p, P 0.1). The aim is to investigate how our classifier can complement the limitations of truncation sampling ($3).
We train our classifier in a teacher-forcing regime by passing the inputs to the model together with target sampling risks. We use binary cross-entropy loss to train the classifier. We discuss the training details in the Appendix A. Our implementation is released as open-source.?
5

---

Published as a conference paper at COLM 2025
5 Experiments
We conduct a comprehensive evaluation of selective sampling, comparing it to existing sampling methods across multiple tasks. Our experiments aim to show the effectiveness of selective sampling in improving both the quality and diversity of the model's outputs, particularly at higher temperatures.
5.1 Tasks
Following Grattafiori et al. (2024); OpenAl et al. (2024); Chen et al. (2024b), we evaluate on commonly used mathematical reasoning benchmarks:
GSM8K (Cobbe et al., 2021): A grade school math problem-solving benchmark.
GSM-Symbolic (Mirzadeh et al., 2023): An extended variant of GSMSK with sym- bolic templates, designed to provide more reliable assessment of reasoning ability.
Minerva MATH (Hendrycks et al., 2021): A dataset of competition-level mathemat- ical problems. We conduct experiments on the PreAlgebra subset.
We follow the standard prompting formats implemented in Evaluation Harness (Gao et al., 2024). For GSM8k and GSM Symbolic, we use the standard 8-shot chain-of- chain-of-thought prompt- ing configuration. For Minerva MATH, we use the 4-shot thain-of-though prompting configuration. Additionally, we experiment on MMLU-Pro question answering task (Wang et al., 2024) with multiple choice answers and CoT (Appendix F.1).
5.2 Model
We use the instruction-tunnd
(8B) model (Grattafiori et al., 2024), selected for its strong performance across various reasoning tasks.
5.3 Evaluation Metrics
Quality-diversity trade-off evaluation. We evaluate the model's output based on two key aspects: quality and diversity, using 25 samples per prompt. Quality is measured by the accuracy of the final answer, indicating whether the model generates correct solutions. For diversity, we compute the averaged distinct n-grams (Li et al., 2016), which quantifies the proportion of unique 1-grams relative to the total number of n-grams in the generated responses, details in Appendix We follow Minh et al. (2025) and measure diversity only over correctly generated samples in order to be less biased towards low-quality outputs.
To present results, we vary the temperature parameter and plot the quality metric on the x-axis and the diversity metric on the y-axis. The best method would have both high-quality and diverse predictions. To compare different methods, we look at the gap between the diversity-quality plots, e-g, which method is better in terms of quality given certain diversity value and vice versa. In order to provide an aggregated metric integrating out the temperature parameter, we also report the area under the quality-diversite plot (AUC) using the trapezoidal rule implemented in scikit-learn (Pedregosa et al., 2011).
Fluency evaluation. To demonstrate the effectiveness of our method in generating diverse yet coherent samples, we evaluate the noisiness of samples generated at high temperatures. We use perplexity
proxy for noise, with higher perplexity indicating a higher likelihood of incoherent or nonsensical text, as suggested by previous studies (Zhang et al., 2024a; Ankner et al., 2025; Marion et al., 2023). Perplexity scores were computed using the Llama- 2-7B-chat-hf model. For each task, we analyze a random subset of 100 instances, generating 25 samples per instance and calculating the average perplexity of the samples the score
We take the theoretical (0,0) point to belong all curves even it not realized by any hyperpa- rameters in the experiment. httpps: http:/ggggggagcccco/////// htt////ttaaaaaaaae www.llllaaaaaaaaaaaaaaaaaa 2-

---

Published as a conference paper at COLM 2025
Symbolic
Minerva Prealgebra
0.6
Dhvensty ever connuE
0.6
0.-
0.4
0.2
Our
Diverstty
Oura
Diversity
02
0.-
0.4
6.8
0.5
0.2
Accuraty
Accurecy
Arcunacy
(a) GSM8K
(b) Symbolic GSM
(c) Minerva (Prealgebra)
Figure 4: Selective sampling with our classifier improves the diversity- trade-off compared to the strong min- truncation baseline. On the x-axis, we report the accuracy, and on x-axis, we report the Diversity over correct samples. Size and color of the circles mark the temperature parameter.
for that instance. The overall perplexity score is then computed as the average of the per-instance scores.
6 Results
6.1 Diversity-Quality Results
We compare our selective sampling with baseline methods, including min-p sampling (Minh et al., 2025), top-p sampling, top-k (Fan al., 2018), (Holtzman al., 2020), c sampling (Hewitt et al., 2022), 1/ sampling (Hewitt et al., 2022) and EDT Zhang et al. (2024b). The full plots are listed in Figures 13 and 14 (Appendix G), where we demonstrate that the baselines methods usually follow the same diversity-quality trajectory, and that there is no clear winner among the baseline methods. In Appendix D, we analyze the effect of the hyperparameters for the baselines, and we choose min-p, with p 0.1, the main baseline for comparison. In Table 1, we compare the cumulative aggregated diversity- scores (AUC metric) for the quality-diversity plots.
In Figure 4, we demonstrate that the using selective sampling for all three tasks. When the temperature is lower (I 0.5), the performance gap between min-p and ours is small. However, our method performs better than min-p when the temperature value is increased.
trade-off of min-p can be improved
Moreover, we estimate the average percentage of token positions where selective sampling chooses greedy over temperature sampling, as shown in Table We observe that selective sampling tends to choose greedy decoding more often under higher temperature values and harder tasks, such as Minerva. This observation is expected, higher temperatures or more challenging tasks tend to degrade the quality. Using greedy decoding more frequently in such cases results in a more reliable strategy to maintain the output quality.
6.2 Fluency Results
Figure 5 illustrates the average perplexity scores across different temperature settings for GSM8K, Symbolic GSM and Minerva Prealgebra. It also highlights the average accuracy of the samples at each temperature value. As shown in the figure, our method consistently produces significantly less noisy and more accurate samples, even at high temperatures, compared to the min-p sampling method. Examples of samples generated by our method versus min-p at t 2.0 are presented in Appendix G.
6.3 Ablation Studies
Classifier features. Above, we demonstrated that our classifier trained top of the last hidden states can improve the quality-diversity trade-off. While our classifier uses more information from the context, compared to the baselines, it is built on top of the model's
7

---

Published as a conference paper at COLM 2025
GSMBk
Symbolic GSM
Minerva Math Prealgebra
1750
Average Perplexity
40
curs
Average perplexity
40
curs
Average Perplexityy
1500
curs
1250
30
30
1000
25
750
20
500
40
250
-750
15
Temperature
Temperature
Temperature
(a) GSM8K
(b) Symbolic GSM
(c) Minerva (Prealgebra)
Figure 5: Our approach produces low perplexity samples and maintains better quality than the p baseline even at high temperatures. Numbers inside the plots represent accuracies (as percentages).
Symbolic GSM
Minerva Prealgebra
Minerva Math Prealgebra
0.6
Method GSM8k
Diversity correct
0.5
top-p 0.32
0.32
0.21
0.4
min-p
0.38
0.40
0.25
min-p 0.1
Temperature
top-k
0.38
0.40
0.23
Ours n-gram
ff sampl.
0.37
0.40
0.24
0.2
Ours
sampl.
0.37
0.40
0.24
0.1
EDT
0.35
0.36
0.24
Ours 0.42
0.47
0.30
0.0
0.0
0.2
0.4
0.6
Table 1: The area under the quality- diversity plot of various sampling strate- gies: aggregated metric of the quality- diversity trade-off. Our method outper- forms the baselines this metric.
Accuracy
Figure 6: We show that our n-gram model, which does not use any internal model rep- resentations, can also slightly improve upon the min-p baseline.
Temperature
GSMSk
Symbolic
Minerva
1.0
0.02
0.10
0.37
2.0
0.05
0.12
0.39
3.0
0.17
0.20
0.44
Table 2: Percentage of greedy decoding token positions under different temperatures and tasks for selective sampling. Under higher temperatures harder tasks, selective sampling chooses greedy decoding more than high-temperature sampling.
hidden representations. To complement the main experiment and to check whether can train the classifier solely using the current context as the feature, we introduce a simple n-gram-based model, a classifier trained over the last input embedding representations. We provide the architectural details in Appendix B. For the Minerva evaluation set, the n-gram classifier obtains 79% accuracy and 0.72% ROC AUC, which is worse compared to the classifier trained on the hidden states (85% accuracy, and 0.78 ROC AUC). In Figure 6, we present the results for selective sampling with the n-gram classifier, where we demonstrate that the n-gram classifier is slightly better compared to the min-p baseline, in term of diversity-quality trade-off, for the smaller temperature values range T E (0;2). However, the n-gram classifier does not reach the performance of our hidden states classifier. We think is easier for the model trained on top of the hidden states to generalize to the unseen contexts, especially in the higher temperature regime.
Classifier sensitivity. To assess the sample complexity of training the classifier, in Ap- pendix E2 we compare performance with random subsamples of the training set. We observe that performance convergens at around 500 training prompts, suggesting that variations in training data do not pose problems to the framework.
8

---

Published as a conference paper at COLM 2025
Symbolic GSM
Minerva Math Prealgebra
0.6
Diversity correct
0.6
Tomparature value
0.5
Terr Terrerr ceneerauure value
0.4
2
0.3
29
HM-
min-p 0.1
Diversity
-tl
min-p 0.1
0.2
Ours
Ours tall tasks)
0.1
Ours (all tasks)
(from symbolic to minerval
0.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
Accuracy
Accuracy
Figure 7: Generalization experiment with evaluation on Symbolic GSM and Minerva Math (Prealgebra) tasks. "Ours (all tasks)" is a classifier trained on all 3 datasets. "Ours (from X to Y)" denotes transfer experiment, where we train the classifier on task and test on Y.
Diversity metric choice. To verify that our results are not overdependent on the choice of the diversity metric, we follow the RFT (Yuan et al., 2023) and use the diversity metric as the average normalized Levenshtein distance between all pairs of correct responses and compare our approach to min-p sampling. We find that our method improves the quality- diversity trade-off, in line with the findings from the n-gram-bass bbaed diversity evaluation (see Figure 11 in Appendix F.3).
6.4 Entropy-based (EDT) baseline.
In this experiment, we modify the EDT (Zhang et al., 2024b) approach, where binary threshold similar to our approach to switch between greedy and high-temperature settings. From the additional results (see Figure 12 in Appendix F:4), we observe that the threshold-based entropy baseline does not outperform the entropy-based dynamic temperature sampling (EDT), and our method outperforms both variants of entropy-based sampling approaches, which highlights the benefit of the trained classifier head versus using entropy.
6.5 Task Generalization of the Selective Sampling Classifier
In this experiment, we ask (1) whether we can transfer a classifier trained on one task to another task, (2) whether we can train single classifier to be applied on many tasks. Regard- ing (1), we observe that selective sampling trained on the Minerva dataset outperforms the min-p baseline for the GSM Symbolic task on diversity-quality, suggesting that our classifier can generalize between these tasks (see Figure 7). Regarding (2), we train a single classifier, Ours (all tasks), on 800 examples from each of the 3 datasets: GSMSK, GSM Symbolic, and Minerva, and evaluate the model on the GSM Symbolic task. We observe that the quality of the Ours (all tasks) model closely matches that of Ours trained only on GSM Symbolic (see Figure 7, left). The same effect is observed when we evaluate the on the Minerva dataset (see Figure 7, right). This shows that we can use the same single classifier multiple tasks.
same all tasks model
Related work
In this section, we review prior work relevant to our study, focusing on sampling techniques from large language models (LLMs), methods for adjusting sampling parameters, and approaches to mitigating hallucinations, These areas collectively informed our approach for improving the quality and diversity of LLM outputs.
Sampling from LLMs. In the context of sampling from LLMs, various techniques (Holtz- man et al., 2020; Basu et al., 2021; Hewitt et al., 2022; Minh et al., 2025) are proposed to improve the fluency, coherence, reduction of repetitiveness, and diversity of LLM responses.

---

Published as a conference paper at COLM 2025
Renze (2024) finds that in context of problem solving through LLMs there is little difference between the existing sampling methods for [0, temperature range. Wang & Zhou (2024) observe that LLMs can reason through sampling by considering alternative decoding paths. Minimum Bayes Risk (MBR) decoding (Kumar & Byrne, 2004) improves the quality of the model by countering the pathological behavior of MAP inference (Freitag et al., 2022; Suzgun et al., 2023; Wu et al., 2025; Jinnai et al., 2024).
Adjusting LLM sampling. There are several relevant approaches to improve sampling quality and creativity by adjusting the temperature of the model. Liu et al. (2024) propose to dynamically choose sampling hyperparameters based on the context, where they prompted LLM to predict the hyperparameters at each step in a zero-shot regime. They assume LLMs already understand sampling risk. Their approach requires maintaining two models during the inference, which doubles the computation. Chang et al. (2023) develop dynamic temperature sampling to improve the context grounding for question answering tasks, and Zhang et al. (2024b) improve upon this work by dynamically adjusting the temperature parameter based the base model entropy. In our work, we use the approach of Zhang et al. (2024b) as an entropy-based baseline. In the context of code generation, Zhu et al. (2024) notice that challenging code tokens tend to appear the beginning of a code block and propose to adjust the temperature based on the model confidence.
Mitigating LLM hallucinations. A broader related line of work improves the quality of LLMs output by mitigating hallucination (Guerreiro et al., 2023; Duan et al., 2024; Mahaut et al., 2024; Chen et al., 2023). To detect hallucinations, Duan et al. (2024) and Mahaut et al. (2024) use LLM hidden states to estimate its reliability. Chen et al. (2024a) and Meng et al. (2025) adjust the next token prediction distribution to improve the factuality and overall quality by using the model entropy. Kossen et al. (2024) propose semantic entropy probes to approximate semantic entropy from the hidden states to mitigate hallucination. Moreover, Li et al. (2023b) introduce extra models to guide the decoding while Li et al. (2023a) shift the activation during the inference improve the truthfulness of LLMs.
Conclusion
Our work investigates temperature sampling from LLMs in the context of reasoning tasks. We highlight that fully relying on model confidence might lead to low-quality outputs. To complement the confidence estimate of a base model, we introduce the sampling risk metric to estimate the risk of choosing sampling action instead of greedy, based on the expected future task-specific reward. We then propose selective sampling approach that switches between sampling and greedy decoding at each decoding time step based on the predicted sampling risk. To achieve this, we train simple classifier to estimate the risk during inference. We demonstrate that our selective sampling strategy can improve the quality-diversity trade-off compared to the commonly used baselines, such as min-p truncation sampling. We hope our work will encourage further improvements on the important quality-diversity tradeoff in language modeling.
9 Limitations
Our work exhibits a few limitations. First, while our classifier for selective sampling is easy to implement, the classifier is model-dependent, which makes our approach not directly transferable between different models compared to the baseline methods. Second, we focus on the mathematical reasoning tasks, because they provide a simple way to estimate correctness using verifiable rewards (accuracy of the final answers). We do not incorporate the correctness of the CoTs as part of the quality metric, which would be much harder to evaluate. We are looking forward to exploring more open-ended and creative writing tasks in the future, with more subtle variations in sampling risks. Third, for the diversity evaluation, we only use commonly used and intuitive n-gram-ban based diversity measure.
10

---

Published as a conference paper at COLM 2025
Reproducibility Statement
Our code is based on open-source software libraries for training and evaluation of LLMs: vLLM (Kwon et al., 2023), Evaluation Harness (Gao et al., 2024), Hugging Face (Wolf al., 2020). We conduct the experiments on commonly used datasets available via Hugging Face (Wolf et al., 2020). We use the prompt formats defined in Evaluation Harness (Gao et al., 2024). Regarding the used datasets, the GSM8K dataset (Cobbe et al., 2021) is openly available under the MIT license. Minerva MATH Prealgebra subset is part of the MATH dataset (Hendrycks et al., 2021), and by using it, we refer to the Fair Use case discussed in Hendrycks et al. (2021). Due to a recent controversy about the copyright status of the only release data or models trained GSM8K and GSM-symbolic. We will revisit the release of the Minerva MATH part of the data if this situation changes. This may affect the reproducibility of part of our experiments (mainly those in which we compare our classifier train on top of the hidden states with its n-gram ablation). We call for more alternative evaluation datasets designed with the intention of being used for research and released with open licenses.
MATH dataset,
Ethics Statement
While we mitigate the degradation in quality for high temperature sampling, we have not measured whether this mitigation also affects toxicity or other undesirable properties of text. We therefore recommend the same care that users should take with any other LLM generation systems. Our method depends on the definition of task reward, which does not account for all important characteristics users might value. Since sampling risk is estimated with classifier, it could possibly be that by suppressing certain outputs our method affects the fairness and broader diversity of the outputs.
Acknowledgments
This publication is part of the project VI. Veni..22..2222 212.228 of the research program 'Veni', which is financed by the Dutch Research Council (NWO); and is part of 'Hybrid Intelligence: augmenting human intellect' (https:/ htttss//hhbrid-iiie 024.004.022 of the research program 'Gravitation' which is (partly) financed by the Dutch Research Council (NWO). Moreover, it is funded in part by the Netherlands Organization for Scientific Research (NWO) under the project number VI.C. 080. It also part of the UTTER project, supported by the European Union's Horizon Europe research and innovation programme via grant agreement 101070631.
with project number
We thank Wilker Aziz, members of LTL and CLTL labs, and the reviewers for fruitful discussions and feedback.
References
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models. In ICLR, 2025. URL / http::/:////ooeeeeeeeeeeee/t/ttt www.neee/orummid-1GGAAneeeee
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. preprint, 2016. URL https: //arxiv http:///axxww..o//b////// otgg/bs//////// 06450.
Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. Uncertainty in natural language generation: From theory applications, 2023, URL https: org/abs/ 1307.15700
Sourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varshney. Mirostat: A neural text decoding algorithm that directly controls perplexity. In ICLR, 2021. URL htttp:://operreviiii http:///////poooooooeee/////////
11

---

Published as a conference paper at COLM 2025
Sébastien Bubeck and Nicolò Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends in Machine Learning, 2012. URL http://sbubeck. com hemm/SureyBBB1.ppp.
Chung-Ching Chang, David Reitter, Renat Aksitov, and Yun-Hsuan Sung. KI-divergence guided temperature sampling, 2023. URL httpp://arxx..oorr/a/////66 01286.
Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation. In ICML, 2024a. URL https://openrevieww net d
Xiusi Chen, Jyun-Yu Yu Jiang, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, and Wei Wang. MinPrompt: Graph-based minimal prompt data augmentation for few-shot question answering. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.254 254-266, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653 /224.//.....o.ogg..o.. 16. URL https: ptttps://aclanes http://aaaaaatthhh http:p/s///caanooooggy..ooo org ogg. /ttp://cl.aantollooor////////2 acl-long. 16.
Yuyan Chen, Qiang Fu, Yichen Yuan, Zhihao Wen, Ge Fan, Dayiheng Liu, Dongmei Zhang, Zhixu Li, and Yanghua Xiao. Hallucination detection: Robustly discerning reliable answers large language models. In CIKM, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint ar 0rXiv:2110.14111 2021.
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, and J. L. Cai al. Deepseek-v3 technical report, 2025. URL https://arree http:///wwwww.../ab///////ss abrg/222b2212. 19437.
Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation with unidirectional reward model. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11781-11791, Singapore, December 2023. Association for Computational URL https: https://aclantt hatp:///aaaaaaoogyyyooo org/
Linguistics. doi: 2t16633/11222..mmmppppp 2023. emn.eemainmmin..
Hanyu Duan, Yi Yang, and Kar Yan Tam. Do iims know about hallucination? an empirical investigation of Ilm's hidden states, 2024. URL https: http:s:///xxxvvva/ org/ abtg////////..aa.///// 89733.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, An- thony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Bap tiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Cantón Ferrer, Cyrus Nikolaidis, Damien Al- lonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Carcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, Elina Lobanova, and Emily Dinan et al. The llama 3 herd of mod- els. ArXiv, abs/2 21783, 2024. URL https:/ //api. www.stiiscccotttiei woree.rg//oooo org/CCrpusID: 271571434.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 889-898, Melbourne, Australia, July 2018 Association for Computational Linguistics. doi: 10.18653 (6653/vl/// /v1/P1--10S// URL https://accanthont https://aclanthologg.oor orp/
12

---

Published as a conference paper at COLM 2025
Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguisties, 10:811-825 2022. doi: 10.1162 //a////cll./ a 00491.
URL attps://aclanthiee htwphhoolgloooooooogooogoo org/ tacl-1.47.
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopi, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo. http://eodd.ooggrerrrrrooo org/records// / 2608602.
Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, and Barbara Plank. What comes next? evaluating uncertainty in neural text generators against human production variability. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, PP. 14349-14371, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2022 org.g/y3332233333 wwwp///...mmmpmmmama.........a emnlp-main. 887.
emnlp-main.887. URL
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and Abhinav Pandey et al. The llama 3 herd of models, 2024. URL httpt:::///axiioorrabbbbs./
21783.
Nuno M. Guerreiro, Elena Voita, and André Martins. Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1059-1075, Dubrovnik, Croatia, May 2023, Association for Computational Linguistics. doi: 11116663/1112222.acclmm// https://aclanthologe org/2023. eacl-main 75.
URL
Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Associa- fion for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1689-1701, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18633/ /1/119-11NN199/
URL https: / http://aaaaattyy.../o/ org/N19-1181
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). preprint, 2023. URL http:///axxxv..o//bbbba/ https/6666.0084555 08415.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
John Hewitt, Christopher Manning, and Percy Liang. Truncation sampling language model desmoothing. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Find- ings of the Association for Compulational Linguistics: EMNLP 2022, pp. 3414-3427, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10.18653 /v1/3//vv/// /1/2022..2fmfcm www....iiningg-eemmll... 249. URL https:://cllantholog/ /022.. findings-emnlp 249.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2020. URL https: //tp///evee....e/e////e net//forumiddrry@@yrfF
Yuu Jinnai, Ukyo Honda, Tetsuro Morimura, and Peinan Zhang. Generating diverse and high-quality texts by minimum Bayes risk decoding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 8494-8525, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: (0.18633/vi /2////112222////// findiigg 503. URL //allat////llnhholoo//// org/ 2024. findings-ace acl. 503.
Diederick P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. URL https: http///xxvvvv////// wit.ii.or/////////////// 6980.
13

---

Published as a conference paper at COLM 2025
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity. In ICLR, 2024.
Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth A. Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in llms. ArXiv, abs/2406.15927, 2024, URL https: //api.//ppp.pp seematttccccolla..rr///rrrsoop2p wtgg/oorpusD::20088111
Shankar Kumar and William Byrne. Minimum Bayes-risk decoding for statistical ma- chine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computationnl Linguistics: HLT-NAACL 2004, pp. 169-176, Boston, Massachusetts, USA, May 2 May 7 2004. Association for Computational Linguistics. URL https: http:/:://aaaaahhhlaaaa wttphhooooooooo///-1/22222 org/N04-1022.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, Pp. 611-626, 2023.
Florian Le Bronnec, Alexandre Verine, Benjamin Negrevergne, Yann Chevaleyre, and Alexan- dre Allauzen. Exploring precision and recall to assess the quality and diversity of LLMs. In Lei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 11418- 11441, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ /10003//11/200 222024.acl-oong..oooomo.
URL https: httt:::/aacaattholleee
2024. acl acl-long. 616.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110- 119, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/ /v/l v1/N16-1014. URL https: http:///aaaantology.ooooo
1014.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. In NeurIPS, 2023a. URL https:// wwtp://wwweeeeoommmdddaaaaaattte
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text gener- ation as optimization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: 'Long Papers), pp. 12286-12313 Toronto, Canada, July 2023b. As- sociation for Computational Linguistics. doi: 10.18653 20663/vv/2022 h1t/0/2..allooog....... acl-long.687. URL https://aclanthologg wwweee....rr///// org/2023..22222 acl -ong.lon..600 687.
Mingyue Liu, Jonathan Frawley, Sarah Wyer, Hubert P. H. Shum, Sara Uckelman, Sue Black, and Chris Willcocks. Self-regulated sample diversity in large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1891-1899, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/ 11/33v11// findings-naacl.112. orgg/22244
URL
wiwdinnnnnnidadde 122.
Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Müller, and Lluis Marquez. Factual confidence of LLMs: on reliability and robustness of current estimators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4554 4570, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 v1/ w00033/wl2222..eee www.///2o.....o......a. 250. URL https: http::::///aclaahhlllooo/o acl-long.22 250.
2024.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale. In NeurIPS
14

---

Published as a conference paper at COLM 2025
Workshop on Attributing Model Behavior at Scale (ATTRIB), 2023. URL https://opennreview.. net/foruuniddNuIYnnjjj...
Clara Meister, Tiago Pimentel, Luca Malagutti, Ethan Wilcox, and Ryan Cotterell. On the efficacy of sampling adapters. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 1437-1455, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 1011655/v1/2022aacloongg.. URL https://aclanthholog]/ wwwpmnnooooooooooo...eoeoe org/2023 acl 1oog-llog.80
Yan Meng, Di Wu, and Christof Monz. How to learn in a noisy world? self-correcting the real-world data noise in machine translation. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 7451-7467, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-195-7, doi: 10.18653/ v1 w022025.... 2tt..22......snaaaaaeaee nindings--aacl.46. URL https://aclanthologg.. org.:///ooo.///222333// findings-naacl 4166/
Nguyen Nhat Minh, Andrew Baker, Clement Neo, Allen G Roush, Andreas Kirsch, and Ravid Shwartz-Ziv. Turning up the heat: Min-p sampling for creative and coherent LLM outputs. In ICLR, 2025. URL https: /ttppr/ie//e wttpee/wwwwentttfffooor////////e
Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-symbolic: Understanding the limitations of mathematical reasoning large language models. In ICLR, 2025. URL id=AjxkRZIvjB.
wttpeeewweeeeeeeouueeeeee net/forum?
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmaa Beirami. Controlled decoding from language models. In ICML, 2024.
URL ptttts::/openreeett wttp:/wwwwwe/ffoo/////eee nett/forummiccciIIcc770...
OpenAI, Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, and et al. Gpt-4 technical report, 2024. URL https:
www.oogg///////// aabp/2303.00. 08774.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. JMLR, 12(0ct):2825-2280, 2011.
Matthew Renze. The effect of sampling temperature on problem solving in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 7346-7356, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https://aclanthhooog org/2024 findings mmnlp.433.
Claude E Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379-44) 423, 1948.
Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujuu Yang, and Wai Lam. A thorough examination of decoding methods in the era of LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp. 8601-8629, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL https: /acaantooaalgyyy..o./ org/2024. emnlp-main, 489.
15

---

Published as a conference paper at COLM 2025
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NeurlPS, 2020. ISBN 9781713829546.
Mirac Suzgun, Luke Melas-Kyriazii and Dan Jurafsky. Follow the wisdom of the crowd: Effective text generation via minimum Bayes risk decoding. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Compudational Linguistics: ACL 2023, pp. 4265-4293, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ 1105//11/2000 fin.inngs--c..2222222 URL https: /atp:///aaaaaalaoyyccooo org/ 2023. findings acl. 262.
Sergey Troshin, Vlad Niculae, and Antske Fokkens. Efficient controlled language generation with low-rank autoregressive reward models, 2024. URL htttp::/arxii.org/abb///// 04615.
Xuezhi Wang and Denny Zhou. teaihoo without prompting. In NeurIPS, 2024. URL http:: / nett/forumiicc44777000))
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Dalasets and Benchmarks Track, 2024. URL htttps://operrvviiie http:///wneeeei.eeeeeeoooomnnddo 00MM62233.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing, 2020. URL https://arxiv. org/abs/1910. 03771.
Ian Wu, Patrick Fernandes, Amanda Bertsch, Seungone Kim, Sina Khoshfetrat Pakazad, and Graham Neubig. Better instructioonfollloiing through minimum bayes risk. In ICLR, 2025. URL htt.://////ppoeeeeeeeeeee//////r
Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang. Qwen2.5 technical report. ArXiv, abs/2412.151155, 2024. URL https: //api. semanticscholar www.eecoooa.rrrrroooooiiiuee Org/CorppsII::: 2748594421
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: deliberate problem solving with large language models. In NeurIPS, 2023.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models, 2023. URL httpp://arxiiv org/abs/2308 01825.
Feipeng Zhang, Wenyu Jiang, Jun Shu, Feng Zheng, Hongxin Wei, et al. On the noise robustness of in-context learning for text generation. NeuriPS, 2024a.
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. Trading off diversity and quality in natural language generation. In Anya Belz, Shubham Agarwal, Yvette Graham, Ehud Reiter, and Anastasia Shimorina ), Proceedings of the Workshop on Human Evuluation of NLP Systems (HumEoul), pp. 25-33, Online, April 2021. Association for Computational Linguistics. URL https: https://aclantthology wwwpey....ggoeeee org/2021 humeval- 1ameval-1..
16

---

Published as a conference paper at COLM 2025
Shimao Zhang, Yu Bao, and Shujian Huang. EDT: Improving large language models' generation by entropy-based dynamic temperature sampling, 2024b. URL https://arxis org/abs/2403. 14541.
Yuqi Zhu, Jia Li, Ge Li, YunFei Zhao, Zhi Jin, and Hong Mei. Hot or cold? adaptive temperature sampling for code generation with large language models. AAAI, 2024.
Appendix
A Classifier Training
We train our classifier using Hugging Face.² We train our classifier with the Adam optimizer (Kingma Ba, 2015) with $1 = 0.9, 82 0.95, e le-12. We use learning rate 0.001, weight decay 0.01, and batch size 100, number of epochs 50. We use binary cross-entropy loss. We freeze all parameters of the base model and only train classifier weights. To create the labelled dataset, the dataset creation, which is equivalent to using top-k filtered p distribution for the regret estimation, and
only k 8 next token candidates to limit the complexity of
the temperature
3.0) to estimate the sampling risk.
For preliminary experiments, we measure the quality of the binary classification with our hidden states classifier using the validation set (100 samples from the training set). For CSMBk/Minerva our classifier obtains 93%/85% accuracy, and 0.73/0.78 ROC AUC. We visualize the predictions of our model along with original labels in Figure 3. Overall, observe that our classifier approximates well the labels in the dataset. We hypothesize that further data refinement and noise reduction can benefit classifier training. We report dataset statistics in the Table (for training, we leave only the samples with a correct greedy solution).
Subset GSM8k
GSM Symbolic
Minerva
train
4487
2300
893
val
100
100
100
test
1319
2000
871
Table Dataset sizes for GSM8K, GSM Symbolic, and Minerva
B N-gram Classifier
We implement the n-gram classifier as feed-forward network applied on top of the last input embeddings (we chose n 10 based on the classification quality on the validation set). The network consists of the 1D Convolution layer with 256 filters, which aggregates the last n embeddings followed by the LayerNorm (Ba et al., 2016), and the GeLU nonlinearity (Hendrycks & Gimpel, 2023). Then we apply LayerNorm and the GeLU, and finally followed by the last linear layer to project into single number.
x linear layer following by the the
C Sampling Methods
C.1 Truncation Sampling Methods
Temperature samping modifies the logits by introducing the temperature parameter 7: I[M = T (-M(-1x). Truncated temperature sampling as implemented in commonly used frameworks (vLLM; Kwon et al., 2023) works by considering only short-list of token
17

---

Published as a conference paper at COLM 2025
candidates V S V at each decoding step (Minh et al., 2025; Holtzman et al., 2020; Basu et al., 2021; Hewitt et al., 2022)
[ if V',
z(oxx)
(3)
100,
otherwise.
and the next token is sampled from the categorical distribution:
exp(z(v|x))
(=(x = Σ exp(z(v"|x))) N'EN'
(4)
Min-p sampling. To maintain coherence at high temperatures, Minh et al. (2025) intro- duced min-p sampling, a dynamic method that adapts its truncation threshold according to the model's confidence at each decoding step. At each step, min-p identifies the maximum probability token in the distribution:
Pmax m p()/2).
(5)
The truncation threshold is subsequently determined by scaling base parameter (0,1) by Pmax: Pscaled = x Pmax, and V (eq. (3)) is determined by V {u E V : 2
Pscaled
Top-p sampling. Also known as nucleus sampling, this method samples from the "nu-
cleus" of high-probability tokens by restricting the sampling pool to the top tokens whose cumulative probability exceeds hyperparameter (Holtzman et al., 2020): V' = fu V Σp(r|x) 2 p}
Top-k sampling. The sampling pool in this method consists of the most probable top k tokens (Fan et al., 2018): V {v E rank (p(olx)) S k]
e-sampling. This method allows any token with probability greater than threshold (Hewitt et 2022): = {u E V p(u|x) > e}
C.2 Entropy Sampling Methods
For language models, Shannon entropy, defined as - p(v) log(p(v)) measures the uncertainty in predicting the next token in sequence (Shannon, 1948). Truncation sampling methods impose fixed thresholds to limit the set of candidate tokens during generation. While these methods can improve output quality by filtering out low-probability tokens, they may also over-restrict choices in low entropy distributions, potentially reducing diversity. Entropy sampling methods adjust the sampling process based the model's entropy, aiming to improve the quality-diversity trade-off.
--smmlinng. 1/-sampling truncates words below an entropy-dependent probability thresh- old (Hewitt et al., 2022): y min(c, a exp(-H using E [0,1] and the hyperparameter Token shortlist is then determined as V (u V p(v|x) > n7)
EDT sampling. Entropy-based Dynamic Temperature sampling method dynamically ad- justs the temperature parameter according to the model's entropy and can be combined with truncation sampling methods as an initial step (Zhang et al., 2024b). With the recommended hyperparameters a = 0.8 and 0 E [0, 1), they shrink the original temperature 7 by (H(()))
namely T(T, H(p)) t a Hip), Intuitively, when the entropy is large, the shrinkage factor ((H(p) approaches 1, and when the entropy is small, it approaches
D Sampling Hyperparameter Settings
For comparing sampling methods: top-k, top-p, min-p, / and entropy-based dynamic sampling (EDT), report their optimal results with conducted tests different hyperpa-
18

---

Published as a conference paper at COLM 2025
rameters in Table 4. Results are selected based the AUC scores over the quality-diversity plot, reflecting the overall quality-diversity trade-off.
For top-p sampling, report top-p 0.7 and conducted tests on p = 0.7,0.8,0.9, For min-p sampling, we report min-p = 0.1 for all experiments and conducted tests on p = 0.1, 0.3. For y and sampling, we test and y/ values 0.0002 and 0.0009, found 0.0009 to score better for both values and report this in our experiment results.
For EDT sampling, we report 0.1 for all experiments, and conducted tests 0 0.1,
E Averaged Distinct N-gram Diversity
==) each with a response set Rx = {r] Following Minh et al. (2025), we first filter out incorrect samples for each instance order to be less biased towards low-quality outputs, and we end up with a set of C correct samples R = {r) per instance, where C may vary across instances. Then, for E (1,2,3,4,5) we calculate the averaged distinct n-gram diversity for each instance as follows:
Given a set of test instances $ = {
5 set(n-gram (Rc))
averaged distinct n-gram (s)
n-gram (Rc)
The total test-set diversity is the average diversity over all instances:
diversity(Rec (Rc)
Σ averaged distinct n-gram (s;).
F
Additional Experiments
E1 MMLU (Social Tasks)
We perform an additional experiment on the MMLU-Pro question answering task (Wang al., 2024) with multiple choice answers and COT. We choose the subset of following tasks: law, philosophy, history, psychology, which are more different from the math tasks from our main experiments. As we can see from the results Figure 8, and Figure 9, (a) our method slightly outperforms the min-p baseline in terms of divarsyty-quality trade-off and (b) better preserves the quality for higher temperatures, which broadens the potential scope for our method.
F2 Sensitivity to the training dataset
To verify that our classifier training is not too sensitive for the variations in training data, we perform an additional experiment for classifier training by randomly subsampling 100, 500, 1000 examples of the training set for the GSM Symbolic task. We measure the accuracy for sampling risk classification on the validation set. From Figure 10, we observe that subsampling 1000 or 500 examples reduces accuracy marginally from 0.9 to 0.89. Taking 100 examples leads to slight overfitting and 0.87 accuracy.
E3 Ablation of the diversity metric
Following RFT (Yuan et al., 2023), use diversity metric as the average normalized Levenshtein distance between all pairs of correct responses and compare our approach to min-p sampling. We find that our method improves the quality-diversity trade-off, in line with the findings from the n-gram-based diversity evaluation (see Figure 11).
E4 EDT ablation experiment
In this section, we perform an additional ablation experiment for the modified the EDT (Zhang et al., 2024b) approach, where binary threshold similar to our approach to
19

---

Published as a conference paper at COLM 2025
MMLU Pro (History)
MMLU Pro (Law)
Diversity over correct
Diversity over correct
0.6
0.6
0.4
-
min-p 0.1
Temper stture
0.4
-
min-p 0.1
Ours
Ours
Temper
0.2
0.2
0.0
0.1
0.2 Accuracy
0.4
0.0
0.1
0.2
0.3
Accuracy
MMLU Pro (Philosophy)
MMLU Pro (Psychology)
Diversity over correct
Diversity over correct
Temperature
Temperalure
0.4
min-pp 0.1
0.4
min-p 0.1
0.2
0.2
0.0
0.0
0.0
0.3
0.4
0.0
0.2
Accuracy
Accuracy
Figure 8: Additional result on MMLU Pro task on 4 social subsets.
MMLU Pro (Social)
0.6
0.5
Temperature value
0.4
Accuracy
0.3
min-p 0.1
Ours
0.2
0.1
0.0
3
Temperature
Figure 9: Additional result on MMLU Pro task on social subsets: quality-temperature trade-off.
switch between greedy and high-temperature settings. In particular, given a threshold I, we evaluate the threshold-based entropy baseline (if H[p] t, then r'=0.0, else r'=T), using t. from [0.5, 1, 2]. From the additional results Figure 12, we observe that the threshold-based entropy baseline does not outperform the entropy-based dynamic temperature sampling (EDT). Our method outperforms both variants of entropy-based sampling approaches, which highlights the benefit of the trained classifier head versus the entropy.
G Examples
In Figure 15, we provide a high sampling risk example from the Symbolic GSM task. In Figure 16, we provide additional examples with 2.0 for the samples for GSM8k. Overall, we observe that for higher temperatures, our methods produces more fluent outputs.
20

---

Published as a conference paper at COLM 2025
Accuracy vs. Training Set Size
0.900
Accuracy
0.885
0.880
0.875
0.870
500
1000
1500
2000
Training Set Size
Figure Training data sensitivity analysis on the GSM Symbolic task: accuracy vs the size of the training set.
Symbolic GSM
Levenshtein over correct
0.6
Temperature value
0.4
min-p 0.1 Ours
0.2
0.0
0.0
0.2
0.4
0.6
0.8
Accuracy
Figure 11: Quality-diversity trade-off plot with Levenshtein diversity metric.
Symbolic GSM
Diversity over correct
0.6
Temperal aarurauree value
0.1
Ours
0.4
EDT Theta 0.1 (binary thres=0.5)
EDT Theta 0.1 (binary thres=1)
(binary thres=2)
0.2
0.0
0.0
0.2
0.4
0.6
0.3
Accuracy
Figure 12: Quality-diversity trade-off plot with Levenshtein diversity metric.
21

---

Published as a conference paper at COLM 2025
Method
Jnversity-Quality AUC
GSMSK
Minerva
top-k 20
0.38
0.23
top-p 0.7
0.28
0.20
top-p 0.8
0.31
0,20
top-p 0.9
0.32
0.21
min-p
0.38
0.25
min-p 0.2
0.37
0.25
min-p 0.3
0.37
0,25
e-sampl. 0.0002
0.35
0.23
c-sampl. 0.0009
0.37
0.24
1-sampl. 0.0002
0.36
0.23
s--samp. 0.0009
0.37
0.24
EDT 0 0.1
0.35
0.24
EDT 1 1.0
0.32
0.24
Table 4: Baselines hyperparameter values. Diversity-quality AUC Scores for GSM8K and Minerva Prealgebra. Bold denotes the chosen hyperparameter value for main results.
GSMBB
GSMBK
GSMBK
Diversit over Conrect
Diverday over cornect
2.
25
0.4
2.
mmn 2.2
mmng 2.3
2.1
0.2
b.3
Ipp-0005 0.45
tsp-2
mini
0.2
0.2
0.
0.0
0.2
0.4
0.8
Acouracy
GSM8k Min-p
GSM8k Top-p
GSM8k Top-k
GSMBK
GSMBk
GSMBK
Diversty Over CORNECT
Diversity Deer cornect
D.4
2.0
-35
minp ens 018
mino C.
0.3
epoiion 0380
0.3
For Theis 6.1
0.2
sta 0 0050
Diveriity
0.2
uguil 0350
2.
-c- cor Thete
5.0
5.0
50
00
Arcuracy
Accuracy
GSM8k 1
GSMBk e samp..
GSM8k EDT
Figure 13: We compare different hyperparameters for the baseline methods the GSM8k task.
22

---

Published as a conference paper at COLM 2025
Hinerva Math Prealgebra
Minerva Math Prealgebra
Minerva Math Prealgebra
Diversty ooeer connect
Diverity ooeer connect
0.
-25
-.5
-3.
-7.0
-2.0
0.5
minpdd
m.
0.2
tops
222
0.2
5.5
0.1
Accuracy
Accuracy
Accuracy
Minerva Min p
Minerva Top-p p
Minerva Top-k
Minerva Math Prealgebra
Minerva Math Prealgebra
Minerva Math Prealgebra
1.0
Divesity over correct
Diverity over cornect
-25
0.3
mipip
0.3
mié P.
-1.5
upal
1.5
Thets
-38
-100
Diversity
0.2
0.6
0.4 Accuracy
0.5
0.
0.2
0.6
Accuracy
Accuracy
Minerva 11 sampl...
E e-mmpl.
Minerva EDT
Figure 14: We compare different hyperparameters for the baseline methods on the Minerva task.
23

---

Published as a conference paper at COLM 2025
Question: Yora qualified for a spot on the saccer toan, so she vent shopping for some athletic gear, She bought Jersey for V24, pair of seccer leggings for v13 rare than the Jersey cost, and pair clests that were originally V38 but vere on sale for two-thirds price. She had a coupon for vi off the package or Yi1 athletie headbands that she also bought. Finally, she got duffle bag for V42 but with 50% discount, and water bottle for 13, How such did she spend on athletic gear? Target:
Risk Token Position:
Yara bought, jersey for 426, The legg cost v13 more then the jersey, so Lhey cos 24 + 13 = 37, The cleats were originally Y30 but were on sale for tuo-thirde price. tho-tsirrs of 38 is (2/3) 38 20. So the cleats cost
(20 Prefix) 0.68
[30 Prefix) 0.32
Vara bought Jersey for V24 Tuo-thirds of 30 is 12/3)
Yara bought Jersey for W24 Twa-thirds of 38 is (2/3) 30 28. the cleats cost 30
30 20. 5o the cleats coST
Correct Answer
Incorrect Answer
bought jersey
Tug-thirds The
bought a jersey and she ect discount, they discount, originally 942
Twu-thards
30 20. So
30
headbands The
dutrie originally Lyy 942 but she 5e% discount, it
V1 discount,
duffle bag
cost 42 [1/2] 21. The sater bottle cost M3. 115.
it cast 42 (42/2) 42 21. The water bottle cost 185.
Now ve add up the costs: 24
x3. 5o the total cast is 24 37 18 10
The rinal answer 115
The final answer 185
(a) Example
0: Chen operates the cash register esactly seven times as fast as her 1ess-experiene reeced catleague Usa. Roya, being wore expertenced than lsaa but tess experienced than Chon, operates at the came spoed as the average of uha and Chen. daily, lnn processes 22 customers. What is the total weakly production for the three if they work all days of the week? 1848
Position:
Since custorers. Chon operates times as fast Uma, Chen's dasly production is 22 154 customers. Chen's weekly production is 154 + 7 1878 customers.
custoners
and there
weekly production
Roya uperates at the sate speed as the everage of Une and Chen, The average of Usa and Chen's deily production is
P (22 Prefix) 0.88
(154) Prefix) m 0.12
processes production
average
processes production
average
Chen's daily
Correct Answer
Incorrect Answer
Since Use processes 22 .. The average Use end Chen's
Since Uha processes 22 The average of Usa and Chen's daily product son is (154 + 154) / 2 306 / = 154 customers. So Reya 's daily production is also 154 custorers. Roya't wrrkly production i5 154 x 7 1870 custo custoners.
daily production (22 + 154 custorers, Chen's weekty toners..
product
Roya operates
speed daily production is Roya's daily production production
average
Chen. average
The total, weekly production for the three is Una's production 1078
154)
production Royo's production. The tinal answer is 2310
The total weekly production for the three is 154 (Unal 1878 (Chen) 616 (Roya) 1848 custowers. The final answer
is 1848
(b) Example
Figure 15: High-Risk sampling examples from GSM-Symbolic. Left: Top-1 Top-1/Greedy genera- tion on high-risk decoding position. Right: Non-greedy generation on high-risk decoding position.
24

---

Published as a conference paper at COLM 2025
G, Kylar went to the store to buy glesees for his new spertment, One glass costs $5, butl every second glass costs only 60% of the price. Kylar wants glasses. How much does need pay for them?
A
Ours
min-p
Rist, let's find out which glazses cost 60%. There are 16 glasses in total and every 2nd goss
First, let's find out wil kch g glassos cost 80% There are 16 glasses in botal and
will be coew of the segganoc one 60%, 60% Starinn with oot the fins one.
every i case the sequenoe,
wee have a rntel of of the 10054 glecesss Smnce (8 Stc 540 For ewery (ind) one there's is
every
99% diagouan a 100% price minas (80 S) that
appyy
(85- 180
ton two Off
Ciloss num number is the posiiion of gloss when courting list is on place 2. On
Add those togothor $40 Se we get o fhna $48 For of those set total that means for
begin again second oost with counter function. List begin lione glasses bought
this oot 8 glasses are that 948 Thore is a tocal of 16 grassce
some rotoo
29 of (100%) and also $5 and the other se cownooselly calis beieed.
100%
350) (5S)
the othert then $48 total together
and since
position Nammterr fiom start cree....
position count Countor- glass counter' Countor numb
So ovenall thooe parns they equad 120 5) (e11)) $58 and in a of (S600) HI (868 for and that
$352
gleasse price sequide (So8))
communicaiing winne dostoon begine sovet upgreding prosen
plapemmnt powtry
contisting
co combinations practical corrying fossil popu root ed echood divengence atfocting exparsion screened hightight moniin ng clothing nemberrs siiver roets exomption driving cancer desenving sought developments
Kyler wants buy pilcoress For avery awcond glaas, which are the swer rumbeer glessane the price is 90% 0.50 '55 = So. For every uold mumber gless the price stays the serree si
rotired
We first original oel) odd num mumbers from to 10: 1.5.7.88:11 11,11.15.
classee for
(88). Those
the
even nu numbere coet the taait ($3), oo S3 $26, or numbers commiined
Ptyer wante buy nossee For sepond glass, which am the even number glasosss the price 90% or 0.60 585 S1. For every odd number glass
the (55),
New
$40 should be
price stays same,
Commining those sums, you have 364
(96). Those from the or nal setl bocause from 1 through is hal which includes the number of glass price ecusling live dallars, cod and oven glass together then ar purchases on additionl oven glasses
which
One glans in S5, and the second glose is 60% of the prion, which $5 x 8. a S3. To tind the number glanoes the sequence
making the lowing
glasses firtt glans is second glnes the
glass
t6 Wumbe and Price Czee a iPs dolleackul price cdet
whichh (1555) #2551) ety50n- #453)) and forts. From thete, find that $5
priced for all odd numbers while 85 priced for the oven numbers in the countin of the sot 85; 96, forth ing th #15 861 and 50,
Kylor
orai 31 glass the cast aeniiionnll glacsees
30) even numbers
$5; solution)
Number2 Glasses using Numbers Kylors store price ordered comecc
land ending 815, and 816 (86 and rospoctivelyl glasses at and 58 prices and Klylrr plane buy glneses wach.
both
glaxoses (Bdy (41, 5, 7, 11, 13, 15), Court 8 gloex.
(ovon Cost G remaiing remaining another J...
(55) ii (55) Tooral solki of 56% prioed glandes Sas. Jrz.
dullea nigher
divided
socond set plasses
16),
then less the ler they leam medde war aller stil wl laal seem full conn disn
Addiing value 845 (odd pricect, of glass and B24 (oven price al glase to the total maki
sudii varly tarm now butt watiltion doni by buok information new thene suptroot using numbers found gave fairyy over will price
the sum
possible
add adding
problem
The
$500
proper above atil o1 will way diearly lowsst second division bated pay all gmotter known come we took agath so examplei get cinar great present usually meeting
aral mostly correclly prioe additin amountt under compare below right shar nelated buy future move we enough end information good in sarfor induvi divididd Jully today four
need find Normal plaso cools dollars scoft
discounted suporately
Normal cost: 8* 40
Diccmnted kosses 8'3 24
only sinos the prioe these will
Adding normel and daccunted glease:
lotal gbea cost: Normal cost diecounted glace crst 40
Figure 16: Example from GSM8k for Ours with temperature 2.0 versus min-p with temper- ature 2.0. Our model is more stable with higher temperatures compared to uncontrolled min-p, which fails co produce coherent outputs for some prompts with temperature 2. We skip part of the output for min-p since it is too long.
25


Reviews:
[{"review_id": "9PBDZCSLQN", "reviewer": "Reviewer_s7Xi", "summary": "The paper studies a novel sampling approach that seeks to better balance the trade off between quality (accuracy) and the creativity (diversity) of LLMs’ generations. The authors first illustrated the limitations of current temperature-based sampling approaches, such as min-p and top-p, through the lens of sampling risk. Sampling risk measures the risk of error in taking a non-greedy sampling approach at a particular decoding time step. This observation motivates the proposed approach of selective sampling, where a classifier is trained to take in the final LLM hidden states to predict the sampling risk (S-risk) at a particular time step, to toggle greedy decoding when risk is high and temperature-based sampling when risk is low, based on a threshold value. Experiments were conducted on math reasoning-based datasets, including GSM8K, GSM-symbolic and Minerva MATH.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed approach is well motivated, backed by analyses on existing temperature-sampling approaches.\n\nThe paper is well-written and easy to follow.", "reasons_to_reject": "The experiments are conducted only on math reasoning tasks. It remains to be shown if this approach is effective for other reasoning tasks. From Figure 3, it seems that the parts with high S-risk are concentrated at tokens related to digits and math operations, which suggests that this approach is relevant and effective for math-based reasoning but it is less clear how well this approach would apply to non-math questions which may not involve math-related tokens as much.", "questions_to_authors": "", "extra_scores": {"ethics_concerns_details": "NA"}}, {"review_id": "MSB6zWqXjB", "reviewer": "Reviewer_vo4Q", "summary": "This paper addresses the challenge of maintaining high-quality outputs while promoting diversity in LLM generations, particularly in tasks requiring precision like mathematical reasoning. Authors show that prior temperature-based methods, while increasing diversity, often degrade reasoning quality by sampling incorrect continuations in sensitive, high-entropy positions. The authors propose a selective sampling method that dynamically switches between greedy decoding and high-temperature sampling, which is guided by a sampling risk metric, estimating the likelihood of errors if high-temperature sampling is applied at a given token position. A classifier is trained on a samll subset of verifiable problems to predict this sampling risk. Experiments on mathematical reasoning demonstrate that selective sampling improves the quality-diversity trade-off, especially in high-temperature settings, and produces less noisy samples compared to existing methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is generally well-written, and the authors are upfront about several limitations of their work.\n\nThe paper tackles the well-recognized trade-off between generation quality and diversity in LLMs, which is a critical area of research.\n\nThe concept of \"selective sampling\" based on a learned \"sampling risk\" metric is an innovative and intuitively appealing solution.\n\nWithin the domain of mathematical reasoning, the experiments demonstrate improvement in the quality-diversity trade-off and fluency compared to strong baselines like min-p sampling.", "reasons_to_reject": "The experiments are focused solely on mathematical reasoning tasks. While suitable for evaluating precision, the paper's claims would be significantly strengthened by demonstrating efficacy on a broader range of tasks.\n\nThe method's current framework appears to require training a separate classifier for each specific task to estimate risk. This limit generalization and usability. A dedicated generalization study showing the classifier's capabilities across different tasks, or a direction towards task-agnostic risk predictor would be very important for broader impact.\n\nWhy are different types of uncertainty important for sampling? The motivation for distinguishing between different types of uncertainty (citing Baan et al., 2023 ) and its direct impact on the necessity of selective sampling versus general uncertainty awareness could be more explicitly connected and justified.\n\nThe experimental details surrounding Figure 2, which illustrates sampling risk versus entropy, are not clear and should be expanded. It's unclear how many examples or positions are represented and the precise methodology for selecting these instances for analysis.\n\nThe rationale for estimating sampling risk by sampling only a single token $v_j$ and then completing the sequence with greedy decoding needs further discussion. While this isolates the immediate impact of $v_j$, it doesn't fully represent the behavior of continuous temperature sampling throughout a trajectory.\n\nThe sampling risk classifier is trained on \"a small subset of verifiable problems\". This raises questions about the method's applicability to tasks where such verifiable rewards are not easily obtainable or are more subjective.\n\nThe trained classifier is model- and task-dependent, potentially limiting its direct transferability between different base LLMs and tasks without retraining.\n\nWhile EDT sampling is used as an entropy-based baseline, a more direct baseline that uses entropy as a real-time proxy to switch between greedy and temperature sampling (without the dynamic temperature adjustment of EDT but rather a binary switch similar to the proposed method) can provide a better measure of the added benefit of the trained classifier head.\n\nThe paper uses the accuracy of the final answer as the quality metric. While understandable for complexity reasons, not incorporating the correctness or plausibility of the Chain-of-Thought (CoT) itself is a limitation in fully assessing reasoning quality.\n\n\nSome further concerns are in questions to authors.", "questions_to_authors": "Could you elaborate on the characteristics and size of the \"small subset of verifiable problems\" used for training the sampling risk classifier? How sensitive is its performance to variations in this training data?\n\nCould you provide more context for Figure 2? Specifically, how were the examples and token positions selected for this analysis, and does it represent a few illustrative cases or a broader trend?\n\nFor the fluency experiments (Figure 5), what was the accuracy of your method compared to min-p sampling at the different temperature points shown, to better understand the quality-fluency trade-off?\n\nThe experiments use 25 samples per prompt. Given that prior research (e.g. Smaller, Weaker, Yet Better -- Bansal et al., 2024, Large language monkeys -- Brown et al., 2024) suggests diversity can continue to increase with a higher number of samples, what was the rationale for this specific number, and have you explored the impact of more samples?\n\nCould you provide an analysis or statistic on how frequently the selective sampling model chooses greedy decoding versus high-temperature sampling across the evaluated datasets?\n\nThe classifier uses hidden states from all layers of the base model. Have alternative feature sets (e.g., a subset of layers, attention-based features) been explored, and how did they compare in performance and efficiency?\n\nhave you considered comparing selective sampling against a simpler heuristic that directly uses an entropy threshold (perhaps learned or set empirically) to switch between greedy and temperature sampling, to further isolate the benefit of the trained classifier head?\n\nIn Figure 4, the color map for temperature values and the colors used for the different methods are very similar. It would be better to use more distinct color scheme for enhanced clarity.\n\nFor GSM8K, it is insightful to analyze the diversity of the generated solutions not just by n-grams in the full CoT, but also by looking at the diversity of the underlying mathematical steps or equations (RFT -- Yuan et al., 2023). Could you explore this as well?\n\nDo you have some insights on the \"sampling risk\" metric and classifier training for tasks lacking easily verifiable rewards, such as open-ended creative writing?", "extra_scores": {}}, {"review_id": "oCUOGE6fh5", "reviewer": "Reviewer_YhHJ", "summary": "The paper proposes a decoding strategy in which the temperature is low for parts of the output that need to be exact, and high for parts where creativity in the output is appreciated.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* clearly motivated paper\n* very simple and straight-forward idea\n* good experimental setup", "reasons_to_reject": "* the experiments are only conducted on one data set, which is a math solving resource. I am wondering what the impact of the proposed strategy would be on purely creative tasks. \n* it is not really clear to me why, in such setups of math solving, a high temperature should generally be preferable. Perhaps the motivation could be improved in this direction.", "questions_to_authors": "* How does the work relate to constraint decoding? Couldn't the selective sampling also be modeled as a setup with high temperature but additional constraints? It would be nice to understand the relation between these perspectives.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach that dynamically switches between greedy sampling and temperature sampling to obtain a good balance between accuracy and diversity of the model's responses. All the reviewers generally liked the paper, and the only marginally negative reviewer had no substantial concerns. The other two reviewers provide good scores. Therefore, we recommend acceptance of the paper.
Decision: Accept