Optimizing Temperature for Language Models with Multi-Sample Inference
Authors: Weihua Du
Yiming Yang
Sean Welleck
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Inference Time Compute
Primary area: deep_learning->large_language_models
TL;DR: By analyzing the behavior of various LLMs across different temperatures and tasks, we propose an entropy-based algorithm for automated temperature optimization in multi-sample aggregation strategies, eliminating the need for labeled validation data.

Abstract: Multi-sample aggregation strategies, such as majority voting and best-of-N sampling, are widely used in contemporary large language models (LLMs) to enhance predictive accuracy across various tasks. A key challenge in this process is temperature selection, which significantly impacts model performance. Existing approaches either rely on a fixed default temperature or require labeled validation data for tuning, which are often scarce and difficult to obtain. This paper addresses the challenge of automatically identifying the (near)-optimal temperature for different LLMs using multi-sample aggregation strategies, without relying on task-specific validation data. We provide a comprehensive analysis of temperature’s role in performance optimization, considering variations in model architectures, datasets, task types, model sizes, and predictive accuracy. Furthermore, we propose a novel entropy-based metric for automated temperature optimization, which consistently outperforms fixed-temperature baselines. Additionally, we incorporate a stochastic process model to enhance interpretability, offering deeper insights into the relationship between temperature and model performance.

Optimizing Temperature for Language Models with Multi-Sample Inference
Weihua Du Yiming Yang'
Sean Welleck
Abstract
(a)
Entropy Curve of OpenMath2-88
Multi-sample aggregation strategies, such as ma- jority voting and best-of-N sampling, are widely used in contemporary large language models (LLMs) to enhance predictive accuracy across various tasks. A key challenge in this process is temperature selection, which significantly im- pacts model performance. Existing approaches either rely on fixed default temperature or re- quire labeled validation data for tuning, which are often scarce and difficult to obtain. This paper addresses the challenge of automatically identifying the (near)-optimal temperature for different LLMs using multi-sampll aggregation strategies, without relying on task-specifie vali- dation data. We provide a comprehensive anal- ysis of temperature's role in performance opti- mization, considering variations in model archi- tectures, datasets, task types, model sizes, and predictive accuracy. Furthermore, we propose a novel entropy- metric for automated tem- perature optimization, which consistently outper- forms teed-teeperature baselines. Additionally, we incorporate a stochastic process model to en- hance interpretability, offering deeper insights into the relationship between temperature and model performance. Our code is available at wttp:ghhuhb.oootSSSggdddduulalaa
LogiToken-Leve
Leg[Token- Lelken--evell Entropy)
Entropy Tumning [EntP):
Tokan-Levei Entropy
Emtroyy
Convex
Concave
-3
Spike
-4
0.2
0.4
0.6
0.5
1.0
1.2
1.4
Temperature
(b)
Accuracy at EntP vs. Best from Grid Search (MATH)
Conelation confiicett 6.5900
shopword
decuuracy
6.4
Eetrroo To
Wirra7 intrunt-0
0.-
033
35
06
0.7
Accuracy EndP
Figure I. (a) The entropy turning point (EntP) (green star) is defined the temperature point where the log-scale of the token- level entropy of generation (red line) shifts from concave to convex, implying sudden spike the entropy curve (blue line). (b) The accuracy tested at EntP is highly correlated with the best accuracy from grid search over temperatures on MATH dataset.
reasoning (Ahn et al., 2024; Sun et al., 2024; Lin et al., 2024; Wu al., 2024). A fundamental research question in gener- ative models is how effectively sample solutions from learned distribution and perform inference-time reasoning.
1. Introduction
Large language models (LLMs) have demonstrated remark- able capabilities across various domains, including question answering (Kamalloo al., 2023), intelligent agents (Wang et al., 2024b; Zhang et al., 2023), scientific discovery (Ma et al., 2024; Romera-Paredes et al., 2024), and mathematical
Recently, multi-sample aggregation strategies have gained increasing attention. These strategies involve generating multiple solutions from the underlying distribution and ag- gregating them into a final prediction (Wei et al., 2022; Yao et al., 2024). Common aggregation techniques, such as majority voting, weighted majority voting, and best-of-N selection, have demonstrated significant performance im- provements in benchmark evaluations of L.MSs (Welleck et al., 2024; Wang et al., 2024a).
'Language Technologies Institute, Carnegie Mellon University. Correspondence to: Weihua Du <weihuad www.ecc..eeuue Yiming Yang syyimin@@cs.cmu.elll <wellecks@ emu.edu>.
Scan Welleck
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Despite the promising success of multi multi-samp sample aggregation strategies, there remains lack of deep understanding regard-

---

Optimizing Temperature for Language Models with Multi-Sample Inference
ing how to optimize the sampling process to enhance LLM performance under different conditions, including variations in training datasets, task types, and model sizes. A crucial open question is how to tune temperature, key hyperpa- rameter that controls the smoothness of the system-learned distribution. Intuitively, increasing the temperature leads to a smoother distribution, enhancing the diversity of sampled outputs. However, excessively high temperatures can intro- duce many low-quality samples, making aggregation more challenging (Holtzman et al., 2019; Renze & Guven, 2024). Conversely, lowering the temperature results in a highly concentrated distribution, reducing diversity and potentially omitting bigh-quality samples. Striking the right balance between over-sampling and under-sampling therefore es- sential for optimizing LLM performance.
temperature for each LLM using multi-sample aggregation strategies. We introduce TURN, our proposed approach for automated temperature optimization. Through extensive ex- periments, TURN has demonstrated strong generalizability across diverse tasks (e.g., mathematical problem-solving. code generation), model sizes, and aggregation strategies (e.g,, majority voting, best-of-N). consistently outper- forms bascline methods using fixed temperature, yielding significant performance improvements. Additionally, our approach enhances the interpretability of temperature's role in model performance by analyzing EntP. Moreover, our analysis explores how the optimal temperature is influenced by the divergence or similarity between model training and tasks (Section 3).
In summary, TURN provides novel, efficient, and princi- pled method for optimizing temperature in LLM inference with multi-sample aggregation. It eliminates the need for labeled validation data and significantly improves perfor- mance across a wide range of applications.
A common practice in prior evaluations is to the same temperature across all methods despite variations in training datasets, task types, model sizes, and aggregation strategies. This practice is clearly suboptimal. An alternative approach is to empirically tune the temperature using labeled valida- tion data for each task, dataset, model size, and aggrega- tion strategy (Zhang et al., 2024a; Dhuliawala al., 2024). However, such process is tedious and time-consuming and heavily dependent on the availability of labeled validation data, limiting its applicability when such data are scarce.
2. Preliminary & Related Work
Before moving to our main contributions, we first review how language models typically generate samples and pro- vide an introduction to multi multi-sample aggregation strategies.
In this paper, we present the first systematic investigation of how temperature affects LLM performance under multi- sample aggregation strategies across various conditions. Fur- thermore, we propose a principled algorithmic solution for automated temperature optimization without requiring la- beled validation data. Our key idea is as follows:
Language Model Sampling Language models typically generate output for generative tasks by autoregressively sampling from the conditional probability distribution over the next token, given both the input context and previously generated tokens. Formally, for an input sequence X and output sequence Y (V1 .N the probability of producing Y is given by the following:
),
1. We use the confidence score of each model as a self- assessment measure.
N
P(Y I = II P(ys Veis X).
(1)
2. If this self-assessment measure is highly correlated with model accuracy on test data, it can serve as a surrogate metric for tuning temperature in the absence of labeled validation data.
To compute the probability distribution, the model obtains set of logits 2; and then divides them by a temperature hyperparameter T before applying the softmax function and a regularization function F:
A surprising finding from our temperature tuning experi- ments is the discovery of a phenomenon we term the en- tropy turning point (EntP) in the self-assessed performance curve. As illustrated in Figure ((a), the token-level en- tropy (y-axis) of an LLM varies with temperature values (x-axis), shown by the blue curve, while its log-scale rep- resentation appears as the red curve. Notably, there is a transition point (EntP) where the red curve shifts from con- cave to convex. Figure 1(b) shows that the accuracy scores at EntP for a set of LLMS are strongly correlated with their highest accuracy scores obtained through grid-based based temper- ature tuning. This finding supports our intuition that EntP can be leveraged to automatically determine the optimal
P( \ye X) = FF F(soflmax
4 T
(2)
where 2; is the logit corresponding to token y:- The temper- ature T controls how peaked or flat the resulting probability distribution will be. The regularization function J is used to reschedule the sampling process (e.g., Top-k (Kool et al., 2019), Top-p (Holtzman et al., 2019), Min-p (Nguyen et al., 2024) and Locally Typical Sampling (Meister et al., 2023)).
Multi-Sample Aggregation Strategy Since different ran- dom seeds can produce varying outcomes, a common ap- proach to mitigate sampling variance is to draw multiple
2

---

Optimizing Temperature for Language Models with Multi-Sample Inference
(a) Accuracy Heatmap of Mistral-7B-lnstruct--0..
(b)
Midpoint of Optimal Temperature Range Number of Samples
0.03
0.16
0.20
1.0
0.04
0.05
6.07
0.09
0.13
0.16
0.19
0.23
0.26
0.9
0.8
Temperature
0.7
0:30
9.11
0.14
0.19
0.44
0.28
0.30
0.30
0:32
Temperaturr
0.6
0.3
0:36
0.16
0.19
0.24
0.25
0.31
0.32
0.33
0.32
0.4
0:36
0.17
0.19
0.23
0.26
0.27
0.27
0.27
0.20
0.3
Mistrai-BB-inttee OnstructionFinetuned Mocal)
0.2
Mistrai-78 SFT (Tase-Finetuned on MATH Datasel)
Number Samples
Number of Samples
128
256
under majority voting across different temperatures. The best temperature for each sampling size highlighted in bold white, and the optimal temperature range shaded white. The green line shows the temperature predicted by our method. (b) Midpoint of Optimal Temperature Range vs, Number of Samples. The optimal temperature range varies by model; those with training data more closely matching the task tend favor higher temperatures.
Figure 2. (a) Accuracy Heatmap.
Performance of Mistral-7B- - -nnt--t-ttttu---..333
samples and aggregate their results. In practice, it leads to substantial performance improvements and has been widely adopted achieve state-of-the-arr performance in math reasoning (Sun al., 2024; Jaech et al., 2024), code genera- tion (Wang et al., 2024a), and many other domains.
Some studies have investigated selecting a temperature for a single-sample method (Zhang et al., 2024b; Li al., 2024; Kumar Sarawagi, 2019; Xie et al., 2024; Dhuliawala et al., 2024) multi-sample aggregation with validation set (Zhang et al., 2024a). Our method has two key differ- ences: (1) we focus on state-of-the-arr multisample aggrega- tion strategies rather than single-sample inference, and (2) we find the optimal temperature without validation data.
Specifically, a set of candidate outputs Y = (Y,,.... YN} is generated and then aggregated into a final answer. Two standard aggregation methods are typically employed:
The final answer is the output that appears most frequently among the candidates, i.e.,
Majority Voting:
3. Correlation Between Model Training and Optimal Temperature
N Σ T(Y =
Multi-sample aggregation caategies-comonnl used in problem-solving, code generation, and related do- mains mains--lverage information from multiple samples, which helps escape local minima and improve robustness. In these settings, sample diversity becomes crucial: diverse set of candidate samples increases the likelihood that the correct solution appears in the pool, rather than repeating the same mistake. The temperature parameter is a primary lever for controlling this diversity.
ÿ = =arg
max (E Y
where TT.) is the indicator function, which returns if its argument is true and 0 otherwise. This method is frequently used where evaluating whether two outputs are equivalent is relatively easy. The method is also called self-consistency (Wang et al., 2022).
Best-of-N: Each sample is scored by a reward function G, and the final answer is the one with the highest score:
We hypothesize that how model is trained impacts the optimal temperature for multi-sample inference strategies. In particular, more specialized or fine-tuned model can safely explore higher temperatures without drifting into low- quality outputs. In contrast, general-purpost purpose model typi- cally benefits from a lower temperature to remain focused on relevant content.
J arg
max
G(y)-
yE (VI....,
The reward function G can be defined in various ways, such as a separate language model's likelihood, or trained or verified reward model.
Choosing Temperature in Multi-Sample Aggregation Despite the widespread use of multi-sample aggregation strategies state-of-the-ar systems, the question of choos- ing the important temperature parameter remains under- explored.
We investigate this in two steps: In Section 1. we show that the optimal temperature varies for base, instruction-tur -uunud,, and fine-tuned model. Then in Section 3.2, we establish general relationship between model's proximity to the
3

---

Optimizing Temperature for Language Models with Multi-Sample Inference
Midpoint of Optimal Temperature Range vs. Distance (MATH)
temperature range. This range encompasses temperatures T where the accuracy A(T) is no less than A(T*) e, with A(T*) representing the peak accuracy. Given the curve's s single-peak nature, this range forms an interval around T*. For our analysis, we set < = 0.02, effectively capturing the temperatures close to the peak where the accuracy remains relatively high.
Lama-332
2-30
0.30
Vigmme To
Liama 2-38 instrutt Mistra--B-instt-tt--
Distance (D)
0.20
coopscok math 7a b2s2
Lama 3.1 rectruct
0.13
Oponlggth2 jama311 Bb Instruct
We then plot the midpoint of this optimal temperature range for each model variant and various sample sizes (Fig- ure 2(b)). We observe that the pretrained model has the low- est midpoint, the instruction instructiio-fiincueed model has a higher midpoint, and the finetfinttuedd model has the highest. An- other observation is that optimal temperature ranges change slowly once beyond sample size of 32. Therefore, we choose sample size of 128 in our following experiments to ensure stable performance in the rich-sample setting.
0.3
parriigt-mal--eene nict math 50000000 misaat-b- torma fo st mel molyraghh if mb
0.05
Rt: Du-00.266--0 Corr 095
04
05
U.
US
11
Temperature (T)
Midpoint of Optimal Temperature Range vs. Distance (MBPP)
0.25
code25-
Usma 3.1 88
0.250
7b-10.
0.225
CodaUama 7b
Distaice (D)
0.200
Secessma 70 Pyshon
Gwen2 7-Code--7-iisttuut
From these observations, we hypothesize general relation- ship between how closely a model is tuned to a particular task and the temperature that yields the best accuracy. We discuss this hypothesis further in the next section.
Matral Liama-3 288-hratt
0.175
deepseer codee TE base
0.550
Toissruce
0125
deesseek-coder- Sosuuutt--
0.00
Dww.0.211--0p. Corr -0.77
11 coder
Y coder chat 30
3.2. Correlation Between Training- Task Similarity and Optimal Temperature
0.7
3.8
09
1.0
11
122
Temperature (T)
Our goal is to establish a general relationship between a model's learned distribution and its optimal temperature for a task. Our key intuition is that token-level entropy can serve proxy for how distant a model is from target task and that this distance helps identify the optimal temperature.
Figure 3. Plot of midpoints of optimal temperature ranges (x-axis, sample size 128) vs. distances between models and tasks (y-axis). strong negative correlation is observed on the MATH and MBPP datasets, with correlation coefficients of -0.895 and -0.777.
Specifically, we define a distance metric that measures how similar a model's 's training data is to a given task. Let T = XI....X++ be the task with k problem instances. We define this distance D(M, TT as the average of token-level entropy H(-) of the language model M when generating the answers A (Yi,..., Yı for the problems in T:
target task and its corresponding optimal temperature. Our key insight is that token-level entropy is a proxy of distance from a task, which motivates our cntropy-based method for automatic temperature selection in Section 4.
3.1. Optimal Temperature Range Varies
We first demonstrate that the optimal sampling temperature varies by model type. We test three Mistral-7B variants: the pretrained base model, the instruction-fietture sion DB-stral-7B-nnstrect and task-finetuned model for MATH' (Wang et al., 2024c). Each model is evaluated us- ing multi-sample aggregation across different temperatures. Figure 2(a) presents the accuracy heatmap for the Mistral- 7B-Instruct model on the MATH dataset. At smaller sample sizes, lower temperatures tend produce better accuracy. However, higher temperatures can yield better results as the sample size increases. For a fixed sample size, the accuracy curve follows single-peak pattern: it rises as temperature increases and peaks, and then gradually declines, staying relatively steady near the peak.
k
Y
D(M,T)
H(pm(-1 Xi, Yi<il)
= k
ver-
(3)
where
H(p) =
> (pu)logp(1) logp(u).
(4)
vEP
To avoid bias toward ground-truth references, we use model- generated sequences (Y:) instead of official gold solutions. Meanwhile, the distance measured at a low temperature T 0.5 to ensure the generation stability.
We evaluated several language models on the MATH and MBPP datasets, including pretrained, instruction- -inetuuddd..iieuuedd and task- fisktfinetuned models. Figure plots the midpoint of the optimal temperature range against our distance metric, demonstrating strong negative correlation. Specifically,
Since the single-peak behavior, we define the e-optimal
Model link: http://hwuginngaaecccooeettiippp
4

---

Optimizing Temperature for Language Models with Multi-Sample Inference
across our model set, the correlation on MATH is -0.895, while on MBPP it is 007777
algorithm in Section 4.3.
4.1. Entropy Spike as an Indicator of Quality Collapse
In practice this suggests using higher temperature (e.g., T = 0.9- for fosk-finetuned models and lower temper- ature (e.g., T = 0.5-0.7) for more seneral-purpose models (pretrained or caaructtio-finetuned
First, we discover a surprising phenomenon that we call the entropy spike. Specifically, increasing the temperature smoothly increases the model's entropy, until dramatic spike where the entropy rapidly increases. We believe the spike is a good indicator of sample quality collapse.
(a)
Entropy Curve of Llemma-7b
Entropy
As illustrated in Figure 4(a), we calculate the token-level entropy at different temperature levels (solid blue line). To reduce computational overhead, we compute the entropy only over the top- K tokens (with the highest probabilities) each step, setting K 1000 in all subsequent experiments. The entropy curve remains stable for lower temperatures but then shows a sudden rise. One might attribute this behavior to temperature's role in flattening the distribution (Equation 1). However, the following analysis indicates that this spike reflects a substantial change in the model's next-token distribution.
Entropy Roed Generatin
6-
Tuming
1
Token-Level Entropy
LegiEntropy)
Convex
4-
Concave
3-
gap
-2
1-
0.2
0.4
0.6
1.2
1.4
(b)
Temperature
Temperature vs. Accuracy
Lenmaa-
Lemma-2 78 turning point
Specifically, we constrain the model to generate the same outputs produced by greedy decoding while evaluating en- tropy under higher temperature (dotted blue line). If tem- perature alone were responsible for the entropy spike, these fixed outputs would yield a similarly high entropy. How- ever, as shown in Figure 4(a), we observe a significant gap between these two entropy curves, indicating that the actual sampling distribution undergoes large shift.
0.50
Jemma-77 78-4::
Lemma:27 78 turnnng point
Accurecy (Mv8120)
0.45
0.40
0.35
0.30
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Temperature
Thus, we infer that the sudden rise in the entropy curve implies a substantial drop in sample quality. Setting the temperature around this sudden rise can balance sufficient diversity without a large quality drop, which is suitable for multi- saltle aggregation strategies.
Figure 4. Entropy Curve Characteristics. (a) The token-level entropy H. (solid blue line) increases slowly lower temperatures and then jumps sharply at a critical turning point. In contrast, the entropy for fixed (greedy) generation stays low (dotted blue line). log(H) (red line) reveals a transition from concavity to convexity that aligns with the sharp increase in H. marking the entropy turning point (EntP). (b) EntP aligns with the best temperature and varies across different models.
4.2. Turning Point Temperature Selection (TURN)
Given the token-level entropy curve of a language model on a specific task, how can we identify suitable temperature for multi-sample aggregation strategies? Inspired by the difference in the shapes of the entropy curve: When the tem- perature remains low, the entropy increases flatly. However, when the sampling temperature is near the spike, the entropy increases (super)-exponenttallly implying a quality drop in samples. Therefore, after taking the logarithm of the entropy curve (shown in Figure 4(a), red line), the flat part becomes concave while the exponentially-incressii part becomes convex. We define the entropy turning point (EntP) as the temperature where the log entropy curve becomes convex. Figure 4(b) tests the llemma-76 7 base model and its task- finetuned variant (Sun et al., 2024), and EntP matches the position with the highest accuracy and varies between dif-
4. Entropy-Based Automatic Temperature Selection
Determining an optimal sampling temperature is crucial in multi-sample aggregation strategies, yet existing approaches often rely on labeled data or tuning on a validation set. This reliance becomes problematic when no such data are available. In this section, we show how to leverage token- level entropy as an intrinsic property to pinpoint suitable temperature without labeled data. We first demonstrate spike on loken-level entropy as a signal of quality collapse in Section 4. Then develop a method that automatically selects temperature using an entropy turning point ( (EntP) derived from the spike in Section 4.2. Finally, we applied a stochastic process model to explain the mechanism of our
Model link: http://hwggiigaaaccccooccaaaaaaa meeeemttt-leee--ltoae
5

---

Optimizing Temperature for Language Models with Multi-Sample Inference
ferent models. Based EntP, we develop a new method for automatic temperature prediction in multi-sample aggrega- tion strategies, called Turning Point Temperature Selection (TURN).
Temperature Entropy and Log Entropy Simulation
Entrnyy
Timing
Erbrayy prow2)
Log Ertropy
Entropy
The optimal temperature should be around EntP to achieve both sample quality and diversity. At the same time, we found that some aggregation methods may be more tolerant of quality drops (e.g., for best-of-N, only one sample is enough be correct). So we added small adaptation factor | based on the aggregation function, and it is set to 0 and +0.1 for majority voting and best-of-N, respectively. The aggregation adaptation for best-of-N is calculated in the MATH dataset but can be directly applied to other tasks. Refer to Appendix C for more details.
08
(b)
Temporature
Temperature v.s. Improper Takens Simulation
Temperoture Impraper Tikers
Jurning Point: 0.73
Improper Tokenss (%a
0.000
Specifically, given language model M, a task T {X....., } with i imput instances, and an aggregation method A. To estimate token-level entropy, we randomly sampled N times. In each time, we randomly choose an input instance X; and generate one sample by M under each candidate temperature t, = j. (with t being the tem- perature interval and j = 0, J, where J = /t)x//)) These entropies are then aggregated calculate the average entropy H((t) at each temperature ty. Taking the logarithm, we obtain E(tz) = log H((t)).
0.50
020
Temperature
Figure 5. Stochastic Process Model. We run our process model the setting: No = 10, Ni 30000, Lo 0, ao 1, L 100. and a 2. (a) The entropy curve similar to that the real language model: flat at first, and then sharply increases. (b) We calculate the relation between temperature and the percentage of improper tokens in the simulation, and the percentage of improper tokens quickly increases after EntP.
Next, we identify the EntP index j", where the second derivative of f changes from negative to positive, and select its corresponding temperature j" t. Then we add the aggre- gation adaptation factor 3 to form the final prediction. The pseudocode for our algorithm listed in Algo. 1.
4.3. Stochastic Process Model
We applied a stochastic process model to explain why the entropy curve exhibits a sudden spike and what that spike signifies.
Algorithm 1 TTunning Point Temperature Selection (TURN)
1: Input: Language Model M, task T = (X1,.., (its...,x)]] Xx), temperature interval f, maximum temperature maxx sample size N, aggregation method A.
Because inference is sequential, when the language model makes eπror (for example, by sampling an improper to- ken), it increases the likelihood of further mistakes. Mean- while, the model may occasionally recover and return to a correct trajectory.
2: Output: Predicted Temperature T pred-
3: Compute J = [maaa/ /t] (Number of choices)
4: Initialize entropy list 8
5: for n = 1 to N do
To simulate this process, we adopt stochastic process model with K steps in sequential, generating token in each step. the start, the model has initial error rate p = Puni representing the probability of selecting an improper token. At each step, if the model selects an improper token, the likelihood of further errors increases to (1 - p)),, where a > 1 is called the noise tolerance rate. Conversely, if the model selects proper token, the error probability decreases to pd (but cannot be smaller than Pinit).
6: Randomly select X; from T
7: for j = to do
8: Generate sample Y using M with 7 j
9: Compute token-level entropy of Y, add to Eli]
10: end for
11: end for
12: Compute H((j) Mean (E(j)) for all j
13: Compute {(i) = log H(j for all j
14: Find jr = arg min,
J drf > 0)
To build a bridge between the temperature T and the initial error rate Pinit, we propose an estimation. All tokens are labeled proper or improper irrelevant to contexts, and the number of improper tokens (N,) is much larger than that of proper tokens (N₂). In the beginning, proper tokens have high logits To with variance N NO0 af) to reflect
15: Compute 1-
16: Add adaptation factor A:: Tpee = + BBA
17: Return T pred
6

---

Optimizing Temperature for Language Models with Multi-Sample Inference
the nature that there may be several proper next tokens with similar logits. Improper tokens have uniformly low logits L1. Then, the initial error rate Pinit is determined as the probability of selecting an improper token based on the logits and temperature During inference, all improper tokens equally share the error rate p. while proper tokens account for the remaining probability based on their logits.
Table I. The prediction from our algorithm TURN (Pred.), the optimal lemperature ranges (Opt. Range) from grid search, and the performance drop (PD) for various models tested in the MATH and MBPP datasets. TURN achieved hit rates of 12/13 and 11 /13, average temperature gaps 0.023 and 0.015, and average performance drop of 0.32% and 0.59%.
MATH with Majority Voting (Hit Rate: 92.3 %)
Using this setup, we can estimate the token-level entropy. As shown in Figure 5(a), the simulated entropy curve (blue line) aligns well with the observed entropy curves of real language model (Figure 4(1) solid blue line). Meanwhile, Figure 5(b) shows the relationship between the temperature and the percentage of improper tokens, which rises quickly after EntP. This observation suggests that, before EntP, in- creasing the temperature can help explore the proper tokens. However, after EntP, the increase in the percentage of im- proper tokens makes the model uncertain and creates errors, implying a quick drop in sample quality. The behavior of the stochastic process model is consistent with our observa- tions of language models, proving that token-level entropy a good indicator of sample quality. Detailed formulas and experiments can be found in Appendix B.
Model Name
Pred.
Opt. Range
PD (4)
mistral-7b-sft
0.9
0,5 1.5
0.75%
math- hiiis-shhherdmissl-lhhe
0.9
0,5-1.3
0%
Misvral-7B-v0.33
0.7
0.3-0.7
0.37%
Mistral 7B Instruct-vO.
0.7
0.5-0.7
0%
deepscek-mathy satbcekk-mtt--b---aa
0.6
0.5-0.7
0.5%
doepscek-math- -btt--b--i-t-ru
0.8
0.5-1.3
0.75%
llemma-7b
0.7
0.3-0.7
0%
lemmnn 7b-mmn-----de 7b- be-stt-ettmmatb-bl
1.1
0.7-1.5
0%
Tamm- 3.1.1.1.... Instrct
0.6
0,3-0.7
0%
Llama-3.1 1-8B
0.6
0.5
0.5%
Liama 3.2-3B 1-2-tnt-
0.7
0.5-0.7
0%
Llama-3 3.2-3B
0.6
0.3
1%
Open MamnMah22-11mm3...
0.8
0.5-1.1
0.25%
MBPP with Best-of-N (Hit Rate: 84.6%)
5. Evaluating TURN
Model Name
Pred.
Opt. Range
PD (L)
We want to answer the following research questions about our approach TURN for selecting the optimal temperature:
deepseek-coderr 7b-buse-v11555
1.0
0.7-0.9
2.70%
deepseek-soder- weweeeee-soode----iiseeeee 7h-insstrictt-11..5
1.0
1.1-1.3
1.77%
CodeL 7b hf
0.8
0.7-0.9
0%
RQ1: How is the accuracy of TURN in automatic temperature prediction?
Codel Jawaa-a--yyhoonnnnn
0.9
0,7-0.9
0.71%
Boolellama-b-instrueee
0.9
0.9-1.1
0%
Qwen2.5 Coder:7B
0.9
0.7-1.1
0%
RQ2: How efficient is TURN regarding the number of samples (the parameter N Algo. 1)?
Qwen2.5 Drven25-CCdder-P7 II-Inttu
0.9
0.7-1.1
0%
Vi-coder-9R
0.7
0.7-1.3
0.97%
Yi-Codex-9B-chat
0.9
0.9-1.5
0%
Through experiments, TURN proves effective across mod- els, aggregation strategies, and tasks while remaining effi- cient, requiring only few samples for temperature predic- tion.
Llama-3.1
0.9
0.5-0.9
1.08%
Llama-3. I-st-ct
0.8
0.7-1.1
0.39%
Mistral v0.
0.8
0.5-0.9
0%
Misttal-7B-Instreecs
0.7
0.7-0.9
0%
5.1. Experiment Setup
et al., 2024)), and fine-tuned models (Math-Shepherd (Wang et al., 2024c), Easy-to-Hard (Sun et al., 2024)).
We evaluate our methods in two scenarios where sampling- based inference is widely used: Math Problem Solving with Majority Voting and Code Generation with oest-of-N. The datasets and models are as follows:
Code Generation: For code generation, we use the MBPP dataset (Austin et al., 2021), selecting the first 100 pro- gramming problems. Accuracy is measured using pass@ K, where correctness is determined by passing provided unit tests. We regard the unit tests as the best-of-N strategy with perfect reward model to rank answers. Besides general-purpose models, we evaluate code-specific mod- els, including Deepseek-Coder (Guo et al., 2024), Codel.- lama (Roziere et al., 2023), Qwen2.5-Codes (Hui et al., 2024), and Yi-coder (01.AI, 2024).
We assess language models" rea- soning abilities using the MATH dataset (Hendrycks et al., 2021), which consists of competition-level math problems. To accommodate multiple models, randomly select 200 test problems (40 per difficulty level). Accuracy is measured based on majority voting. We test pereral-purpose models (Llama (Dubey et al., 2024), Mistral (Jiang et al., 2023)), domain-specific models (Llemma (Azerbayev et al., 2023), OpenMath2 (Toshniwal et al., 2024), Deepseek- Matsh (Shao
Math Problem Solving:
Implement Details: For both tasks, we sample 256 times
7

---

Optimizing Temperature for Language Models with Multi-Sample Inference
Table Comparison Between TURN and Fixed Temperatures. compared TURN to various fixed temperatures under two metrics: The sum of Temperature Gap and the average Performance Drop. '-Ada.' means removing the aggregation adaptation factor B. Although some temperatures are generally suitable for multi- sample aggregation strategies (i.e. ''' 0.7 or ''' 0.9). TURN can outperform any single fixed temperature across any dataset, highlighting the strong performance of TURN in automatic temperature selection. The underline means not inferior to the best fixed temperature, and the bold the best result.
Fixed Temperature
TURN
TURN -Ada.
0.1
0.3
0.5
0.7
0.9
1.1
Sum Gap (L)
4.6
2.0
0.4
0.4
2.0
3.6
0.3
0.3
MATH
Avg. Drop (!)
8.6%
3.5%
1.0%
0.8%
3.1%
7.2%
0.3
0.3 %
Sum Gap (L)
8.2
5.6
3.0
0.8
0.2
1.2
0.2
0.6
MBPP
Avg. Drop (L)
22.5%
10.7%
5.1%
1.5%
0.9%
4.2%
0.5%
1.1%
Sum Gap (U)
6,4
3.8
1.7
0.6
1.1
2.4
0.25
0.45
Average
Avg. Drop (4)
15.55%
7.1%
3.05%
1.15%
2.0%
5.7%
0.4%
0.7%
per question each temperature level and compute accuracy across different sampling sizes. For temperature prediction in TURN, we use an interval of = 0.1 and set N 8 x dataset size (an excessive sample size, see Section 5.4 for discussion). Additional inference configurations are detailed in Appendix A.
5.3. Baseline
As no existing method automatically adjusts tempera- tures in multi-sample aggregation strategies, compare against fixed temperature baseline. We search for {0.1, 0.5, 0.7, 0.9, 1.1] and select the temperature that maximizes the overall accuracy. This mimics a common, yet suboptimal practice where developers apply a single temperature across all models, disregarding variations in model behavior and task requirements.
Table 3. Variance of Entropy Estimation. We report the aver- age variance the entropy curve and the variance of estimated temperature feed under different sample sizes with 50 trials Llama3.1-88 1-taaa3.----Imsss -nsttuuct on MATH. A small sample size (e.g.. 40) is sufficient for entropy estimation in TURN for its low prediction variance and small performance drop.
5.4. Results
We evaluated 13 models on two tasks (with ma- jority voting) and MBPP (with Best-of-N) and present the results in Table 1. Recall Figure I(b), the correlation coefficient between the accuracy of the predicted tempera- ture and the best accuracy from grid search is 0.9998 for MATH (and 0.9913 for MBPP). TURN achieves a Hit Rate of 12/13 on MATH and 11/13 on MBPP, indicating strong performance across most models. The Temperature Gap remains minimal even when the predicted temperature falls outside the e-optimal range (0.023 for MATH and 0.015 for MBPP). Compared to the best temperatures found through the grid search, TURN incurs only small drop in average performance (0.32% and 0.59%, respectively). Full per- model results and predicted turning points are provided in Appendix D.
Sample Size (LV
10
40
100
800
Mean ( (Var (H(-)))
0.084
0.022
0.010
0.001
Var (Teee)
0.020
0.005
0,003
0.001
Performance (r)p(L)
0.9%
0.2%
0.1%
0.0%
5.2. Evaluation Metrics
To assess the performance of our algorithm for automatically selecting the optimal sampling temperature, we define the following key metrics (all the metrics are calculated under a large sample size of 128, refer to Section 3.1 for discussion):
Metries: We use the following metrics to evaluate the accu- racy and reliability of our temperature prediction algorithm:
Comparison with Fixed Temperatures: We next compare TURN to fixed temperature baseline. Specifically, we sample temperatures from 0.1 to 1.1 at intervals of 0.2 and report the Temperature Gup (TG) and Performunce Drop (PD) in Table 2. Our method outperforms the best of fixed temperatures by 0.5% on MATH and 0.4% on MBPP in average accuracy. When both tasks are combined, the mar- gin increases to 0.75% highlighting the benefit of adaptive temperature selection over uniform fixed temperature.
Hit Rate (HR): The frequency with which TURN selects a temperature within the e-optimal range", indi- cating practical reliability.
Temperature Gap (TG):
The absolute difference be- tween the predicted temperature and the nearest bound- ary of the e-optimal temperature range.
Performance Drop (PD):
The accuracy loss compared to the best temperature found by grid search.
Number of Samples for Temperature Estimation: Fi-
DDefined Section 3.1.
8

---

Optimizing Temperature for Language Models with Multi-Sample Inference
nally; we assess the efficiency of TURN by examining the prediction variance under different sample sizes for Llama- 3. 3.1-8B-Ine ---t-unnt on MATH. As shown in Table 3, we report the average variance of the entropy curve in all choices of T, the variance of the predicted temperature, and the aver- age performance drop. We find that even with a moderate sample size (e.g., 40 samples), the variance remains low and the performance drop is tiny (0.2%), suggesting that small sampling budget is sufficient for accurate temperature estimation and thus proves the efficiency of our algorithm.
Azerbayev, Z.,, Schoelkopf, H., Paster, K., Santos, M. D.,
McAleer, S., Jiang, A. Q.. Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for math- ematics. arXiv preprint arXiv:2310 10631, 2023.
Ben Allal, L., Muennighoff, N., Kumar Umapathi, L., Lipkin, B., and von Werra, L. A framework for the evaluation of code generation models. wtt.:/ccc////iia. wttpbiiggooopppjjjj.....p #iseodd--vvlllaaioon--aaeeee
2022.
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. P. P.D.O. O., Kaplan, J., Edwards, H., Burda, Y., Joseph. N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint 100.i:2107.03374 2021.
6. Conclusion
In this paper, we investigated the critical role of temperature in multi-sample aggregation strategies. We observed that the optimal temperature varies significantly across models due to differences in training strategies and data distributions. By analyzing the relationship between training-testing dis- tribution similarity and the optimal temperature range, we identify a strong correlation that provides valuable insights into model behavior. Furthermore, we proposed the first method for automatically predicting optimal temperatures across diverse tasks, achieving this without labeled data. deeper understanding of tem-
Dhuliawala, S., Kulikov, I, Yu, P., Celikyilmaz, A., Weston, J., Sukhbaatar, S., and Lanchantin, J. Adaptive decod- ing via latent preference optimization. arXiv preprint arXiv:241 1.09661, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint 247722240..21783, 2024.
Our findings contribute perature's s impact on language model performance and offer practical approach for optimizing inference settings.
Guo, D., Zhu, Q. Yang. D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401. 14196, 2024.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv preprint arXiv:2 0et:221033030 03874, 2021.
Acknowledgments
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint TTt4:1904.09555 2019.
SW would like to thank Convergent Research and the Lean Focused Research Organization and the Microsoft Acceler- ating Foundation Models Research Initiative.
Hui, B., Yang. J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
References
Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
0I.AI. Meet yi-coder: A small but mighty 11m for code, September 2024. URL hitps: //huggingfacc. co/blog/lor/ iimanima//// /yider
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., and Yin, W. Large language models for mathematical reasoning: Pro- gresses and challenges. arXiv preprint arXiv:2402.00157, 2024.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C.,, Chaplor, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.., et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2 (oo:2108.000 108.07732. 2021.
Kamalloo, E., Dziri, N., Clarke, C. L., and Rafici, D. Eval- uating open-domain question answering in the era of large language models. arXiv preprint arXiv:2305.06988 2023.
9

---

Optimizing Temperature for Language Models with Multi-Sample Inference
Kool, W. Van Hoof, H., and Welling, M. Stochastic beams and where find them: The gumbel-top-k trick for sampling sequences without replacement. In Interna- tional Conference on Machine Learning. pp. -409- PMLR, 2019.
Sun, Z., Yu, L.., Shen, Y., Liu, W., Yang, Y., Welleck, S., and Gan, C. Easy-to-hard generalization: Scalable alignment beyond human supervision. arXiv preprint arXiv:2403.09477 2024.
Toshniwal, S., Du, W., Moshkov, I., Kisacanin, B., Ayrapetyan, A., and Gitman, I. Openmathinstruct-22 Accelerating ai for math with massive open-source in- struction data. arXiv preprint arXiv:2410.01560 2024.
Kumar, A. and Sarawagi, S. Calibration of encoder decoder models for neural machine translation. arXiv preprint arXiv:1903.00800 2019.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Wang, E., Cassano, F., Wu, C., Bai, Y., Song, W., Nath, V., Han, Z., Hendryx, S., Yue, S., and Zhang, H. Planning in natural language improves I1m search for code generation. arXiv preprint arXiv:2409.0373. 2024a.
Wang. L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J.,
Li, Y., Mi, F,, Li, Y., Wang, Y., Sun, B., Feng, S., and Li, K. Dynamic stochastic decoding strategy for open-domain dialogue generation. arXiv preprint arXiv:2406.07850, 2024.
Chen, Z.,, Tang, J., Chen, X., Lin, Y., et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, (8(6):18634) 2024b.
Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Verify and reinforce Iims step-by-step without human annotations. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9426-9439, 2024c.
Lin, H., Sun, Z., Yang, Y., and Welleck, S. Lean-star: Learn- ing to interleave thinking and proving. arXiv preprint arXiv:2407.10044 2024.
Ma, P. Wang, T.H.., Guo, M., Sun, Z., Tenenbaum, J. B., Rus, D., Gan, C., and Matusik, W. Llm and simulation as bilevel optimizers: A new paradigm to advance physical scientific discovery. arXiv preprint arXiv:2405.09783, 2024.
Wang. X., Wei, J., Schuurmans, D., Le, Q., Chi, E.,, Narang, S., Chowdhery, A., and Zhou, Self-consistency im- proves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 2022.
Meister, C., Pimentel, T., Wiher, G., and Cotterell, R. Lo- cally typical sampling. Transactions of the Association for Computational Linguistics, 1 111102-121, 2023.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of- of-thought prompting elicits reasoning large language models. Advances in neural information processing systems. 35:24824-24837, 2022.
Nguyen, M. Baker, A., Neo, C., Roush, A., Kirsch, A., and Shwartz-Ziv, R. Turning up the heat: Min-p sampling for creative and coherent Ilm outputs. arXiv preprint arXiv:2407.01082, 2024.
Welleck, S., Bertsch, A., Finlayson, M., Schoelkopf, H., Xie, A., Neubig, G., Kulikov, I., and Harchaoui, Z. From de- coding to :etaageneration: Inference-time algorithms for large language models. arXiv preprint arXiv:2406. 16838, 2024.
Renze, M. and Guven, E. The effect of sampling tempera- ture on problem solving in large language models. arXiv preprint MXXi:22402.052011 2024.
Pareeee, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J., Ellenberg, J. S., Wang, P., Fawzi, O., et al. Mathematical discoveries from program search with large language models. Nature, 625 (7995):468-477) 475, 2024.
Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. Infer- ence scaling laws: An empirical analysis of compute- optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.00724, 2024.
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. Code llama: Open foundation models for code. arXiv
Xie, J., Chen, A. S., Lee, Y., Mitchell, E., and Finn, C. Calibrating language models with adaptive temperature scaling, arXiv preprint vrXiv:2409.19817 2024.
preprint ar arXiv:2308. 12950, 2023.
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
10

---

Optimizing Temperature for Language Models with Multi-Sample Inference
Zhang, H., Du, W., Shan, J., Zhou, Q., Du, Y., Tenenbaum, J. B., Shu, T., and Gan, C. Building cooperative embodied agents modularly with large language models. arXiv preprint 23x.v:2307.0248... 2023.
Zhang, K., Zhou, S., Wang, D., Wang, W. Y., and Li, L. Scaling IIm inference with optimized sample compute allocation. arXiv preprint
2024a.
Zhang. S., Bao, Y., and Huang, S. Edt: Improving large language models' generation by entropy-based dynamic temperature sampling. arXiv preprint arXiv:2403. 14541, 2024b.

---

Optimizing Temperature for Language Models with Multi-Sample Inference
A. Inference Configuration
A.1. Software
Our experiments build upon two open-source projects: Easy-to-Hard Generalization (Sun et al., 2024) for the MATH dataset and bigcode-evaie evaauation--aanne (Ben Allal et al., 2022) for the MBPP dataset. We employ vLLM (Kwon al., 2023) to accelerate inference. All experiments be reproduced single L40S A6000 GPU.
A.2. Sampling Hyperparameterss
We use zero-shot inference for models fine-tuned specifically for each dataset. For general-purpose models, we use four in-context examples (few-shot inference) to ensure correct output formatting. The maximum output length is set to 1024 tokens for all tasks. For the MATH dataset, we top-k sampling with k: 20. No additional sampling constraints imposed for the MBPP dataset.
A.3. Metric Calculation
To compute the majority vote results for the MATH dataset, consider two samples have the same answer if they match after normalization. For the pass @ K metric. we follow the definition in Chen et al. (2021). Let N be the total number of samples and C' be the number of correct samples. Then pass@ K is defined as:
pass@K =
(5)
B. Details of the Stochastic Process Model
We introduce stochastic process model to explain that (1) the token-level entropy increases steadily at the beginning but rises rapidly when the sampling temperature reaches certain threshold. (2) The optimal temperature is near the turning point when using multi-sample aggregation strategies.
The stochastic process model has two underlying assumptions: (1) Every token can be labeled as 'proper' or 'improper" at each decoding step. Generally, proper tokens have relatively higher logits than improper tokens, while the number of improper tokens is much higher than that of proper tokens. (2) When an improper token is generated, improper tokens have higher generation probability in the next step, and vice
Under these two assumptions, we can calculate the token-level entropy under different sampling temperatures, and the temperature-entrrr curve fits that of real language models. Meanwhile, the percentage of improper tokens quickly increases after the turning point, implying quick drop in sample quality in real language models.
B.1. Model Setup
1.1.1. INITIAL CONDITIONS
We consider discrete-time process {2, ] where each E [0. 1] represents the model's probability of producing an improper token at time step L. We start with an initial error rate:
Zo Zinit [0, 1].
Conceptually, Pma corresponds the model's baseline 'error propensity' the start. This value is related to the sampling temperature T' of the language model: higher T' typically yields flatter probability distribution over tokens, increasing the chance of selecting improper token and thus increasing Finit (See Section B. 1.4 for heuristic link between temperature and initial error rate.)
B. 1.2. INTERPRETING THE ERROR RATE
At each step f, the language model chooses a single token, and each token classified as proper or improper. Although in practice, the correctness of token depends on the context and is not truly binary, we approximate this by treating correctness as Bernoulli trial:
12

---

Optimizing Temperature for Language Models with Multi-Sample Inference
Probability of producing an improper token: Zizi
Probability producing a proper token: Ztz.
Define random variable E, that indicates whether an error occurred time step
{1
with probability 2;, (improper token), with probability 1 - 2t, (proper token).
Es =
B.1.3. ERROR RATE UPDATE RULES
After each step, the error rate of the next step 2++1 updated based on whether the token this step / is proper or not:
If an error occurs (E, = 1): The error rate of the next step r++1 increased. Intuitively, making a mistake can make the model more likely to continue making errors. Formally, we update:
++
x4
can be seen that 21+1 2 2, for n, [0. 1] and > (a is hyperparameter). Here, a can be considered as the noise tolerance rate, measuring how stable the model is when it experiences unexpected noise, and we try different a in experiments.
If proper token is produced (E, 0): The error rate of the next step T++1 is reduced, reflecting a 'reinforcement of correct behavior. We do this by:
Tz++1 = max( Finit)-
It generally makes 1 xy smaller, this update lowers the error rate. In particular, do not allow the error rate drop below the initial baseline Winit-
B.1.4. 1.4. LINKING INITIAL ERROR RATE AND TEMPERATURE
At time step f, the token probability mass of the model divided into:
Improper tokens with total probability P1iimproprr 21;
Proper tokens with total probability Po.proper 1 Tt--
Therefore, by definition, we have Zinit = immipproeer Under higher temperatures T, the softmax distribution flattens, increasing Pi,improper because the number of improper tokens is large but their logits are low. Thus, 2iit increases as T increases.
B.1.5. TYPE OF TOKENS DURING DECODING
The probability of tokens when decoding is usually multi-peak (i.e., except for the token with the highest logit, some other tokens have reasonably high logits and are also acceptable during decoding), so it is natural to consider a scenario with three categories of tokens:
Proper tokens:
Small number of tokens with high logits. Let No be the number of proper tokens. To capture the multi-peak behavior, the logits of proper tokens .o.,, lo,N, are sampled from the Gaussian distribution A VLLo oo).
Low Probability Improper tokens: Many low-logit tokens where language models seldom choose them. Let N1 be the number of tokens of this type, and their logits are set to L, for simplicity.
High Probability Improper tokens:
Due to insufficient training or mistakes in training data, some tokens may have exceptionally high logits but are logically improper in (e.g., the token 3 in + 1 = 3). Since different language models behave differently, we only consider decoding without sigh-probability improper tokens in our discussion.
For the first two types of tokens, we have Lo > In, No & Ni.
13

---

Optimizing Temperature for Language Models with Multi-Sample Inference
B. 1.66. TOKEN PROBABILITY DURING SAMPLING
At each step L, the probability of producing improper tokens is It. Let Pt.pooprimpooprr be the probability of the j-th proper/improper tokens at step t. For improper tokens, have:
It
immpoorr
Ni
For the proper tokens, we allocate the remaining probability x, according to their relative logits:
proppop = (1 - zz st)ssoftmax((........ lo.o Vj E [1,
This ensures that the relative order the probabilities for the proper tokens remains determined by their logits, while the total mass allocated to the improper tokens It-
B. 1.7. ENTROPY CALCULATION
We define token-level entropy H. on a sequence of decoding steps:
H
Ptj Pt.j
P log lo propperr
Pt. improperr log imppoperr
K
Here, K is the total number of decoding steps considered.
B.2. Experiment
B.2.1. MODEL HYPERPARAMETER
The stochastic process model has the following hyperparameters:
The numbers of proper and improper tokens: No. Ni:
The logits of proper and improper tokens: (o.( (1....... where:
los NNoo, 0o); h L1;
The number of total steps K;
The noise tolerance rate a.
The input is the temperature and the output is the average token-level entropy H. over 500 seeds. In our experiment, the hyperparameters are set to be:
No = 10, N₁ = -3000, Lo = 0, I = -100 do 1, K = 512.
Furthermore, we tested different noise tolerance rates a E 2..5, to show the behavior under different noise tolerances.
B.2.2. RESULT
Figure is the temperature entropy curve derived from the toy model different noise tolerance rates The curves with different noise tolerances have similar shape. Generally, the curve be divided into two parts: (sub)-linear increase and (super)-exponential increase. In the first part, the increase is due to the various choices among the proper tokens, while the sharp rise in the second part is due to the loss of control (i.e., the model frequently chooses improper tokens and then makes the error rate extremely high).
In particular, the curve very similar to the behavior real language models, and some reference entropy curves and log-entropy curves are shown Figures 10 and 11.
14

---

Optimizing Temperature for Language Models with Multi-Sample Inference
Temperature v.s. Entropy in Simulation
alpha=1.5
alpha=2
alpha-2.5
alpha-3
Entropy
Turning Point: 0.73
Temperature
Figure The temperature- curves.
Temperature Log Entropy in Simulation
alpha-1.5
alpha-2
alpha=2.5
Log Entropy
alpha=3
Turning Point: 0.73
0.
94
0o
0.5
2.2
L2
Temperature
Figure The temperature-log entropy curves.
Figure 8. The curves derived from the stochastic process model under different a.
Relation to Improper Token Rate It natural to consider that proper tokens can lead to correct final answers and that improper tokens will result incorrect final answers, so we measure the percentage of improper tokens. As shown in Figure 9, when the temperature exceeds the turning point, the percentage of improper tokens increases quickly, implying quality drop in the samples. Interestingly, the difference in noise tolerance rates has little inference the turning point but controls the improper token increasing speed after the turning point. However, the percentage of improper tokens increases rapidly in all tested a.
B.3. Conclusion
Our stochastic process model provides simplified but insightful framework for understanding how lemperature-depeeddent sampling dynamics can lead to characteristic shifts in the model output distribution. The model predominantly chooses from the proper tokens the low-temperature (or sublinear growth) regime, resulting in relatively stable and controlled outputs. The distribution flattens, and nonsense tokens gain significant probability mass as temperature increases beyond a certain threshold. This transition leads to a sudden and steep increase in entropy-mirrorin observations actual language models-ate corresponding drop the correct rate. Therefore, increasing the temperature can initially increase generation diversity (sampling among proper tokens) with small correctness drop. However, the performance suffers quick drop after reaching certain threshold (i.e., the turning point EntP).
C. A Aggregation Adaptation Calculation
The choice of aggregation function affects the optimal generation temperature. For example, in majority voting, the final answer must selected by the majority, whereas best-of-N, only a single correct instance out of the N samples is required.
15

---

Optimizing Temperature for Language Models with Multi-Sample Inference
Temperature v.s. Improper Token (%) Simulation
alpha=1.5
Improper Token (%6)
alpha-2
6.150
alpha=2:5
alpha=3
Tumning
0623
0.2
04
06
0.6
1.5
12
Temperature
Figure 9, The Temperaluure-mmproerr Token (%) curves.
Plat of Entropy "emperature
Entropy of misttal---- domaii)
Enerpy mestti77--toon somain
Eneoay mesta/77-boot
Entray misrall7--- 7b-stt off domain
Entray drepsoes 76 stt (off domai))
Fermay dreppenc 76 have
Enimps of Lammme-- tame
Endrups of Uemmee-e all (un domain)
thought ertnps
Figure 10. The temperature-entrooy curves.
Not of Lag Entropy us. Temperature
Lop Thoupit Crteey
Frempy of nistral theth fun drow
Frempy of muttral tn consni Enempyy ot mettral Theusse Enerppy of metrrl- Theth font do meen! Ent of Seepseek Th-dh (ott domentt Eir now of deepseek Th bese Eim eetropy of Jomens 7b how
Em of Jumma 7b doman)
Lri et Jame abe
Tangendur
Figure 11. The temperature-log entropy curves.
In the case of majority voting, the turning point on the entropy curve aligns with the optimal temperature, so we set its adaptation to O. For best-of-N, we computed an adaptation on MATH and then tested it on MBPP to confirm generality. Specifically, we averaged the difference between the midpoints of the optimal temperature ranges for best-of-N and majority voting across 13 models on MATH. This difference was 0.092 on average. Therefore, for simplicity, we set the aggregation adaptation for best-of-N 0.1.
16

---

Optimizing Temperature for Language Models with Multi-Sample Inference
Table Aggregation Adaptation for Best-of-N: we calculate midpoints optimal temperature ranges on Majority Voting Best-of-N for MATH. The difference between the average of midpoints is 0,092, so set the adaptation factor to 0,1.
Aggregation
Individual Models (Models are listed the same order as Table 1)
Average
Best-of-N
0.6
0.8
0.6
0.6
0.7
0.5
0.6
1.1
1.2
0.5
0.6
1.3
1.0
0.7769
Majority Voting
0.6
0.9
0.6
0.5
0.3
0.6
0.5
1.1
0.9
0.5
0.6
1.0
0.8
0.6846
D. Results of All Tested Models
We present the accuracy heatmaps and entropy estimations for all tested models. Figure 12 shows the heatmaps of model accuracy for the MATH dataset, while Figure 13 displays the heatmaps for the MBPP dataset. Additionally, Figure 14 illustrates the entropy curve estimations for the MATH dataset, and Figure 15 provides the entropy curve estimations for the MBPP dataset.
17

---

Optimizing Temperature for Language Models with Multi-Sample Inference
Accursty/ leetmmp of Ulama-3 3-t---::-0mnmnn 1-00-insstucct
Accureky youurryy Teelimeg of Llame-3.1 1-00
Pocuretyy Destmmp of Llemm33 2-30-instrucct
0:00
2:44
642
030
207
GU4
5:18
6:22
4.12
02F
6:1
312
1::0
8.1
640
0.57
041
0:43
6:45
0.35
6:50
944
0.41
8.5
0.7
644
847
241
8/44
821
0.71
6.24
6:28
6:10
6.44
0.41
3.50
1.44
037
Serrples
Accuracy Hestmap Liama
Accuracy Heatmap of Mistrol Instructiv. V0.3
Accuracy Hostmop Mistrai vO.3
01L
8:22
3:23
6:30
033
ORZ
6:2
042
6X2
8-02
035
C.M
4.04
C144
6:05
3:07
623
4114
035
044
0.2
400
8:00
064
CM
4:10
A31
312
0.5
322
6:00
5:18
0:31
4.14
332
6:21
0:00
2.44
9.28
0.21
0.11
5.1
5:22
1.
0.77
0:31
612
113
025
C15
A37
A17
313
0.2
822
033
4:55
A11
020
Nameerdd of Samples
Marber Screples
Namber Samples
Accuracy Heatmap of DoenMath2- waw.thh---aamaaaa 1a.. BB
Accuracy Heatmap deepsook math- 70--77----a
Accur acy Heatmap of deapseck math- 7b- Pl-instruct
6:11
200
021
6/1
095
100
675
611
413
X15
6:30
1.14
941
041
042
0:40
4:54
000
6.
059
1.24
6:22
075
7.55
0.26
4:2
4:22
X27
335
041
144
AA3
052
0447
4:44
8.4
5.64
0.55
640
471
632
0:1
1:20
0.42
0448
5.49
550
0:00
6:44
427
037
6.10
8.6
8.20
34L
8:39
Namrer Samplix
Martar Surrpho
Numthur Samplec
curacy /eeetmmo of Iemmm 2b sft metamath Itvel 1003 hf
Accurecy Hestmmp of liermaa 7b
Accuracyy leetmap meth shepherd mistrel 7b
6:1
144
114
0:2
6:51
E10
3/D
6:M
TXB
A11
034
AM
6.7
144
345
8.41
C:X
3:48
6:3
51Y
414
Toype dire
020
6:0
0.44
041
6.17
332
0.00
0.18
6:4
6:44
3.6
DAT
04F
KCS
8:38
0,7
6:32
Accuracy Hoatmap mistral-7D
oar
G/G
032
021
1/3
Torperder
0.43
200
GM
437
0.20
Nammor of Samples
Figure/ 12. The accuracy heatmap for all tested models on the MATH datasct. The green line is our predicted temperature.
18

---

Optimizing Temperature for Language Models with Multi-Sample Inference
Acturacy Ieeetmep of Codellemma--ae herme----innntutt---e -hff
Accurssy lestrop of Cottel -e-p--l--PPyyuuree hrunhd
Aucunatty lestmop of Clstee Jema-7t--rr
0.94
E-I
6.D
6:10
230
0.08 0.04
672
0440
0.53
367
0.70
0:43
0.56
6:02
347
071
0.5
544
047
9.45
1.40
164
027
Serrpins
Accuracy Hestmap Uamo Instruct
Accuracy Heatmaa Liama BB
Accuracy Heatmap Nistral Instruct
6.0
6:91
342
0.44
X12
0.00
6:30
9.45
3.54
044
4:75
C42
405
6.00
9.14
022
6:31
0.55
CM
414
4:14
334
033
(44
0.72
311
6.00
3.7
0:20
6:1
542
848
8:21
9.78
0.30
0.42
0.48
6.35
9.79
0.00
6:33
215
0.9
000
367
902
8:47
0.31
204
242
0.9
071
4.73
X74
4.45
648
853
0.54
0.5
6:57
0:4
6.47
343
055
C55
Nameer of Samples
Mamber Screpies
Namber Samples
Accuracy Heatmap of Mistral- 78 v0.3
Accuracy Heatmap QwenZ.5 S S-Cadar--8 Instnctt
Accuracy Heatmap of Qwan2 5 Cader 78
6:31
c:00
EXE
032
0.5
6:90
340
0.10
3.0
304
039
0.33
6.93
284
0:41
0:32
365
043
6/2
5/1
C40
6/2
617
087
021
CH1
358
052
0.44
6.54
612
6:40
6:01
6:01
5.7
375
037
Namrzr Samplex
Marder Sumppho
Namtbur Sarppics
Accurasy /eetmap of "I Coder 58 chet
Accurscy /leetmp Coder 9B
Accurasy Hestmep of deepseet cocer 7o base v1.5
ART
A77
6AT
643
0:16
E45
075
0.00
6:14
317
8.25
0.44
0.5Y
C-4
631
8.34
347
Toype
072
0.71
304
066
6.91
6:00
0.90
612
A37
031
AIB
044
8:59
RAT
6.73
6.22
080
+:77
Marbee
Accuracy Heatmap deassook: wodar.7b. wtt.ruuuttttttt......
0.2
2:5
C.
Torperdure
0.72
317
C1
200
0.30
304
Namoor of Sarplis
Figure 13. The accuracy heatmap for all tested models on the MBPP dataset. The green line is our predicted temperature.
19

---

Optimizing Temperature for Language Models with Multi-Sample Inference
Entropy Estimation of Uame-3.1-4 3.1e-33------- Insttttt leathereyt
Entropy Estimation Lame 3.1-00
Entropy Estimation of Llame-33 3-2---3--9-- instract Leghteoup
Ing Kaoo
Pimperanure
Terpersture
Terperouue
Ertropy on U..m-3322.8
Entrogy Estimation insttruuttvv..33
Entropy Estimation Mistral-78.v0.
0.0
Suning
Loor meneeee
Tomporature
krrperature
Torpessture
Entropy Estimationn of 1.o
Entrogy Estimation sonpsseskmath-7--bbaa
Entrozy Estimation Rospoosk-matth-b-mnttee
Temparature
Sarperature
Remparature
Entropy Estimatioo Ilemma- 7b sft metamath level 1to3
Entropy Estimetioo Temma, 76
Entropy Estimation of math-hhhsphenn-iitrrl
Deney
Erency
07
Korpouture
Krrpersture
mation of mibstall-b-sft
Saring
04
Figure 14, The entropy curves and turning points of language models when testing on the MATH dataset.
20

---

Optimizing Temperature for Language Models with Multi-Sample Inference
Entropy Estimation CodeLJame- 7b Pi-Iintructt-i
Entropy Estimation of CodeLlame lame /b Python Hd
Entropy Estimation Codellamme 7b ht
Legthereyt
Ing
Logbbrowryy
Loa Sagiinsnrr
L000 Matorooooo
E20000
Trmperauue
Terpersture
Temperouur
Entropy Extimation Ulma-3311 Instruct
Entrogy 3.1.08
Entropy Estimation of Nistral-78 Instructivo.3
Cintrr
0.30
Tarding
Tomporature
Krrpereture
lorpessture
Entropy Ectmation Mistral-7B.v0.3
Enrropy Estimation of Qwen2 S-Coder:7-iinstrue
Entropy Ediimation of Qwen2 3-Coder-7B
Mytatoool
Temparature
Tarppnndere
Remporature
Entropy Extiratiion of Yi-Coder-9B.co
Entropy Ecpimettoon of Coder-9B
Entrogy Estimation deepseek coder base
Deney
Erency
Uog txeneeyp
Lo0 10000000
Komporuture
Kerpersturr
1orpssoure
Estimation degoseck code::7--nstrurrl.5
Saring 6.9
Ungttnireps
Figure 15. The entropy curves and turning points of language models when testing on the MBPP datasct.
21


Reviews:
[{"review_id": "9nymPp6Ljo", "reviewer": "Reviewer_3z2D", "summary": "The paper investigates the role of temperature when using best-of-n and majority-voting inference strategies. The authors discover an intriguing phenomenon coined \"The Entropy Turning Point\" that can accurately predict the change-point between generating diverse high-quality samples and generating low-quality samples. This phenomenon is used to devise an algorithm for automatic temperature selection. In addition, the authors investigate the optimal temperature for pre-trained, instruction-tuned, and domain-fine-tuned LLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3z2D,\n\nThank you for the insightful suggestions. We are glad you find our research problem important and provide positive reviews. Please find our detailed responses below:\n\n> *it'd be interesting to see both code and math models on each of the Figure 3 plots. A good distance measure should identify that code models are further than instruct models from the math task and vice versa.*\n> \n\nTo further investigate this, we conducted additional experiments comparing the distances between different model types and tasks. Specifically, we measured:\n\n- The distance between the **code-finetuned model** (*code-llama-7b-hf-instruct*) and the **MATH** task,\n- The distance between the **math-finetuned model** (*OpenMath2-Llama3.1-8b*) and the **MBPP (code)** dataset,\n- And the corresponding distances for a **general-purpose model** (*Llama-3.1-8B-Instruct*) for comparison.\n\nThe results, shown in the table below, align with expectations: the code-finetuned model is furthest from the MATH task, while the math-finetuned model is furthest from the MBPP task. The general-purpose model falls in between for both. This result implies the robustness of our distance metric.\n\n| Distance | OpenMath2-Llama3.1-8b (finetuned on math) | Llama-3.1-8B-Instruct (general purpose) | code-llama-7b-hf-instruct (finetuned on code) |\n| --- | --- | --- | --- |\n| MATH | **0.0867** | 0.1847 | 0.2152 |\n| MBPP (code) | 1.0159 | 0.1901 | **0.1477** |\n\n> *Could you please provide additional details regarding the prompts used in the experiments? If CoT was not used, I recommend conducting additional experiments with CoT to validate the method in a more practical setting.*\n> \n\nFor the **MATH** task, we **did** use chain-of-thought (CoT) prompting. Specifically, we adopted prompts from the official codebase for models already finetuned on the MATH training set with typical CoT reasoning formats (e.g., *llama-7b-sft-metamath-hf*). For general-purpose instruction-tuned models, we added a **four-shot CoT prompt** containing step-by-step reasoning examples.\n\nFor the **coding task**, models were prompted to directly generate code solutions, following the default configuration from the *bigcode-evaluation-harness* benchmark.\n\nWe hope these clarifications and additional experiments address your concerns. Thank you again for your thoughtful feedback — we welcome any further questions or suggestions.", "claims_and_evidence": "For the most part:\n- Regarding the proposed distance between the model's train data and a given task, to support the claim that it is indeed an informative distance measure, it'd be interesting to see both code and math models on each of the Figure 3 plots. A good distance measure should identify that code models are further than instruct models from the math task and vice versa.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "- The improvements with the temperature chosen by TURN are relatively modest in comparison to the empirically popular t=0.7. However, given the simplicity of adopting the method, even modest improvements are welcome.\n- Best-of-n and majority voting are typically used with CoT-style prompts to achieve the best performance. Could you please provide additional details regarding the prompts used in the experiments? If CoT was not used, I recommend conducting additional experiments with CoT to validate the method in a more practical setting.", "relation_to_broader_scientific_literature": "As test-time scaling gains popularity, choosing temperature is an important consideration.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please see my questions regarding CoT-prompting and the proposed distance. I'd be happy to increase the score if these questions are addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cu0O4yIp9l", "reviewer": "Reviewer_NxtV", "summary": "This paper proposes a method to estimate the right temperature LLM. The method relies on  \"multi-sample aggregation strategies\" which has the advantage to spare the costly and task-specific validation data. The idea is based on an extensive  analysis of the impact of temperature temperature while varying model architectures, datasets, task types, ...  The outcome is a novel entropy-based metric for automated temperature optimization, which allows the model to outperform the  fixed-temperature baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NxtV,\n\nThank you for your valuable and positive feedback. We are pleased to hear that you find our research problem compelling and appreciate the novelty of our proposed algorithm. Please find our detailed responses below:\n\n> *For instance, Llama models commonly use a temperature of 0.7 (see the technical reports), which generally aligns with the authors' reported optimal range across tasks. This is why clearer formatting and additional comparisons could help make this point more evident.*\n> \n\nYou're absolutely right that T=0.7 is often used as a default for general-purpose models like LLaMA, and in many cases, it performs reasonably well. However, when examining the optimal temperature across different models and tasks, we observe clear deviations from this default, particularly for task-finetuned and pretraining-stage models.\n\n- **Task-finetuned models** often require **higher temperatures** to encourage greater diversity in generation.\n- **Pretraining-stage models**, by contrast, tend to perform better with **lower temperatures** to reduce deviation and improve focus.\n\nFor example:\n\n- On the **MATH** dataset, the optimal temperature for *LLaMA3.2-3B* is **T = 0.3**, which results in a **3% performance gain** over the default T = 0.7.\n- Conversely, the optimal temperature for *DeepSeek-Math-7B-Instruct* is **T = 1.0**, outperforming T = 0.7 by **2%**.\n\nThese results highlight the importance of adaptive temperature selection across varying model-task combinations. We’ve improved the formatting in our final submission to make these comparisons clearer and more accessible.\n\n> *The improvements achieved by the method over a fixed temperature setting are marginal*\n> \n\nOne of our method's key strengths is its ability to operate **without any labeled validation data**, making it directly applicable to real-world test sets. As Reviewer 3z2D also noted, even modest gains can be valuable, particularly given this level of generality and ease of use.\n\nOur method achieves an **average performance improvement of 0.75%** over the best fixed temperature across various models and tasks. Notably, **individual models** see gains of up to **2–3%**, which can be significant in competitive or high-stakes settings.\n\n> *The claim that adaptive beta does not require specific tuning lacks sufficient empirical support, as its generality is demonstrated on only one additional dataset.*\n> \n\nThank you for highlighting this concern. To further assess the generality of adaptive beta, we conducted an additional experiment on the GSM8k dataset. Due to the time constraints of the rebuttal period, we evaluated only one model (Llama3.1-8B-Instruct, temperature grid size=0.1).\n\n| GSM8k | Majority Voting | Best-of-N | adaptive beta |\n| --- | --- | --- | --- |\n| Llama3.1-8B-Instruct | 0.9 | 1.0 | 0.1 |\n\nOur results show that the adaptive beta calculated for GSM8k (0.1) is consistent with that of MATH (0.092, reported in Appendix C), providing initial evidence that adaptive beta can generalize to another dataset. However, we acknowledge that its behavior may vary across different aggregation functions, and we leave a more comprehensive investigation for future work.\n\n> *The evaluation process is computationally expensive, and it remains unclear how many samples are actually needed to reliably estimate the optimal temperature.*\n> \n\nWe address this concern in Section 5.4 (Table 3) of the paper, where we analyze how sample size affects temperature estimation. Our findings show that the method is **robust even with small sample sizes**. For instance, with only **40 samples**, the estimated optimal temperature shows a **variance of only 0.005** and a **performance drop of only 0.2%**. Meanwhile, calculating the entropy for 40 samples at a temperature interval costs approximately 2 minutes on an A6000 GPU for an 8B model. This makes the method practical and cost-efficient, even in low-resource scenarios.", "claims_and_evidence": "- The paper is well-structured and easy to follow.\n- The problem tackled is relevant and interesting.\n- The identification of the concave-to-convex shift in entropy as a function of temperature and its connection to model accuracy is particularly insightful.\n- The proposed method enables temperature tuning without requiring labeled validation data.\n\nOverall, the demonstration that the proposed method provides a clear advantage over a fixed temperature baseline. This is the central issue since in practice, when using certain models, a quick online search often yields general temperature recommendations, which tend to fall within the presented optimal range. For instance, Llama models commonly use a temperature of 0.7 (see the technical reports), which generally aligns with the authors' reported optimal range across tasks.\nThis is why clearer formatting and additional comparisons could help make this point more evident.", "methods_and_evaluation_criteria": "- The experiments appear somewhat disorganized. See remarks for specific concerns.\n- The improvements achieved by the method over a fixed temperature setting are marginal, especially given the additional  computational costs of generating multiple samples. See remarks for further discussion.\n- The claim that adaptive beta does not require specific tuning lacks sufficient empirical support, as its generality is  demonstrated on only one additional dataset.\n- The evaluation process is computationally expensive, and it remains unclear how many samples are actually needed to  reliably estimate the optimal temperature.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See the remark above", "relation_to_broader_scientific_literature": "Seems OK", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "See the remark above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9OwkQYyvuI", "reviewer": "Reviewer_TgFe", "summary": "The paper tries to find the optimal temperature on various question answering tasks.\n\nThey notice that the optimal temperature is higher for models that are fine-tuned on the task, and lower for more general models.\n\nMotivated by that, they set out to find a way to set the temperature that doesn't require a validation set.\n\nTo do that, they look at the entropy of generations.  They notice that as you increase the temperature, there is a point at which entropy explodes.  They hypothesise that this is the point at which performance breaks down, which does seem reasonable.\n\nThey therefore set the temperature using this point, which they find by looking at the inflection point of log(entropy).\n\nThey show modest performance improvements using their temperature selection scheme.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer TgFe,\n\nThank you for your supportive review and thoughtful suggestions. Please find our detailed responses below:\n\n> *It would be good to see performance before/after rather than performance drops (as in Table 1). These performance drops are quite difficult to interpret.*\n> \n\nWe appreciate this suggestion. Our goal in presenting performance drops was to highlight how close our estimated temperature comes to the true optimal temperature. The complete results are in **Appendix D.** We will add more explanation about performance drops in our revision.\n\n> *The performance drops look relatively modest in the Tables, but very large in Figure 4A. What's going on here?*\n> \n\nThank you for pointing this out. We believe there may have been a mix-up in the figure reference—**Figure 4A** is unrelated to performance drops. The figure showing drops in accuracy is **Figure 4B**, which visualizes the sensitivity of model performance to temperature.\n\nTo clarify:\n\n- **Table 1** compares the performance of our **estimated temperature** to the **best temperature**, resulting in small performance drops due to the accuracy of our estimation.\n- **Table 2** shows performance drops when using a **fixed temperature** (e.g., T=0.1), which can lead to significant degradation (e.g., up to 15.5%) when the temperature is poorly suited to the task.\n- **Figure 4B** aligns with these findings, illustrating how performance deteriorates when the temperature is too low or too high.\n\n> *I wonder whether what's going is less \"temperature is related to closeness of the task\", and more \"fine-tuned models can in general tolerate higher temperatures than pre-trained models\". Not sure if you did this, but one way to disambiguate would be to take a model fine-tuned on one task, and look at the optimal temperature for a different task.*\n> \n\nThis is a very insightful observation. Our primary claim is that **the optimal temperature correlates with task closeness**—that is, models perform best with higher temperatures when the test task closely matches their training data. However, we added experiments and found that **task-finetuned models applied to unrelated tasks tend to underperform**, making temperature analysis in these **task-mismatch** settings less meaningful.  For example, *CodeLlama-8B*, fine-tuned on code tasks, achieves only **8% accuracy** on the MATH dataset. In such cases, the model’s poor task fit overshadows any insights from temperature tuning.\n\n> *Figure 1A is basically repeated as Figure 4A. I wasn't able to understand Figure 1 on my first reading, so I'd recommend dropping Figure 1.*\n> \n\nThank you for this constructive feedback. We agree that **Figure 1A** may be hard to grasp before reading the introduction. While it provides early motivation, we recognize the potential redundancy with **Figure 4A**. In our revision, we have refined Figure 1 with clearer captions and context, which can be found at [link](https://drive.google.com/file/d/1O9DOpEW0z3GRjID6AQeUHT-Z869Tyrqe/view?usp=sharing).\n\n\n> *Perhaps the one weakness of the paper is that they find lots of interesting phenomena, such as the explosion in entropy at a particular temperature. But there isn't that much analysis of these phenomena. e.g. is there some easy way to see the generations go \"off the rails\"? Is it evident in samples?*\n> \n\nThis is an excellent point. When the temperature exceeds a certain threshold, we frequently see sharp **entropy spikes**, which come from **erratic or degenerate outputs**. For example, consider the following excerpt from a high-temperature generation on a MATH problem:\n\n> **Step 1: Step 1:** In order to find the probability that the product of the two numbers is a multiple of 5, we will find the total number of possible outcomes, and the number of outcomes that are a multiple of 5.  \n… …  \nStep 5: Case 1: The first integer is a 5. In that case, the second integer has a **$1\\\\in\\\\frac{6}{6}$** probability to be the second integer. Thus the number of ways this can occur is $6$.  \nStep 6: Case 2: The second integer is a 5. In that case, the second integer has a **$1\\\\in\\\\frac{6}{6}$** probability to be the first integer. Thus the number of ways this can occur is $6$.  \nStep 7: Therefore, the probability to obtain the desired product is **$$1-\\\\frac{6+6}{36}=\\\\boxed{\\\\frac{5}{6}}$$**\nThe answer is: \\\\\\\\boxed{\\\\frac{5}{6}}  \n**In the provided example, the answer to each step in Question 1 is a numerical value or a mathematical expression that … ...**\n> \n\nIn this generation, we observe several breakdowns:\n\n- **Repetitions**, such as “Step 1: Step 1”\n- **Logical and numerical inaccuracies**\n- **Unprompted continuation into irrelevant content**\n\nThese symptoms clearly illustrate how poor temperature selection, particularly at extremes, can degrade output quality. They also **complement** our entropy-based analysis qualitatively, reinforcing the idea that high-entropy generations often go “off the rails.”", "claims_and_evidence": "The claims made are appropriate to the evidence.  They include data such as plots of temperature vs token entropy for Llemma-7b (Fig. 1a is this a typo?), and plots of the best accuracy, as obtained from a grid search, against the accuracy at the turning point (Fig 1b).  Figure 2a analyses performance vs temperature and number of samples, while 2b looks at the midpoint of the optimal temperature range vs the number of samples for Mistral-7B-Instruct.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs are appropriate to the claims made, and thorough, e.g. Table 1 looks at multiple models on the MATH and MBPP datasets.", "relation_to_broader_scientific_literature": "Yes.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Overall, I think the paper has a number of interesting insights, and should be published.\n\nSome other thoughts:\n* It would be good to see performance before/after rather than performance drops (as in Table 1).  These performance drops are quite difficult to interpret.\n* The performance drops look relatively modest in the Tables, but very large in Figure 4A.  What's going on here?\n* I wonder whether what's going is less \"temperature is related to closeness of the task\", and more \"fine-tuned models can in general tolerate higher temperatures than pre-trained models\".  Not sure if you did this, but one way to disambiguate would be to take a model fine-tuned on one task, and look at the optimal temperature for a different task.\n* Figure 1A is basically repeated as Figure 4A.  I wasn't able to understand Figure 1 on my first reading, so I'd recommend dropping Figure 1.\n* Also, Figure 1A/4A has two different y-axes, which is usually regarded as bad-practice.  But I can see why it makes sense here.\n* Perhaps the one weakness of the paper is that they find lots of interesting phenomena, such as the explosion in entropy at a particular temperature.  But there isn't that much analysis of these phenomena.  e.g. is there some easy way to see the generations go \"off the rails\"?  Is it evident in samples?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dqAIZGffzu", "reviewer": "Reviewer_3FXt", "summary": "This paper introduces TURN, an entropy-based approach for automatically determining optimal sampling temperatures for large language models using multi-sample aggregation strategies. The authors identify that different models require different temperature settings based on their training and observe an \"entropy turning point\" that strongly correlates with optimal temperature values. Their method outperforms fixed-temperature baselines without requiring labeled validation data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3FXt,\n\nWe appreciate your insightful recommendations. Our point-by-point responses are detailed below:\n\n> *Minimal discussion of computational overhead for entropy calculation*\n> \n\nTable 3 in our paper presents the computational overhead associated with sampling. The results indicate that as few as 40 samples are sufficient for temperature estimation. For entropy computation, we leverage VLLM, which supports efficient inference and provides token probabilities. Calculating the entropy for 40 samples at a temperature interval for an 8B model costs approximately 2 minutes on an A6000 GPU.\n\n> *Investigate scaling laws of optimal temperature with model size would be great*\n> \n\nThank you for the insightful suggestion. Due to the limited rebuttal period, we may not be able to complete this analysis, but we acknowledge its importance and leave it for future work.\n\n> *Analyze whether the optimal temperature varies by difficulty level within tasks1.*\n> \n\nThank you for this valuable suggestion. We extended our analysis to test whether the optimal temperature varies by difficulty level on MATH. The table below reports the accuracy across difficulty levels (Model: Llama3.1-8B-Instruct, sample size: 128):\n\n|  | T=0.1 | T=0.3 | T=0.5 | T=0.7 | T=0.9 | T=1.1 |\n| --- | --- | --- | --- | --- | --- | --- |\n| level 1 | 0.90 | **0.91** | **0.91** | 0.90 | 0.88 | 0.86 |\n| level 2 | 0.70 | **0.76** | **0.76** | **0.76** | **0.76** | 0.70 |\n| level 3 | 0.68 | 0.70 | 0.73 | **0.69** | 0.68 | 0.68 |\n| level 4 | 0.45 | 0.51 | 0.54 | **0.59** | 0.46 | 0.36 |\n| level 5 | 0.21 | 0.31 | 0.27 | **0.32** | 0.28 | 0.21 |\n\nWhile no consistent trend is observed in the optimal temperature across difficulty levels, temperature selection becomes more critical for harder problems. For example, the performance difference between T=0.1 and the optimal temperature is 1% for level-1 problems, but rises to 11% for level-5 problems. This highlights the increasing importance of temperature tuning in complex scenarios.\n\n> *1) testing on more diverse domains beyond mathematical reasoning and code generation \nLimited task diversity (only math and code)\nHow does TURN perform on generative tasks with more subjective quality metrics?*\n> \n\nThank you for these suggestions. We primarily focus on problem-solving domains like math and code, where answers from language models can be aggregated and verified objectively. These domains have demonstrated the most success with multi-sample inference.\n\nReward models or human evaluations would be required for generative tasks with subjective metrics. However, these evaluation tools may suffer from reward hacking or be expensive. Therefore, to minimize external factors and focus on the language models themselves, we currently limit our experiments to domains with objective quality metrics.\n\n> *2) including newer/different aggregation methods (e.g., weighted majority voting)*\n> \n\nThank you for the insightful comment. We initially did not experiment with weighted majority voting for two reasons:\n1. Introducing a reward model could introduce biases, such as reward hacking.\n2. If the reward model were perfect (i.e., assigning +1 for a correct answer and 0 for an incorrect one), best-of-N and weighted majority voting would yield identical results.\n\nAdditionally, to address this within the given time constraints, we conducted an experiment using the log-scale probability of generation as the reward, minimizing external bias. We then applied weighted majority voting and report the results below (Model: Mistral-7B-SFT, Sample Size: 128, Dataset: MATH).  \n\n| Temperature | 0.2 | 0.4 | 0.6 | 0.8 | 0.9 | 1.1 | 1.3 | 1.5 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Majority Voting | 0.408 | 0.450 | 0.465 | 0.473 | 0.463 | 0.465 | 0.470 | 0.465 |\n| Weighted Majority Voting | 0.400 | 0.453 | 0.468 | 0.465 | 0.458 | 0.463 | 0.470 | 0.463 |\n\nEmpirically, our results indicate that weighted majority voting, when using a reward model with similar performance as the generator, has minimal impact on performance. The optimal temperature range remains almost the same. Furthermore, since entropy estimation in TURN is independent of aggregation methods, TURN remains applicable in this setting.\n\nFuture work could explore whether a stronger reward model would provide meaningful benefits in this context.\n\n> *Have you extended your method to tuning other sampling parameters like top-p*\n> \n\nWe appreciate this suggestion. We conducted experiments involving other common sampling parameters (Model: Llama3.1-8B-Instruct, sample size: 128, Dataset: MATH):\n\n|  | 0.5 | 0.7 | 0.9 | 1.1 |\n| --- | --- | --- | --- | --- |\n| None | 0.625 | **0.650** | 0.597 | 0.482 |\n| top-p=0.9 | 0.645 | **0.653** | 0.635 | 0.623 |\n| top-k=20 | 0.650 | **0.653** | 0.613 | 0.563 |\n\nThese findings suggest that while alternative sampling parameters can slightly reduce temperature sensitivity, they do not significantly impact the optimal temperature.", "claims_and_evidence": "The paper presents two main claims:\n1. The optimal temperature varies across models and correlates with training-task similarity\n2. The entropy turning point (EntP) can predict optimal temperature without labeled data\n\nBoth claims are supported by extensive experimental evidence across multiple models (13 models on two distinct tasks). The correlation between model-task similarity and optimal temperature is compellingly demonstrated.", "methods_and_evaluation_criteria": "The methodology is sound and the evaluation criteria appropriate. The authors analyze temperature's impact on model performance using three well-defined metrics: Hit Rate, Temperature Gap, and Performance Drop. The comparison against fixed-temperature baselines is comprehensive and fair. One limitation is the restriction to majority voting and best-of-N strategies. Exploring weighted voting would strengthen the work.", "theoretical_claims": "The stochastic process model provides a theoretical foundation for the observed entropy spike, but lacks rigorous mathematical proof. While intuitive and empirically grounded, the connection between the toy model and actual LLM behavior could be more formally established.", "experimental_designs_or_analyses": "The experiments are thorough, examining 13 models across two distinct tasks. The temperature range exploration is methodical, and the sample size analysis demonstrates TURN's efficiency. However, the work would benefit from 1) testing on more diverse domains beyond mathematical reasoning and code generation, 2) including newer/different aggregation methods (e.g., weighted majority voting)", "relation_to_broader_scientific_literature": "The authors adequately position their work within the literature on sampling strategies for LLMs. However, connections to related fields like ensemble methods in traditional ML could be strengthened.", "essential_references_not_discussed": "NaN", "other_strengths_and_weaknesses": "Strengths:\n- Novel insight connecting entropy characteristics to optimal temperature\n- Method requires no labeled validation data\n- Consistent performance gains across diverse models\n\nWeaknesses:\n- Limited task diversity (only math and code)\n- Minimal discussion of computational overhead for entropy calculation", "other_comments_or_suggestions": "- Investigate scaling laws of optimal temperature with model size would be great\n- Analyze whether the optimal temperature varies by difficulty level within tasks1.", "questions_for_authors": "1. How does TURN perform on generative tasks with more subjective quality metrics?\n2. Have you extended your method to tuning other sampling parameters like top-p", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yvsvUi1hr7", "reviewer": "Reviewer_JAaC", "summary": "Authors explores how to automatically determine the optimal temperature for large language models (LLMs) in multi-sample inference settings, without relying on labeled validation data. The authors analyze temperature’s role in balancing diversity and quality in generated samples and propose TURN (Turning Point Temperature Selection), an entropy-based method for automatic temperature tuning. A key insight is the Entropy Turning Point (EntP)—the temperature at which the log-entropy curve shifts from concave to convex—which strongly correlates with the best-performing temperature found via grid search. Through extensive experiments across diverse models and tasks (e.g., mathematical reasoning and code generation), TURN demonstrates robust generalization and outperforms fixed-temperature baselines, offering a practical solution for optimizing LLM inference.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer JAaC,\n\nThank you for acknowledging our efforts. Our detailed responses are as follows:\n\n> ***Essential References Not Discussed:** Not strictly related but also a multi-sample or more specifically multi-domain scaling approach: Robust Calibration with Multi-domain Temperature Scaling. Yaodong Yu, Stephen Bates, Yi Ma, Michael I. Jordan.*\n> \n\nThank you for pointing out this reference. *Yu et al.* present a method for training a temperature prediction model with labeled data applicable across multiple tasks for confidence calibration. We will cite and discuss it in our paper. The key distinction is that our study focuses on **accuracy improvement in multi-sample inference** using **label-free temperature selection** in **natural language processing**, whereas the referenced work centers on **confidence calibration** in **computer vision**.\n\n> *I am concerned about the novelty and evaluation here because temperature scaling is common in calibration. Now the paper only measures the entropy turning point and accuracy, it is unclear whether this can apply to canonical setting in calibration - optimizing for a temperature using a holdout validation set to see how the model is better calibrated for decoding.*\n> \n\nWhile temperature tuning is a well-studied technique in the context of calibration, the novelty of our approach lies in its methodology. As noted by other reviewers (3z2D, NxtV, TgFe, and 3FXt), our method, TURN, determines temperatures based on the **intrinsic probability distribution, without requiring a validation set**. In contrast, traditional temperature calibration relies on **validation data**.\n\nIt is also worth highlighting that we include **grid search on test sets** as a baseline in our evaluation. This serves as an **upper bound** for achievable performance. TURN exhibits a **high correlation** between its predicted temperatures and those identified via grid search (Figure 1B), and it **outperforms fixed-temperature baselines** (Table 2).\n\n> *The evaluation does not include a wider range of metrics such as fluency, diversity etc as in the adaptive calibration paper. It is crucial because having a small temperature would*\n> \n\nOur evaluation is focused on **multi-sample inference** strategies like *best-of-N* and *majority voting*, primarily in domains such as math and code, where **accuracy** is an appropriate metric. These tasks are less suited for evaluating fluency or diversity, which are more relevant in open-ended generative tasks.\n\nAs for your reference to an \"*adaptive calibration*\" paper, though without a specific citation, we identified the relevant works:\n\n*[1] Xie et al. Calibrating Language Models with Adaptive Temperature Scaling*\n\n*[2] Huang et al. Calibrating Long-form Generations from Large Language Models*\n\nHowever, we did not find fluency or diversity as metrics in them. If a different work was intended, we would appreciate a specific reference so we can address it more thoroughly.\n\n> *Another concern is it is not common that tuning temperature won't affect model performance a lot if the model size is large enough. [1]*\n> \n> \n> *[2] The Effect of Sampling Temperature on Problem Solving in Large Language Models. Matthew Renze, Erhan Guven*\n> \n\nThank you for raising this point. We add an evaluation of a relatively large model (Qwen2.5-32B-Instruct) on MATH at two temperatures using majority voting (sample size = 16):\n\n| Temperature | 0.1 | 0.7 |\n| --- | --- | --- |\n| Accuracy | 0.825 | **0.870** |\n\nNotably, increasing the temperature from 0.1 to 0.7 improved performance by 4.5%, a significant margin given the already high baseline. \n\nWhile *Renze & Guven* focus on **single-output accuracy**, our study evaluates **multi-sample inference**, where **sample diversity** and **accuracy** are crucial. Thus, our findings do not contradict their conclusions.\n\n> *Please include more takaways into fig.1 legend as the concept of EntP is introduced late.*\n> \n\nThank you for the constructive suggestion. We have revised Figure 1 and its caption to better introduce and clarify the EnTP concept. The updated version can be found at https://drive.google.com/file/d/1O9DOpEW0z3GRjID6AQeUHT-Z869Tyrqe/view?usp=sharing\n\n> *Usually for math and coding, a good temperature is low, e.g. 0.5, so how does the found optimal temperature vary across different tasks?*\n> \n\nYou make a great point. When generating only a few samples (e.g., 4), lower temperatures (0–0.3) often perform better (see Figures 12 and 13). However, with more samples (e.g., 32), low temperatures lead to repetitive outputs, while higher temperatures promote diversity, which is beneficial in multi-sample settings. Additionally, the optimal temperature depends on the task and aggregation methods. For example, best-of-N favors diversity since only one strong candidate is needed, leading to a higher optimal temperature compared to majority voting.\n\nWe are looking forward to hearing your further comments!", "claims_and_evidence": "TURN estimates optimal temperatures without relying on external labels by using token-level entropy. This is a clear advantage over existing methods that require validation data. Also empirical correlations (Fig. 4) between EntP and grid-searched optimal temperatures, showing strong alignment. However, while the results are consistent across datasets, the explanation of why EntP works theoretically is based on a stochastic process model, which is a simplification of real LLM behavior.", "methods_and_evaluation_criteria": "The method leverages token-level entropy to estimate an optimal temperature without requiring labeled validation data, which is a practical and scalable approach. The evaluation is conducted on two benchmark datasets (MATH for reasoning and MBPP for code generation), which are reasonable choices since they rely heavily on multi-sample aggregation strategies like majority voting and best-of-N. The experiments systematically test different models, including general-purpose and task-finetuned variants, to assess robustness. Key metrics—hit rate, temperature gap, and performance drop—provide a clear measure of how well TURN approximates the best temperature compared to grid search.", "theoretical_claims": "There is no theorem that is proved.", "experimental_designs_or_analyses": "observed correlation between Entropy Turning Points (EntP) and optimal temperatures supports the method’s validity. The stochastic process model is an interesting theoretical tool to explain entropy behavior, but it remains a simplified approximation that may not capture all aspects of LLM sampling dynamics. The sample efficiency analysis (Table 3) is a strong aspect, demonstrating that TURN requires relatively few samples to make accurate predictions.", "relation_to_broader_scientific_literature": "The paper's key contributions align with broader research on temperature calibration in language models, multi-sample inference strategies, and self-assessment metrics for optimization. Prior work has established that temperature tuning significantly impacts generation quality, affecting the trade-off between diversity and coherence (Holtzman et al., 2019; Renze & Guven, 2024)", "essential_references_not_discussed": "Not strictly related but also a multi-sample or more specifically multi-domain scaling approach: Robust Calibration with Multi-domain Temperature Scaling. Yaodong Yu, Stephen Bates, Yi Ma, Michael I. Jordan.", "other_strengths_and_weaknesses": "I am concerned about the novelty and evaluation here because temperature scaling is common in calibration. Now the paper only measures the entropy turning point and accuracy, it is unclear whether this can apply to canonical setting in calibration - optimizing for a temperature using a holdout validation set to see how the model is better calibrated for decoding. The evaluation does not include a wider range of metrics such as fluency, diversity etc as in the adaptive calibration paper. It is crucial because having a small temperature would \nAnother concern is it is not common that tuning temperature won't affect model performance a lot if the model size is large enough. [1] \n\n[2] The Effect of Sampling Temperature on Problem Solving in Large Language Models. Matthew Renze, Erhan Guven", "other_comments_or_suggestions": "Please include more takaways into fig.1 legend as the concept of EntP is introduced late.", "questions_for_authors": "Usually for math and coding, a good temperature is low, e.g. 0.5, so how does the found optimal temperature vary across different tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WAEkZIVT52", "reviewer": "Reviewer_tFFz", "summary": "The authors introduce a data-free way to automatically find the best sampling temperature for multi-sample generation in LLMs. By detecting a sharp “turning point” in the model’s token-level entropy, they find a good balance between quality and diversity.-.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer tFFz,\n\nThank you for your thoughtful feedback and for recognizing the significance of our work. We appreciate your positive remarks on our approach and presentation. Our detailed responses are as follows:\n\n> *It is not entirely clear why the authors do not include a comparison to a baseline that has access to labels. Although I understand that TURN does not require labels, selecting the optimal temperature typically occurs before deployment, where labels are commonly available. Could you provide a rationale for excluding this baseline? Additionally, could you elaborate on scenarios where selecting the optimal temperature is necessary but labels are not accessible?*\n> \n\nWe compared TURN to grid search using the test set as validation, which serves as a strong upper-bound baseline that has access to labels. Our results show that TURN correlates very highly with grid search (Figure 1B) and even outperforms the best fixed-temperature baseline (Table 2).\n\nIn real-world scenarios, labeled data is often scarce or expensive to obtain. For example, in domains like robotics or drug discovery, labeled samples are difficult to collect. Furthermore, when training models that go beyond human performance or require super alignment (e.g., for scientific discovery or working with new or unsolved math problems), ground-truth labels may no longer exist, making label-free methods essential.\n\n> *Do reasoning models follow the same paradigm? It would be helpful if you could provide results using available reasoning models (e.g., Gemma-3, Qwen).*\n> \n\nThank you for your insightful question about whether reasoning models follow the same paradigm. Evaluating temperature sensitivity in reasoning models is indeed a valuable consideration. At the time of our original submission, the first well-known open-source reasoning model, DeepSeek-R1, was released on January 20, 2025—approximately 10 days before the ICML deadline. This timing prevented us from including such an evaluation in our initial work.\n\nSince then, we have conducted an additional experiment using **DeepSeek-R1-Distill-Qwen-7B** on the MATH dataset. Below are the results (content length = 1000, sample size = 128, majority voting):\n\n| Temperature | 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | 1.1 | 1.3 | 1.5 |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Accuracy | 0.847 | 0.860 | 0.855 | **0.874** | 0.860 | 0.830 | 0.830 | 0.800 |\n\nOur findings suggest that temperature has a relatively low impact on the accuracy of **DeepSeek-R1-Distill-Qwen-7B** on the MATH dataset, with performance remaining consistently high across the tested range (peaking at 0.874 at T=0.7). This stability could be attributed to the model being heavily optimized for the MATH dataset and then overfitting it, or the dataset is relatively simple for reasoning models. Due to time constraints, we are unable to explore additional reasoning models or datasets, but we recognize this as a promising direction for future work.\n\nWe appreciate your thoughtful review and welcome any further questions or suggestions.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are not theoretical claims.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Research on LLMs is extensive; understanding the impact of temperature is a significant contribution to the field.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "#### Strengths\n- The paper is clearly written and well presented.\n- It compares multiple LLMs, identifying critical turning points for each. Additionally, it demonstrates that the same LLM can behave differently depending on whether it has been specialized for a given task.\n\n#### Weaknesses\nSee Questions.", "other_comments_or_suggestions": "See Questions.", "questions_for_authors": "1. It is not entirely clear why the authors do not include a comparison to a baseline that has access to labels. Although I understand that TURN does not require labels, selecting the optimal temperature typically occurs before deployment, where labels are commonly available. Could you provide a rationale for excluding this baseline? Additionally, could you elaborate on scenarios where selecting the optimal temperature is necessary but labels are not accessible?\n\n2. Do reasoning models follow the same paradigm? It would be helpful if you could provide results using available reasoning models (e.g., Gemma-3, Qwen).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes to optimize the temperature parameter for decoding from language models in inference scaling applications (using best-of-n or majority voting), The paper observes an intriguing phenomenon: The accuracy vs temperature plot seems to go through a turning point going from concave to convex on a certain temperature. The proposal then is to use that temperature for decoding. Subsequent experiments show that the performance is on par with optimizing temperature on a validation set. The key selling point of the proposal is that it is solely based on the entropy of the generation of the model and there is no need for a validation set to choose the temperature. Besides that the paper makes uncovers intriguing phenomena about the optimal temperature in generative models beyond the default 0.7. Overall, I think the concave / convex hypothesis that the paper identifies is quite interesting, and all reviewer and the AC agree with this fact (which is the basis of the positive scores). However, the AC also identified some issues that need to be addressed for the camera ready to the extent that is possible. Congratulations to the authors! 

- The message of this paper is slightly misleading. The authors pitch their method against validation over a temperature set of {0.3, 0.5, 0.7, 0.9, 1.1} which might be too coarse given that TURN uses an interval of size 0.1 and show that it has a smaller gap. This gap would likely evaporate and become unfavorable to TURN if both sets use an interval of size 0.1. I am actually not convinced that when there is a validation set to choose temperature, TURN would be any useful. I think the real usefulness is in cases where no such validation set exists for using the model in the wild for new problems.

- The relative metrics are confusing. I think the temperature selection metrics are not too informative. The main important outcome of the process is actually accuracy. I suggest that the authors simply report the raw numbers (e.g., accuracy) and compare the accuracy of using TURN vs optimal vs grid search on the same interval of 0.1. 

- The proposed concave / convex hypothesis is based on a heuristic observation. The fact that the algorithm actually returns a valid outcome suggests that the concave / convex hypothesis is likely valid but I think more investigation on that front is needed on why that happens, what is needed for it to happen, etc. Also in practice, the algorithm is at the mercy of numerical stability of estimating the second order statistics. To this end, the authors propose a stochastic process model in Section 4.3, which I think is half baked and I was not convinced that it captures the dynamics of real language models.
Decision: Accept (poster)