The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
Authors: Shivam Agarwal
Zimin Zhang
Lifan Yuan
Jiawei Han
Hao Peng
Venue: NeurIPS 2025 poster ()
Keywords: large language model
alignment
finetuning
Primary area: applications

Abstract: Entropy minimization (EM) trains the model to concentrate even more probability mass on its most confident outputs. 
We show that this simple objective alone, without any labeled data, can substantially improve large language models’ (LLMs) performance on challenging math, physics, and coding tasks. We explore three approaches: (1) EM-FT minimizes token-level entropy similarly to instruction finetuning, but on unlabeled outputs drawn from the model; (2) EM-RL: reinforcement learning with negative entropy as the only reward to maximize; (3) EM-INF: inference-time logit adjustment to reduce entropy without any training data or parameter updates. 
On Qwen-7B, EM-RL, without any labeled data, achieves comparable or better performance than strong RL baselines such as GRPO and RLOO that are trained on 60K labeled examples. Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the challenging SciCode benchmark, while being 3x more efficient than self-consistency and sequential refinement. Our findings reveal that many pretrained LLMs possess previously underappreciated reasoning capabilities that can be effectively elicited through entropy minimization alone, without any labeled data or even any parameter updates.

The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, Hao Peng University of Illinois Urbana-Champala (shivama2, (aopeng)@eillnggg
Abstract
Entropy minimization (EM) trains the model to concentrate more probability mass on its most confident outputs. We show that this simple objective alone, without any labeled data, substantially improve large language models' (LLMs) performance on challenging math, physics, and coding tasks. We explore three approaches: (I) EM-FT minimizes token-level entropy similarly instruction finetuning, but on unlabeled outputs drawn from the model; (2) EM-RL: reinforce- ment learning with negative entropy the only reward maximize; (3) EM-INF: dnference-time logit adjustment to reduce entropy without any training data or parameter updates. On Qwen-7B, EM-RL, without any labeled data, achieves comparable or better performance than strong RL baselines such as GRPO 68 and RLÔO [ that are trained on 60K labeled examples. Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the challenging SciCode benchmark (78), while being 3x more efficient than self- self-consistency and sequential refinement. Our findings reveal that many pretrained LLMs possess previously underappreciated reasoning capabilities that can be effectively elicited through entropy minimization alone, without any labeled data or even any parameter updates.
1 Introduction
Entropy minimization (EM) trains the model to concentrate even more probability mass on its most confident outputs without any labeled data external supervision [23 60 (2]]. It operates on key
assumption and simple intuition: if model is reasonably capable, is more likely to be correct
when it is confident [56). Modern large language models (LLMs) are pretrained
massive data and
often undergo a dedicated annealing phase to enhance downstream task performance [59 It should be, therefore, fairly reasonable to treat them capable models even before any supervised finetuning (SFT) or reinforcement learning (RL) [90]. This naturally invites the questions: Can entropy minimization alone improve their performance? And how far can they go without any labeled data? This work answers both: yes, and surprisingly far, at least for reasoning tasks.
77].
We present systematic exploration of post-pre post-pretraining adaptation and inference-time scaling of LLMs, using entropy minimization alone without any labeled data. We propose three novel methods, each aligned with established post-pretraining stage.
(1) Unsupervised finetuning by directly minimizing token-level entropy (EM-FT) mirrors SFT and minimizes token level loss, unlabeled outputs sampled from the model conditioning on the input prompts [46]. We find that EM-FT achieves surprisingly strong performance on math and coding tasks, and can even outperform labeled GRPO and RLOO on LeetCode (26 (coding) and Minerva 42 (math).
'Code
25/EM_PT
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(2) Reinforcement learning with a negative entropy reward (EM-RL) uses a reward signal based solely on entropy: the negative sum of token-level entropy across a rollout, adjusted by a constant baseline. This is analogous to the REINFORCE algorithm [76 I but with entropy as the only supervision without any labeled data. We find that without any labeled data EM-RL can achieve competitive performance to RLOO and GRPO on most math and coding tasks while outperforming it on LeetCode, Minerva and AMC (math) 43).
(3) Inference- time scaling through entropy minimization (EM-INF) optimizes the logits during each decoding step reduce the entropy of the LLM's distribution without any parameter update. We find that EM-INF works best in complex tasks with high uncertainty (e.g. AIME math [3), UGPhysics 88 and SciCode (78). We observe that Qwen 32B (77] can outperform frontier models like GPT-4o on Scicode [78] and is 3x more efficient than inference scaling through self-consistency and sequential refinement.
While the results are promising, they must be interpreted with care ($5) First, EM is most effective when model confidence correlates with correctness, as in the experiments above. It less suited for tasks like aligning with human values (35), where confidence alone is not a reliable proxy for quality, Through experiments on individualistic value reasoning [35], we observe that Qwen-2.5 struggles on the task without finetuning on labeled data, and further EM leads to no improvements, Second, the effectiveness of EM hinges on the assumption that the pretrained model is already capable in the tasks of interest. We observe that the improvements from EM-FT and EM-RL on Llama-3. 1-8B are less substantial on math reasoning tasks compared to those on Qwen-2.5. This may be due to 1ama-3.1-8BB is less capable on these reasoning tasks.
The success and limitations of EM highlight the importance of the capabilities of the pretrained models, which is sometimes underappreciated, at least for reasoning tasks. We call for the inclusion of EM as baseline in the evaluation of future pest-pretraining and inference-time scaling algorithms, to better separate the contribution of algorithmic innovations from the intrinsic capabilities of the model itself. Besides, as we show, an approach that performs surprisingly well on certain models and tasks can fail in trivial ways on others. Broad evaluation across diverse pretrained models and tasks can help ensure that the conclusions generalize and are not artifacts of specific model or dataset.
2 Background and Related Work
2.1 Related Work
Engraie-re RL augments the expected reward objective with a regularization term aiming to increase entropy (98, in order to promote exploration by adding uncertainty into the actions I8, 41). Entropy-regullrizee RL usually learns a policy from scratch in grid world situations (e.g. robot simulations) where adding perturbations can lead to exploring various paths to get to the goal. In contrast, we investigate if reducing the uncertainty of the policy can improve pre- trained language models capabilities, with the hypothesis that they already acquire some capabilitics for the tasks of interest during pretraining. Instead of learning from scratch, we finetune LLMs to reinforce confident outputs. Therefore, we draw inspiration from cntropy-regularizzan design reward function (EM-RL) that minimize the entropy of the model encouraging it to commit to a fewer more confident paths, instead of prompting exploration like in previous works.
64
33
27
in RL to
Entropy minimization (EM) has been extensively used as regularization in emmi-supervise learning 23 40 g (1], domain adaptation (60, 62 80), and few-shot learning [25] 39] 38 20 apply
EM for reinforcement learning with semi-supervised scenarios, where entropy is used to estimate
the rewards on unlabeled data. [81] 94. [56] use EM for unsupervised test-time adaptation, where pre-trained classifiers are adapted to new data at test time without using any labels. Despite its 15| 61 7T) EM has rarely been explored as a standalone training objective for LLMs. Our key hypothesis is that EM can reinforce LLMs strong capabilities acquired during pretraining. Instead of applying entropy minimization for test time-domain adaptation or as a regularizer, we study if it can be used for LLM finetuning and inference time scaling.
well-studied history [53 52 48
Unsupervised and few-shot RL for LLMs has raised the possibility of self-improvement Here a model learns via self-training in which it critiques its own generations without external feedback 22 47
35
57
93 or refines generated text [I 96 70). Building on self-improvement,
self-verification
45 has been applied for RL tuning 995 84 99 concurrently. [32 uses the most

---

frequent answer as reward (majority voting) for finetuning, since optimizing the likelihood of most consistent answer can reinforce pretraining priors. [99] applies this idea to unsupervised RL tuning at test time by optimizing on test dataset as whole. We work on the same intuition of reinforeing model confidence, however, we study post-training where EM-FT and EM-RL. are used for finetuning L.MSs on large general math and coding data. Further, consistent answers and output extraction is not always possible in complex tasks like scientific coding I0 82 73). On the other hand, EM-FT and EM-RL can be applied on tasks like code generation where output extraction is non-trivial. Unlike [99] which updates the model parameters on each test set individually, inference scaling via EM-INF optimizes the models' entropy without updating parameters and does not need task- specific optimization. EM-INF can be deployed and served like regular text decoding 30, 58 19 requiring neither assumption on task
model optimization. [95] proposes to estimate rewards by computing the frequency of extracted answers for a prompt. Here, the model is incentivized to prefer certain answers, which makes the answer distribution more concentrated. While it reduces the entropy, is not directly enforced and emerges from learning to repeat most frequent solutions. EM-FT and EM-RL target uncertainty in models' token level distribution, whereas the frequency- based approaches target alignment with desired outputs where entropy reduction is not an inherent outcome and task-dependent. [84] investigates RL from few-shot output verified labeled data and observe that direct entropy maximization as a standalone objective can be used for finetuning to small extent. They find that carefully selecting a few output-verifial maximization as regularizer can lead to improvements. We find that only negative entropy rewards (entropy minimization) are sufficient for RL with similar effects without requiring any data selection. Standalone entropy maximization does not converge and eventually collapses [84], but EM-RI. and EM-FT via EM converge and can be used to train on large datasets without output verification.
problems along with entropy
2.2 Preliminaries
Problem setup:
Let Ty denote an autoregressive LLM policy: Ta <)(y) = II, na To (V [x, Bes). where denotes the input prompt and y = y1 ... M/ I the output trajectory. For clarity, will suppress whenever it is clear from the context. Let r(y) denote reward function,
the conditioning which provides a score for an output sequence. RL learns a policy that produces output sequence to maximize the expected reward Eyn,,, [ry)].
Entropy minimization minimizes the Shannon entropy H(TT)) Ey Tog To policy 23 We consider two methods for empirically estimating the entropy.
67 of the
Trajectory-level entropy estimator 77 is computed based on the trajectory distribution of the output sequences produced by the policy. In practice, it samples full sequences and computes the total log- probbabiii.y.. For an autoregressive policy.
N Σ log To (1')
N ly
(To) = N
log no (vElv(r))
N
(y' T are N full trajectories sampled from the policy.
Token (action) level entropy estimator 27 18 computes the entropy of the policy based on the
token (action) distribution at each step. In practice, sums per-token entropy across time steps. At generation step é, the model produces a probability distribution Te Te(-lee (-ly over a vocabulary V. The entropy of the policy using token-level estimation is then
N
Iv'l
Hank (TO) =
H H(mo y
(2)
N
where H(mo(- y<+)) Ejev To(i y<t) log no(j y-v)). While both trajectory-level and token-level estimators are used to compute entropy, they lead to different behaviors when used for training. Minimizing trajectory entropy leads to policies with lower entropy over trajectories whereas minimum token-leved entropy leads to policies with low entropy at each generation step. Furthermore, the token-level estimate is commonly used in maximum entropy- regulaiee RL frameworks, such as
soft actor-critic due its low variance [14 27 64 41)
In the following subsections, we introduce three post-training methods based on entropy minimization as standalone objective. (1) EM-FT ( 3.1 directly minimizes the token-level entropy (Eqn.
3

---

similar to supervused finetuning, but on unlabeled outputs drawn from the model; (2) EM-RL (3.2))
reinforcement learning with negative trajectory-level entropy (Eq. | or neagtive token level entropy (Eqn. ( as the only reward to maximize; (3) EM-INF ) inference-time logit adjustment to reduce
entropy using the token level estimate (Eqn. 2 without any training data or parameter updates.
Entropy Minimization for Post-training
3.1 EM-FT: Direct Entropy Minimization through Token-Level Finetuning
Supervised finetuning methods such as rejection sampling finetuning [6 91 46 sample data from the policy and use reward function to select high-quality data. Good performance on these tasks has been attributed to post-training labels (e.g. output supervision) and pretraining task mixtures 665 [89] Building on entropy minimization and on-policy finetuning we test if directly minimizing the entropy of the sampled trajectories can enable learning without labels. Given batch of N sampled trajectories {y- ~ To J= we directly minimize the the empirical total token-level entropy Hock (Eqn.
2). We present this objective and corresponding gradient in Table
3.2 EM-RL: Entropy Minimization with Negative Entropy Rewards in RL
We now minimize the entropy of the policy by using negative entropy as the only reward in RL (EM- RL). We design a reward function to assign outcome rewards to trajectory y based on: 1) negative trajectory level entropy (EM-RI --suuene-eueecce:: 2) negative token level entropy (EM-RL-tokee))
E.M-LL-eeueecce minimizes the entropy based on the trajectory level estimator of entropy (Eqn. D. This estimate favors trajectories that receive high probabilities but does not care about if the reasoning paths are concentrated or not to few options. Minimizing this entropy estimate useful in situations where multiple but limited chains of thought are preferred, for example, shorter math tasks where some plausible solutions can be explored but not too many. We use the joint probability of the sequence reward ruaj to minimize the trajectory level entropy estimate in expectation, By [fimj(() (y)) (Te) The reward rimj is,
Tinj (y)
Σ log To (B( = log (T(
EM-RL-token minimizes the entropy based on the token level entropy estimate. This estimate prefers trajectories that are more deterministic and confident at each generation step, focusing the probability mass on fewer possible reasoning paths (chains of thought). For example, problems requiring complex long chains of thought where being more deterministic at every step helps prevent the model from losing track of the reasoning process. Based the token level entropy estimate (Eqn 2.
we define the reward Fiok to minimize the token level entropy estimate expectation, Beg=rolTind [rio = - Hook (TTO).
y
Tonk (3) = Σ M(To (ly)))
While both the objectives EM-FT, EM-RL aim to minimize the entropy of the policy, they are optimized differently. EM-FT minimizes the entropy by directly differentiating it, whereas EM-RI. uses policy gradients (64 Table summarizes the objectives and the corresponding gradients. In practice, we subtract the RLOO baseline [I] in EM-LL, which does not change the expectation of the gradients but helps with optimization 75 87). In addition, we use KL-regularizer weighted by /3 between the policy ho and the initial base model गe to avoid large deviations from the base model and aid optimization, i.e., -((TT) - 3KL (T Tell = -((TT) - B( ((-H(xx) - "M(TT, T [4 [7]]. The KL regularization can increase the entropy when the base model is too uncertain (higher entropy), however when 33 < 1 (we use 0.001) the dominant gradient is from the reward, in our case entropy minimization.
Connecting to entropy regularized RL: Fntropy-reeullaiized RL by previous works 27 I8 uses entropy maximization term (as regularizer) along with the reward signal (policy loss) to

---

Table 1:
Comparing various entropy minimization estimates and their corresponding gradients. N denotes the number of trajectories. Min. denotes minimizing an objective and Max. maximizing.
Objectives
Gradients wrt. 0
Finetuning
EM-FT $31
Min. Hack
VoH (ne(- v(r)
RL
EM-Runeeeeeece 3.2
Max. Hunj
D
log (o y Vo (o
RL
EM-RI-ttoken $3.2
Max. Hook
N
t=1 H(TT(( y Vo log
Inf-Scaling
EM-INF
Min. Heok
N/A
encourage stochasticity in action selection and promote exploration. 84 build on this idea and use few labeled examples for the policy loss along with entropy maximization. They find that using entropy maximization only can boost performance initially, however, the optimization with only entropy maximization does not converge and eventually collapses the policy. In contrast, EM-RL aims reinforce pretraining biases by learning a more deterministic policy via entropy minimization without requiring any labeled examples. We find that EM-RL can converge on large datasets and improve model performance on math and code generation.
3.3
Training Setup
We use the following setup for all experiments, and provide additional details in Appendix D
For our training experiments, we evaluate on math and coding tasks using [13 Math- 500 229] AMC [43) AIME 2024 [43]. Minerva math (Minerva) (42) Olympiad Bench (Olymp.)
Evaluation:
22 LeetCode (LeetC) [26]. and LiveCodeBench-v2 (LiveC) 34 For inference time scaling, we
additionally evaluate on Scicode
and UGPhysics [8]] Following 63 37 we estimate the
training FLOPs as 6PD and inference FLOPs as 2PD where P is the number of parameters in the model and D is the number of tokens (Appendix $D.4)
Training data, models, and methods: We construct the training data for math and coding by randomly sampling 35K prompts from Numina math (43] and 25K prompts from Eurus-2 coding split [13. respectively. We use Qwen2.5-Math-77 (89 and Furus-2-7B-SFT [13] model for training on math and coding, respectively. We use Verl 66 to train. For all RL methods, use early stopping based on the validation set. We set the number of rollouts to N = 4. batch size to 512, learning rate 1e- We use KL regularizer with small coefficient 8 = 0.001 which does effect entropy minimization. We use RLOO and GRPO with output verification only without reward models. We use the same prompts for EM-FT but without the labels. Instead it samples solutions to estimate the entropy. All experiments are done using 4xGH200 Nvidia GPUs.
Baseline: We consider additional rewarding scheme based the frequency of the final answer- Self consistency (SC-RL). It assigns rewards to each trajectory y; based on majority voting [85 32]. An answer 0; is extracted for the trajectory y; and the frequency of the answer is used estimate the reward, r(y:) = Σ I(a==))) where I is the indicator function. Self-consistency assigns hard rewards to trajectories where answer extraction is possible (e.g. math), however inapplicable on tasks like code generation. Such rewards are useful on tasks where several consistent answers can be generated (c.g. MATH-500). However, challenging prompts (c.g. scientific coding) often admit multiple plausible solutions with no answer extraction possible, consequently making majority voting inapplicable. [99] apply this idea for training on the test set, however, we use it on the training set.
3.4 Results: Entropy Minimization for Post- Dost-Training
Table compares the performance of using entropy minimization as a standalone post-training objective to supervised finetuning and output-verified RL.
Can unsupervised finetuning by directly minimizing token-level entropy improve LLMs? We first observe that directly minimizing the entropy (EM-FT) can improve improve performance over the base model by 8% on average on math and coding without using any labels. This observation indicates that pretraining priors can be further capitalized to squeeze more performance and entropy can
5

---

Table 2: Performance comparison of unsupervised finetuning (EM-FT) and various rewarding methods in EM-RL with supervised finetuning and RL. Italics, Bold indicates performance improvement over GRPO and SC-RI. (self-consistency RI.). respectively. Dash line ("-") denotes that self-consistency
is inapplicable. FLOPss are reported 10 (DD.4).
Math
Coding
Math AMC
AIME Minerva Olymp. Avg.
LeetC
LiveC
Avg. FLOPs
Qwen2.5-7b
43.8
31.3
15.6
14.7
19.0
24.9
26.1
18.4
22.3
Trained using 60K labeled prompts
w/ SFT N=1
48.2
30.2
10.0
17.6
22.4
25.7
18.3
18.3
18.3
1.0
w/ RLOO N=4
73.0
57.8
23.3
31.2
34.2
43.9
28.3
26.7
27.5
13.1
GRPO N=4
71.8
56.6
21.1
25.0
35.9
42.1
25.0
25.8
25.4
13.1
Our unsupervised methods trained on 60K unlabeled prompts
EM-FT N=1
67.2
51.8
14.4
33.1
34.4
40.2
28.3
17.2
22.8
1.0
SC-RL N=4
73.2
51.8
15.6
26. I
36.7
40.7
13.1
-SSUUI-ST N=4
67.2
53.0
21,1
30.9
35.6
41.6
31,1
21.7
26.4
13.1
EM-RL-TOKE N=4
70.8
57.8
18.9
30.9
35.9
42.9
29.5
24.5
27.0
13.1
independently contribute to performance gains. We also note that with only N = trajectory sampled from the model, EM-FT can achieve competitive performance GRPO and RLOO (outperforms on Minerva and LeetCode by 5% on average) which require output verification over multiple sampled
(N = 4) trajectories. EM-FT better contextualizes the improvements from supervised methods and better uses pretraining capacity, while maintaining smaller computational footprint. We also observe no improvements on LiveCode likely because LiveCode regularly updated with new problems that might be different from the pretraining distribution 34 This observation indicates that entropy minimization can potentially boost performance on unlabeled scenarios if it's not significantly
different from the pretraining task mixture [8 73 31).
Can we improve learning from entropy minimization through RL? In Table we observe that unsupervised RI. methods (SC-RI,EEM-RL)) improves base model performance by 11% on average and maintains competitive performance as compared to output-verified GRPO and RLOO (outperforms on AMC, Minerva and LeetCode by 4.5% on average) under similar compute constraints. This observation suggests that model confidences in the form of token and sequence level entropy estimation can be used as pseudo-rewards to learn from unlabeled scenarios and better use pretraining priors. This observation also ties up with (55] 20 31 who show that model confidences can be a proxy for reward. Among the various rewarding schemes, we observe that token level and sequence level entropy estimation can be flexibly applied to variety of tasks without assumptions on the problem structure (e.g. answer extraction) unlike self-consistency which is not applicable for code generation. On average, we observe that EM-RL has better performance via entropy rewards, indicating that fewer deterministic CoT's is more useful certain reasoning tasks (LiveCode, AMC). We also notice that EM-RL improve model performance on LiveCode by 6% where direct entropy minimization (EM-FT) did not work, indicating that careful reward design to learn from unlabeled experience better enhance learning and pretraining capabilities. Through EM-FT and EM-RL, we show that EM can elicit pretraining priors necessary for reasoning without any labeled data. While supervised training is expected to scale better, we put forward EM-FT and EM-RL as important baselines to better contextualize improvements from post-training methods.
4 EM-INF: Inference-time Logit Optimization
Inference-ting scaling methods suggest that enabling LLMs to improve their output generations at test time by scaling compute critical for improving performance on a complex tasks [86] 72 66 (50 [ Practical LLMs need to adapt to tasks online, i.e., we should be able to adapt to user-queries while the requests are being received. As a result, online LLM adaptation should unsupervised. Self-consistency (parallel generations) [83 and sterative-refinemet (sequential generations) are widely used for reasoning when external verification is not available multiple answers from the model and then selects the solution with the largest frequency. Whereas
47 Self-consistency samples

---

Table 3: Performance comparison of EM-INF against test-time scaling methods. Bold & Italics indicates best and second best performance, respectively. We use temperature of = 0.1, sample n trajectory for all methods unless specified and report the average of three runs.
Math
Coding
Science
Math
AMC
AIME
Minerva
Olymp.
Avg.
LeetCode
UGPhysics
80.7
50.6
11.9
35.7
41.1
44.0
0.6
18.5
Greedy Decoding (f = 0)
79.0
49.4
10.0
34.6
39.9
42.6
1.1
18.8
lierative-refinmment (N 3)
73.0
39.7
7.5
32.4
33.2
37.2
0.4
17.4
Self-consistency (NV 4)
80.7
50.9
13.3
36.7
41.0
44.5
0.8
18.5
ADAPTIVE TEMP
81.9
53.1
10.7
36.3
39.8
44.4
2.2
22.5
EM-INF
81.3
55.2
14.8
34.8
40.6
45.4
0.6
21.4
Qwen2.5-775 Bren2.5-7B-Instruune
74.3
44.6
9.1
35.2
38.4
40.3
47.2
22.7
Greedy Decoding (1 0)
73.2
42.2
8.8
33.8
37.9
39.2
46.1
23.7
Iterative-refinmment (N 3)
72.3
40.9
8.8
34.8
38.9
39.1
50.0
21.5
Self-consistency (N 4)
74.4
43.7
35.1
40.5
23.6
ADAPTIVE TEMP
74,9
49.8
10.0
38.4
37.5
42.2
49.4
24.1
EM-INF
74.0
46.7
11.7
36.5
38.7
41.5
51.7
25,4
Llama-3. 1orma-3.1-B---- 1-8B-Instruct
41.9
19.2
3.3
21.1
16.5
20.4
13.1
17.1
Greedy Decoding (f 0)
40.6
16.9
3.3
21.0
16.0
19.6
12.8
16.1
Iterative-refinemmnt (N 3)
41.3
18.7
2.5
20.3
16.0
19.8
9.6
18.3
Self-consistency (N 4)
43.3
20.2
2.8
20.1
18.1
20.9
16.9
ADAPTIVE TEMP
43.1
22.7
3.3
20.0
17.3
21.3
16.3
19.9
EM-INF
43.9
23.5
3.3
21.3
16.2
21.6
14.1
17.6
feeds the previously generated solution into the model as in-context examples inorder to refine the solution. However, online adaptation should not be tailored to certain problem types sypes-self-come self-consistency only works when output extraction possible and iterative refinement is bottle-necked by context length. To tackle these challenges we propose EM-INF that uses entropy minimization at inference time for logit adjustment. EM-INF treats the model's output logits as if they were parameters, and gradient descent to update them minimize the entropy of the distribution they induce; no gradient wrt. or the model parameters or parameter update is needed.
Let 2, Eº be a the logit vector produced by the model generation step from its last layer. We freeze the model parameters and optimize the logits 2; to minimize the entropy of the output distribu- tion it induces via gradient descent. We treat the logits as free parameters and never backpropagate through the model or update its parameters. To prevent verroptimmzztion, which would effectively lead to greedy decoding, works the best). Our inference time objective for each generation steps is defined as,
minimum entropy threshold 5 (we empirically find 0.1 5 < 0.5
LEM-INE max
Ea(z)) log ()) jeV
(5)
where a denotes the softmax function. After logit optimization, we use sampling based decoding to select the next token, since this optimization does not change the top logit and greedy decoding will change the output post-optimization (Prop. T 31. The above optimization operates over z, and is equivalent to optimizing one-layer neural network with [V] parameters with softmax activation. empirically find that to 15 gradient steps are enough. The vocabulary size (~ 150K in Qwen2.5 (77) is much smaller than number of parameters (7B) in the model, making the optimization overhead negligible compared to a forward pass the full model. We require only O(n) (ri sequence length) forward passes from the language model, equivalent to regular decoding. Whereas, other inference time scaling methods like self- consistency and sequential refinement require ONN) forward passes where N is the number of trajectories.
Baseline: We use adaptive temperature scaling to reduce the entropy of the models' output distribution. Starting from logits 2, we gradually decrease the temperature in the softmax until the entropy reaches target value. We search for the largest r > 0 such that H(a((z/T)) S max(o where E (0, 1) controls the fraction of entropy to reduce. We find (0.4, 0.6) works the best. Proposition 1. (Informal) Both adaptive temperature scaling and logit optimization reduce the entropy of model's output distribution. However, logit optimization can change the order of non-top
8))

---

logits, while temperature scaling preserves the order of logits and merely sharpens or flattens the distribution proportionally. Both temperature scaling and logit optimization will keep the token with the top logit and increase its probability, making them less likely to change the model', s behavior when it is highly confident (Appendix A).
EM-INF for scientific coding: We apply EM- INF to scientific coding in Table SciCode is challenging code generation benchmark requir- ing significant domain knowledge and reasoning capabilities where even frontier models such as GPT4o struggle 78). We observe that test time entropy minimization (EM-INN, adaptive tem- perature) improve performance of the base models consistently on easier subproblems and eventually the more difficult main problem by 2.5%. We note that logit optimization (EM- INF) better than adaptive temperature scaling on SciCode by 3%. Owing to the difficulty of SciCode, the models' output distribution has large entropy. Under this scenario. EM-INE INF can change the relative order of the logits which likely leads to better reasoning chains. Whereas, adaptive temperature scaling will never change the relative ordering of the logits each genera- tion step - it only makes the distribution peakier (Proposition Next, we observe that scaling through feerativerrffinement [7 leads to some improvements over the base model, however, it is bottlenecked by the max context length of the model making it harder to refine long chains of thought. Whereas, EM-INF does not require re- finement and maintains the same computational complexity as the base model when used with regular decoding. We note that EM-INF with Qwen2.5- ---.-----BB--- In--rustt can outperform several frontier models on the main problem without requiring refinement or editing model parameters by 3% and over the base model by 6%. Building on these observations, we put forward EM-INF practical, compute efficient inference time scaling method that does not require any supervision and makes no assumption on the nature of the problem.
Table 4: Performance comparison of inference time scaling with ADAPTIVE TEMP and EM-INF with API models and inference time scaling meth- ods on Scicode. Sub and main indicate subprob- lem and main problem mode in Scicode, whereas w/G refer to with background knowledge in the prompts. Bold.Italics indicate the best and second best performance over methods.
Sub w/ BG
Main w/ BG
GPT-4o
9.2
-0144--ubb--2224-01-999
33.7
9.2
Claude3.5-Sonnet
35.4
12.3
Claude3-Opus Claude3-Somet
Gemini 1.5 Pro
30.6
7.7
Qwen2.5-77 100222777-Innnnune
11.5
0.0
Greedy decoding (f = =0)
13.4
3,0
Ieerativereffiemment (n 2)
12.1
0.0
ADAPTIVE TEMP EM-INF
13.2
0.0
16.7
1.5
(reen2.5-3bbInstiie
25.4
4.6
Greedy decoding (f 0)
25.4
4.6
Ierative-refinement (n = 2)
24.7
7.6
ADAPTIVE TEMP EM-INF
28.8
7.6
27.1
10.7
EM-INF for math and coding: Table 3 compares the performance of EM-INF for test time scaling. We observe that minimizing the entropy of the model at test time without updating the parameters (EM- INF, adaptive temperature) improves the performance of the base model for almost all tasks and model classes by 3% average, indicating that EM an useful method for test time scaling. We also note
50
Accuracy (%)
46
Self-consistency
EM-INF
that EM can be applied all tasks (math, coding and ugphysics) without assumptions such as output extraction in self-consistency or output scoring. Next, we note that EM competitively performs against self- consistency while outperforming on tasks requiring significant reasoning capabilities, e.g. UGPysics by 3.6%. Unlike self-consistency which requires sam- pling multiple trajectories (N > 1), EM-INF re- quires only one trajectory leading to lower computa-
40
Adap. Temp.
20
40
Time Consumed (hours)
Figure 1: Computational efficiency of EM-
INF compared to self-consistency with N 4, 8, 16 on AMC.
tional requirements. These observations collectively indicate that pretraining performance can be improved via FM, and making the model more deterministic at inference time via unsupervised EM in-order to prefer fewer chain of thought options can lead to better performance.

---

Table 5: The limitations of EM on tasks and models where model confidences do not help. Left shows performance of Llama-3.1 1--BB on math tasks trained with the setup in Sec. 3.3 Right compares the performance of Qwen-2.5-70 instruct on individualistie value alignment.
Math
AMC
AIME
Minerva
Olymp.
IndVal
Llama-3.1 --8--nsttucttt
40.6
18.1
1.0
22.4
15.7
Qwen2.5-7B-Instme
65.9
W/ RLOO
52.0
25.3
3.3
26.8
20.7
EM-FT (N=1)
66.0
EM-FT (N=l)
42.2
21.7
3.3
18.8
16.4
EM-RL-token (N=4)
66.0
MM-RL-tokken (N=4)
41.0
18.1
5.6
19.1
15.6
EM-RL-sequents (N=4)
66.3
-M-Rll-seueence (N=4)
41.6
18.1
7.8
18.4
16.6
compress the compute requirements of EM-INF with inference scaling baselines. We observe that EM-INF can achieve better performance while using third of time. This observation suggests that EM-INF is practical inference scaling method in compute bound scenarios, for online adaption without updating model parameters [8]]). EM-INF can also be combined with self-consistency and sequential refinement to scale up (Appendix CC.4
Computational efficiency: Figure compares
Limitations: Dependence on Base Model Capability and Confidence
To contextualize the applicability of EM, we probe its performance by 1) training on tasks with different bias as compared to pretraining; and 2) finetuning models lacking target reasoning behaviors.
Table T evaluates EM individualistic value reasoning (IndVal) (5 Individualistic value reasoning is a difficult alignment task that evaluates LLMs' ability to reason about an individual's value preferences in new situations. Given known statements about certain individual, the task is to predict if the user would or would not associate with a new situation (measured using accuracy). We observe that Qwen-2.5-7B struggles with the task, likely because LLMs' biased to coarse values acquired from pretraining which may not generalize to a large pluralistic audience [97 35)
As expected, note that EM is not effective in this task, likely because model confidences incorrectly biased to training priors that do not benefit the target task, as also observed in [99 31]
Furthermore, Table (left) compares entropy minimization (EM-FT, EM-RL) with supervised RL (RLO0) using the same training data, setup and evaluation as Sec 3.3 with Llama-3, 1aama-31-&b-Instruue the base model 24 We observe that while supervised RL can improve model performance, im- provements from entropy minimization is less effective as compared to Qwen-2.5 and can sometimes hurt performance may lack reasoning behaviors initially as compared to Qwen models, as also observed in [84). The lack of certain reasoning behaviors for math in the initial policy possibly leads unreliable model confidences and eventually inaccurate entropy rewards 112. This observation suggests that success of entropy minimization hinges on the assumption that the pre-trained model's 's initial reasoning behaviors are critical for the capacity of improvement. Overall, the success of EM-FT and EM-RL must be interpreted with care, as it can be less effective depending the task and base model choice.
math reasoning. We tie this observation to (1 who show that Llama models
6 Conclusion
We show that entropy minimization through unsupervised finetuning (EM-FT), RL with standalone negative entropy reward (EM-RL) and inference time scaling through logit optimization (EM- INF) can improve LLMs on complex math, physics, and coding tasks without using any labeled data. We observe that through EM-FT and EM-RL, Qwen-2.5-7B can achieve comparable or better performance than RL methods like GRPO and RLOO. Through test time logit optimization, EM-INF enables Qwen-2.5-2 2.5--22. to outperform proprietary models like GPT4o on complex scientific coding without training or updating model parameters. Our findings suggest that many pretrained LLMs already possess strong reasoning ability which can be enhanced using EM alone. While successful, EM is less beneficial on tasks and models with different inductive biases as pretraining. We call for the inclusion of EM-FT, EM-RI and EM-INF as a baseline the evaluation of future post- pretrrinngn and inference-tine time scaling algorithms, to better attribute the source of improvements in them.

---

Acknowledgments and Disclosure of Funding
This research used the DeltaAl advanced computing and data resource, which supported by the National Science Foundation (award OAC 2320345) and the State of Illinois. DeltaAl is joint effort of the University of Illinois Urbana- Tabana-Champaigee and its National Center for Supercomputing Applications.
This work was also in part supported by research awards from Apple and the Allen Institute for AI.
Research was also supported in part by US DARPA INCAS Program No. (IR0011-21-CO11 and BRIES Program No. HR001 1-24-3-0325, National Science Foundation IIS-19 19-11956,,,, the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environment (I-GUIDE) by NSF under Award No. 21 2118329. Any opinions, findings, and conclusions recommendations expressed herein those of the authors and do not necessarily represent the views, either expressed or implied, of DARPA or the U.S. Government.
References
[1] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce reinforce style optimization for learning from human feedback in 1lms. arXiv preprint vrXiv:2402.147400 2024.
[2] Jihyun Janice Ahn and Wenpeng Yin. Prompt-reverse inconsistency: Lim self-inconsistency beyond generative randomness and prompt paraphrasing. arXiv preprint arXiv:2504.01282, 2025.
[3] Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, al. Ext5: Towards extreme multi-task scaling for transfer learning, arXiv preprint arXiv:2111.10955 2021.
[4] Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, and Breck Baldwin. Llm stability: A detailed analysis with some surprises. arXiv preprint arXiv:2408.0400 2024.
[5] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022.
[6] Vidhisha Balachandran, Jingya Chen, Lingjiao Chen, Shivam Garg, Neel Joshi, Yash Lara, John Langford, Besmira Nushi, Vibhav Vineet, Yue Wu, et Inference-time scaling for complex tasks: Where we stand and what lies ahead. arXiv preprint Nixii::25040002944 2025.
[7] Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D' O'Amour, Jacob Eisenstein. Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint arXiv:2401 01889, 2024.
[8] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems, 32, 2019.
[9] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 8344-8353, 2022.
[10] Sébastien Bubeck, Varun Chadrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
[11] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, In The Twelfth International Conference on Learning Representations, 2024. URLhttppr /tttps:://ppereeiiii
net htt///ommmmdduuiiiiiiooa
[12] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
10

---

[13] Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, al. Process reinforcement through implicit rewards. arXiv preprini arXiv:2502.01455, 2025.
[14] Yuhao Ding, Junzi Zhang, Hyunin Lee, and Javad Lavaei. Beyond exact gradients: Convergence of stochastic soft-max policy gradient methods with entropy regularization. arXiv preprint arXiv:21 10. 10117, 2021.
[15] Mario Döbler, Florian Marencke, Robert A Marsden, and Bin Yang. Diversity-aware buffer for coping with temporally correlated data streams in online test-time adaptation. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7665-7669, IEEE, 2024.
[16] Hanze Dong Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
ISSN 2835-88855
[17] Laura Ekroot and Thomas M Cover. The entropy of markov trajectories. IEEE Transactions Information Theory, 39(4):1418-142)) 2002.
[18] Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust problems. arXiv preprint arXiv:2103.0622 2021.
[19] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In
Anmual Meeting of the Association for Computational Linguistics, 2018. URL https
peapissemanticschol... org/ (oorpusID:4413422:
[20] Chelsea Finn, Tianhe Yu, Justin Fu, Pieter Abbeel, and Sergey Levine. Generalizing skills with semi-supervised reinforcement learning. arXiv preprint arxiv:1612.00429 2016.
[21] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint 15Vv:25033.03000 2025.
[22] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint 2r0v 1738, 2023.
[23] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
[24] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[25] S Dhillon Guneet, Chaudhari Pratík, Ravichandran Avinash, and S Stefano. A baseline for few-shot image classification. In International Conference on Learning Representations (ICLR), volume 10, 2020.
[26] Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseck-coder. When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401.14194 2024.
[27] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMlr, 2018.
[28] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: challenging benchmark for promoting AGI with olympiad-level bilin- gual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computa- Bangkok, Thailand, August
tional Linguisties (Volume I: Long Papers), pages
11

---

2024. Association for Computational Linguistics. doi: 10. 10.1865/ww1/2224cchloooggg URL https://aclarthollogy .rg.ilgg..orrooooooe.e wwwgg/222...m 21124.acl-oooo long.2111
[29] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
[30] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.0997 09751, 2019.
[31] Audrey Huang, Adam Block, Dylan Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jor- dan T Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharpening mechanism. arXiv preprint arXiv:2412.01990 2024.
[32] Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint 1rXiv:2210.11110, 2022.
[33] Shiyu Huang, Hang Su, Jun Zhu, and Ting Chen. Svqn: Sequential variational soft q-learning networks. In International Conference Learning Representations, 2020. URL https 7 //openreview. weee//oorummiddr11h22112
[34] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodehench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403, 07974, 2024.
[35] Liwei Jiang, Taylor Sorensen, Sydney Levine, and Yejin Choi. Can language models reason about individualistic human values and preferences? arXiv preprint 1r.0v22410.03868, 2024.
[36] Liwei Jiang, Taylor Sorensen, Sydney Levine, and Yejin Choi. Can language models reason about individualistie human values and preferences?, 2024. htttssss/s//////.rrrrrrr 03868)
[37] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wxXv:20011.8861, 2020.
[38] Hilbert J Kappen, Vicenç Gómez, and Manfred Opper. Optimal control as a graphical model inference problem. Machine learning, 87:159-182, 2012.
[39] Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12912 12921, 2022.
[40] Dong-Hyun Lee. Pseudo-label The simple and efficient semi-supervised learning method for deep neural networks. 2013. URL https://api semanticsmhen wwwpacholaa..ooooooussssss/. 18507866
[41] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, arXiv preprint arXiv: 100.00099000 2018
[42] Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL 1.t.d--FFXXEXXXXX
https://openrevaee wtt.:/ww..eeeoooeeeoeeeeeeeeeeee
[43] Jia LI, Edward Beeching. Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. com/
htt..///.....//aaaaaaaaaaaaoaaa. project-numina aimo p http://www.rrrspsprr/r//aaaaaaar dataset. 2024.
Ihttps://hugg.
12

---

[44] Yiwei Li, Ji Zhang, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang. Jiayi Shi, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Yao Hu, al. Revisiting sonfssoon from dynamic distributional alignment perspective on answer aggregation. arXiv preprint arXiv:2502. 2225500.99390 19830, 2025.
[45] Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504. 02495, 2025.
[46] Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning, arXiv preprint arXiv:2401. 241.024000..967, 3, 2024.
[47] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang, et al. Self-refine: Iterative refinement with self- feedback.... Advances in Neural Information Processing Systems, 36:46534 46594, 2023.
[48] Robert A Marsden, Mario Döbler, and Bin Yang. Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2555 2555-2565, 2024.
[49] Jincheng Mei, Wesley Chung, Valentin Thomas, Bo Dai, Csaba Szepesvari, and Dale Schuur- mans. The role of basclines in policy gradient optimization. Advances in Neural Information Processing Systems, 35:17815 18 17830, 2022.
[50] Niklas Muennighoff, Zitong Yang. Weijia Shi. Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393 2025.
[51] Gang Niu, Bo Dai, Makoto Yamada, and Masashi Sugiyama. Information- theoretic semi- supervised metric learning via entropy regularization. Neural Computation, 26:1717-1762, 2012. URL httpsstts://ppp www.:/acccaaannncccccaaoaaaaamae
ID: 15064396
[52] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Real- istic evaluation of deep semi-supervised learning algorithms. Advances in neural information processing systems, 31, 2018.
[53] Daniel Jarne Ornia, Giannis Delimpaltadakis, Jens Kober, and Javier Alonso-Mora. Pre- dictable reinforcement learning dynamics through entropy rate minimization. arXiv preprint arXiv:2311.18703, 2023.
[54] Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong- Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self-improvement by reinforcement learning contem- plation. In The Twelfth International Conference on Learning Representations, 2024. URL
htttp:://openrviee. wwt.:e/eoooooddd-e/eeeeeeee
[55] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Picter Abbeel, and Kimin Lec. Surf: Semi-supervised reward learning with data augmentation for feedback-efffcien preference-based reinforcement learning. arXiv preprint arXiv:2203. 10050, 2022.
[56] Ori Press, Ravid Shwartz-Ziv, Yann LeCun, and Matthias Bethge. The entropy enigma: Success and failure of entropy minimization. arXiv preprint arXiv:2405.0501 205.00012.. 2024.
[57] Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. Advances in Neural Information Processing Systems, 37:55249-55285, 2024.
[58] Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018. htt.://ww...aa/////...a.aaaaaaa. www.eeannicschoo..rrrro//oooaaaa
493132445
[59] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
13

---

[60] Subhankar Roy, Aliaksandr Siarohin, Enver Sangineto, Samuel Rota Bulo, Nicu Sebe, and Elisa Ricci. Unsupervised domain adaptation using Beature-whitening and consensus loss. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 1471-9480, 2019.
[61] Evgenia Rusak, Steffen Schneider, George Pachitariu, Luisa Eck, Peter Gehler, Oliver Bring- mann, Wieland Brendel, and Matthias Bethge. If your data distribution shifts, use self-learning. arXiv preprint arXiv: 29228, 2021.
[62] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. Proceedings of the IEEE/CVE international conference on computer vision, pages 8050-8058, 2019.
[63] Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.0044, 2023.
[64] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXiv:/707.06344 2017.
[65] Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for IIm reasoning. arXiv preprint arXiv:2410.08146, 2024.
[66] Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or rl is suboptimal. arXiv preprint (xXix22502122188 2025.
[67] Claude E Shannon. mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
[68] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2002:22020033000. 2024.
[69] Guangming Sheng, Chi Zhang, Zilingfeng Ye. Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
[70] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8888 8652, 2023.
[71] Manli Shu, Weili Nie. De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in Eision-language models. Advances in Neural Information Processing Systems, 35:14274-14289, 2022.
[72] Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 11m test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv. org/abs/2408.033144 14, 11, 2024.
[73] Kaya Stechly, Karthik Valmeekam. and Subbarao Kambhampati. On the self-verification limitations of large language models on reasoning and planning tasks. arXiv preprint arXiv:2402.08// 080002008/133 15, 2024.
[74] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. 2022. URL htRl.https:///atl
//axxiv. org/ httgg/bb//0000.. 200bb//200001255
[75] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradi- ent methods for reinforcement learning with function approximation. In Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999 URL httg9//U///tttpppp/p//eddngggnn neurips. www.see...pppppp.e.........e..e. http:/perr.fillespeerr///// FiL1://ssts0000000eeeeeeeee00000
.df
14

---

[76] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[77] Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:// quenlm. github. io/blog//gg hw/blog///en2.....//.o/ 5/
[78] Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongehon, Yao Li, al. Scicode: A research coding benchmark curated by scientists, 2024. URL https://arxiv. org/abs/2407.133
Friedrichs. Ralf Helbig, Behmoush Hajian, Dawid K. Fiejtek, Florence Wagner. Holly H. Soutter, Ashlee M. Earl, Jonathan M. Stokes, Lars D. Renner, and James J. Collins. Discovery of structural class of antibiotics with explainable deep learning. Nature, 626:177- 2024.
[79] Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov, Pierre Perrault, Yunhao Tang, Michal Valko, and Pierre Menard. Fast rates for maximum entropy exploration. In International Conference on Machine Learning, pages 34161-34221. PMLR, 2023.
[80] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Ad- versarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2517-2526, 2019.
[81] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006, 10726, 2020.
[82] Julian Junyan Wang and Victor Xiaoqi Wang. Assessing consistency and reproducibility in the outputs of large language models: Evidence across diverse finance and accounting tasks. arXiv preprint arXiv:2503. 16974, 2025.
[83] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou, Self-consistnncy improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 111711 2022.
[84] Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.2057,, 2025.
[85] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions. arXiv preprint 22..::2222222.00000 2022.
[86] Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ili Kulikov, and Zaid Harchaoui. From decoding meta- meta-generation: Inference-time algorithms for large language models. arXiv preprint arXiv:2406. 16838, 2024.
[87] Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with aetion-dependent factorized basclines. arXiv preprint arXiv:1803.077. 072446 2018
[88] Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, and Yang Wang. Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models. arXiv preprint Stti::20020003444 2025.
[89] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via celf-improvement. arXiv preprint XXX::2409.11222 2024.
[90] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang,
15

---

Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo. Tianhao Li. Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. htttps://a// org/ abs 2505. 09388
[91] Zheng Yuan, Hongyi Yuan. Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.0168 01825, 2023.
[92] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng. Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.0241 14, 2022.
[93] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong. and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.1282; 2023.
[94] Qingyang Zhang, Yatao Bian, Xinke Kong, Peilin Zhao, and Changqing Zhang. COME:
Test-time adaption by conservatively minimizing entropy. In The Thirteenth International Conference on Learning Representations, 2025. URL hnttps://openreviet wwwpurvviiennneffouun
id-506BjJ1ziZ
[95] Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised IIm reasoning incentivization. arXiv preprint arXiv:2504.05812, 2025.
[96] Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. Tempera: Test-time prompting via reinforcement learning. arXiv preprint arXiv:2211./1800 2022.
[97] Minjun Zhu, Yixuan Weng. Linyi Yang, and Yue Zhang. Personality alignment of large language models, arXiv preprint orXiv:2408. /779, 2024.
[98] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, IL. USA, 2008.
[99] Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint a5Xiv:2504. 16084, 2025.
16

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract, Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Introduction Sec. Results Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide data, hyperparameter and code details in Sec. 3.5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: Code is released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuia wwwiiicgguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details provided in Sec 3.3 Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Multiple training runs are computationally expensive
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide all setup in Sec. 3.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://...........tttt...s...aa.
Answer:
[Yes]
Justification:
We follow the code of conduct.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: No societal impact
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
20

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all open-sourced datasets and models used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.reeswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
21

---

Answer:
[NA]
Justification: No new assets introduced
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
22

---

Justification:
We do not use LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httees////earrppcccc////o/rrrees for what should or should not be described.
23

---

A Proof for proposition I
(Informal) Both adaptive temperature scaling and logit optimization reduce the entropy of a model's 's output distribution. However, logit optimization can change the order of non-top logits, while temperature scaling preserves the order of logits and merely sharpens or flattens the distribution proportionally. Both temperature scaling and logit optimization will keep the token with the top logit and increase its probability, making them less likely to change the model's 's behavior when it is highly confident.
Part 1: Let p=)) softmax(z)) and H(p) denotes the entropy of p: H. - P: log P:. 3z
R/V 17 > and a, b E [I,...,/MM [VD. such that za < Zb and yo > yb where y: = 24 + mp, (log p. p.pp7. H),
i i= 1.......11 |/.
Proof: Choose with two top-top indices satisfying Pa < Pb < e (H (H+1), and therefore za < 2b. Consider ya yb Za 2/ + 179a - 9b) where 9. = p. (log p: H). g is the negative gradient of entropy H w.r.t. 2. We show that 9a > 9b-
By Mean-Value Theorem 3e (Pa, Pb) such that 9a 9b (Pa pbb (log c H 1). We then have
pb e (H-1)
log H + 1 <0- 7 Ja 9b
(6)
Thus, we can set an arbitrary large y such that,
Za 26 + 11 96) > 0
Part 2: b cannot be arg max; 2;-
Proof: Let j arg max; 2;, i.e. the index of the maximum logit. Since pj max; p: can bound it using the inequality:
H(p) - --=--m ps log 2 - log Pj
Pj 2 e H(p) 2 e (H(p)+1)
(8)
We now analyze how 9: behaves as a function of pt. Let g(,)) = p: + H) and g'(p)) = log Pt + H 1. Thus, ((:)) is monotonically increasing whenever g' (p:) > 0, i.e. when,
Pc (H+1)
(9)
So no lower-probability index i can have fi > gj. i.e the entropy gradient cannot reduce y, relative to others. Therefore, * j. leading to the following behavior
0 <
9a S 96
Pa E (e (e-* ,pb]
(10)
Ja
9b
Pa E -H)
(11)
B Adaptive Temperature Scaling
The algorithm of adaptive temperature scaling for EM-INF is presented Algorithm
C Additional Results
C.1 EM-RL with DeepSeet-Mathh-B-Innttee
To further understand initial model choice, we finetune another base model- MeepSeek-Math-77 7B- Instruct on the same data and training settings as Dw (Section (.33). Deepseek-T7 Math-Instruct has been finetuned on math-related tokens for 500B tokens on top of Deepseck-coder-7b
68 We use the instruction tuned version which has been further finetuned using chat data. We fine tune Deepseek-765 Booeeeekk-b-aaaee Instrutttt using entropy minimization (EM-RL.) and evaluate it in Table 6 We observe that entropy minimization improves performance over the Deepseek model by 2.5% on average math tasks.
24

---

Algorithm Inference-time Adaptive Temperature
Input: Logits 2;; Initial maximum temperature Tmar initi Initial minimum temperature
Tmin_init 0; Maximum iterations M; Tolerance étol > O; Target entropy reduction ratio
a E (0, 1); Target entropy threshold 8 >> 0.
Output: Scaled logits z/.
1: function COMPUTEADAPTVEETEPPEEERA
Tmaz_init init, fnits M, Ecols at 8)
2:
Pinitial - Softmax (2+)
3:
Hinitial &- Entropy P Pinitial)
4:
Htarget max(ō, a Hinstial)
5:
Tfinal f- 1.0
6:
if Himitial > 5 then
7:
Tlow Tmin_in inet.
8:
Thigh t- Tmaz_init
9:
for iteration f- 1 to M do
10:
Tmid + (Thigh + Tlow)) //2w
11:
Peurrent f- Softmax 24/ Tmidd
12:
H Heurvent f- Entropy P Peurrent)
13:
if Heurrent Htrrget| Ctol then
14:
Tfinal E Tiidd
15:
break
16:
else if Heurrent < Htarget then
Noww Tmid
18:
else
Thigh Tridd end if Tfinal
20:
-Taid
22:
end for
end if
24:
2 f Ze/Tfinal
25:
return =t.
26:
end function
Table 6: Performance comparison of various entropy based rewarding methods in EM-RL against RL with output verified rewards using Deepseek- Maasscck-Mtt--65 Mlath-7b-Instruce as the base model. We use the same setup Qwen-2.5 2.5--ath--B (Section 3.3). Bold indicates performance improvement over RLOO.
Math
Math
AMC
Minerva Olymp.
Avg.
Deepseek math- 7b-instruct
42.0
16.9
19.1
12.6
22.7
Trained using 60K labeled prompts
w/ RLOO N=4
42.4
24.1
20.6
13.2
25.1
Our unsupervised methods trained on 60K unlabeled prompts
EM-RI-SSQUUENCE N=4
42.8
21.7
21.7
14.4
25.2
FM-RI-TOKEN N=4
43.4
18.1
22.8
13.6
24.5
C.2 Sensitivity to KL regularization coefficient 3
We provide results with different 33 for EM-RL token below in Table 7 We observe that a small B value is better for entropy minimization because it weighs EM more in the overall loss function.
Theoretically, the KL regularization can increase entropy when the base model is too uncertain, i.e,, it deviates from the pre-trained checkpoint causing it to visit high entropy states. However when < (we use 0.001) the dominant gradient the policy gradient from entropy minimization. Since set 3 to small value the overall objective minimizes the entropy.
25

---

Table Performance variation of EM-RL-token using different KL regularization coefficient, 3. We
use Qwen-2.5-Math-76 as the base model with the same setup in Sec 3.3
Math
Model
Math
AMC
AIME
Minerva
Olymp.
Avg.
Swen-2.55Matth-
43.8
31.3
15.5
14.7
19.0
24.9
EM-RL-TOKEN / 1.5
45.0
39.8
13.3
9.6
21.2
25.8
EM-RL- -----OTTONNNN / 0.5
50.8
39.8
16.7
10.7
23.0
28.2
EM-RE RL-TOKEN $ 0.001
70.8
57.8
18.9
30.9
19.0
39.5
EM- RL-TOKEN 8
70.9
57.5
18.2
29.6
19.0
39.0
Table 8: Sensitivity to number of rollouts N -ES-RELSEQUENCE We use Qwen-2.5-NM and use the training setup defined in 3.3
Math
Model
Math
AMC
AIME
Minerva
Olymp.
Avg.
EM- RE-SEQUENCE N=4
67.2
53.0
21.1
30,9
35.6
41.6
EM- RL-SEQUENCE N=8
71.4
54.2
13.3
31.6
35.0
41.1
EM RMM-RL-SQQUE RL-SEQUENCE N=16
70.6
51.9
15.5
23.9
35.1
39.4
C.3 Sensitivity to number of rollouts N
We measure the impact of the number of rollouts N on EM-RL-SEQUENCE in Table g that AM-RL-SEQUENCE benefits with larger on MATH-500, suggesting that larger number of rollouts might reduce the variance. However, we note that a larger N does help AMC, Minerva and Olympiad bench, and reduces performance on AIME. Overall, we observe consistent performance of AMMRL--EQUEECE on sweeping from 16. We set N building on existing RL fine tuning works to make sure that the training overhead is consistent
We observe
C.4 Combining EM-INF with Self-consistency and Iterative Self-refinement
We combine EM-INF with self-consistency and iterative self-refinement in Figure 2a 2b Since, EM-INF only optimizes the logits, can be plugged into both methods easily. We observe that EM-INF get the same performance self-consistency (N=4) while using time less FLOPs AMC with Qwen 2.5 7B. As expected, EM-INF reduces the diversity of the generation, which causes it's benefits to diminish over iterations. Self-consistency benefits the most when several diverse generations can be produces which lead to the same answer, however, EM-INF reduces diversity through entropy minimization 4 / leading to smaller benefits with larger compute. EM-INF is most suitable for inference time scaling on high uncertainty tasks under compute bound scenarios.
C.5 Qualitative analysis of EM-INE on SciCode
We conduct qualitative analysis of EM-INF on SciCode through case study where Qwen2.5- 7B-Instruct with EM-INF successfully generates the correct code, while the base model fails. The problem and corresponding model generations are illustrated in Figure 3 The prompt requires generating a symmetric matrix with increasing diagonal values, where each element is modified by the product of user-defined noise factor and normally distributed random number, followed by symmetrization via averaging with the matrix's 's transpose.
The base Qwen2.5-77 2-BB Instruct model produces partially correct implementation of the init mni__matrix function. It correctly initializes the increasing diagonal and applies the symmetrization formula (A = (A A. T) 2). However, it fails to apply the noise factor to all elements of the matrix. Specifically, the noise modification is applied only to the off-diagonal elements, leaving the diagonal entries (e.g., i+1) unaltered by the stochastic process prior to symmetrization. This error occurs because scientific coding a difficult task where both coding and domain knowledge required, therefore making the model's output distribution uncertain.
26

---

51
Accuracy (%)
Accuracy (%)
30
02
(2-1233
EMANE MeratbreBettimmeee (2-1,23441
0.2
0.5
0.6
0.11
1.0
1.2
14
15
1.5
2.0
2.5
3.0
3,5
4.0
4.5
50
FLOPs
FLOPs
1x15
(a) Accuracy FLOPs for combining EM-INF and self-consistency at inference time on AMC.
(b) Accuracy vs. FLOPs combining EM-INF and
iterative refinement at inference time on LeetCode.
Figure 2: Performance of EM-INF combined with various inference-ting timee scaling methods using Qwen2.5-7B-Inn EM-INF can effectively improve performance when combining with other inference-time scaling methods while requiring negligible additional computation.
By contrast, the generation using EM-INF produces the correct implementation that adheres to all constraints specified in the prompt. EM-INF reduces the entropy the output distribution, which leads to more deterministic generations. The generated code by EM-INF, expected, reduces the diversity in generation, leading to more concise implementations and omitting additional comments. These results suggest that EM-INF is particularly beneficial for scientific coding tasks like SciCode, where correctness and determinism are critical.
SciCode Problem 16.1
Question: Wrlle a function generate a symmetric matrx with incressing values along its diegonal All elements in the matrix should be modified based on the product of normally distributed random number generated numpy and inpul given by the user. Symmetrize the matrix by laking the average of the the and tranpose.
Qwen2.5 8-7B-Instruct
EM-NF
inst (dte, noses]:
Gererate syenetric mateis incevasing values along its diagonal.
def init, matris(din, noise): A ap.aaess((din, dia)) for cangu(dia):
Inputs:
noisw: Roisw Level (fleet).
dia: dimension of the natric (int).
A[s,5] 1 1
Output:
array shere each element is flast, representing the synnetric natris.
(A /
noise mp.randee rdndn dis)
Initialize din dia watrix with zeess
A Cole, dia])
FITL the satrix with Increasing values along the diagenal
For In range(dtin::
Fer in zonge(dte]:
IF 1 =
Als][s] =
stse:
A[s][1] up candee normal(a, noixe)
Symmetrize the netris by taking the avecage the sus of the eatris and its tramspose A CA A.T) /
reture
Figure 3: Qualitative analysis of EM-INF SciCode. 7at.2.557B-Inntruree generated code (Left) fails to add random noise to all elements the matrix, whereas the code generated with EM-INF (Right) performs the correct operation. We observe that EM-INF (I) reduces the diversity of generated tokens and is more deterministic as expected; (2) generates correct code; (3) produces more concise outputs; and (4) is more helpful in scientific coding tasks like SciCode where LLMs highly uncertain.
27

---

D Experimental Setup
D.1 Evaluation Benchmarks
D.1.1 Math Reasoning
We adopted the data and evaluation pipeline from PRIME-RL codebase [33 for all math reasoning ovaluations. The final answer of LM the string inside the last \bozed{]. All math benchmarks is evaluated on Pass@I I accuracy. Here, we will introduce each math benchmarks in detail.
MATH 500 MATH 500 contains 500 prompts sampled from the 5,000 test problems in the orig- inal MATH dataset 29 The test of MATH comprises challenging competition math problems drawn from sources such as the AMC and AIME exams. These problems span seven subjects- Prectlsbrr,, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus- and are labeled with difficulty levels from (casiest) to 5 (hardest), Each test problem includes final boxed answer and full step-by-step solution in LaTeX, enabling exact match evaluation and providing interpretable reasoning paths for model analysis.
We use the validation set from AI-MO's ' AMC dataset 43 which contains 83 problems extracted from AMC 12 (2022 and 2023). The AMC dataset comprises math competition problems from the American Mathematics Competitions (AMC), covering a broad range of topics including arithmetic, geometry, combinatorics, and algebra. It is designed evaluate language model's capacity for mathematical reasoning, abstraction, symbolic manipulation, and logical deduction.
AMC
AIME We use the validation set from AI-MO's AIME dataset 443, which includes 90 problems drawn from AIME 22, AIME 23, and AIME 24. The AIME (American Invitational Mathematics Examination) dataset features high-difficulty problems intended for top-performing students from the AMC competitions. These problems typically require advanced mathematical reasoning across topics such as algebraic manipulation, number theory, combinatorics, and geometry. Each problem involves multiple intermediate steps and provides minimal guidance, making the dataset well-suited for evaluating a language model's ability to perform multi-step symbolic reasoning and maintain logical coherence throughout extended problem-solving chains.
Minerva Math Minerva Math refers to the OCWCourses evaluation dataset introduced in the Minerva paper (42] This dataset comprises 272 undergraduateellvvl STEM problems sourced from MIT OpenCourse (OCW), covering topics such as solid-state chemistry, information theory and entropy, differential equations, and special relativity. Each problem is self-contained and accompanied by clearly defined final answer-either numeric (191 problems) or symbolic (81 automatic evaluation. The dataset is designed to assess language model's ability to perform multi-step scientific reasoning, including the application of domain-specific knowledge, symbolic manipulation, and precise quantitative computation.
OlympiadBench OlympiadBench (28 is large-scale bilingual multimodal benchmark comprising 8,476 Olympiad-level mathematics and physics problems sourced from international competitions such the IMO and IPhO, as well as elite Chinese contests and Gaokao mock exams. The dataset evaluates LLMs on advanced mathematical and physical reasoning tasks, including multi-step logical deduction, symbolic manipulation, equation solving. theorem proving, and visual-spatial understanding from diagrams. Its high difficulty, multimodal diversity, and expert-level content make it rigorous benchmark for assessing scientific reasoning capabilities. In this work, we use the open-ended, English, text-only subset of math competition problems as our evaluation set, which contains total of 674 problems.
D.1.2 Coding
LeetCode benchmark is introduced in the technical report of DeepSeek-Coder [26]. This benchmark consists of 180 recent competition-level programming problems from LeetCode, collected between July 2023 and January 2024. Each problem is paired with 100 test cases to ensure comprehensive coverage, and prompts are constructed in standardized format to simulate realistic coding challenges. This dataset evaluates language model's ability to perform code
LeetCode
28

---

generation in competitive programming settings, testing a wide range of capabilitiies such as problem comprehension, algorithm design, logic reasoning, and implementation precision. The inclusion of varying difficulty levels (Fasy, Medium, Hard) allows for a thorough assessment of model's generalization and problem-solving skills under pressure. During evaluation, the python code will be extracted from model's 's response and run the test cases. Performance is reported using the Pass @ metric, indicating the proportion of problems for which the model's first generated solution passes all test cases.
LiveCodeBench LiveCodeBench [3] is large-scale, contamination free benchmark designed to holistically evaluate the code-related capabilities of large language models. It comprises 612 recent coding problems collected from competitive programming platforms such as LeetCode, AtCoder, and Codeforces between May 2023 and August 2024. Unlike traditional benchmarks that focus solely code generation, LiveCodeBench assesses four key skills: (1) code generation from natural language, (2) self-repair using error feedback, (3) code execution via output prediction, and (4) test output prediction without implementation. This comprehensive setup evaluates not only functional correctness but also model's 's ability to debug, reason about program behavior, and construct tests. Its time-segmented evaluation protocol further supports rigorous detection of training data contamination. In this work, we adopt the V2 release, which includes 511 problems released between May 2023 and May 2024. We evaluate methods on the code generation ability. The model is given a natural language problem description and tasked with generating correct program. Correctness is evaluated by passing all test cases. The Pass @ metric is used, measuring the fraction of problems where the first generated solution passes all tests.
D.1.3 Scientific Reasoning
UGPhysics UGPhysics 88 is a large-scale bilingual benchmark designed to evaluate undergraduate-Ievel physics reasoning in large language models. It comprises 5,520 problems (11,040 including both English and Chinese versions) spanning 13 physics subjects and 59 topics, featuring diverse answer formats such as numerical values, symbolic expressions, equations, and intervals. Each problem is annotated with one of four reasoning skill types: Knowledge Recall, Laws Application, Math Derivation, and Practical Application. Unlike prior benchmarks that primarily target multiple-choice or high-school-level content, UGPhysics emphasizes deeper problem-solving and derivation skills, requiring LLMs to perform symbolic reasoning, apply physical laws, and handle real-world quantities. This dataset is designed to assess LLMs' ability to carry out multi-step physics reasoning beyond simple mathematical manipulation. In this work, we evaluate on the Semiconductor Physics subset, which consists of 372 problems. During evaluation, models answer physics problems using prompts tailored to different answer types for improved extraction and judgment. A rulc-based correctness assessment then extracts answers from the model's output and reports accuracy.
SciCode SciCode (78 is challenging, scientist-curated benchmark designed to evaluate large language models (LLMs) on real-world scientific coding tasks across 16 subfields in mathematics, physics, chemistry, biology, and materials science. It comprises 80 complex main problems, each decomposed into multiple subproblems (338 in total), requiring models to integrate domain-specific scientific knowledge, reasoning, and Python programming. Each problem includes gold-standard solutions and test cases, with optional expert-authored background context. SciCode assesses LLMs' abilities in multi-step problem decomposition, code synthesis from scientific instructions, and integration of partial solutions. By emphasizing scientific reasoning and simulating real-world workflows, it provides a rigorous benchmark for evaluating LLMs as scientific assistants. In our paper, we evaluate methods on the test set, which contains 65 main problems and 288 subproblems in total. Models generate Python functions for each subproblem and integrate them into a complete solution. Evaluation is fully automatic, using gold test cases for both subproblems and main problems. The evaluation conducted with and without scientific background information, as well as using gold or generated prior solutions, to measure realistic performance. Success requires all subproblems and the main solution to be correct. Final accuracy is reported as the percentage of problems achieving success.
29

---

D.1.4 Individualistic Value Reasoning
IndVal The INDVAL dataset [36] is derived from the World Values Survey (WVS) and consists of natural language statements expressing individual human values and preferences from over 93,000 participants worldwide. Each participant responds to 253 questions grouped into 13 distinct categories, including Social Values, Economic Values, and Science & Technology, among others. The dataset standardizes these diverse survey responses into over 900 value-expressing statements per individual, enabling evaluation of language models' ability to reason about individualistic human values in a bottom-up, granular manner. This dataset particularly valuable it challenges models to understand and predict nuanced personal value judgments beyond demographic stereotypes. Consequently, reveals limitations in current language models' capacity for individual value reasoning and promotes more equitable, personalized Al alignment that respects individual diversity. In our work, we randomly sampled 200 individuals excluded from training, uniformly distributed across all continents. For each individual, we randomly selected 4 statements per question category to form an individualistic value profile consisting of 52 known statements. Then, for each question category, we selected statements not included in the known profile and asked the language model to select the most appropriate statement from the given candidates based the individual profile. This process resulted in an evaluation dataset containing 5,400 prompts. During evaluation, the selected statement is extracted from the model generation and compared with the ground truth. Final performance is reported as accuracy.
D.2 Training Data
Math Reasoning We randomly sampled 35K prompts from Numina math training set 43 as our math training prompts. The NuminaMath dataset contains 860,000 problems ranging from high-school to advanced competition levels, including Olympiad, AMC, AIME, and Chinese K-12 math problems.
Coding We randomly sampled 25K prompts from Eurus-2-RL training dataset's coding split [3 as our coding training prompts. The Eurus-2-RL training dataset a high-quality reinforcement learning dataset consisting of about 457, 7,000 math problems and 27,000 coding problems with outcome verifiers such as LaTeX answers for math and test cases for coding. The math problems are sourced from the NuminaMath-Cun Math dataset, which covers wide range from Chinese high school math to International Mathematical Olympiad-leve questions. The dataset is carefully cleaned, filtered, and reformatted into direct question-anse annseer format ensure high quality and solvability. is especially useful for training because provides diverse, verified, and challenging problems covering different level of difficulties and tasks.
Individualistic Value Reasoning The training dataset is derived from the IndVal dataset [36]. We sampled 900 individuals uniformly across seven continents. For each individual, we randomly selected 4 statements per question category to form an individualistic value profile comprising 52 known statements. Then, we sampled questions per question category as queries, resulting in total of 70,200 training prompts.
D.3 Models
Qwen2.5 Family For the entropy minimization for post-training experiments, we use two models derived from the Qwen2.5 family: Qwen2.5-Math-7B [8) and Eurus-2-7B-SFT I3. Qwen 7B is selected as the base model for math reasoning tasks. We apply various post-training methods using the math training datasct and evaluate their performance on math reasoning benchmarks. This model serves complex mathematical reasoning problems. Eurus-2-77 2-7B-SFT a SFT checkpoint obtained by further fine-tuning Qwen2.5 5-tth-- the Eurus-2-SFT-D
excellent starting point for experiments due to its strong capability in tackling
from PRIME-RL [13] which is action- centric train-of-thougee reasoning dataset. Since Eurus-2-7B. 2--B-T further enhances the reasoning abilities of 1wwe2.5-Mathh- we select it for coding experiments, applying different post-training methods with the coding training dataset and evaluating on coding benchmarks.
For entropy minimization during inference time experiments, we select --th.----- and D-wen.55-7BIInstreee as the base models for math reasoning, coding, and scientific reasoning tasks because of their inherent followiwn and reasoning abilities. Specifically, for SciCode,
30

---

Table 9: Performance comparison of EM-INF against test-time scaling methods. Bold & Italics indicates the best and second best performance, respectively. We report the mean = standard deviation of four independent runs with different seeds. We use temperature, I 0.1 and sample N ==1 trajectory unless specified.
Math
Coding
Science
Math
AMC
AIME
Minerva
Olymp.
Avg.
LeetCode
UGPhysics
Qwen2.5-Mir 1aa2.-5Mtth-7BB5 Maath-7B-Instruct
80.7 + 1.2
50.6 1.6
11.9 = 0.7
35.7
+ 1.4
41.1 = 0.8
44.0
= 0.3
18.5 + 0.9
Greedy Decoding 0)
79.0 0.0
49.4
0.0
0.0
34.6
0.0
39,9
0.0
42.6
0.0
llerative-refnemmnt (N
73.0
19.7:0099
7.5 1.3
324
33.2
08
37.2
0,4 0.2
17.4 1.5
Self-consistency (N 4)
10.7.1.33
1.1
13.3
1 0.8
36.7
1 0.8
41.0
0,3
44.5
0.8
0.4
18.5 1 0.3
ADAPTIVE TEMP
81,9 + 1.4
5.71
10.7
= 1.6
36.3
39.8
44.4
2.2 0.2
22.5 1.0
EM-INF
0.
5 1.3
14.8
1.2
34.8 0.8
40.6
0.6
45.4
0.6 0.2
21.4 1.2
Qwen2.5-72 11-225-7B-Insse --I-nstruct
74.3
44.6 1.1
9.1
+ 1.8
35.2
38.4 0.4
40.3
47.2
1.3
22.7
Greedy Decoding (! 0)
73.2 0.0
42.2
0.0
8.8
+ 0.0
33.8 + 0.0
37.9 + 0.0
39.2
46.1
0.0
23.7 + 0.0
lleraivverrffiemment
72.3
40,9
8.8
2.2
0.4
38.9
39.1
500
21.5
Self-consistency (N
74.4 0.8
43.7
10.0
1 0.5
35.1
1.2
39.2
1.2
40.5
48.8
23.6
ADAPTIVE TEMP
74.9 1.3
49.8 = 1.1
10.0
= 1.0
38.4
1.0
37.5 0.9
42.2
49.4
0.9
24.1 + 1.0
EM-INF
0.9
46.7
+ 0.6
11.7
= 1.7
36.5 1.2
38.7 = 0.8
41.5
51.7
= 0.7
25.4
1.2
Llama-3.1 1-8B-Instruce
41.9
1922
3.3 1.6
21
16.5
0.6
20.4
13.1
+ 0.8
17.1
Greedy Decoding (1 0)
40.6
16.9
3.3
0.0
21.0 0.0
16.0
0.0
19.6
128
1 0.0
16.1
Ierstive-refinement (N
41.3 0.3
1.7
25 1.1
20.3 0.4
16.0 0.6
19.8
0.6
18.3
Self-consisenncy (N 43.3 0.4
20.2
2.8
20.1
18.1
0.7
20,9
12.8
1.7
0.0
ADAPTIVE TEMP
43.1 + 1.5
22.7 1.1
3.3
0.9
20.0 +
1.6
17.3
+
1.0
21.3
16.3
1.5
19.9 + 1.8
EM-INF
43.9 1
1.2
23.5 1
1.7
3.3
1
1.4
21.3 1
0.8
16.2
1
0.4
21.6
14.1
1
0.3
17.6
1.6610 0.7
include more powerful model, Qwen2.5-32DD IIstruct, due to its superior capability in handling complex problems, allowing us to evaluate the performance of EM-INF on stronger models.
Llama-3.1 Family Llama-3.1 24 is the second model family included in this work ensure the generalizability our methods across different models. For both the entropy minimization post-training and inference-time experiments, we select Llama-3.1-8B
as the base model.
D.4 FLOPs Computation
we estimate the training FLOPs as 6PD and inference FLOPs as 2PD where P is the number of parameters in the model and D is the number of tokens. We use early stopping based on the validation set accuracy for all methods, which significantly changes the FLOPS utilized.
Following 63
EM-FT: On each training step, we sample one trajectory from the model costing 2Pn where n is the sequence length followed by training costing 6Pn. The total cost per step is thus 8Pn. We set sequence length to 4096, batch size to 512 and the training early stops at 40 gradient steps costing 1.01 27 FLOPs for a 7B parameter model.
EM-RL,, GRPO, RLOO: None of the methods use a reward model. We use GRPO and RLOO with output verification only. We sample N = 4 trajectories costing 8Pn, then we make a forward pass on the reference model costing another 8Pn followed by training costing 24P The total training cost per step is thus 40Pn. We set sequence length to 4096, batch size to 512 and the training early stops at 100 gradient steps costing 13 x 17 FLOPs for 7B parameter model.
31


Reviews:
[{"review_id": "VhAanbktbn", "reviewer": "Reviewer_AvWR", "summary": "The paper demonstrates that Entropy Minimization can effectively improve the performance of large language models on complex math, physics, and coding tasks—even without using any labeled data.", "questions": "1. What is j in Formula 5 (Lines 265 -266)", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n\n1. The paper is well-structured, and the core idea is both compelling and original, offering a promising new direction for unlocking the potential of language models in post-training.\n\n2. The greatest strength of Entropy Minimization lies in its independence from large amounts of high-quality labeled data, eliminating the need to collect and curate such annotations. This makes it a more flexible and efficient approach to post-training, with significant potential for broader application.\n\n\n**Weaknesses**:\n\n1. Why do these improvements appear only on the Qwen model? Could the authors provide more insights by analyzing the internal parameter characteristics or pretraining data of Qwen? Why does the same approach not work as well on LLaMA?\n\n2. Beyond math and coding tasks, have the authors evaluated performance on other domains, such as memory-intensive tasks?\n\n3. Regarding the KL regularization coefficient β, how does it affect training outcomes? It would be helpful to see additional training results and analysis related to its impact.\n\n4. I would like to see more CoT examples generated after Entropy Minimization training. How do these differ in style and structure from those produced by RL training with GRPO? And will EM fail to activate the model's self-verification capabilities as effectively as GRPO does?\n\nIf these concerns are properly addressed, I would be willing to reconsider my evaluation.", "was_revised": true, "final_justification": "Most of my concerns during rebuttal have been addressed, so I have updated my score accordingly.", "rebuttal": "**Impact of initial model choice**\n\nThe reviewer raised a great point on the impact of the choice of the pretrained model on the performance improvements achieved through entropy minimization. We observe that the improvements from EM-FT and EM-RL on Llama-3.1-8B-Instruct are less substantial on math reasoning tasks compared to those on Qwen-2.5 (Sec. 5). The intuition behind using EM is based on the observation that models tend to be more accurate on examples for which they make predictions with higher confidence [6,7,8] (L19, L76). However, [9] reveals that EM based methods improves performance as long as the finetuning data align closely with the pretraining data. As this fine-tuning data diverges from the training distribution, the model's performance diminishes. Building on this observation, we conjecture that EM is most beneficial when there is reasonable alignment between the pre-training and fine-tuning data. \n\nA common assumption in RL, and test time scaling for LLMs is that it is expected that the base models can already solve a problem in N trials, i.e., we require at least one successful trajectory to reinforce and compute the advantage [10]. In output-verified RL, this assumption is applied by using accuracy based filters during training, which filters out un-solvable problems based on the success rate over N trials [11,12]. This hypothesis suggests that the base model must generate high-quality responses with reasonable probability and coverage over the downstream task. Extrapolating this assumption, entropy minimization via EM-RL, EM-FT and EM-INF sharpens the probability distribution over all responses based on the model's confidence [13]. We conjecture that this sharpening mechanism is expected to be more useful on base models which show better scaling under best of N. Table 1 (rebuttal) shows the percentage gain under best of n sampling (N=4) compared to greedy decoding (N=1) averaged over MATH, AMC, Minerva and OlympiadBench. We observe that Qwen-2.5-math-7b shows larger gains compared to LLaMA-3.1-8B-instruct when the number of responses is increased from N=1 to N=4, indicating that Qwen-2.5 is more likely to produce a correct solution. We hypothesize that better inference time scaling (e.g. best of n) is a likely cause for improvements from EM-RL, since we observe larger improvements on Qwen2.5 compared to LLaMA on math (Sec.5).\n\nTable 1: Average performance on MATH, AMC, Minerva and OlympiadBench with greedy decoding (N=1) and best of N (N=4). We select the best response by using self-consistency, i.e., we pick the answer with the highest frequency.\n\n|Model|Greedy (N=1)|Best of N (N=4)|% Gain\n|---|---|---|---|\n|Qwen-2.5-math-7b|23.2|26.4|13.4|\n|LLaMA-3.1-8B-instruct|19.5|21.1|7.5|\n\nTo further understand initial model choice, we finetune another base model- DeepSeek-Math-7B-Instruct on the same data and training settings as Qwen-2.5-7B-Math (Sec.3.3). Deepseek-7b-math-instruct has been finetuned on math-related tokens for 500B tokens on top of Deepseek-coder-7b by the model creators [1]. We use the instruction tuned version which has been further finetuned using chat data. We fine tune deepseek-7b-math-instruct using entropy minimization (EM-RL) and evaluate it in Table 2 (rebuttal), we observe that EM improves performance over the Deepseek model by 2.5% on average on math tasks. Compared to LlaMA both, Qwen-2.5-math and DeepSeek-7b-math have undergone domain specific math finetuning [2,1]. This observation suggests that gains from EM-RL are not only limited to Qwen, and can be seen on stronger models finetuned for the downstream task.\n\nTable 1: Performance comparison of unsupervised EM-RL against supervised RLOO. Bold indicates performance improvement over RLOO. We use the same training setup defined in Sec. 3.3.\n\n|Model|Math|AMC|Minerva|Olympiad|Avg.|\n|---|---|---|---|---|---|\n|Deepseek-math-7b-instruct|42|16.9|19.1|12.6|22.65|\n|RLOO|42.4|24.1|20.6|13.2|25.07|\n|EM-RL-token|43.4|18.1|22.8|13.6|24.47|\n|EM-RL-sequence|42.8|21.7|21.7|14.4|25.15|\n\nThe success and limitations of our method highlight the importance of the capabilities of the pretrained models. We call for the inclusion of EM as a baseline in the evaluation of future post-pretraining and inference-time scaling algorithms, to better separate the contribution of algorithmic innovations from the intrinsic capabilities of the model itself (Line 58). We are unable to analyze the pre-training data of either LLaMA and Qwen models, since they have not been released and acknowledge that studying the impact of base model choice in RL is of great importance. We will update Sec.3.4 with these analyses and results to better contextualize model choice. We will also rephrase (Line 48, Sec. 6. Conclusion) to better acknowledge the impact of initial checkpoints on RL tuning. \n\n\n**Evaluation on knowledge intensive tasks**\nWe evaluate EM-INF and EM-RL on knowledge intensive tasks- UGPhysics and MMLU and observe that EM can improve performance UGPhysics, but maintains the same performance on MMLU.\n\nUGPhysics is a question answering benchmark for Physics which requires undergrad level physics knowledge [3]. We observe that entropy minimization with logit optimization (EM-INF) improves Qwen-2.5-7B-Instruct, Qwen-2.5-math-7B-Instruct, and Llama-3.1-8B-Instruct by 4% on average, indicating that unsupervised EM-INF helps improve pre-training performance by making the model more deterministic (Table 3). We will add this description about UGPhysics in Sec.4 to clarify. \n\nWe additionally evaluate MMLU-stem-COT [4] performance for our RL tuned models (EM-RL) in Table 3 (rebuttal). We observe that EM-RL did not impact its performance on knowledge intensive MMLU.\n\nTable 3: Performance of EM-RL compared with GRPO and RLOO on the MMLU-stem-COT dataset [4].\n|Model|MMLU-stem-CoT|\n|---|---|\n|Qwen2.5-Math-7B|63.6|\n|RLOO|63.2|\n|GRPO|64.1|\n|EM-RL-Sequence|63.7|\n|EM-RL-Token|64.4|\n\n\n**Sensitivity to KL regularization coefficient $\\beta$**\n\nWe provide results with different $beta$ for EM-RL token below in Table 4 (rebuttal). We observe that a small $\\beta$ value is better for entropy minimization because it weighs EM more in the overall loss function (L172). \n\nTheoretically, the KL regularization can increase the entropy when the base model is too uncertain, i.e., it deviates from the pre-trained checkpoint causing it to visit high entropy states. However when $\\beta<1$ (we use $0.001$) (L172) the dominant gradient in the policy gradient is from entropy minimization. Since we set $\\beta$ to a small value the overall objective minimizes the entropy. \n\nTable 4: Sensitivity of EM-RL-token to various values of the KL regularization coefficient. \n|Model|Math|AMC|AIME|Minerva|OlympiadBench|Avg.|\n|---|---|---|---|---|---|---|\n|Qwen-2.5-math-7b|43.8|31.3|15.5|14.7|19|24.86|\n|EM-RL-token-$\\beta=1.5$|45|39.8|13.3|9.6|21.2|25.78|\n|EM-RL-token-$\\beta=0.5$|50.8|39.8|16.7|10.7|23|28.2|\n|EM-RL-token-$\\beta=0.001$|70.8|57.8|18.9|30.9|19|39.48|\n|EM-RL-token-$\\beta=0$|70.9|57.5|18.2|29.6|19|39.04|\n\n\n**What is j in Formula 5**\nThank you for pointing this out! $j$ indexes over the vocabulary, i.e., $j$ ranges from 1 to the vocabulary size. We will correct the notation to $L_{EM-INF} = max (-\\sum_{j=1}^{|{V}|}\\sigma(z_t)_j\\log \\sigma(z_t)_j,\\delta)$\n\n**COT example**\n\nWe provide example generations with entropy minimization at inference time (EM-INF) via logit optimizations in the Appendix (Figure 4). For EM-INF, we observe that the model produces more deterministic and concise outputs, since entropy minimization optimizes the model to seek a specific mode. This behaviour helps improve on complex/highly uncertain tasks such as scientific coding. \n\nWe provide EM-RL generations below and compare it with regular RLOO model generations. We observe that both EM-RL and RLOO invoke the self-verification step in their generations. These observations indicate that the entropy minimization makes the model more deterministic and generates a more concise outputs while preserving self-verification. We will add these qualitative analysis to the results section. \n\n**RLOO**\n[omit] **Let's check if this solution works.** The five integers are 1, 8, 10, 10. [omit] The median is 2 greater than the arithmetic mean, [omit] So, the solution works. The least possible value for the mode is 10.\n\n**EM-RL-seq**\n[omit] **We should verify that this solution works by checking the values of a, b, c, d and e**\nThe median is [omit] 2 greater than the median. [omit] So, the solution is correct. Thus, the least possible value for the mode is 10.\n\n[1] Shao, Zhihong, Peiyi Wang et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models\n\n[2] Yang, An, Beichen Zhang, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement\n\n[3] Xu, Xin, Qiyun Xu, et al Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models\n\n[4] Hendrycks, Dan, Collin Burns, et al. Measuring massive multitask language understanding\n\n[5] MMLU-Stem-COT subset from Deepseek evaluator: https://github.com/deepseek-ai/DeepSeek-Math\n\n[6] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization\n\n[7] Subhankar Roy, Aliaksandr Siarohin, et al. Unsupervised domain adaptation using feature-whitening and consensus loss\n\n[8] Wang, D., Shelhamer, E., Liu, S., et al, T. Tent: Fully test-time adaptation by entropy minimization\n\n[9] Press, Ori, Ravid Shwartz-Ziv, et al. The entropy enigma: Success and failure of entropy minimization\n\n[10] Beirami, Ahmad, Alekh Agarwal, et al. Theoretical guarantees on the best-of-n alignment policy\n\n[11] Yu, Qiying, Zheng Zhang, et al. Dapo: An open-source llm reinforcement learning system at scale\n\n[12] Cui, Ganqu, Lifan Yuan,  et al. Process reinforcement through implicit rewards\n\n[13] Huang, Audrey, Adam Block, et al Self-improvement in language models: The sharpening mechanism"}, {"review_id": "uNnySNG9nL", "reviewer": "Reviewer_mT9o", "summary": "This paper explores entropy minimization in the context of LLMs, particularly, as a way to improve performance without additional labeled data. The authors propose three methods to do so: minimizing token-level entropy directly, using entropy as a reward in reinforcement learning, and optimizing the logits to reduce entropy at inference time. Their experiments show that entropy minimization improves performance both on math and coding tasks.", "questions": "What decoding method do you when sampling responses from the base LM? \n\nl162-4: this sentence is unclear to me. What does “lost in the sauce” mean?\n\nIs the KL regularization term used for both EM-RL-token and EM-RL-sequence or just the former? If only the former, which it seems like from 3.2, why?", "limitations": "Appropriately addressed.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n* I think this is an overall strong paper. It makes connections between an extensively studied ML method and large language models, which is interesting in itself, and additionally demonstrates promising results. \n* The empirical evaluation is extensive; it is done on a diverse set of datasets and across several LLMs. \n* Entropy minimization turns out to be useful both for math and coding tasks, which I think is an exciting finding given that it is an unsupervised method. \n* The paper also discusses limitations and settings where the method does not work, namely, for tasks where confidence is not a good indicator of performance and for less capable base models. \n* There is appropriate discussion of related work. \n\n\n**Weaknesses:**\n1. The method assumes that we take random samples from the LLM during decoding (I take it that entropy minimization won’t make much of a difference if you do greedy decoding), so naturally there is variance in the estimates. Yet, the authors only report point estimates in their experiment, i.e., there are no error bars. It would be great if that could be added to have more confidence in the results and ensure reproducibility.\n2. Again, regarding variance: Trajectory-level entropy estimator must have very high variance, and N=4 rollouts therefore seems very small. I’m surprised that it works with such a small number of rollouts, but it would nevertheless be interesting to see an analysis on how this impacts performance, and to understand if there is a trade-off between performance and runtime.\n3. Tab 3 shows that adaptive temperature, which is a simpler method, often outperforms the proposed inference method based on entropy. I think that should be conveyed more clearly in the text in order not to mislead. \n4. It's a bit odd that Prop. 1 is only given in an informal manner. I don't think it would be too complicated to write out formally what it is you prove.", "was_revised": true, "final_justification": "The (minor) weaknesses I pointed out in my review have been resolved and I maintain my recommendation to accept.", "rebuttal": "**Error bars**\n\nWe thank the reviewer for the great suggestion. We provide all result tables with error bars below by running the evaluation three times with a different random seed. We report the mean $\\pm$ standard deviation for all models. We observe that our trend is still consistent with EM-RL, without any labeled data, achieving comparable or better performance than RL baselines such as GRPO and RLOO that are trained on 60K labeled examples. We will add these error bars to aid clarity and reproducibility in the paper. Furthermore, our trends with inference time logit optimization (EM-INF) is still consistent with improving the performance of the base model for almost all tasks and model classes by 3% on average.\n\nTable 1: Performance comparison of unsupervised finetuning (EM-FT) and various rewarding methods in EM-RL with supervised finetuning and RL. Italics, Bold indicates performance improvement over GRPO and SC-RL (self-consistency RL), respectively. \n\n|Model|Math|AMC|AIME|Minerva|Olymp.|Avg.|LeetC|LiveC|Avg.|FLOPs|\n|-|-|-|-|-|-|-|-|-|-|-|\n|**Qwen2.5-7b**|43.8($\\pm1.1$)|31.3($\\pm2.7$)|15.6($\\pm3.7$)|14.7($\\pm1.0$)|19.0($\\pm1.5$)|24.9|26.1($\\pm1.8$)|18.4($\\pm0.9$)|22.3|–|\n|\n|w/SFT N=1|48.2($\\pm2.4$)|30.2($\\pm2.6$)|10.0($\\pm0.6$)|17.6($\\pm2.3$)|22.4($\\pm1.7$)|25.7|18.3|18.3|18.3|1.0|\n|w/RLOO N=4|73.0($\\pm0.9$)|57.8($\\pm2.9$)|23.3($\\pm4.8$)|31.2($\\pm0.6$)|34.2($\\pm0.7$)|43.9|28.3|26.7|27.5|13.1|\n|w/GRPO N=4|71.8($\\pm0.6$)|56.6($\\pm2.0$)|21.1($\\pm2.4$)|25.0($\\pm3.6$)|35.9($\\pm1.0$)|42.1|25.0|25.8|25.4|13.1|\n|\n|EM-FT N=1|67.2($\\pm0.6$)|51.8($\\pm1.0$)|14.4($\\pm2.0$)|*33.1($\\pm0.7$)*|*34.4($\\pm0.2$)*|40.2|*28.3($\\pm1.1$)*|17.2($\\pm0.4$)|22.8|1.0|\n|SC-RL N=4|73.2($\\pm0.6$)|51.8($\\pm2.2$)|15.6($\\pm1.6$)|*26.1($\\pm1.0$)*|*36.7($\\pm0.5$)*|40.7|–|–|–|13.1|\n|EM-RL-SEQUENCE N=4|67.2($\\pm1.4$)|**53.0($\\pm2.1$)**|**21.1($\\pm2.8$)**|***30.9($\\pm2.3$)***|35.6($\\pm0.9$)|**41.6**|*31.1($\\pm1.4$)*|21.7($\\pm1.1$)|*26.4*|13.1|\n|EM-RL-TOKEN N=4|70.8($\\pm0.8$)|***57.8($\\pm2.6$)***|**18.9($\\pm3.1$)**|***30.9($\\pm1.0$)***|35.9($\\pm0.6$)|**42.9**|*29.5($\\pm0.9$)*|24.5($\\pm0.7$)|*27.0*|13.1|\n\nTable 2:Performance comparison of EM-INF against test-time scaling methods. Bold & Italics indicates the best and second best performance, respectively. Dash line (\"–\") denotes that selfconsistency is inapplicable. We use temperature, t= 0.1 and sample N = 1 trajectory unless specified\n|Model|Math|AMC|AIME|Minerva|Olymp.|Avg.|LeetCode|UGPhysics|\n|-|-|-|-|-|-|-|-|-|\n|Qwen2.5-Math-7B-Instruct|79.2($\\pm0.7$)|48.2($\\pm1.6$)|8.9($\\pm0.6$)|**43.4($\\pm1.2$)**|**41.3($\\pm0.6$)**|44.2|0.6($\\pm0.4$)|18.6($\\pm1.0$)|\n|Greedy Decoding($t=0$)|79.0|53.0|*11.1*|*42.6*|39.9|45.1|1.1|18.8|\n|Iterative-refinement($N=3$)|79.2($\\pm0.0$)|48.2($\\pm0.0$)|8.9($\\pm0.0$)|**43.4($\\pm1.1$)**|**41.3($\\pm0.6$)**|44.2|*1.7($\\pm0.2$)*|16.4($\\pm0.8$)|\n|Self-consistency($N=4$)|78.8($\\pm0.0$)|53.0($\\pm0.0$)|*11.1($\\pm0.0$)*|**43.4($\\pm1.2$)**|40.7($\\pm0.7$)|*45.4*|0.8($\\pm0.4$)|19.6($\\pm0.8$)|\n|Adaptive Temp|**80.8($\\pm0.9$)**|**55.4($\\pm0.9$)**|10.0($\\pm1.9$)|41.5($\\pm2.1$)|38.7($\\pm1.2$)|45.3|**2.2($\\pm0.3$)**|**23.7($\\pm0.7$)**|\n|EM-INF|*80.2($\\pm0.6$)*|*54.2($\\pm2.5$)*|**14.4($\\pm1.6$)**|42.3($\\pm0.8$)|*40.8($\\pm0.6$)*|**46.4**|0.6($\\pm0.4$)|*23.1($\\pm1.3$)*|\n|\n|Qwen2.5-7B-Instruct|74.0($\\pm0.9$)|41.0($\\pm1.8$)|8.9($\\pm0.9$)|*41.2($\\pm0.7$)*|37.5($\\pm0.2$)|40.5|47.2($\\pm1.5$)|23.4($\\pm1.3$)|\n|Greedy Decoding($t=0$)|*74.2*|44.6|7.8|40.4|37.2|40.8|46.1|23.7|\n|Iterative-refinement($N=3$)|73.8($\\pm0.0$)|37.3($\\pm0.0$)|*10.0($\\pm0.0$)*|41.5($\\pm0.7$)|37.8($\\pm0.2$)|40.1|*50.0($\\pm0.5$)*|21.5($\\pm1.2$)|\n|Self-consistency($N=4$)|73.4($\\pm0.0$)|43.4($\\pm0.0$)|*10.0($\\pm0.0$)*|33.5($\\pm2.9$)|**41.3($\\pm0.8$)**|40.3|48.3($\\pm0.9$)|23.4($\\pm1.4$)|\n|Adaptive Temp|**74.4($\\pm0.8$)**|**50.6($\\pm1.8$)**|*10.0($\\pm1.6$)*|**41.9($\\pm0.9$)**|*39.7($\\pm0.7$)*|**43.3**|49.4($\\pm0.9$)|*25.8($\\pm1.0$)*|\n|EM-INF|*73.8($\\pm1.1$)*|*45.8($\\pm0.9$)*|**11.1($\\pm0.5$)**|*41.2($\\pm0.7$)*|38.2($\\pm0.5$)|*42.0*|**51.7($\\pm0.7$)**|**26.9($\\pm1.1$)**|\n|\n|Llama-3.1-8B-Instruct|40.6($\\pm0.5$)|18.1($\\pm1.5$)|1.1($\\pm1.6$)|22.4($\\pm1.0$)|15.7($\\pm0.8$)|19.6|10.6($\\pm1.2$)|17.5($\\pm0.7$)|\n|Greedy Decoding($t=0$)|40.6|16.9|3.3|21.0|16.0|19.6|12.8|16.1|\n|Iterative-refinement($N=3$)|41.0($\\pm0.0$)|19.3($\\pm0.0$)|1.1($\\pm0.0$)|22.4($\\pm1.5$)|15.7($\\pm0.8$)|19.9|11.7($\\pm1.1$)|18.3($\\pm0.6$)|\n|Self-consistency($N=4$)|41.2($\\pm0.0$)|20.5($\\pm0.0$)|*4.4($\\pm0.0$)*|*20.2($\\pm0.8$)*|**19.4($\\pm0.4$)**|21.1|14.9($\\pm1.1$)|*20.2($\\pm1.1$)*|\n|Adaptive Temp|**43.6($\\pm0.6$)**|**25.3($\\pm1.8$)**|**5.5($\\pm1.2$)**|**24.3($\\pm1.1$)**|*16.6($\\pm0.6$)*|**23.1**|**17.2($\\pm1.7$)**|19.9($\\pm1.9$)|\n|EM-INF|*43.0($\\pm1.2$)*|*22.9($\\pm2.3$)*|3.3($\\pm1.4$)|*22.8($\\pm0.7$)*|16.4($\\pm0.4$)|*21.7*|*14.4($\\pm0.3$)*|**20.4($\\pm1.3$)**|\n\n\n**Sensitivity to number of rollouts $N$**\n\nThe reviewer has a great observation that the trajectory level entropy estimator has a high variance. We set $N=4$ building on existing RL fine tuning works to make sure that the training overhead is consistent [1]. We measure the impact of the number of rollouts, N on EM-RL-sequence in Table 4 (rebuttal). We observe that EM-RL-sequence benefits with larger $N$ on MATH-500, suggesting that larger number of rollouts might reduce the variance. However, we note that a larger N does not help AMC, Minerva and Olympiad bench, and reduces performance on AIME. Overall, we observe consistent performance of EM-RL-sequence on sweeping N from 4 to 16. We will add these results to Sec. 3.4 of the paper as a plot. \n\nTable 4: Sensitivity to number of rollouts N of EM-RL-sequence. We use Qwen-2.5-Math-base and use the training setup defined in Sec. 3.3. \n|Model|Math|AMC|AIME|Minerva|OlympiadBench|Avg|\n|---|---|---|---|---|---|---|\n|EM-RL-sequence-N=4|67.2|53.0|21.1|30.9|35.6|41.56|\n|EM-RL-sequence-N=8|71.4|54.2|13.3|31.6|35.0|41.1|\n|EM-RL-sequence-N=16|70.6|51.9|15.5|23.9|35.1|39.4|\n\n\n**Effectiveness of adaptive temperature**\n\nThank you for this suggestion. We will update Sec.4 results by adding the following:  Adapting the temperature (adaptive temperature) at every decoding step to reduce the entropy boosts model performance by 3% on average. Adaptive temperature is a simple method which is broadly applicable to all tasks while being computationally efficient.\n\n**Formal proof for prop 1**\n\nThank you for pointing this out! A more formal proof is provided below. We will add this to Sec. 4. \n\nProposition: Both adaptive temperature scaling and logit optimization reduce the entropy of a model’s output distribution. However, in high-uncertainty settings (i.e., when the model’s predictive distribution has high entropy), logit optimization can change the order of non-top logits. In contrast, temperature scaling preserves the order of logits and merely sharpens or flattens the distribution proportionally. Both temperature scaling and logit optimization will keep the token with the top logit and increase its probability, making them less likely to change the model’s behavior when it is highly confident.\n\nProof: \n**Part 1:** $\\forall \\textbf{z} \\in \\mathbb{R}^{|\\mathcal{V}|}$ (logits) let $p(\\textbf{z}) = \\text{softmax}(\\textbf{z})$. Let $\\mathcal{H}(p)$ denote the entropy of $p$;  $\\mathcal{H}=-\\sum_{i=1}^{|\\mathcal{V}|}p_i \\log p_i$.\n$\\exists \\textbf{z}$, $\\eta>0$ and $a,b\\in[1,\\dots,|V|]$, such that \n$\\textbf{z}_a<\\textbf{z}_b$ and $\\textbf{y}_a>\\textbf{y}_b$ where $\\textbf{y}_i = z_i + \\eta p_i(\\log p_i + \\mathcal{H})$, $i=1,\\dots,|\\mathcal{V}|$.\n\n\nConsider $\\mathbf{y_a}-\\mathbf{y_b} = \\mathbf{z_a}-\\mathbf{z_b} + \\eta (g_a-g_b)$ where $g_i = p_i (\\log p_i + \\mathcal{H})$. $g$ is the gradient of entropy $\\mathcal{H}$ w.r.t. $\\mathbf{z}$. \n\nWe now show that $\\exists \\mathbf{z}, a,b$ such that $z_a<z_b$ and $g_a>g_b$.\n\nBy Mean-Value Theorem $\\exists c \\in (p_a,p_b)$ such that $g_a-g_b = (p_a-p_b)(\\log c + \\mathcal{H} + 1)$. \nWhen $p_a-p_b < e^{-(\\mathcal{H}+1)}$ we have \n\n$c<p_b<e^{-(\\mathcal{H} + 1)} \\implies \\log c + \\mathcal{H} + 1 < 0 \\implies g_a-g_b > 0$\n\nThus, we can set an arbitrary large $\\eta$ such that, \n$z_a-z_b + \\eta(g_a-g_b) > 0$\n\n\n\n**Part 2**: $b$ cannot be $\\arg \\max_{i} z_i$. \n\nProof: Let $j = \\arg\\max_i \\mathbf{z_i}$, i.e. the index of the maximum logit. \nSince $p_j = \\max_i p_i$ we can bound it using the inequality: \n   \n$\\mathcal{H} (p) = -\\sum_i p_i \\log p_i \\geq -\\log p_j \\quad \\Rightarrow \\quad p_j \\geq e^{-\\mathcal{H}(p)}\\geq e^{-(\\mathcal{H} (p)+1)}$\n\nWe now analyze how $g_i$ behaves as a function of $p_i$. Let $g(p_i) = p_i (\\log p_i + \\mathcal{H})$ and $g'(p_i) = \\log p_i + \\mathcal{H} + 1$.\nThus, $g(p_i)$ is monotonically increasing whenever $g'(p_i)>0$, i.e. when, \n$p_i > e^{-(\\mathcal{H} + 1)}$\nSo no lower-probability index $i$ can have $g_i>g_j$, i.e., the entropy gradient cannot reduce $\\mathbf{y_j}$ relative to others. Therefore, $b\\neq j$, leading to the following behavior\n\n$0<g_a\\leq g_b  \\quad p_a \\in (e^{-\\mathcal{H}},p_b]$\n\n$g_a < 0 < g_b \\quad p_a \\in (0,e^{-\\mathcal{H}})$\n\n\n**Decoding method for sampling?**\n\nWe use sampling based decoding with a temperature of 0.1 for Table 2. We will add this to the Table caption to aid reproducibility. \n\n**l162-4: this sentence is unclear to me. What does “lost in the sauce” mean?**\n \nThank you for pointing this out. We will rephrase it as the following to make it more formal- “​​For example, problems requiring a complex, long chain of thought, where being more deterministic at every step helps prevent the model from losing track of the reasoning process.”\n\n**Is the KL regularization term used for both EM-RL-token and EM-RL-sequence or just the former?**\n\nThe KL term is used in both EM-RL-token and EM-RL-sequence. We will add this to Sec. 3.2 more explicitly.  \n\n[1] Cui, Ganqu, Lifan Yuan,  et al. Process reinforcement through implicit rewards"}, {"review_id": "9jPcgXyA6l", "reviewer": "Reviewer_3pzP", "summary": "This paper presents a suite of related techniques for boosting the reasoning\nperformance of post-pre-training, pre-fine-tuning LLMs using entropy\nminimization (EM).  In particular, it introduces two train-time methods,\nEM-based \"supervised\" fine-tuning and EM-based RL fine-tuning as well as an\nEM-based inference time algorithm.  The intuition is that after pre-training\nLLMs already have strong priors and that decreasing entropy (uncertainty) will\nlead stronger performance, particular in tasks that have definite answers and\nrequire sequence of intermediate steps.  The results show that the EM-based\nmethods can indeed improve LLM performance reasoning tasks with lower compute\nrequirements and no need for labelled data.", "questions": "The paper is clear enough such that I do not have any major questions.\n\n- How does this relate to the general principle the neural networks are\n  over-confident and calibrating them usually involve strategically decreasing\n  their confidence?\n  - How does relate to hallucinations and over-confidence? I suppose reasoning\n    tasks are maybe less hallucination-prone compared something like\n    question-answering which it seems like EM-based methods are less likely to\n    improve.\n- Does this suggest that pre-training itself could benefit from incorporating\n  some sort of entropy-maximization, or does EM only serve to benefit the model\n  after pre-training has finished?\n- For EM-INF, by \"optimizing the logits\" do you mean optimizing the layer that\n  produces the logits?  By my understanding, the logits themselves are just\n  values, and adjusting the values directly does not make sense (as opposed to\n  adjusting, say weights/parameters).\n\n### Minor points\n- ? [Tab. 2] Does 60k unlabeled prompts mean the prompts from the particular task just without the provided answers?\n- It seems like self-consistency could still be applied to long-output tasks if you use a fuzzy matching function instead of just an indicator (e.g., which output has the lowest average Levenshtein distance to all other outputs?).\n- [Line 198] By \"1e-6\", do you mean \"1 * 10^-6\" as opposed to \"1 * exp(-6)\"?\n- \"chain of thoughts\" -> \"chains of thought\"\n- \"lost in the sauce\" is a bit too informal.", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths\n- The results of the apply methods demonstrate frequent (maybe not consistent)\n  improvements without the use of labelled data.  Insofar as this is shown to\n  match performance to standard methods, it represents a Pareto-improvement\n  since it eliminates the need for labeled data.\n- A variety of related methods are presented which is not only useful insofar\n  as each method has slightly different applicability, but also suggests\n  something deeper is at play with entropy minimization.\n- The presentation of the methods is clear and easy to follow.\n### Weaknesses\n- (minor) The experiments and analysis do not give a wholly clear picture of\n  when EM-based methods will work versus when they won't.  These details could\n  help indicate whether the proposed methods will be game-changers versus\n  a niche way to squeeze out performance in particular scenarios.", "was_revised": true, "final_justification": "I maintain my positive assessment of the paper after the discussion and clarifications offered by the authors.", "rebuttal": "**When is entropy minimization helpful?**\n\nThe reviewer raised a great point on when EM works. The intuition behind using EM is based on the observation that models tend to be more accurate on examples for which they make predictions with higher confidence (L19, L76). However, [1] reveals that entropy minimization based methods improves performance as long as the finetuning data align closely with the pretraining data. As this fine-tuning data diverges from the training distribution, the model's performance diminishes. Building on this observation, we conjecture that EM is most beneficial when there is reasonable alignment between the pre-training and fine-tuning data, as supported by model performance on downstream tasks. \n\nFurthermore, a common assumption in RL, and test time scaling for LLMs is that it is expected that the base models can already solve a problem in N trials, i.e., we require at least one successful trajectory to reinforce and compute the advantage [2]. In output-verified RL, this assumption is applied by using accuracy based filters during training, which filters out un-solvable problems based on the success rate over N trials. This hypothesis suggests that the base model must generate high-quality responses with reasonable probability and coverage over the downstream task. Extrapolating this assumption, EM-RL, EM-FT and EM-INF sharpens the probability distribution over all responses based on the model's confidence [4]. We conjecture that this sharpening mechanism is expected to be more useful on base models which show better scaling under best of N. \n\nWe will update Sec.3.4 with this discussion to better contextualize when entropy minimization can help. We will also add this to our future work and limitations section and acknowledge that the success and limitations of entropy minimization is a great research direction. \n\n\n**Over-confidence and calibration**\n\nThank you for raising this interesting point on over-confidence and calibration! Entropy minimization (EM-RL, EM-FT) encourages the model to produce low-entropy (i.e., highly confident) predictions. We conjecture that entropy minimization can both improve or hurt calibration. If entropy minimization helps the model become more confident in the right direction- that is, it becomes more confident on correctly generated solutions, then both performance and calibration improves. We expect that when the pre-trained model’s data distribution closely aligns with the fine-tuning task and the pre-trained model produces the correct response with a reasonable probability, entropy minimization should benefit performance and calibration. On the other hand, entropy minimization can harm calibration when it reinforces highly confident (low entropy) incorrect solutions to be generated with higher probability. \nWe measure the expected calibration error (ECE) of both LLaMA and Qwen 2.5 models in Table 2 (rebuttal). We observe that the ECE for Qwen models improves, likely because EM-RL substantially benefits Qwen 2.5 math on math reasoning (Sec.3.4) . On the other hand, we notice that entropy minimization harms LLaMA’s calibration likely because we noticed smaller improvements on using EM-RL with LLaMA. We conjecture that the impact of entropy minimization on calibration is tied to the base model capability and expect that entropy minimization should likely benefit or have minimal impact on model performance and calibration. \n\nTable 2: Expected Calibration Error for EM-RL-sequence and EM-RL-token compared to the base model on the AMC math dataset. Lower ECE is better. \n|Model|ECE on AMC|\n|-|-|\n|Qwen2.5-Math-7B|64.9|\n|EM-RL-SEQUENCE $N=4$|43.5|\n|EM-RL-TOKEN $N=4$|44.7|\n|\n|Llama-3.1-8b-Instruct|79.0|\n|EM-RL-SEQUENCE $N=4$|81.5|\n|EM-RL-TOKEN $N=4$|83.3|\n\n\n**Non-reasoning tasks**\nWe evaluate EM-INF and EM-RL on knowledge intensive tasks- UGPhysics and MMLU and observe that EM can improve performance UGPhysics, but maintains the same performance on MMLU.\n\nUGPhysics is a question answering benchmark for Physics which requires undergrad level physics knowledge [3]. We observe that entropy minimization with logit optimization (EM-INF) improves Qwen-2.5-7B-Instruct, Qwen-2.5-math-7B-Instruct, and Llama-3.1-8B-Instruct by 4% on average, indicating that unsupervised EM-INF helps improve pre-training performance by making the model more deterministic (Table 3). We will add this description about UGPhysics in Sec.4 to clarify. \n\nWe additionally evaluate MMLU-stem-COT performance for our RL tuned models (EM-RL) in Table 3 (rebuttal). We observe that EM-RL did not impact its performance on knowledge intensive MMLU. This observation along with calibration errors, indicates that entropy minimization does not necessarily make the model too confident on non-reasoning tasks.\n\nTable 3: Performance of EM-RL compared with GRPO and RLOO on the MMLU-stem-COT dataset.\n|Model|MMLU-stem-CoT|\n|---|---|\n|Qwen2.5-Math-7B|63.6|\n|RLOO|63.2|\n|GRPO|64.1|\n|EM-RL-Sequence|63.7|\n|EM-RL-Token|64.4|\n\n**Entropy maximization during Pre-training**\nEntropy maximization during pre-training is an interesting idea to explore! [5] find that supervised RL finetuning in its default setting (e.g. GRPO) drops entropy during the early stages of training. Meanwhile, the policy’s validation performance presents an inverse trend, where it rises rapidly when training starts, and then saturates at a certain level. They find that entropy needs to be effectively maintained during training to avoid saturation of policy performance. This observation suggests that, for a given performance level, a policy with higher entropy is preferred, since low entropy policies are difficult to further improve. Building on these observations, we conjecture that if the pre-trained model has converged and is at a higher entropy level, further entropy minimization through EM-RL can help as a post-training step can benefit, since there is more entropy to consume by EM-RL. We will add this insight to Sec. 3.4 of the paper. \n\n\n**Logit optimization**\n\nEM-INF treats the model’s output logits as if they were free parameters, and uses gradient descent to update them to minimize the entropy of the distribution they induce; no gradient wrt. or the model parameters or parameter update is needed (Line 260). This optimization is performed during each decoding step to reduce the entropy of the LLM’s distribution without any parameter update. After logit optimization, we use sampling based decoding to select the next token. EM-INF can change the order of non-top logits (proposition 1), thereby generating different sequences. Empirically, we find that EM-INF works on complex tasks with high uncertainty (e.g. AIME math, UGPhysics and SciCode)(Table 3). By using EM-INF, we observe that Qwen 32B can outperform frontier models like GPT-4o on Scicode and is 3x more efficient than inference scaling through self-consistency and sequential refinement (Table 4, Figure 1).\n\n**[Tab. 2] Does 60k unlabeled prompts mean the prompts from the particular task just without the provided answers?**\n\nYes, the 60k unlabeled prompts is a collection of prompts without the ground truth answers. \n\n**Self-consistency on long output tasks**\n\nThank you for pointing out this interesting experiment! We agree that self-consistency can be applied using more sophisticated measures of consistency beyond exact matching, such as, fuzzy string match, embedding based similarity and using another LLM to measure consistency. We leave experimentation with these techniques as future work and we will clarify this throughout our paper to acknowledge this. \n\nWe experiment with fuzzy matching for self-consistency for EM-INF in the Table 4. We find that our trend that entropy minimization (EM-INF, adaptive temperature) improves inference time scaling methods (self-consistency and iterative refinement) by 3% still holds. We will add these results to our paper and acknowledge that self-consistency can be performed on long outputs using methods like fuzzy string matching. \n\n\nTable 4: Performance comparison of EM-INF against test-time scaling methods on LeetCode. Bold & Italics indicates the best and second best performance, respectively. We use temperature, t= 0.1 and sample N = 1 trajectory unless specified. For self-consistency on long outputs, we pick the output which has the lowest average Levenshtein distance to all other outputs.\n| Model|LeetCode|\n|-|-|\n|Qwen2.5-Math-7B-Instruct|0.6|\n|Greedy Decoding ($t=0$)|1.1|\n|Iterative-refinement ($N=3$)|*1.7*|\n|Self-consistency ($N=4$)|0.8|\n|Adaptive Temp|**2.2**|\n|EM-INF|0.6|\n|\n|Qwen2.5-7B-Instruct|47.2|\n|Greedy Decoding ($t=0$)|46.1|\n|Iterative-refinement ($N=3$)|*50.0*|\n|Self-consistency ($N=4$)|48.3|\n|Adaptive Temp|49.4|\n|EM-INF|**51.7**|\n|\n|Llama-3.1-8B-Instruct|10.6|\n|Greedy Decoding ($t=0$)|12.8|\n|Iterative-refinement ($N=3$)|11.7|\n|Self-consistency ($N=4$)|14.9|\n|Adaptive Temp|**17.2**|\n|EM-INF|*14.4*|\n\n\n**\"1e-6\"**\n1e-6 means 1*10^-6. We will replace L198 with this expression for clarity. \n\n**Typo and language**\nThank you for pointing these out! We will fix the typo and replace \"lost in the sauce” with: For example, problems requiring a complex, long chain of thought, where being more deterministic at every step helps prevent the model from losing track of the reasoning process.”\n\n\n[1] Press, Ori, Ravid Shwartz-Ziv, et al. The entropy enigma: Success and failure of entropy minimization\n\n[2] Beirami, Ahmad, Alekh Agarwal, et al. Theoretical guarantees on the best-of-n alignment policy\n\n[3] Xu, Xin, Qiyun Xu, Tong Xiao, et al Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models\n\n[4] Huang, Audrey, Adam Block, et al Self-improvement in language models: The sharpening mechanism\n\n[5] Cui, Ganqu, Yuchen Zhang et al. The entropy mechanism of reinforcement learning for reasoning language models."}, {"review_id": "jNYMFfOXPm", "reviewer": "Reviewer_PKyh", "summary": "This paper investigates entropy minimization as a standalone training and inference objective for improving large language models' performance on reasoning tasks without labeled data. The authors propose three approaches: EM-FT , EM-RL, and EM-INF. They demonstrate competitive performance with supervised methods on math, physics, and coding benchmarks.", "questions": "See weakness", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Authors present a very interesting observation that entropy minimization is an effective way to improve model performance on reasoning tasks.\n2. Based on this discovery, authors propose three ways to improve the model, FT, RL, and inference time augmentation. Experimental results show that all these methods can significantly outperform the base model, and be comparable to supervised RL training. \n\nWeakness:\n1. While the empirical results are strong, the paper lacks deep theoretical analysis of why entropy minimization works so effectively. The core assumption that \"confident models are more likely to be correct\" is intuitive but not rigorously validated across different domains and model architectures.\n2. The paper could benefit from discussing how entropy minimization can interact with supervised RL training, e.g. will combining two rewards give better results?\n3. Some existing works such as [1] show that doing entropy maximization during RL can also improve the model's performance, which seems contradictory to this paper. It'd be helpful if authors can discuss this phenomenon.\n4. The paper posits that EM is effective only when a base model possesses sufficient \"latent reasoning capabilities,\" as evidenced by the performance gap between Qwen and Llama models. However, the argument is supported primarily by the end-task performance differences. The paper would be more impactful if it included a more direct analysis of what these capabilities are and how to quantify them.\n5. Code is not provided, so it could be hard to reproduce the results.\n\n[1] https://arxiv.org/abs/2504.20571", "was_revised": true, "final_justification": "This paper provides interesting insights into RL training. However, there remains questions on reproducibility and what's the underlying reason of the phenomenon.", "rebuttal": "**Base model capabilities**\nThe reviewer raised a great point on the impact of the choice of the pretrained model on the performance improvements achieved through entropy minimization. We observe that the improvements from EM-FT and EM-RL on Llama-3.1-8B-Instruct are less substantial on math reasoning tasks compared to those on Qwen-2.5 (Sec. 5). The intuition behind using entropy minimization is based on the observation that models tend to be more accurate on examples for which they make predictions with higher confidence [4,5,6]. However, [7] reveal that entropy minimization based methods improves performance as long as the finetuning data align closely with the training data. As the fine-tuning data diverges from the training distribution, the model's performance diminishes. Building on this observation, we conjecture that entropy minimization is most beneficial when there is reasonable alignment between the pre-training and fine-tuning data. \n\nA common assumption in RL, and test time scaling for LLMs is that it is expected that the base models can already solve a problem in N trials, i.e., we require at least one successful trajectory to reinforce and compute the advantage [2]. In output-verified RL, this assumption is applied by using accuracy based filters during training, which filters out un-solvable problems based on the success rate over N trials. This hypothesis suggests that the base model must generate high-quality responses with reasonable probability and coverage over the downstream task. Extrapolating this assumption, entropy minimization via EM-RL, EM-FT and EM-INF sharpens the probability distribution over all responses based on the model's confidence. We conjecture that this sharpening mechanism is expected to be more useful on base models which show better scaling under best of N [8]. Table 1 (rebuttal) shows the percentage gain under best of n sampling (N=4) compared to greedy decoding (N=1) averaged over MATH, AMC, Minerva and OlympiadBench. We observe that Qwen-2.5-math-7b shows larger gains compared to LLaMA-3.1-8B-instruct when number of responses is increased from N=1 to N=4, indicating that Qwen is more likely to produce a correct solution. We hypothesize that better inference time scaling (e.g. best of n) is a likely cause for improvements from EM-RL [8].\n\nTable 1: Average performance on MATH, AMC, Minerva and OlympiadBench with greedy decoding (N=1) and best of N (N=4). We select the best response by using self-consistency, i.e., we pick the answer with the highest frequency \n|Model|Greedy (N=1)|Best of N (N=4)|% Gain\n|---|---|---|---|\n|Qwen-2.5-math-7b|23.2|26.4|13.4|\n|LLaMA-3.1-8B-instruct|19.5|21.1|7.5|\n|\n\nTo further understand initial model choice, we finetune another base model- DeepSeek-Math-7B-Instruct [11] on the same data and training settings as Qwen-2.5-7B-Math (Sec. 3.3). Deepseek-7b-math-instruct has been finetuned on math-related tokens for 500B tokens on top of Deepseek-coder-7b by the model creators. We use the instruction tuned version which has been further finetuned using chat data. We fine tune deepseek-7b-math-instruct using entropy minimization (EM-RL) and evaluate it in Table 2 (rebuttal), we observe that entropy minimization improves performance over the Deepseek-math-7b-instruct by 2.5% on average on math tasks. Compared to LlaMA both, Qwen-2.5-math and DeepSeek-7b-math-instruct have undergone domain specific finetuning [11,12]. This observation suggests that gains from EM-RL are not only limited to Qwen, and can be seen on stronger models tuned for the downstream task.\n\nTable 2: Performance comparison of unsupervised EM-RL against RLOO for DeepSeek-Math-7B-Instruct. We use the setup defined in Sec. 3.3. Bold indicates performance improvement over RLOO. \n|Model|Math|AMC|Minerva|Olympiad|Avg.|\n|---|---|---|---|---|---|\n|Deepseek-math-7b-instruct|42|16.9|19.1|12.6|22.65|\n|RLOO|42.4|24.1|20.6|13.2|25.07|\n|EM-RL-token|**43.4**|18.1|**22.8**|13.6|24.47|\n|EM-RL-sequence|42.8|21.7|**21.7**|**14.4**|25.15|\n\nThe success and limitations of our method highlight the importance of the capabilities of the pretrained models. We call for the inclusion of EM as a baseline in the evaluation of future post-pretraining and inference-time scaling algorithms, to better separate the contribution of algorithmic innovations from the intrinsic capabilities of the model itself (Line 58). We will update Sec.3.4 with these analyses and results to better contextualize model choice. We will also rephrase (Line 48, Sec. 6. Conclusion) to better acknowledge the impact of initial checkpoints on RL tuning. \n\n**Entropy minimization and Supervised RL**\nThe reviewer raises an interesting parallel of combining entropy minimization with supervised RL. [3] shows that typical maximum entropy reinforcement learning techniques (e.g. PPO, GRPO, Soft actor-critic) which apply entropy maximization struggle on challenging control tasks where a precise subset of actions are required, and the remaining actions will lead to non-recoverable states and consequently low final rewards. They find that maximum entropy regularization can encourage the policy to select multiple similarly mediocre actions rather than the best one for short-term entropy benefits. Extrapolating this observation, reasoning tasks such as math and coding, require a precise sequence of actions to arrive at the correct solution. We conjecture that entropy minimization (EM-RL) along with supervised RL can encourage the policy to take the most precise action over the several mediocre choices. \n\nFurthermore, [1] find that supervised RL finetuning in its default setting (RLOO/GRPO) drops entropy during the early stages of training. Meanwhile, the policy’s validation performance presents an inverse trend, where it rises rapidly when training starts, and then saturates at a certain level. They find that entropy needs to be effectively maintained during training to avoid saturation of policy performance, as also observed in [11]. This observation suggests that, for a given performance level, a policy with higher entropy is preferred, since low entropy policies are difficult to further improve. Building on these observations, we conjecture that if the pre-trained model has converged and is at a higher entropy level, further entropy minimization through EM-RL can help as a post-training step, since there is more entropy to consume by EM-RL. We will add this insight to Sec. 3.4 of the paper. \n\n**Entropy maximization vs minimization:**\n\nThank you for pointing out this interesting work! Similar to [9], we tried to maximize entropy, but observed that entropy maximization alone without the policy loss collapses the policy after 10 gradient steps while using Qwen-2.5-7B-Math, this observation was also noted in [9]. While [9] observes performance improvements, we conjecture this is because they train on a single carefully selected example.  Furthermore, [9] trains performed on test-set examples where making the model more random (larger entropy) could be desirable, similar to best of n [2]. We find that only negative entropy reward (entropy minimization) is sufficient for RL with similar effects without requiring any data selection. Standalone entropy maximization does not converge and eventually collapses, but EM-RL and EM-FT via entropy minimization converge and can be used to train on large datasets (60K prompts) without output verification. We will add this discussion to Sec.3 of the paper. \n\n\n**Code release**\nWe will release the code for reproducibility. Since the rebuttal guidelines do not allow external links, we will attach it to our final version. \n\n[1] Cui, Ganqu, Yuchen Zhang, Jiacheng Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models\n\n[2] Beirami, Ahmad, Alekh Agarwal, et al. \"Theoretical guarantees on the best-of-n alignment policy\n\n[3] Zhang, Ruipeng, Ya-Chien Chang, and Sicun Gao. When Maximum Entropy Misleads Policy Optimization\n\n[4] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization.\n\n[5] Subhankar Roy, Aliaksandr Siarohin, et al. Unsupervised domain adaptation using feature-whitening and consensus loss.\n\n[6] Wang, D., Shelhamer, E., Liu, S., et al, T. Tent: Fully test-time adaptation by entropy minimization.\n\n[7] Press, Ori, Ravid Shwartz-Ziv, et al. The entropy enigma: Success and failure of entropy minimization\n\n[8] Huang, Audrey, Adam Block, et al  Self-improvement in language models: The sharpening mechanism\n\n[9] Wang, Yiping, Qing Yang, Zhiyuan Zeng, et al. Reinforcement learning for reasoning in large language models with one training example\n\n[10] Yang, An, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu et al. Qwen3 technical report.\n\n[11] Shao, Zhihong, Peiyi Wang et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models\n\n[12] Yang, An, Beichen Zhang, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement"}]

Author rebuttal:

Meta-review: This paper provides an in-depth investigation of Entropy Minimization (EM) for enhancing large language model performance, with its core advantage being independence from additional labeled data. The authors construct a comprehensive methodological framework encompassing:  EM-based "supervised" fine-tuning,  EM-based RL fine-tuning, and EM-based inference time algorithm, accompanied by exhaustive comparative analysis. Experimental results convincingly demonstrate EM's significant effectiveness in mathematical and coding tasks, offering a novel approach to strengthening LLM reasoning capabilities. The proposal to incorporate EM as a new baseline in future LLM reasoning evaluations represents a valuable contribution. With rigorous arguments, well-supported evidence, and effective responses to reviewers' concerns during discussion that demonstrated strong persuasiveness, the paper has earned unanimous approval. Therefore, I recommend accepting this paper.
Decision: Accept (poster)