Title: The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language model

alignment

finetuning
Abstract: Entropy minimization (EM) trains the model to concentrate even more probability mass on its most confident outputs. 
We show that this simple objective alone, without any labeled data, can substantially improve large language models’ (LLMs) performance on challenging math, physics, and coding tasks. We explore three approaches: (1) EM-FT minimizes token-level entropy similarly to instruction finetuning, but on unlabeled outputs drawn from the model; (2) EM-RL: reinforcement learning with negative entropy as the only reward to maximize; (3) EM-INF: inference-time logit adjustment to reduce entropy without any training data or parameter updates. 
On Qwen-7B, EM-RL, without any labeled data, achieves comparable or better performance than strong RL baselines such as GRPO and RLOO that are trained on 60K labeled examples. Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the challenging SciCode benchmark, while being 3x more efficient than self-consistency and sequential refinement. Our findings reveal that many pretrained LLMs possess previously underappreciated reasoning capabilities that can be effectively elicited through entropy minimization alone, without any labeled data or even any parameter updates.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates entropy minimization as a standalone training and inference objective for improving large language models' performance on reasoning tasks without labeled data. The authors propose three approaches: EM-FT , EM-RL, and EM-INF. They demonstrate competitive performance with supervised methods on math, physics, and coding benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Authors present a very interesting observation that entropy minimization is an effective way to improve model performance on reasoning tasks.\n2. Based on this discovery, authors propose three ways to improve the model, FT, RL, and inference time augmentation. Experimental results show that all these methods can significantly outperform the base model, and be comparable to supervised RL training. \n\nWeakness:\n1. While the empirical results are strong, the paper lacks deep theoretical analysis of why entropy minimization works so effectively. The core assumption that \"confident models are more likely to be correct\" is intuitive but not rigorously validated across different domains and model architectures.\n2. The paper could benefit from discussing how entropy minimization can interact with supervised RL training, e.g. will combining two rewards give better results?\n3. Some existing works such as [1] show that doing entropy maximization during RL can also improve the model's performance, which seems contradictory to this paper. It'd be helpful if authors can discuss this phenomenon.\n4. The paper posits that EM is effective only when a base model possesses sufficient \"latent reasoning capabilities,\" as evidenced by the performance gap between Qwen and Llama models. However, the argument is supported primarily by the end-task performance differences. The paper would be more impactful if it included a more direct analysis of what these capabilities are and how to quantify them.\n5. Code is not provided, so it could be hard to reproduce the results.\n\n[1] https://arxiv.org/abs/2504.20571\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a suite of related techniques for boosting the reasoning\nperformance of post-pre-training, pre-fine-tuning LLMs using entropy\nminimization (EM).  In particular, it introduces two train-time methods,\nEM-based \"supervised\" fine-tuning and EM-based RL fine-tuning as well as an\nEM-based inference time algorithm.  The intuition is that after pre-training\nLLMs already have strong priors and that decreasing entropy (uncertainty) will\nlead stronger performance, particular in tasks that have definite answers and\nrequire sequence of intermediate steps.  The results show that the EM-based\nmethods can indeed improve LLM performance reasoning tasks with lower compute\nrequirements and no need for labelled data.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The results of the apply methods demonstrate frequent (maybe not consistent)\n  improvements without the use of labelled data.  Insofar as this is shown to\n  match performance to standard methods, it represents a Pareto-improvement\n  since it eliminates the need for labeled data.\n- A variety of related methods are presented which is not only useful insofar\n  as each method has slightly different applicability, but also suggests\n  something deeper is at play with entropy minimization.\n- The presentation of the methods is clear and easy to follow.\n### Weaknesses\n- (minor) The experiments and analysis do not give a wholly clear picture of\n  when EM-based methods will work versus when they won't.  These details could\n  help indicate whether the proposed methods will be game-changers versus\n  a niche way to squeeze out performance in particular scenarios.\n\nQUESTIONS:\nThe paper is clear enough such that I do not have any major questions.\n\n- How does this relate to the general principle the neural networks are\n  over-confident and calibrating them usually involve strategically decreasing\n  their confidence?\n  - How does relate to hallucinations and over-confidence? I suppose reasoning\n    tasks are maybe less hallucination-prone compared something like\n    question-answering which it seems like EM-based methods are less likely to\n    improve.\n- Does this suggest that pre-training itself could benefit from incorporating\n  some sort of entropy-maximization, or does EM only serve to benefit the model\n  after pre-training has finished?\n- For EM-INF, by \"optimizing the logits\" do you mean optimizing the layer that\n  produces the logits?  By my understanding, the logits themselves are just\n  values, and adjusting the values directly does not make sense (as opposed to\n  adjusting, say weights/parameters).\n\n### Minor points\n- ? [Tab. 2] Does 60k unlabeled prompts mean the prompts from the particular task just without the provided answers?\n- It seems like self-consistency could still be applied to long-output tasks if you use a fuzzy matching function instead of just an indicator (e.g., which output has the lowest average Levenshtein distance to all other outputs?).\n- [Line 198] By \"1e-6\", do you mean \"1 * 10^-6\" as opposed to \"1 * exp(-6)\"?\n- \"chain of thoughts\" -> \"chains of thought\"\n- \"lost in the sauce\" is a bit too informal.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores entropy minimization in the context of LLMs, particularly, as a way to improve performance without additional labeled data. The authors propose three methods to do so: minimizing token-level entropy directly, using entropy as a reward in reinforcement learning, and optimizing the logits to reduce entropy at inference time. Their experiments show that entropy minimization improves performance both on math and coding tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n* I think this is an overall strong paper. It makes connections between an extensively studied ML method and large language models, which is interesting in itself, and additionally demonstrates promising results. \n* The empirical evaluation is extensive; it is done on a diverse set of datasets and across several LLMs. \n* Entropy minimization turns out to be useful both for math and coding tasks, which I think is an exciting finding given that it is an unsupervised method. \n* The paper also discusses limitations and settings where the method does not work, namely, for tasks where confidence is not a good indicator of performance and for less capable base models. \n* There is appropriate discussion of related work. \n\n\n**Weaknesses:**\n1. The method assumes that we take random samples from the LLM during decoding (I take it that entropy minimization won’t make much of a difference if you do greedy decoding), so naturally there is variance in the estimates. Yet, the authors only report point estimates in their experiment, i.e., there are no error bars. It would be great if that could be added to have more confidence in the results and ensure reproducibility.\n2. Again, regarding variance: Trajectory-level entropy estimator must have very high variance, and N=4 rollouts therefore seems very small. I’m surprised that it works with such a small number of rollouts, but it would nevertheless be interesting to see an analysis on how this impacts performance, and to understand if there is a trade-off between performance and runtime.\n3. Tab 3 shows that adaptive temperature, which is a simpler method, often outperforms the proposed inference method based on entropy. I think that should be conveyed more clearly in the text in order not to mislead. \n4. It's a bit odd that Prop. 1 is only given in an informal manner. I don't think it would be too complicated to write out formally what it is you prove.\n\nQUESTIONS:\nWhat decoding method do you when sampling responses from the base LM? \n\nl162-4: this sentence is unclear to me. What does “lost in the sauce” mean?\n\nIs the KL regularization term used for both EM-RL-token and EM-RL-sequence or just the former? If only the former, which it seems like from 3.2, why?\n\nLIMITATIONS:\nAppropriately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper demonstrates that Entropy Minimization can effectively improve the performance of large language models on complex math, physics, and coding tasks—even without using any labeled data.

STRENGTHS AND WEAKNESSES:
**Strengths**:

1. The paper is well-structured, and the core idea is both compelling and original, offering a promising new direction for unlocking the potential of language models in post-training.

2. The greatest strength of Entropy Minimization lies in its independence from large amounts of high-quality labeled data, eliminating the need to collect and curate such annotations. This makes it a more flexible and efficient approach to post-training, with significant potential for broader application.


**Weaknesses**:

1. Why do these improvements appear only on the Qwen model? Could the authors provide more insights by analyzing the internal parameter characteristics or pretraining data of Qwen? Why does the same approach not work as well on LLaMA?

2. Beyond math and coding tasks, have the authors evaluated performance on other domains, such as memory-intensive tasks?

3. Regarding the KL regularization coefficient β, how does it affect training outcomes? It would be helpful to see additional training results and analysis related to its impact.

4. I would like to see more CoT examples generated after Entropy Minimization training. How do these differ in style and structure from those produced by RL training with GRPO? And will EM fail to activate the model's self-verification capabilities as effectively as GRPO does?

If these concerns are properly addressed, I would be willing to reconsider my evaluation.

QUESTIONS:
1. What is j in Formula 5 (Lines 265 -266)

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03