Title: Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Reasoning

Reinforcement Learning

Unsupervised Learning
Abstract: Existing methods to enhance the reasoning capability of large language models predominantly rely on supervised fine-tuning (SFT) followed by reinforcement learning (RL) on reasoning-specific data. These approaches critically depend on external supervisions--such as labeled reasoning traces, verified golden answers, or pre-trained reward models. In this work, we propose Entropy Minimized Policy Optimization (EMPO), which makes an early attempt at fully unsupervised LLM reasoning incentivization. By minimizing the semantic entropy of LLMs on unlabeled questions, EMPO achieves competitive performance compared to supervised counterparts. Specifically, without any supervised signals, EMPO boosts the accuracy of Qwen2.5-Math-7B Base from 33.7\% to 51.6\% on math benchmarks and improves the accuracy of Qwen2.5-7B Base from 32.1\% to 50.1\% on MMLU-Pro. Primary analysis are also provided to interpret the effectiveness of EMPO. Code is available at https://github.com/QingyangZhang/EMPO.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nReasoning models are growing in prominence. However, training LLMs for reasoning often requires substantial supervised data. It remains unclear how to elicit reasoning in LLMs in an unsupervised manner. Here, the authors develop a new method (EMPO) to encourage models towards better reasoning in an unsupervised manner.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper was strongly motivated. The authors do a good job of motivating that they take on cases without a clear \"ground truth\" answer -- many real problems/queries are like that, and we want our reasoning systems to be robust to handling them! To my understanding, EPMO is an original method (though I'm not too familiar with methods for incentivizing reasoning so may have missed something). The overall idea is clever, intuitive, and well-explained: all strengths to the paper.  \n\nI also thought the authors had a good discussion on the relative role of pre-training vs. reasoning elicitation.\n\nWith that said, I found the experimental results quite challenging to work through. The large tables were not easy to interpret. I appreciated that the authors included the amount of supervising information provided (that was helpful for assessing the methods), but it would have been helpful for more discussion and work on the presentation of results. (more noted in the Questions below).\n\nIf the experiment section were conveyed more clearly, I think this could be a very strong paper.\n\nQUESTIONS:\nThe authors primarily report accuracy. Is this averaged over multiple seeds? It would be helpful to see error bars, if possible. \n\nWhat are the error bars in Figure 3 showing? \n\nThe authors filter out by length: what happens if you then test the resulting EMPO-guided model on long prompts? We would want reasoning models to handle difficult tasks! I may have misinterpreted some of the prompt filtering.\n\nLIMITATIONS:\nUnfortunately, the authors did not include an explicit limitations section nor a negative societal impacts section. A limitations section is definitely needed. The authors are good about discussing the relative merits of their approach but could be more upfront on the current weaknesses of the method and experimental paradigm for testing thus far.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

comments: SUMMARY:
This paper introduces EMPO: a variant of GRPO rewarding models for minimizing response "semantic entropy". This is done first by clustering a set of responses $\{s_1,...,s_K\}$ into semantic clusters $\{c_1,...,c_C\}$ and then giving reward $r_i$ for $s_i$ as $p(c_{s_i})$ i.e. the probablity of the semantic cluster $c_{s_i}$. Experiments are conducted on math and free-form reasoning tasks.

STRENGTHS AND WEAKNESSES:
**Strengths:**

- The paper proposes a smart extension of entropy minimization ideas to LLM reasoning
- The paper is mostly clear and easy to follow
- The paper evaluates a good range of model sizes (3B, 7B, 14B) over multiple reasoning datasets

**Weaknesses:**

- The implementation details for how semantic entropy is measured are not present in the main body of the paper. For math problems, only the final answer is used to do clustering, resulting in an objective very similar to self-consistency rewards. Clustering for free-form problems is done using an open-source instruction model (which is fine but should be made clear much earlier).
- Because the final EMPO objective is so similar to self-consistency rewarding approaches, it would be nice to see a self-consistency rewarding baseline to compare to.

QUESTIONS:
- Have you considered alternatives to final answer equivalence classes for reasoning? I think it would be very interesting to investigate the effect of different equivalence class schemes in more detail (this was briefly discussed in the final appendix section).

LIMITATIONS:
The authors have **not** included a limitations and broader impacts section.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Entropy Minimized Policy Optimization, a novel reinforcement learning algorithm aimed at enhancing the reasoning capabilities of large language models without any supervision. Existing methods rely on supervised fine-tuning or external reward signals; EMPO instead minimizes semantic entropy over multiple generations from unlabeled prompts. The authors cluster semantically equivalent outputs and assign higher rewards to more confident clusters, training the model to favor consistent reasoning traces. Empirical results show that EMPO significantly improves performance on both mathematical reasoning (e.g., MATH, Minerva) and free-form reasoning benchmarks (e.g., MMLU-Pro, GPQA), often matching or surpassing supervised approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is well-structured with clear motivation.\n- The paper conduct extensive experiments on both mathematical reasoning and free-form reasoning benchmarks.\n- The Section 5 discussion of the role of unsupervised learning in LLM reasoning is promising.\n\n**Weaknesses**\n- The authors acknowledge the risk of reward hacking under unsupervised objectives and propose a dual entropy thresholding strategy to filter out prompts that are either too uncertain or already overconfident. While this is a reasonable design, the paper does not present any ablation study or sensitivity analysis to empirically justify this choice. In particular, there is no evidence on how different threshold values affect training stability or final model performance. Given the critical role this filtering plays in avoiding trivial solutions and stabilizing training, a deeper empirical exploration would be valuable.\n- Given the rapid pace of development in this area, a number of concurrent works have emerged that also explore label-free RL for language models. While it is understood that these works are likely developed in parallel, a brief discussion comparing EMPO to these methods could strengthen the paper. Adding a discussion section, perhaps in the appendix, highlighting the distinctions in methodology, assumptions, or application scope would help readers better understand EMPO’s specific contributions in the broader context of this growing field.\n- Since EMPO relies entirely on unlabeled data, the quality and diversity of the training prompts may play a significant role in its effectiveness. It would be helpful to understand how the method performs under varying amounts of data or different prompt selection strategies. For example, does performance scale linearly with more data, or is it sensitive to prompt difficulty or domain distribution? Including a brief analysis or reflection on this point would enhance understanding of the method’s robustness and practical applicability.\n\nQUESTIONS:\n- The paper introduces dual thresholds to exclude examples with overly low or high entropy during training. Could the authors explain how these thresholds were selected in practice?\n- In the natural reasoning tasks, the authors utilize a verifier model to determine whether two outputs are semantically equivalent, using a prompt such as \"Please verify if the student’s answer is equivalent to the ground truth answer.\" This raises some questions about the underlying assumptions of the task setup. Specifically, are the natural reasoning prompts used in this work associated with deterministic, canonical answers that can reasonably serve as “ground truth” for such verification? If so, this setup is more straightforward. However, in more open-ended tasks, such as translation, summarization, or rewriting, there may not exist a single correct answer, and the notion of equivalence becomes less well-defined. In such cases, relying on a single model’s binary judgment of equivalence may introduce noise or bias into the clustering process. It would be helpful if the authors could clarify the scope of applicability of their current semantic clustering approach and discuss whether any adaptations would be needed for tasks with inherently diverse valid outputs.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a new unsupervised learning paradigm for RL training of LLM reasoning and proposes Entropy Minimized Policy Optimization (EMPO) to elicit LLM reasoning performance via unsupervised learning without the need for ground truth labels. They show that when trained on mathematical reasoning and free-form natural reasoning tasks, RMPO achieves the same or even better performance than RL methods like GRPO and it does not need a manually labeled reward or even generated pseudo-reward from LLM (for example, by majority vote). \n\nOn a high level, I think this new paradigm is interesting and worth investigating, and has potential for future research. Although I think this research is in the initial stage of unsupervised training for LLM on reasoning tasks, I lean toward accepting this submission because of its novelty and the soundness of the experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. I think the new paradigm this paper proposes about unsupervised training for LLM reasoning is novel and has huge potential, especially when we need a huge amount of data, and labeling these questions is hard, if not impossible. This is also suitable for the reasoning tasks that do not have ground-truth labels, such as some general open-form tasks.\n\n2. The experiment results look good, and the performance of EMPO can match or even outperform the RL methods like GRPO, or other methods like SFT. I think this is surprising to me.\n\n3. The discussion part is pretty clear, I think. When I read sections 1 to 4, I was thinking: does the pass@k rate improve when k is large? And does the total solvable set increase compared to the base model and supervised-RL-trained models? And they answer my questions in section 5, as they know what I want to read.\n\nWeaknesses and Questions:\n\n1. About the formulation 7: They have an entropy filtering applied to the questions that are trained in EMPO. I think their explanation is good, but I have two questions. (a). How many prompts do they filter from this entropy filter? (b). Could you show some experimental results about what happens if we remove the \\delta_low or \\delta_high (I think we will be more interested in the results when removing \\delta_high as it is predictable that when we remove \\delta_low, the peak performance will not change too much but the training will become slower).\n\n2. About the experiment details: It seems to me that you do SFT and GRPO on a different training dataset. (not sure whether I understand it correctly. Please correct me if I am wrong) Why is that? Moreover, you mentioned you filter a part of the data based on the response length of Llama3.3-70B-Instruct? Did you do the same step for SFT and GRPO? I think you should make a clearer comparison of the difference between the EMPO and GRPO implementations.\n\n3. I think the experiments do not show the most advantageous case of EMPO. I think the best case will be: there is a huge dataset with questions only, and labeling them is impossible, or the labels are messy. Will EMPO achieve a good performance on such a dataset?\n\n4. Since EMPO does not rely on the ground-truth labels, there are no such signals. Then, it will be even more important to select a training dataset with moderate difficulty. How will EMPO perform when the dataset is very very hard? For example, if most answers generated are incorrect (but the incorrect answers are diverse so it still passes the entropy filtering), how will the model perform?\n\nQUESTIONS:\nSee the strengths and weaknesses section.\n\nLIMITATIONS:\nSee the strengths and weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}