Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization
Authors: Qingyang Zhang
Haitao Wu
Changqing Zhang
Peilin Zhao
Yatao Bian
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM Reasoning
Reinforcement Learning
Unsupervised Learning
Primary area: deep_learning
TL;DR: We makes an early attempt at fully unsupervised LLM reasoning incentivization.

Abstract: Existing methods to enhance the reasoning capability of large language models predominantly rely on supervised fine-tuning (SFT) followed by reinforcement learning (RL) on reasoning-specific data. These approaches critically depend on external supervisions--such as labeled reasoning traces, verified golden answers, or pre-trained reward models. In this work, we propose Entropy Minimized Policy Optimization (EMPO), which makes an early attempt at fully unsupervised LLM reasoning incentivization. By minimizing the semantic entropy of LLMs on unlabeled questions, EMPO achieves competitive performance compared to supervised counterparts. Specifically, without any supervised signals, EMPO boosts the accuracy of Qwen2.5-Math-7B Base from 33.7\% to 51.6\% on math benchmarks and improves the accuracy of Qwen2.5-7B Base from 32.1\% to 50.1\% on MMLU-Pro. Primary analysis are also provided to interpret the effectiveness of EMPO. Code is available at https://github.com/QingyangZhang/EMPO.

Right Question is Already Half the Answer:
Fully Unsupervised LLM Reasoning Incentivization
Qingyang Zhang Tianjin University
Haitao Wu Tianjin University
Changqing Zhang Tiajjin University
Peilin Zhao
Shanghai Jiao Tong University & Tencent AI Lab
Yatao Bian
National University of Singapore & Tencent Al Lab
Abstract
Existing methods to enhance the reasoning capability of large language models predominantly rely on supervised fine-tuning (SFT) followed by reinforcement learning (RL) crasoning-speciinn data. These approaches critically depend on external supervisions-sur as labeled reasoning traces, verified golden answers, pre-trained reward models. In this work, propose Entropy Minimized Policy Optimization (EMPO), which makes an early attempt at fully unsupervised LLM reasoning incentivization. By minimizing the semantic entropy of LLMs on unlabeled questions, EMPO achieves competitive performance compared to supervised counterparts. Specifically, without any external supervision, EMPO boosts the accuracy of Qwen2.55Math--7 Base from 33.7% to 51.6% on math benchmarks and improves the accuracy of Qwen2.5-7B Base from 32.1% to 50.1% on MMLU-Pro. Primary analysis are also provided to interpret the effectiveness of EMPO. Code is publicly available at https://wuabbboooommmmggmgm/mmzz
Introduction
Large language models (LLMs) have demon- strated exceptional potential in challenging tasks and code generation [4] A prevailing paradigm for train- ing reasoning LLMs involves firstly performing supervised fine-tuning (SFT) and then reinforce- ment learning (RL., or iterative combinations both, applied to reasoning-spedt
+13.2
such as mathematical reasoning
Bac
+25.0
PMIV
+18.0
Accorncc
+25.3
+10.6
+12.9
+13.3
20-
datasets after pretraining (5). Unfortunately, these methods typ- ically depend on large-scale reasoning datasets with various forms of supervised information, such human-labeled reasoning traces, verified golden answers, or an additional pre-trained re- ward model. As consequence, endowing LLMs with powerful reasoning capability through human
MATH
Minerva
Otompial
ATME2
AMC
MMLU Pro
GPCA
Figure 1: Improvement of the proposed method on Qwen2.5-7B and Qwcn2.5-7B-MM model.
"Work done during internship Tencent Al Lab.
I "oorespondence to Changqing Zhang <zhangchangging @tju.edu.cn> Yatao Bian is the project lead.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

experts is becoming increasingly time-consuue consuminn and costly, which greatly limits the scalability and broader adoption reasoning models.
To mitigate this, previous work employs self-consistency to construct pseudo data and deploy supervised finetuning for better performance [6] However, the performance improvement limited and under risks of model collapse [7]. Recent advancements, such as the pioneering work PFPO [8] frame the labeling of solutions as evaluation against test cases and then leverage self-consistency to generate pseudo test cases. Despite the promising results, the proposed method still necessitates supervision from instruction finetuning data and supervision signals from the frontier LLMs to initialize the RL process. Another more recent work [9] introduces two-stage framework to construct self-rewarding reasoning models using self generated data followed by RL. Despite the superior performance, the proposed method relies on ground truth verifier to obtain self-correction reasoning traces by rejection sampling. These approaches inspire our exploration of critical open question: How can we incentivize LLM reasoning capacities in fully unsupervised manner?
LLM Outputs
Rewards
Advantages
Reference Model
O₁
Normalize GAF
Policy Model
Reward Model or Pre-defined Rules
GRPO
0g
Value Model
AG
EMPO
(a) Comparison of different RL methods
LLM Outputs
Clusters
Rewards
Advantages
The capital of France is Paris.
P(C,lo)
A
q: Where the capital of Frunce?
The capital of F France is Paris.
C
P(G]44
A2
It is Paris.
P(G.g)
Normalize
Ag
Pollcy Model
Londen
Te-1 P(C,le)
Ag-1
CM
London France's capital.
P(Cwle)
(b) Overview of EMPO
Figure Overview of the proposed method. (a) Previous method like PPO 10 or GRPO typically relies external supervised signals, e.g., pretrained reward model golden answers. (b) The proposed Entropy Minimized Policy Optimization (EMPO) samples a of responses from the current policy model, and then builds semantic clusters according to their equivalence. By continuously minimizing the entropy at meaning level, our method achieves competitive benchmark performance without any external supervision, i.e., rule-based reward, pre-defined test cases or pre-trained reward model.
Recent advanced DeepSeek-RI -Zeo demonstrates robust reasoning capabilities without depen- dency SFT data. By directly initiating RL from the base model, DeepSeek-R 1-Zero autonomously evolves sophisticated reasoning behaviors such as reflection and self-critic by exploring the reward signals provided by rule-based rewards. i.e., verified golden answers or an additional pre-trained reward model. Inspired by the success of DeepSeck-R1-22 pervised approach for powerful reasoning capability. Specifically, we propose novel reinforcement learning algorithm termed as Entropy Minimized Policy Optimization (EMPO), which incentivizes the reasoning capability of LLMs fully unsupervised manner by minimizing their predictive entropy in latent semantic space. This method optimizes the model to favor reasoning traces yielding consistent answers, enhancing output reliability. The semantic entropy objective we propose to mini- mize is a well-established measurement of L.MSs' uncertainty, which extends beyond mathematical reasoning to free-form cesstioonanssering tasks. We further introduce entropy thresholding to filter unreliable reasoning traces, stabilizing the unsupervised training process. Experiments various
our motivation is to devise fully unsu-

---

tasks including mathematical reasoning and free-form natural reasoning are conducted to validate the proposed method. Our contributions are summarized follows:
We propose an effective and principled strategy called Entropy-Minmmized Policy Optimiza- tion (EMPO) for incentivizing the reasoning capabilities of LLMs in a fully unsupervised manner.
We establish semantic entropy as a potent intrinsic reward signal for guiding LLM reasoning. Our analysis confirms strong negative correlation between semantic entropy and model accuracy, validating its efficacy as a robust, unsupervised optimization objective that drives models towards generating more consistent and reliable outputs.
Experiments on both math reasoning tasks with deterministic golden answers and free- form natural reasoning tasks are conducted to validate the efficacy and versatility of EMPO. Additionally, we provide critical insights into EMPO's mechanism, demonstrating that its effectiveness stems from enhanced ability to consistently select and prioritize strong, pre- existing reasoning pathways learned during pre-training, rather than instilling fundamentally reasoning skills. This underscores EMPO's strength in efficiently eliciting and refining latent capabilities within base models.
2 Related Work
Self-Supervised and Semi-Supervised Reasoning. To address the dependency on labeled data, several self-supervised and unsupervised methods have emerged. Huang et al. [6] propose a self- improvement framework where LLMs generate high-confidence answers using Chain-of-Thought (CoT) prompting and self-consistency, subsequently fine-tuning on these pseudo-labels. However, the performance gains are often limited, and there is a risk of model collapse. noted in 77). Recently, Patel et al. [3 apply self-improvement to web navigation tasks in WebArena, fine-tuning on synthetic data generated by the model itself. Li et al. [4] enhance long-context reasoning via SeaLong, sampling multiple outputs and optimizing with Minimum Bayes Risk. These methods, while reducing reliance on external labels, still involve supervised fine-tuning steps, contrasting with EMPO's 's fully unsupervised RL approach. A concureent work, i.c., test-time reinforcement learning (TTRL) [5]] directly obtains pseudo label by major voting and then conducts RI on test prompts at inference time, whereas our EMPO strictly maintains the separation between training and testing phases for ensuring that the model remains unexposed to any test prompts during training. Furthermore, while TTRL is currently limited to mathematical tasks, our approach is applicable to more general frre-form reasoning tasks.
Solf-Rewarding and RL-based Reasoning. RL has become a prominent technique for enhancing LLM reasoning, often leveraging external self-generated rewards. Yuan et al. [66 propose using the LLM itself via Luddde prompting to provide rewards during training, reducing reliance on human feedback. Similarly, recent works [ [7] incorporate the training of model's 's self- verification capability into RLVR process, which improve the reasoning and verification capabilities at the same time. PFPO [ frame solution labeling as evaluation against test cases, yet still rely on instruction fine-tuning and frontier L.M signals for RI initialization. ETPO [8] augments RI. with entropy bonus to promote exploration, differing from EMPO's entropy minimization focus. DeepSeek-R11 [2 demonstrate robust reasoning via RL from a base model, using rule-based rewards. Seed-GRPO [19 leverages semantic entropy for reward shaping, which enables dynamic adjustment of policy update magnitudes based on question uncertainty. These methods highlight spectrum of supervision levels, positioning EMPO as unique in its fully unsupervised nature, leveraging semantic entropy as an internal reward.
Entropy Minimization and Semantic Consistency. Entropy minimization is well-established technique in supervisd and unsupervised learning, with roots in traditional machine learning. Many previous work (20 21 22] has demonstrated that minimizing entropy on unlabeled data can improve classification accuracy by encouraging model confidence. Test-time adaptation methods like Tent adapt models to new domains by minimizing entropy on test data, filling domain gaps without additional labels. These approaches highlight the potential of entropy minimization as an unsupervised objective, which EMPO leverages for LLM reasoning by extending it to semantic entropy
3

---

23 in latent space. More recent work [24] compute the Semantic Entropy to quantify the diversity of the modelaAZs high-level strategic plans, reinforcing its relevance.
3 Method
We propose an RL-based method to minimize the entropy of LLM generations in a latent semantic space for incentivizing its reasoning capability. We term our method Entropy-Minimizee Policy Optimization (EMPO), which is devised in fully unsupervised manner without any forms of external supervised information.
3.1 Preliminaries
Recent advancements in reinforcement learning have demonstrated remarkable breakthroughs in enhancing the reasoning capabilities of LLMs. Taking the representative RL technique Group Relative Policy Optimization (GRPO) [[] used by DeepSeek-R1 RI-rrro [12] as an example. GRPO first samples group of outputs (01,-.. ,0G) from the policy model ma and then optimizes it by maximizing the following objective:
JCRPO = Big-P(OONdohaarreeoo
1
( (L(Tollare))
G
where 3 a parrame- which avoids the policy model diverge too far away from the reference model Tee.. clips extreme advantages for stability. G is the number of samples in one group. A, is the advantage computed by normalizing the rewards within each group as A, 2' r==meon (fr ral) math reasoning task, the reward can computed by predefined rules:
std(r)," rc)
(1 if verifier(o,, True
F 0
otherwise
where a verifier is used to determine the correctness of o, by comparing it with the golden answer a.
Unlike the above example, we consider fully unsupervised optimization settings where there are no golden answers verify the correctness of model predictions. this circumstance, we only have unlabeled reasoning problems P(Q). Such problems were freely raised by users during the deployment of LLMs. Given pre-training LLM ma parameterized by 0, our goal to enhance its reasoning ability by only utilizing the unlabeled user problems {qi ) which requests minimized cost of data collection.
3.2 Semantic Entropy Minimization Objective
Entropy is a classical unsupervised objective in the traditional semi-supervised and unsupervised learning fields 20 2.5. Previous works in computer vision show that by continuously minimizing the entropy on unlabeled samples after pre-training the classification accuracy of machine learning models can be significantly improved to fill the domain gaps [2]] (22]. The basic intuition behind entropy minimization is that a robust model should not only fit labeled data well but also make confident and consistent predictions unlabeled data. This principle encourages the model to avoid ambiguity and make decisive predictions, thereby enhances generalization. In this work, we choose semantic entropy (23) our unsupervised optimization objective, which is natural extension of classical Shannon entropy specified for large language models. Intuitively speaking, minimizing semantic entropy encourages the LLMs' outputs to be more consistent in semantic level rather than format, and thus the final answers are expected to more reliable.
Specifically, semantic entropy first samples group of outputs (oI.-.., } and then clusters the output sequences according to their meaning. That is, if two outputs share the same meaning (i.e., they are bidirectionally entailed), they should be merged into one same cluster in the semantic space. This can be done without notable computational cost by predefined rules such as N-gram, regular expressions an additional small language model. Once built such set of meaning clusters {c) in

---

semantic space, we then approximate the probability over the meanings as the proportion of sampled answers as
p(cllr) lo//GG
where Cj E {c} is the j-th meaning cluster. lcyl denotes the numbers of outputs that belong to cj- Finally, given question the semantic entropy (denoted as H) over the model's 's output meanings distribution can be estimated follows
Σ p(csl)) logplell)))) cp((c)
As proven by previous work, semantic entropy has strong negative relationship with model accuracy. which can be used as an efficient measurement to detect unreliable LLM generations such as confabulation and hallucination [23 26 27 Motivated by this, propose leverage semantic entropy as an unsupervised optimization objective for incentivizing the reasoning capability of LLM.
3.3 Entropy-Minimized Policy Optimization
We propose Entropy- -Miripy-iiiimzzed Policy Optimization (EMPO), an RL-based method that optimizes the pre-trained large language model Tyy to favor low semantic entropy responses given unlabeled user questions (g: "_1- Given input questions, EMPO incentivizes the outputs that belong to higher probability meaning cluster, and thus minimizes the semantic entropy over the meaning distribution. Specifically, given question
our EMPO first samples group of output (01,..., oG) from the current model ma and then merges them into a set M meaning clusters {cI.... ). As we mentioned before. this can be done without notable computational cost (please refer to the quantitative results Appendix [T) by predefined rules such as N-gram, regular expressions or an additional small language model (SLM)") Once built such meaning set, EMPO approximately minimizes the semantic entropy by maximizing the following objective
Σ (A.), A, r: mean(1r1, rra)) G
1
JEMPO
(5)
std(ri, rG)
where A, is the advantage of output o, calculated by normalizing the rewards. Unlike GRPO in which the rewards is calculated depending on external supervision such as pre-defined rules or an reward model, EMPO, the reward assigned for the i-th outputs o is the likelihood of its meaning cluster, i.e.
r: = p(cylg), where I(o,) = Cj-
where the meaning likelihood p(c; (4) is approximated by Eq. Intuitively, the outputs convey higher-probability meanings are of higher advantages, and are therefore incentivized through training.
How to Mitigate Potential Reward Hacking? Note that different from verifiable rule-based reward, which inherently resists reward hacking risks, optimizing unsupervised entropy objectives may permit trivial solutions. For instance, models could exploit the reward signal by overfitting to high-confident but wrong predictions for the most frequent semantic clusters without carefully reasoning process. To address this, we implement a straightforward entropy thresholding strategy, restricting optimization prompts exhibiting moderate uncertainty via dual threshold criteria. Specifically, two entropy thresholdings are deployed to filter out user queries q that result in overly high or low entropy unreliable answers. Extremely high entropy indicates that the model highly uncertain, and thus its predictions are prone to be unreliable. In addition, continuously optimizing on responses with already low entropy is redundant and at the risk of overconfidence (28). The final optimization objective of EMPO is
JEMPO
[G|
cli(1,, +
S.L.
δtom
H < &righ
"Such a SLM does not provide explicit or direct supervision signals regarding correctness or quality of reasoning for a given query. The "unsupervised" nature of EMPO refers to its independence from labeled (query, correct-answer) pairs (query. valid-reasoning trajeccc)) pairs for learning the reasoning task itself. More discussions are in Appendix
5

---

where H is the semantic entropy defined in Eq. The questions results in highly unreliable answers with entropy greater than dhigh are filtered out. Besides, we also filter out low-entropy answers to maintain the diversity of model outputs and further avoid potential reward hacking. Following previous work [29], we remove the KI. constraint for better performance. 6 clips extremely high or low advantages for stability similar to common practice.
4 Experiments
4.1
Experimental Settings
We conduct experiments on multiple datasets including both closed-form math reasoning tasks and free-form natural reasoning tasks. Our EMPO shows competitive performance by purely RL in a fully unsupervised manner compared to supervised finetuning and RL methods.
Prompt Collection and Data Engineering. For mathematical reasoning, following the common practice 8 we adopt 20,000 prompts randomly selected from NuminaMath-Ca dataset [32] for training without additional data engineering. For free-form natural reasoning tasks, we adopt the prompts from Natural Reasoning a large-scale dataset consisting of diverse reasoning questions from multiple domains (e.g., Physics, Computer Science, Economics, Social Sciences and more). For training efficiency, we filter out the questions with over-long prompt or reference answer. Besides, taking inspiration from [33 351. we the response length of Llama3.3-70B I-ntut-nttt difficulty estimation metric, and filter out samples with response lengths exceeding 4096 tokens. The remaining samples are simpler for stabilizing the training process. The final training subset is consisted of 18,000 questions. More details can be found in Appendix G
Evaluation. For mathematical reasoning, the performance is evaluated diverse suite of benchmarks including Minerva Math, MATH, AMC23, OlympaidBench and AIME24. o For free- form natural reasoning. we evaluate MMLU-Pr [6] and GPQA [37] benchmarks, which consist of challenging reasoning-focused problems across various subjects, e.g., biology, business, chemistry, computer science and within "Aboxed() )" and report the multi-choice accuracy. Without specific clarification, all evaluations conducted using zero-shot prompting and greedy-decoding.
We prompt the model to reason step by step and output the final answer
Model training. For mathematical reasoning tasks, we train Qwen2.5-Math- and 7B Base models with our EMPO. The baselines we consider include supervised finetuning (SFT), online direct preference optimization (ODPO) [11 and the representative GRPO. We also compared with Qwen2.5- Math Instruction models for a more comprehensive comparison, where the instruction model is trained by iteratively supervised finetuning and RL on private data. For free- natural reasoning tasks, we initialize from Qwen2.5-3B, 7B and 14B Base models. Different from mathematical reasoning, it is difficult to adopt rule-based reward for free-form question-answent golden answers. We consider the corresponding Instruct model, the Base model with or without few-shot CoT prompt as baselines. Besides, we also compare with SFT where the Base model is tuned fit the response of Maama3.--70B-Inttuue For more results other model families beyond the Qwen2.5 series (e.g., Llama3), please refer to the Appendix D
tasks without deterministic
SFT: We train models by supervised finetuning via Open-Instruct 38 with a fixed learning rate of 1 x 10-6, global batch size of 128 and train for 1 epoch with a max length of 2048.
GRPO: We implement GRPO via verl [39]. We sample 16 and 12 responses for each prompt for mathematical and natural reasoning tasks respectively. For Qwen2.5-Math model series, we train the model for 300 steps with maximum generation length of 3096. For Thtokhinker model series, we train the model for 100 steps with a maximum generation length of 16K. We adopt train prompt batch size of 256 and mini-batch size of 32. Specifically, we adopt clipping by importance ratio based on sequence likelihood proposed by GSPO [40] for stable off-policy RL training. Following [41), we only use the rule-based accuracy reward and do not adopt format-reward. The accuracy reward is implemented follows: If the response contains the correct final answer within "boxed())" it receives reward of 1.0. If the model prediction is wrong, it receives a reward of 0.0.
bhtpps//hugginggfcccco///o http://nngnccccooo//aassst///// Hwwoo//mmiiiaiaaaaaaaaaaaaaaaaaa hatproowomiiaapopppppepeeee wwwptprropppopeeeee http:://wwgggggiieccooooaaeeeees reasoning

---

Online-DPO Recent advanced Online-DPP first samples set of responses and then verifies and selects the responses with highest reward and lowest reward as a preference pair. We directly copy the results from ΣI]. where the model is trained for 7 iterations. Each iteration involves training epochs and 20K training samples, i.e., 140K training samples in total.
EMPO: Most hyper-parameters of method, e.g., number of generations, max generation length, batch size, learning rate are the same with GRPO. In mathematical reasoning tasks, we use set of regular expressions merge the outputs into meaning clusters. For more general free-form natural reasoning, we leverage General-Verifiee (a compact small language model with 1.5B parameters) to determine whether two outputs of the same meaning or not following [23 26 A concrete example can be found in Appendix C Specifically, if the final predictions (i.e., the contents within "Aboxed ("") of two model outputs are bidirectionally implicating, then we merge them into one semantic cluster ignoring their reasoning traces. More details are in Appendix
4.2 Main Results
4.2.1
Performance on Mathematical Reasoning Tasks.
We conduct experiments mathematical tasks to evaluate our method. The main results are shown in Table EMPO has successfully incentivized the Qwen2.5-Math Base model with reasoning capability without dependency on any external supervision. We observe a substantial improvement in the average performance on commonly used mathematical reasoning benchmarks from 33.0% 42.6% and 33.7% to 51.6% on Qwen2.5-M 1--Ma-hh.5555 and 7B models, respectively. Notably, through fully unsupervised RL training, the 1.5B and 7B model has both achieved competitive performance (42.6% and 51.6%) near to -att2.5--athh-nnss dataset multi-stage iteratively supervised fine-tuning and reinforcement learning.
(42.8% and 51.7%), where the latter depends on private
Table 1:
Accuracy mathematical reasoning benchmarks. We report the pass @ accuracy tested with greedy decoding. The results ODPO are directly copied from [3.. Here r, a denote the dependency questions, human-verified reasoning traces and golden answers respectively.
Minerva Math
Olympiad Bench
Supervision
MATH
AIME24
AMC23
Avg.
frontier model
Llama 3.1-70B Instruct
a a 1g
64.6
35.3
31.9
16.7
30.1
35.7
2arus-2-7B-PRIME
79.2
38.6
42.1
26.7
57.8
48.9
1.5B model
Twen2.55MMail
None
66.4
19.1
33.8
3.3
42.5
33.0
Qwen2.5- Mathh ---tMMMMnttuuct Instruct
rg
75.2
33.8
42.8
6.7
52.5
42.2
Qwen2.5- w/SFT
(g
61.8
26.1
27.1
3.3
37.5
31.2
Qwen2.5 w/Rand
(g.r,a)
65.0
26.1
10.0
55.0
Qwen2.5- w/GRPO
{q.a}
78.0
37.1
39.1
10.0
50.0
42.8
5wen2.5-Math w/EMPO
(4}
77.6
36.0
39.5
10.0
50.0
42.6
7B model
Qwen2.5- 5athhhhh
None
70.2
12.5
30.8
10.0
45.0
33.7
Qwen2.5- Mathh Instruct
rg
80.8
41.9
49.2
13.3
67.5
50.5
Qwen2.5- 5-Math w/SFT
a
72.2
34.6
33.2
10.0
45.0
39.0
Qwen2.5- w/Rand
(g.r,a)
73.0
26.5
37.0
26.7
52.5
43.1
Qwen2.5- Mathh w/ODPO
{g
76.8
30.9
37.9
26.7
62.5
47.0
Qwen2.5-Matte Math w/GRPO
{q.a)
82.4
45.2
47.6
23.3
60.0
51.7
Qwen2.5 Math w/EMPO
{a}
81.4
42.3
46.1
23.3
65.0
51.6
4.2.2 Performance on Natural Free-form Reasoning Tasks.
We present the results on free-form natural reasoning tasks in Table 2 On the MMLU-Pro benchmark, our EMPO improves the accuracy from 32.1% 1% to 50. 1 % and 32.7% to 58.8% on Qwen2.5-7B and 14B Base model respectively. Besides, on more challenging GPQA benchmark, EMPO results in increasing accuracy from 15.9% 28.8% on model, 30.6% 35.3% on 14B model. Notably, we observe that the SFT baseline fails to consistently improve model performance. We hypothesize that this is
'ttpss/ttugginggtac..c/TCCCCC Law.:/Gge-LLbbgggeeeeeleeeeeee

---

due to the noise in the reference responses within the Natural Reasoning training data (as mentioned by [3 This phenomenon further underscores the practical potential our proposed method.
Table Accurascy results on free-form natural reasoning benchmarks. We report pass @ accuracy tested with greedy decoding. Here {g. r, a} denote the dependency on questions, human-verfied reasoning traces and verifiable golden answers respectively.
MMLU Pro Social
GPQA
Supervision
STEM
Humanities
Other
Avg.
3B model
Qwen2.5-Bass
8.32
5.35
7.42
4.13
6.83
11.2
Qwen2.5-Base 5-shot
{4,,,)
34.7
26.2
47.9
35.9
35.3
13.8
InnstrctI
{q.7,
44.8
30.7
56.0
47.1
44.5
28.2
Qwen2.5 Base w/SFT
{g,r,a
19.8
10.4
28.0
18.4
19.1
11.5
Qwen2,5-Baa Base w/Rand
{g.a)
17.0
29.8
49.6
28.4
26.0
18.5
Qwen2.5 Base w/GRPO
{g.a)
32.2
27.7
49.8
38.7
35.2
17.1
Qwen2.5-Base w/EMPO
{q)
31.7
26.2
48.1
36.7
34.1
7B model
Qwen2.5-Baa
30.1
23.8
45.9
34.3
32.1
15.9
Qwen2.5 5-shot
(g
36.3
59.1
49.4
46.8
23.5
Qwen2.5 1rnn2..-Instrrce
{q.r,a
56.9
38.1
64.1
58.6
55.2
35.3
Qwen2.5-Baas 5-Base w/SFT
{9,7,a
32.6
7.1
15.8
30.1
25.6
22.4
Qwen2.5- Base w/Rand
{g
45.8
30.7
60.4
50.9
46.4
25.3
Qwen2.5- Beasee w/GRPO
{q.a}
57.1
36.2
64.4
56.6
54.5
33.8
Qwen2.5 Base w/EMPO
{a)
52.4
34.6
59.0
50.9
50.1
28.8
14B model
Qwen2.5-
30.8
28.0
44.4
33.0
32.7
30.6
Qwen2.5 Base 5-shot
51.9
35.8
63.4
54.4
51.4
33.2
Owen2.5-Instruct 5-Intruct
(q.7.a)
63.6
47.1
73.8
66.7
62.9
42.9
Qwen2.5 5-Bee w/SFT
(g.r,a)
37.0
27.8
40.2
38.0
36.1
28.5
Qwen2.5-Baa w/GRPO
{g.a)
62.9
42.1
68.6
59.8
59.6
35.6
Qwen2.5-Bard w/EMPO
{q)
41.6
68.3
60.0
58.8
35.3
4.2.3 Training Dynamics
We further conduct experiments to investigate the reliability of our unsupervised reward signals. As shown Figure the unsupervised reward signals of EMPO have strongly negative correlation with the true rewards based on golden answers. Thus, by continuously minimizing the semantic entropy objective, the model can boost accuracy in a fully unsupervised manner. Furthermore, our experiments, we observed that during the training process, the response length of Qwen2.5-Math gradually shortened before stabilizing within certain range. In contrast, inference length of Qwen2.5 continuously increased. However, this variation in length did not show clear correlation with performance improvement.
4.2.4 Additional Results on Broader Model Family beyond Qwen2.5
To further validate the effectiveness EMPO on broader model family beyond Swen2.5-Matte conduct additional experiments on starting from the Llama-3 model family. The results are shown in table We sample responses for each prompt and train the model for 100 steps with batch size of 256 and mini-batch size of 32. The maximum generation length 16K. EMPO successfully improve the average accuracy from 7.7% to 27.6%, which demonstrates its effectiveness various model family.
Base [42, which built upon mid-training
5 Discussion and Conclusion: The Role of Unsupervised Learning in Eliciting Reasoning Capabilities
The strong empirical performance EMPO, particularly ability as a fully unsupervised method to match or even slightly outperform supervised counterparts like GRPO (as observed with the 7B model), prompts deeper examination of how such reasoning incentivization mechanisms work. This is especially pertinent given the counterintuitive observation that these substantial improvements

---

Total Entropy
Somantic Probability Reward
Accuracy Reward
0.70
1.7
0.45
1.6
Probability
0.65
0.42
Aw
Entropy
Reward
1.5
0.60
0.39
1.4
0.55
0.36
1.3
0.50
0.33
0.30
20
40
60
80
20
40
60
80
0
20
40
60
80
Step
Step
Step
Figure 3: We visualize the training dynamics when tuning Qwen2.5-Mathh Base model with EMPO on 20K prompts randomly sampled from NuminaMath-Con The left illustrates the running average of semantic entropy (Eq. 4). The middle shows the trend of unsupervised reward defined by Eq. 6 The right shows the model accuracy on training data at each RL steps. Along the unsupervised RL-based training trajectory, EMPO establishes stable learning process with consistently decreased semantic entropy and improved accuracy.
Table 3: Additional results on broader model family beyond Qwen, i.e., OctoThinker-Long:
Minerva Math
Olympiad Bench
Supervision
MATH
AIME24
AMC23
Avg.
3B model
Octo Chinker Base
None
15.8
2.9
7.5
0.0
12.5
7.7
OctoThinker-Zer
{q.a] a
69.6
27.6
32.0
13.3
42.5
37.0
Octo w/GRPO
{g
65.0
23.5
27.3
6.7
32.5
31.0
Thtokhinker w/EMPO
{4)
17.3
23.6
27.6
Minerva Math
AMC23
1.0
Coverage (pass(ik)
0.8
Coverage (pass(@kk
0.9
0.6
0.8
0.5
0.7
0.4
Basc EMPO
0.6
0.3
EMPO
0.2
GRPO
0.5
GRPO
0.1
0.4
256
Number of Samples k
Number of Samples k
Figure 4: Pass @k curves of Owen2.5--Mah-77 Base model and its counterparts trained with GRPO and our EMPO on Minerva Math and OMNI reasoning benchmarks. Pass@k measures the probability that least of the top & generated solutions correct. Pass@) equivalent accuracy. as checks if the single solution is correct. When is small, RL-trained models outperform the original base model. However, as k increases (e.g., into the tens or hundreds), the performance of the base models often converges with, or even exceeds, that of the RI-trrinedd models.
benchmarks are achieved without consistent increase in response length clear evidence of an "Aha moment" - a hypothesized sudden emergence enhanced reasoning capabilities.
To dissect the nature of the improvements conferred by reinforcement learning (RL) post- traininiinnn we investigated its influence on pass@k k accuracy. This metric is crucial as recent studies 43 44 suggest that RL may not fundamentally expand the inherent reasoning capacities of LLMs beyond those already embedded their pre-trained base. As depicted in Figure 4 our findings align with this perspective. Both GRPO and EMPO significantly enhance pass @k scores for small to moderate values of (e.g., k=16 or 32) compared to base model. This demonstrates an improved efficiency in surfacing correct reasoning paths with fewer attempts. However, as k becomes substantially large,

---

the performance of these RL-trained models tends to converge with, and is sometimes surpassed by. that of the base model.
This convergence at high k values, coupled with our qualitative observations that the base mod- els themselves already exhibit sophisticated reasoning behaviors such pausing, self-correction, and backtracking (see Appendix for examples), strongly indicates that the foundational reasoning pathways are largely pre-existing. Consequently, RL post-training, whether supervised or unsuper- vised like EMPO, appears to primarily refine the model's ability to efficiently access, prioritize, and consistently select these latent reasoning patterns, rather than instilling fundamentally novel ones. The observed improvements in pass @1 @1 (accuracy) are thus likely a consequence of this enhanced sampling efficiency.
These empirical insights from the pass@k analysis lend considerable support to the emerging consensus that pre-training shoulders the primary burden of endowing L.MSs with their core abilities. We align our interpretation with prior insights from [45]: "Pretraining does all the hard work. One big bet is that the pretraining phase grants all the abilities to the base LM, and finetuning is simply like a style transfer which positions the model to the right output space. Under this conjecture (or more precisely, an emerging, but not yet unanimously accepted consensus 43 D, we attribute the efficacy of our method to the robust pretraining process of the Qwen2.5 Base model: If base model possesses strong inherent reasoning capabilities, the subsequent challenge not necessarily teach it new reasoning skills from scratch, but rather to effectively elicit and guide these existing skills.
EMPO's success highlights that intrinsic reward signals, derived purely from the model's 's objective to minimize semantic entropy and thus achieve greater consistency in its outputs, can be surprisingly potent for this elicitation process. In well-prrettriiee model, outputs that semantically consistent are more likely to align with correct and coherent reasoning. EMPO leverages this by incentivizing the model favor such consistent outputs, effectively guiding to refine its selection from its collection of existing reasoning strategies without requiring external validation of correctness.
In conclusion, while RL techniques, including EMPO, may not be forging entirely new fundamental reasoning capabilities beyond what pre-trate pre-training provides, their role in significantly enhancing the sampling efficiency and reliability of accessing these pre-trained abilities is of paramount practical importance. Optimizing models for such efficiency is crucial for real-world applications. EMPO, by achieving this through fully unsupervised framework, offers particularly scalable, cost-effective, and practical approach to unlocking and refining the vast reasoning potential embedded within pre-trained LLMs, especially in domains where curated supervisory data is scarce prohibitively expensive to obtain.
6 Limitations
This manuscript presents a new training algorithm that improves the reasoning capability by min- imizing semantic entropy. By encouraging semantically coherent outputs, the model's reasoning capability can be boosted without any external supervision. There exists several noteworthy limitation of EMPO. The unsupervised nature of EMPO removes the dependency of external supervision at the cost of potential reward hacking risks. The model may be highly overconfident on its prediction regardless correctness. Besides, similar to other RL for reasoning methods, EMPO may exhibit undesirable behaviors that stem from pretraining corpus of the Base model, e.g., hallucination, bias and so on. Entropy minimization may not only increase accuracy by encouraging consistency, but also make the bias more severe. Thus, more studies are necessitated to mitigate such issues for large reasoning models.
7 Acknowledgement
This work is supported by the National Natural Science Foundation of China (62376193 and 61925602). YB is supported by the National University of Singapore SoC (grant no: A-0010308-00 00). We thank Ma Huan for the insightful discussion regarding the role of RL incentivization the reasoning capabilities of models and writing, We also thank Zongbo Han (BUPT), Chenqian Gao (MBZUAD,, Qichao Wang (NTU) and Hengyu Liu (CUHK) for their helpful discussion about the project and comments on the manuscript.
10

---

References
[1] Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, al. A survey of reinforcement learning for large reasoning models, arXiv preprint ar arXiv:250900885 08827, 2025.
[2] Ganqu Cui, Lifan Yuan, Zefan Wang. Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456 2025.
[3] Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprini arXiv:2412.01981, 2024.
[4] Dejian Yang Daya Guo, Qihao Zhu. Deepseek-coder. When the large language model meets programming the rise of code intelligence, 2024.
[5] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et Qwen2. 5-math technical report: Toward mathematical expert model via self-improvemmnt, arXiv preprint arXiv:2409. /2/22, 2024.
[6] Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[7] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, (31(2)2)):75-
759, 2024.
[8] Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F Chen, Shafiq Joty, and Furu Wei. Preference optimization for reasoning with pseudo feedback. In International Conference on Machine Learning. PMLR, 2024.
(9] Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self- rewarding correction for mathematical reasoning, arXiv preprint arXiv:2502.19113 2025.
[10] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1.7,::7777003477 2017.
[11] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033: 00002400.03300, 2024.
[12] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arxiv preprint arXiv:2501.12944, 2025.
[13] Ajay Patel, Markus Hofmarcher,, Claudiu Leoveanu- Leovcanu-Condrei, and Sepp Hochreiter. Large language models self-improve at web agent tasks. arXiv preprint arXiv:2405.22 20309, 2024.
Dinu, Chris
[14] Siheng Li, Cheng Yang, Zesen Cheng, Lemao Liu, Mo Yu, Yujiu Yang, and Wai Lam. Large language models self-improve in long-contess reasoning. arXiv preprint arX1v:2411.08147, 2024.
[15] Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuckai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084 2025.
[16] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024.
[17] Wenkai Yang, Weijie Liu, Ruobing Xie, Yiju Guo, Lulu Wu, Saiyong Yang, and Yankai Lin. Laser: Reinforcement learning with last-token self-rewarding. arXiv preprint arXiv:2510. 10..1444, 2025.
11

---

[18] Muning Wen, Cheng Deng, Jun Wang, Weinan Zhang, and Ying Wen. catropy-reuularize token-level policy optimization for large language models. arXiv e-prints, pages urXiv-2402. 2024.
[19] Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Seed-grpo: Semantic entropy enhanced grpo for uncertainty-aware policy optimization. arXiv preprint arXiv:2505. /2346, 2025.
[20] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances neural information processing systems, 17, 2004.
[21] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[22] Qingyang Zhang, Yatao Bian, Xinke Kong, Peilin Zhao, and Changqing Zhang, Come: Test- time adaption by conservatively minimizing entropy. In International Conference on Learning Representations, 2025.
[23] Lorenz Kuhn, Yarin Gal, and Sehastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation natural language generation. arXiv preprint arXiv:2302.09664 2023.
[24] Haozhe Wang, Qixin Xu, Che Liu, Junhong Wu, Fangzhen Lin, and Wenhu Chen. Emergent hierarchical reasoning in lims through reinforcement learning. arXiv preprint 23Xx:2550.00366, 2025.
[25] Ori Press, Ravid Shwartz-Ziv, Yann LeCun, and Matthias Bethge. The entropy enigma: Success and failure of entropy minimization. In International Conference on Machine Learning. PMLR, 2024.
[26] Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625 630, 2024.
[27] Huan Ma, Jiadong Pan, Jing Liu, Yan Chen, Joey Tianyi Zhou, Guangyu Wang, Qinghua Hu, Hua Wu, Changqing Zhang, and Haifeng Wang. Semantic energy: Detecting Ilm hallucination beyond entropy. arXiv preprint arXiv:2508. /4496, 2025.
[28] Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Pri- oritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pages 15630-15649, PMLR, 2022.
[29] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source reinforcement learning system at scale. arXiv preprint arXiv:2503. 14876, 2025.
[30] Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025.
[31] Hanning Zhang, Jiarui Yao, Chenlu Ye, Wei Xiong, and Tong Zhang. Online-dpo-rl: Unlocking effective reasoning without the ppo overhead, 2025. Notion Blog.
[32] Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong. Li Zhou, Yann Fleureau, Guillaume Lample. and Stanislas Polu. Numinamath. [https: /
(https: com/ project- numina/ ailaa http:///mw....aoorpppsr////////p hatte//aannnnnaaattttttte. pdf) 2024.
[33] Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason E Weston, et al. Naturalreasoning: Reasoning in the wild with 2.8 challenging questions. arXiv preprint
2025.
12

---

[34] Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu. Mengfei Zhou, Zhuosheng Zhang, al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. In International Conference on Machine Learning, 2025.
[35] Huan Ma, Jingdong Chen, Guangyu Wang. and Changqing Zhang. Estimating Ilm uncertainty with logits. arXiv preprint arXiv:2502.00290 2025.
[36] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2024.
[37] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman, Gpqa: A graduate-level poogle-proof q&a benchmark. In First Conference on Language Modeling, 2024.
[38] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training. 2024.
[39] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024.
[40] Chujie Zheng, Shixuan Liu, Mingze Li, Xiong Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507. 2025.
[41] Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung-Yeung Shum Xiangyu Zhang. Open-reasoner-zerro: An open source approach to scaling reinforcement learning on the base model. https://gi whw.ggtthubb.. www./owwmemeeeeeeeeee Reasonnr---n--r-eee Zero http://ppeenResoeeereeeeeeee Reasoner Zero 2025.
[42] Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid- Mid-training incentivizes reinforcement learning scaling. arXiv preprint ar arXiv:2506.20512, 2025.
[43] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504. 18377, 2025.
[44] Yuda Song, Hanlin Zhang, Carson Eisenach, Sham Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self-improvemmnt capabilities of large language models. In International Conference Machine Learning, 2025.
[45] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Reft: Representation finetuning for language models. Advances in Neural Information Processing Systems, 37:63908-633 63962,, 2024.
[46] Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train? arXiv preprint arXiv:2505.21444 2025.
[47] Xuandong Zhao Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint wrXiv:2505.19500 2025.
[48] Zitian Gao, Lynx Chen, Haoming Luo, Joey Zhou, and Bryan Dai. One-shot entropy minimiza- tion. arXiv preprint arXiv:2505. 202822 2025.
[49] Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint 2tt::25055033355 2025.
[50] George Casella and Roger Berger. Statistical inference. CRC press, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
There is no result that need to be mathematically proved.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[Yes]
Justification:
reproducible, they In the supplementary meterials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so âAIJNoâÅl is an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. wwc./pullcccccc howlliccggiies.cco
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In the appendix we run multiple times and report the mean accuracy.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //t::://neuiiseeeeeee https////ccpppptttecccccccccdaa
Answer: [Yes]
Justification:
The research is with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
19

---

Justification:
The LLM is used only for writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:s/s//eeerippoccc///o/rreerp for what should or should not be described.
20

---

Appendices
Prompt Templates
B Case Study
23
Implementation Details about Semantic Clustering
24
D Additional Results on Llama3 Model Series
24
Additional Training Details
26
Computational Cost of Semantic Clustering
26
Details of Prompt Collection
27
H
Additional Results about @a
27
Discussion about Concurrent Works
27
The Inffuence of Clustering Quality the Performance of EMPO
28
A
Prompt Templates
We provide the prompt templates used for training and evaluation.
For mathematical reasoning tasks, we adopt the following reasoning prompt template similar to Online-DPO-RI project [31] for both training and testing. During testing, we found that by adding system prompt, the accuracy of Qwen2.5-Math Base model can be better on mathematical benchmarks. However, system prompt would not help in natural reasoning tasks. Thus we use the same test prompt (start with system prompt) for both Base model and finetuned models in mathematical tasks. In natural reasoning tasks, we do not add system prompt for Base models.
Mathematical Reasoning Training and Evaluation Template
<lim_startls system
Please reason step by step, and output your final answer within \boxed }.
<lim_endi>
<lim_staat(>zuse
(Question) Let's think step by step and output the final answer within \boxed ).
<lim_endl>>
<lim starti>assistant
To train models with our EMPO for free-form natural reasoning tasks, we adopt the following reasoning prompt template similar that we used in mathematical tasks for training.
Free-form Natural Reasoning Training Template
clim_start>system
Reason step by step, and output your final answer within \boxed
<lim_endI>
<lim start/zutr
[
[Question] Reason step by step and output the final answer within \boxed ).
<lim_
<lim lim_sttatbaassistan
21

---

Since the MMLU-Pro and GPQA are both close-formed multi-choice benchmark. To evaluate the natural reasoning capability of the models, we use the following prompt template during testing.
MMLU Pro Test Template for Base Models
Question: (Question} Reason step by step and output the final answer (the correct letter choice from A-P) within \boxed(); J. Answer:
Few Shot MMLU Pro Test Template
Question in Demonstration 1) Reason step by step and output the final answer (the correct letter choice from A-P) within \boxed())
Question:
Answer: Let's reason step by step. CoT of Demonstration Therefore, the correct answer is Answer of Demonstration 1.
(Omit more demonstrations for readability)
Question: [Question| Reason step by step and output the final answer (the correct letter choice from A-P) within \boxed(]: J.
Answer:
MMLU Pro Test Template for Finetuned Models (SFT and RL)
djm__sart(>syee
Reason step by step, and output your final answer (the correct letter choice from A-P) within \boxed().
<lim_endl:
<lim startizuser
[Question] / Reason step by step and output the final answer (the correct letter choice from A-P)
within \boxed(). J.
<lim_end|>
<lim atm_staattbassistant
GPQA Test Prompt for Base Models
Question: [Question] Reason step by step and output the final answer (A, B, C, or D) within \boxed(].
Answer:
Few Shot GPQA Test Prompt for Base Models
Quusstion in Demonstration 1} Reason step by step and output the final answer (A, B, C, D) within \boxed
Question:
Answer: Let's s reason step by step. CoT in Demonstration Therefore, the correct final answer Answer Demonstration 1.
(Omit more demonstration for readability)
Question: [Question] Reason step by step and output the final answer (A, B, C, or D) within \boxed[].
Answer:
22

---

GPQA Test Prompt for Finetuned Models (SFT and RL)
<lim, startlsysstem
Reason step by step, and output your final answer (A, B. C, or D) within \boxed }. <lim_endi>
<lim_start>buser
[Question) Reason step by step and output the final answer (A, B, C, or D) within \boxed )
<lim_endl>>
dinnssarrD>assistant
B
Case Study
Qwen2.5-Math- 7 Answer to the 18-th Question AMC23
Question: Janet rolls standard 6-sided die times and keeps a running total of the numbers she rolls. What is the probability that some point, her rumning total will equal 3? The final answer can be written in the form m where and are relatively prime positive integers. What is m + Response: solve this problem, we need to determine the probability that at some point during 4 rolls of standard 6-sided die, the running total will equal 3. We can use dynamic programming to keep track of the probabilities of each possible running total after each roll. Here's s the plan:
1. Initialize a list to keep track of the probability of each possible rumning total after each roll.
2. Start with the initial condition that the probability of running total of 0 after 0 rolls is 1.
3. For each roll, update the probabilities of each possible running total by considering the outcomes of rolling 1, 2, 4, 5, 6.
4. After each roll, check the probability of a running total is non-zero.
5. The final answer will be the probability of running total of 3 after 4 rolls. Let's implement this in Python:
(We omit the python code for readability.)
It looks like there was an issue with the implementation. The probability of the running total being exactly after rolls should not be zero, let's correct the approach and re-implement the solution. Here's the corrected plan:
Initialize list to keep track of the probability of each possible running total after each roll.
2. Start with the initial condition that the probability of a running total of 0 after O rolls is 1.
3. For each roll, update the probabilities of each possible running total by considering the outcomes of rolling 1, 2, 4, 5, 6.
After each roll, check if the probability a running total is non-zero.
5. The final answer will be the probability of running total of 3 after 4 rolls.
Let's ''' re-implement this in Python:
(We omit the python code for readability.)
It seems there is still an issue with the implementation. Let's manually check the logic and correct it step step.
1. Initialize the probability list with the initial condition that the probability of a running total of after rolls is 1.
2. For each roll, update the probabilities of each possible running total by considering the outcomes of rolling 1, 2, 3, 5, or
3. After each roll, check if the probability of a running total of is non-zero.
The final answer will be the probability of running total of 3 after rolls. Let's manually simulate the process: After 0 rolls: Probability of 0 is
The probability of running total of 3 after 4 rolls is 4/1296, which simplifies to 1/324. Therefore, the final answer in the form of m/n is 1/324, and the sum is 1 324 325. The final answer 325
23

---

C Implementation Details about Semantic Clustering
We detail the implementation of semantic clustering in our EMPO. The semantic cluster process is shown Algorithm [ We also provide the simplified python code show how we determine whether two model outputs are equivalent or not in mathematical reasoning (Algorithm 2 and free-form natural reasoning tasks (Algorithm
Algorithm 1: Semantic Clustering
Require question q, a group model response {02...., 0G), verifier Initialize: C= {01}
for 2 < do
for C' do
// Random choose one element from for comparison
0c c(0]
// Is the meaning of old sequence equivalent to new one?
if V(g, Oc, 0c) == True then
// Put into existing class
c (0:) break
end
end
// o, semantically distinct, belongs to a novel cluster. C + Cu
end
Return :C
Algorithm 2: Implementation of verifier for mathematical reasoning tasks.
from math verify math_verify import parse, verify
def are_equivalent (model_output_l. model_output_2)
prediction_l parse (model_output__) prediction_2 parse (mwdel_ootppteetttcm return verify (prediction_l. prediction_2)
Additional Results on Llama3 Model Series
We conduct additional experiments to validate the efficacy of our EMPO on other model series beyond Qwen2.5. The results shown Table 4 Consistent with other concurrent practice, are unable to implement R1-Zero-like training on the Llama series, i.e., directly initializing RL process from the Base model without SFT). Thus, we instead consider a semi- supervised learning approach by initializing from instruct-tuned model and enhance the reasoning capability with our EMPO. As shown Table when initialize from Llama3.2-3B. 3--2-333-Istttrctt model, our EMPO can also substantially improve reasoning capability of instruct-tuned model which have undergone carefully-
post-training.
Consistent with open-source community practices, we found that R R1-Zerollike RL training can only be reproduced unsupervised on Qwen2.5 series Base models. In contrast, Llama3 series model still necessitate "cold-so start". i.c., SFT, before RL. Specifically, in our experiments, the Qwen2.5 Base models demonstrated inherent answer consistency from the initial stages of EMPO training. However, Llama3 series Base models suffer severe inconsistency and fail convergence during training. We hypothesize this divergence stems from Qwen2.5's pretraining strategy. As mentioned in the technical report [5], the pretrain data corpus are mixed with both web text and QA pairs generated by instruct-tuned Qwen2 models. This endows Qwen2.5 Base models with native instruction-followee capabilities. Experimental evidence supports this hypothesis. As shown in Table Qwen2.5 Base models successfully follow the instruction such "put the final answer (A-P) within box" when answering multiple-choice questions from MMLU and achieve an accuracy notably higher than random guess.

---

Algorithm 3: Implementation verifier for natural reasoning tasks.
verifier AutoModel prrp:eaaaaaaa..oorrrrrereree from_pretraineed (...)
tokenizer AutcTokemizer. htt.:/eerr..oooopeaaieeeeeeeee
def @re_equivalent (model_outpiil nodel_output__ question, verifier)
prediction_1 parse (model_output__1) prediction_2 parso (modcl_output__
prompt -
E"User: ### Question: (question]\nln"
f"### Ground Truth Answer: [prediction_1])lnl
f"### Student Answer: (pretiction_2)Alla
"For the above question, please verify if the student's answer is equivalent to the ground truth answer.In
"Do not solve the question by yourself; just check if the student answer is equivalent to the ground truth answer "If correct, output \"Final Decision: Yes If incorrect, output \"Final Decision: No\".An"
"Assistant: Final Decision:
inputs
(modifisd__rooppt
return_tensor=="pC. (self, model device
input_ids
inference for output logits
with infceninn caance__ccell mode
outputs self model foraa ids)
logits = outputs. outputs..ogit.. logits
get next output logits text_tokke_logits logits input_ids. shape [1] 1,
get the teken ID of "Yes" and "No" decision_tokens self.tokenizer (" Yes"" "No")
yes_id deccision_toeenee input_ids [0]
mo_id = becistoo__okens.inptt... innptasinnpt___d_____
calculate probability
probs torth.softmax (next_ _ext__token_logite dim-0)
prob probs yye__ide_ no no_prob - probs| iid_ioooo_d_____ee ()
return yes_prob no_prob
25

---

Table 4: Accuracy on mathematical reasoning benchmarks.
Minerva Math
Supervision
MATH
OMNI
AIME24
AMC23
Avg.
frontier model
Llama-3.117000 3.1--00--n--tt
(g,r,a) {q,r,a}
35.3
31.9
16.7
35.7
Eurus- 2-B-P---
79.2
38.6
42.1
26.7
57.8
48.9
1B model
Llama3.2 Insstruct
None
27.2
5.1
5.6
0.0
10.0
9.6
Llama3 Instruct w/GPPO
{g. a) (g)
29.8
3.7
6.4
0.0
12.5
10.5
Llama3.2 2...222-nnee 2-Instruct w/EMPO
31.0
5.1
7.9
3.3
7.5
11.0
3B model
Llama3. 2- Instruct
None
46.2
19.1
13.3
3,3
20.0
20.8
Llama3.2 2- 2-Instruct w/GRPO
{q.0)
49.2
22.4
17.6
13.3
32.5
27.0
Llama3.2-Instree w/EMPO
[9]
49.8
20.2
18.4
13.3
30.0
26.3
E
Additional Training Details
We provide brief summary of our training recipes in Table Besides, we have release the code in the supplementary materials which contained the full training configurations for re- re-implementation.
Table 5: A brief summary of training recipes of Qwen2.5 Base models.
1.5h-Math
7B-Math
3B
7B
14B
Number of generations
7
12
12
12
Learning rate
3e-7
3e-7
3e-7
Max completion length
2048
2048
1024
1024
768
Batch size per GPU
2
1
1
Computational Cost of Semantic Clustering
Given the number of responses sampled per question G (i.e., the group size) and the training dataset size N, the time complexity of the clustering process is O x N). mathematical reasoning tasks, semantic clustering is implemented by regular expressions which do not involve notable computational cost. For natural reasoning tasks, we rely on an additional compact small language model. To evaluate the additional computational overhead introduced by semantic clustering in EMPO, we conducted comparative analyses of EMPO and GRPO in terms of total training duration and GPU memory utilization. The results of mathematical reasoning and natural reasoning are shown in Table and Table 7 respectively. It is worthy to note that the 14B model experiments requires slightly less computational time than the 7B model. This is because. in 14B experiments, reduced the batch size and maximum response length from and 1024 to and 768, respectively, compared to the 3B and 7B configurations. This adjustment was made to fit the limited GPU memory of single 8x A 100 80G machine.
Table 6: Comparison of total runtime (measured 8x A100 GPU hours) and storage cost (measured by max total GPU memory (GiB) utilization) between GRPO and EMPO. The GPU Memory semantic cluster process requires minimal computation and storage.
Owwn2.5-1.5B-Matth
Qwen2.5-7B-M
GPU Hours
GPU Mem
GPU Hours
GPU Mem
GRPO
11.2
240.4
8.5
501.3
EMPO
11.7
208.2
8.7
532.7
26

---

Table 7: Comparison of total runtime (measured as 8x A100 GPU hours) and storage cost (measured by total GPU memory (GiB) utilization) between GRPO and EMPO. The GPU Memory semantic cluster process requires minimal computation and storage.
Qwen2.5-3B
Qwen2.5-7B
Qwen2.5-14B
GPU Hours
GPU Mem
GPU Hours
GPU Mem
GPU Hours
GPU Mem
GRPO EMPO
9.5
274.8 286.9
12.4 14.6
508.6 532.7
11.0 11.5
588.2
11.1
541.1
G Details of Prompt Collection
For mathematical reasoning, we directly use 20,000 prompts randomly selected from Numina-Mati COT. For free-form natural reasoning tasks, we adopt the prompts from Natural Reasoning by filtering out the questions with over-long prompt, reference answer. Besides, we use the response length of (lama..3-70B-Intttee
difficulty estimation metrie, and filter overly difficult samples with response lengths exceeding 4096 tokens. The data collection python code is demonstrated as follow:
Algorithm 4: Python code of data filtering huggingface- style.
from datasets import load_dataset
dataset=load (" ('facebook " -aturallReasoningg
filtered_dataset
fflaset.filte.
1ambda x:
F no answer
"xefereeneee answer"nsw"""
& over-long answer
and (e[""eeeeee "nsserr 129 overly difficuit questions and
4096
i over-long prompt and proof-oriented
512
and ("prove" not in x "["question") lower ()) and ("proof" not in x xl"question") lower
lowee ())
H Additional Results about Pass@k
We provide additional visualization pass @k results of models trained with EMPO. The results are shown follow. As shown in Figure H the Base model consistently catch up with RL trained models when k large.
Discussion about Concurrent Works
Recent works explore self-supervised RL with various internal reward signals including majority voting (TTRL [5]. SRT (46)), semantic coherence (our EMPO), token-level confidence score (Intuitor 47 One-shot Entropy Minimization (48). and self-play (Absolute Zero [4]). TTRL and SRT generate pseudo label by majority voting, which is limited to close-end mathematical reasoning. Besides, TTRL directly conduct RL on test prompts (test-time training), while EMPO keeps the strict separation between training and testing. Intuitor and One-shot EM use the token level softmax probability confidence. By contrast, EMPO rewards coherent outcomes regardless the token-level probability. Besides, Absolute Zero further remove the dependency of prompts, which only relies on compile environment only.
http::/wwgggiggiee.ooaaasssctttt reasoning
27

---

MATH
OMNI
1,0
0.8
Number of Samples Coverage (passi@k)
Coverage (pass(lk)
0.7
0,9
0.6
0.8
0.5
Das: EMPO
Dosc
0.7
EMPO
0.3
GRPO
GRPO
0.6
0.2
2
57
60
Number of Samples
Figure 5: Trend of pass @k accuracy on Math test-set.
Figure 6: Trend of pass @k accuracy on OMNI
test-set.
J The Influence of Clustering Quality on the Performance of EMPO
In our mathematical reasoning experiments, semantic clustering achieved solely through regular expression matching without introducing additional models. Due to the naturally structured response formats mathematical tasks, regular expression could accurately determine answer equivalence, resulting in relatively high clustering quality.
However, in more general free-form natural reasoning tasks where model responses are free-form much more diverse (e.g., matrix, numbers, few lines of sentences/codes...) the clustering quality can impact EMPO's effectiveness. For instance, our more early practice, we tried DeBERTa (a bert-like model with 300M parameters trained by microsoft) for semantic clustering. Due to the poor quality semantic clustering, our EMPO straggled scale up and suffered from frequent reward hacking. Subsequently, by leveraging the general-verifier released by Tiger-Lab (a fine-tuned Qwen2.5-1.5B. 1..5--att model) for clustering, we successfully generalized EMPO more general free-form reasoning tasks. Noted that even though this small language model undergoes supervised finetuning, it serves within our fully unsupervised framework as a fixed utility function for semantic comparison, rather than serving as external supervisor for task-specific feedback. There are several fundamental difference between cluster model and the reward model used supervised RL:
The cluster model does not evaluate output correctness relative input queries. It just provides pairwise comparisons between the model's own outputs. That is, it only provides binary answer about "whether these two answer is the same?" rather than "which answer is better?".
The cluster model does not provide any guidance, such as gradient information or hints on how refine the reasoning traces.
Compared to reward model or human-verifier golden answers, it can be much easier to implement such cluster model. For example, in mathematical reasoning tasks, only regular expressions are enough for clustering. In natural reasoning tasks, finetuned Qwen2.5-1B model can provide high quality semantic cluster results.
Essentially, this is related to the non- non-identifiability problem in statistical inference [50]. The issue of -on-identiifabilitt arises because multiple, distinct underlying states (potential "truths," or more accurately, different reasoning pathways or different clusters of incorrect answers) could produce the same pattern of relational signals (i.e., the same semantic clustering results).
28


Reviews:
[{"review_id": "RYVSZ1qVwm", "reviewer": "Reviewer_3qKR", "summary": "This paper studies a new unsupervised learning paradigm for RL training of LLM reasoning and proposes Entropy Minimized Policy Optimization (EMPO) to elicit LLM reasoning performance via unsupervised learning without the need for ground truth labels. They show that when trained on mathematical reasoning and free-form natural reasoning tasks, RMPO achieves the same or even better performance than RL methods like GRPO and it does not need a manually labeled reward or even generated pseudo-reward from LLM (for example, by majority vote). \n\nOn a high level, I think this new paradigm is interesting and worth investigating, and has potential for future research. Although I think this research is in the initial stage of unsupervised training for LLM on reasoning tasks, I lean toward accepting this submission because of its novelty and the soundness of the experiments.", "questions": "See the strengths and weaknesses section.", "limitations": "See the strengths and weaknesses section.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. I think the new paradigm this paper proposes about unsupervised training for LLM reasoning is novel and has huge potential, especially when we need a huge amount of data, and labeling these questions is hard, if not impossible. This is also suitable for the reasoning tasks that do not have ground-truth labels, such as some general open-form tasks.\n\n2. The experiment results look good, and the performance of EMPO can match or even outperform the RL methods like GRPO, or other methods like SFT. I think this is surprising to me.\n\n3. The discussion part is pretty clear, I think. When I read sections 1 to 4, I was thinking: does the pass@k rate improve when k is large? And does the total solvable set increase compared to the base model and supervised-RL-trained models? And they answer my questions in section 5, as they know what I want to read.\n\nWeaknesses and Questions:\n\n1. About the formulation 7: They have an entropy filtering applied to the questions that are trained in EMPO. I think their explanation is good, but I have two questions. (a). How many prompts do they filter from this entropy filter? (b). Could you show some experimental results about what happens if we remove the \\delta_low or \\delta_high (I think we will be more interested in the results when removing \\delta_high as it is predictable that when we remove \\delta_low, the peak performance will not change too much but the training will become slower).\n\n2. About the experiment details: It seems to me that you do SFT and GRPO on a different training dataset. (not sure whether I understand it correctly. Please correct me if I am wrong) Why is that? Moreover, you mentioned you filter a part of the data based on the response length of Llama3.3-70B-Instruct? Did you do the same step for SFT and GRPO? I think you should make a clearer comparison of the difference between the EMPO and GRPO implementations.\n\n3. I think the experiments do not show the most advantageous case of EMPO. I think the best case will be: there is a huge dataset with questions only, and labeling them is impossible, or the labels are messy. Will EMPO achieve a good performance on such a dataset?\n\n4. Since EMPO does not rely on the ground-truth labels, there are no such signals. Then, it will be even more important to select a training dataset with moderate difficulty. How will EMPO perform when the dataset is very very hard? For example, if most answers generated are incorrect (but the incorrect answers are diverse so it still passes the entropy filtering), how will the model perform?", "was_revised": true, "final_justification": "Thanks for your detailed rebuttal, and I really appreciate the experiments on entropy thresholding. \n\nAnd also, you addressed my concerns about some experiment details such as the prompt difference for different methods. \n\nThe results on the DeepMath-103K are promising and I really like it.\n\nI believe your response greatly addresses my concerns and questions and will keep my score.", "rebuttal": "We appreciate your thoughtful and thorough review and recognition of the interesting and novel paradigm. We provide detailed responses to the constructive comments.\n- Q1: How many prompts are filtered out by entropy thresholding? Could you show experimental results when we remove $\\delta_{low}$ or $\\delta_{high}$?\n\nAccording to your suggestions, we conduct additional experiments to ablate the entropy thresholding. We train Qwen2.5-7B with $\\delta_{low}=0$ and $\\delta_{high}$ ranges from {0.0, 0.25, 0.5}. The ratio of filtered prompts, mean absolute error between EMPO's unsupervised reward and GRPO's correctness reward (i.e., the noise of EMPO's reward signal), the final accuracy (average over 16 random sampling, i.e., mean@16) on AMC23 testset are as follow\n| $\\delta_{high}$ | Filter ratio   | Reward noise  | Accuracy  |\n| -------------- | ----- | ----- | ---- |\n| $0.0$   | $0.000$  | $0.413$  |  $39.69$    |\n| $0.25$   | $0.503$     |   $0.346$    |  $45.59$    |\n| $0.5$            | $0.613$ | $0.219$ |   $48.12$   |\n\nAs shown above, the thresholding effectively reduces the noise of reward signals and improves the final performance. We will add these results to our revision.\n\n- Q2: Why do SFT and GRPO use different train sets? Did the authors do the same data filter step for both SFT and GRPO? A clearer comparison of the difference between EMPO and GRPO implementations should be provided.\n\nPlease kindly note that the Numina-Math and Natural-Reasoning training sets are labeled with both ground-truth and chain-of-thought reasoning traces. We use the **same** train prompts for SFT, GRPO and EMPO in our experiment. The only difference is the form of supervision. SFT and GRPO use CoT and golden final answer labels respectively. Our EMPO only needs prompts. Besides, for the natural reasoning task, **we do data filtering for both SFT and RL methods (EMPO, GRPO) to make sure they are trained on the same subset**.\n\nThe implementation difference between GRPO and EMPO lies in the calculation of reward signal. While GRPO relies on groundtruth, our EMPO uses the frequency of meaning clusters as internal reward. Besides, as suggested by DAPO [1], we also remove the KL penalty from the reward, which benefits training efficiency.\n\nWe thank the reviewer for these suggestions and will further clairify the implementation details in revision.\n\n- Q3: Will EMPO achieve good performance on a huge dataset with questions only?\n\nYou raise a very interesting comment. Unlabeled questions in the wild arise freely upon deploying LLMs in the open world, of which the amount should be much larger than that of labeled ones. Since reasoning-oriented reinforcement learning is an emerging and rapidly developing field, to our best knowledge, there are no such benchmarks yet. To investigate the scalibility of EMPO, we train Qwen2.5-7B on the very recent DeepMath-103K (released on 2025.04, **5 times larger** than the training set we used in the manuscript). The model accuracy on AMC23 testset steadily increases along with the scale of training samples.\n\n| Training step | 0   | 100  | 200  |  300   | 400  | 500  | \n| -------------- | ----- | ----- | ---- | ----- | ----- | ---- |\n| Total samples   | 0K  | 6.4K  |  12.8K    | 19.2K  | 25.6K  |  32K    |\n| Accuracy   | 27.81     |   35.94    |  38.75    | 39.87     |   44.69    |  48.12    |\n\n- Q4: How will EMPO perform when the dataset is very very hard?\n\nVery insightful question! To investigate this problem, we run OctoThinker-3B (a weaker LLM trained from Llama3-3B-Base) on diverse large-scale DeepMath-103K dataset. In this circumstance, the reward signal is expected to be rather noisy. The model accuracy improves at the beginning, but finally falls into a trivial solution.\n\n| Training step | 0   | 100  | 200  |  300   | 400  | 500  | \n| -------------- | ----- | ----- | ---- | ----- | ----- | ---- |\n| Accuracy   | 14.37     |   26.87    |  31.41    | 24.82     |   12.06    |  0.00    |\n\nThus, if there are too many incorrect answers pass the entropy filtering, the unsupervised learning may become biased and ultimately collapses. This limitation widely exists in unsupervised learning field [2], but still noteworthy! Detecting incorrect but highly confident model answers would be a promising research direction. We promise to transparently discuss this failure mode in the Limitations section in our revision.\n\n[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale, Arxiv Preprint\n\n[2] The Entropy Enigma: Success and Failure of Entropy Minimization, ICML'24"}, {"review_id": "Ozr5fS9sTo", "reviewer": "Reviewer_bd1c", "summary": "This paper introduces Entropy Minimized Policy Optimization, a novel reinforcement learning algorithm aimed at enhancing the reasoning capabilities of large language models without any supervision. Existing methods rely on supervised fine-tuning or external reward signals; EMPO instead minimizes semantic entropy over multiple generations from unlabeled prompts. The authors cluster semantically equivalent outputs and assign higher rewards to more confident clusters, training the model to favor consistent reasoning traces. Empirical results show that EMPO significantly improves performance on both mathematical reasoning (e.g., MATH, Minerva) and free-form reasoning benchmarks (e.g., MMLU-Pro, GPQA), often matching or surpassing supervised approaches.", "questions": "- The paper introduces dual thresholds to exclude examples with overly low or high entropy during training. Could the authors explain how these thresholds were selected in practice?\n- In the natural reasoning tasks, the authors utilize a verifier model to determine whether two outputs are semantically equivalent, using a prompt such as \"Please verify if the student’s answer is equivalent to the ground truth answer.\" This raises some questions about the underlying assumptions of the task setup. Specifically, are the natural reasoning prompts used in this work associated with deterministic, canonical answers that can reasonably serve as “ground truth” for such verification? If so, this setup is more straightforward. However, in more open-ended tasks, such as translation, summarization, or rewriting, there may not exist a single correct answer, and the notion of equivalence becomes less well-defined. In such cases, relying on a single model’s binary judgment of equivalence may introduce noise or bias into the clustering process. It would be helpful if the authors could clarify the scope of applicability of their current semantic clustering approach and discuss whether any adaptations would be needed for tasks with inherently diverse valid outputs.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper is well-structured with clear motivation.\n- The paper conduct extensive experiments on both mathematical reasoning and free-form reasoning benchmarks.\n- The Section 5 discussion of the role of unsupervised learning in LLM reasoning is promising.\n\n**Weaknesses**\n- The authors acknowledge the risk of reward hacking under unsupervised objectives and propose a dual entropy thresholding strategy to filter out prompts that are either too uncertain or already overconfident. While this is a reasonable design, the paper does not present any ablation study or sensitivity analysis to empirically justify this choice. In particular, there is no evidence on how different threshold values affect training stability or final model performance. Given the critical role this filtering plays in avoiding trivial solutions and stabilizing training, a deeper empirical exploration would be valuable.\n- Given the rapid pace of development in this area, a number of concurrent works have emerged that also explore label-free RL for language models. While it is understood that these works are likely developed in parallel, a brief discussion comparing EMPO to these methods could strengthen the paper. Adding a discussion section, perhaps in the appendix, highlighting the distinctions in methodology, assumptions, or application scope would help readers better understand EMPO’s specific contributions in the broader context of this growing field.\n- Since EMPO relies entirely on unlabeled data, the quality and diversity of the training prompts may play a significant role in its effectiveness. It would be helpful to understand how the method performs under varying amounts of data or different prompt selection strategies. For example, does performance scale linearly with more data, or is it sensitive to prompt difficulty or domain distribution? Including a brief analysis or reflection on this point would enhance understanding of the method’s robustness and practical applicability.", "was_revised": true, "final_justification": "I confirm that the authors have addressed my concerns and questions.", "rebuttal": "We sincerely thank the reviewer for your valuable comments and appreciate your recognition of the clear motivation, extensive experiments and promising discussion. We provide detailed responses to address your concerns.\n\n- W1: The paper does not present any ablation study or sensitivity analysis to empirically justify the choice of entropy thresholds.\n\nThanks for raising this comment. We conduct two additional experiments involving Qwen2.5-Math-1.5B, Qwen2.5-7B on the very recent DeepMath-103K dataset (released at 2025.04) to ablate entropy thresholding. Since it is predictable that when we remove $\\delta_{low}$, the peak performance will not change too much but the training will become slower (which is also mentioned by reviewer 3qKR). Thus, due to the high computational cost of LLM ablation studies, we preliminarily ablated the entropy upper bound ($\\delta_{high}$). We report the ratio of filtered prompts, reward noise (mean absolute error between EMPO's unsupervised reward and GRPO's correctness reward), the final accuracy on AMC23 testset (averaged by 16 random sampling, i.e., mean@16).\n\nQwen2.5-Math-1.5B\n| $\\delta_{high}$ | Reward Noise   | Accuracy  |\n| ----- | ----- | ----- |\n| 0.0   | 0.625  | 47.66 (early stopped before reward hacking) | \n| 0.25   | 0.422    |   46.56    | \n| 0.5       | 0.228 | 51.56 |\n\nQwen2.5-7B\n| $\\delta_{high}$ | Reward Noise   | Accuracy  |\n| ----- | ----- | ----- |\n| 0.0   | 0.413  | 39.69 | \n| 0.25   | 0.346    |   43.59    | \n| 0.5       | 0.216 | 48.12 |\n\nAs shown above, the thresholding effectively reduces the noise of reward signals and improves the performance as we expected.\n\n- W2: A brief discussion comparing EMPO to other concurrent works could strengthen the paper.\n\nWe sincerely appreciate your kind understanding and suggestions.\n\nWe extend our discussion in line 72-87 to further explain the distinction between EMPO and other concurrent works. Recent works explore self-supervised RL with various internal reward signals including majority voting (TTRL [1], SRT [2]), semantic coherence (our EMPO), token-level confidence score (Intuitor [3], One-shot Entropy Minimization [4]), and self-play (Absolute Zero [5]). TTRL and SRT generate pseudo label by majority voting, which is limited to close-end mathematical reasoning. Besides, TTRL directly conduct RL on test prompts (test-time training), while our EMPO keeps the strict separation between training and testing. Intuitor and One-shot EM use the token level softmax probability as confidence. By contrast, EMPO rewards coherent outcomes regardless the token-level probability. Besides, Absolute Zero further remove the dependency of prompts, which only relies on a compile environment only. **We will definitely add this discussion in our revision. If there are any representative papers we missed, please let us know.**\n\n- W3: How EMPO performs under varying amounts of data or different prompt selection strategies? Does performance scale linearly with more data, or is it sensitive to prompt difficulty or domain distribution? A brief analysis or reflection on this point would help.\n\nVery insightful question! We appreciate your carefully reading and in-depth suggestions. To investigate the scalibility and sensitivity of EMPO, we conduct additional experiments and train OctoThinker-3B and Qwen2.5-7B with EMPO on the very recent, large-scale DeepMath-103K (more than 5x larger than the training set we used in paper). The test accuracy (mean@16) on AMC'23 of varying training samples are shown as follows\n\n| Training step | 0   | 100  | 200 | 300 | 400 | 500 |\n| ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Training samples   | 0  | 6.4K | 12.8K  | 19.2K | 25.6K  | 32.0K  |\n| Accuracy of Qwen2.5-7B   | 27.81  | 35.94 | 38.75  | 39.69 | 44.69 | 48.12 |\n| Accuracy of OctoThinker-3B   | 14.37  | 26.87 | 31.41  | 24.82 | 12.06 | 0.00 |\n\nAs shown in above, EMPO consistently scales up the performance of Qwen2.5-7B on larger and more diverse training dataset. The model accuracy on AMC23 testset steadily increases along with the amount of data.\nHowever, for OctoThinker-3B [6] which is trained from weaker base model (Llama3-3B), the performance improves at the first 200 steps and reaches a plateau. Scaling up performance with long-term (supervised or unsupervised) RL may still be a challenging systematic engineer, including entropy controlling, data diversity, training efficiency and AI infra [8]. We promise to make this point transparent in our revision.\n\n- Q1: Could the authors explain how these thresholds were selected in practice?\n\nWe suggest two ways to determine the thresholds.\n\n1) We can employ EMPO with a smaller LLM (e.g., qwen2.5-math-1.5B) and **monitor the training-time semantic entropy** to detect \"hacking\". When hacking occurs, the semantic entropy typically drops rapidly to nearly 0 within <20 steps. Through such a pre-experiment, we can then choose a suitable threshold for training larger LLMs (>=7B). To illustrate this, we present the change in semantic entropy during hacking while training Qwen2.5-Math-1.5B on DeepMath-103K without thresholding.\n| Training step |1008  | 1012 | 1016 | 1020 | 1024 |\n| ----- | ----- | ----- | ----- | ----- | ----- |\n| Semantic entropy   | 2.995  | 2.367 | 1.580  | 1.136 | 0.343  |\n\n2) Alternatively, we can **inference offlinely on the unlabel dataset for multiple times and determine the thresholds according to the average semantic entropy**. For example, when training Qwen2.5-7B on DeepMath, semantic entropy has a strong positive correlation with reward noise. Thus we can estimate the reward noise level by semantic entropy, and then choose suitable thresholdings.\n\n\n| $\\delta_{high}$ | Reward Noise   | Semantic entropy  |\n| ----- | ----- | ----- |\n| 0.0   | 0.413  | 3.626 | \n| 0.25   | 0.346    |   3.575    | \n| 0.5       | 0.219 | 2.305 |\n\nBesides, in our experiments, setting the entropy upper threshold to 0.5 * maximum entropy (i.e., log K, where K is the number of semantic clusters) consistently prevented hacking.\n\n- Q2: Are the natural reasoning prompts associated with deterministic, canonical answers that can reasonably serve as “ground truth” for such verification? In more open-ended tasks there may not exist a single correct answer...It would be helpful to clarify the scope of applicability of current semantic clustering approach and discuss whether any adaptations would be needed for tasks with inherently diverse valid outputs. \n\nWe sincerely thank for your carefully reading and in-depth comments. Your question led us to think a lot. EMPO's semantic entropy minimization objective is primarily designed to address \"hallucinated\" false, nonsensical or fabricated answers. Thus such an objective may not benefit tasks that necessitate divergent thinking like creative writing. However, for other tasks that require ensuring factual correctness, such as rewriting, translation, and summarization, our EMPO still has the potential to be adapted. For example, in long-text generation tasks, previous work [7] has utilized external models to first construct several factual questions based on the generated content. The model is then prompted to answer these questions based on generated text, and the semantic entropy calculated individually for each question is aggregated to determine the factual reliability of the entire passage. By leveraging such a **divide-and-conquer** approach, EMPO will have the potential to be adapted for those long-text generation tasks that require guaranteed factual correctness. \n\n**In the revision, we will define the current scope for convergent reasoning and explicitly detail these powerful adaptations for complex, diverse-output tasks as a promising direction for future work.**\n\n[1] TTRL: Test-Time Reinforcement Learning, Arxiv Preprint\n\n[2] Can Large Reasoning Models Self-Train?, Arxiv Preprint\n\n[3] Learning to Reason without External Rewards, Arxiv Preprint\n\n[4] One-shot Entropy Minimization, Arxiv Preprint\n\n[5] Absolute Zero: Reinforced Self-play Reasoning with Zero Data, Arxiv Preprint\n\n[6] OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling, Arxiv Preprint\n\n[7] Detecting hallucinations in large language models using semantic entropy, Nature'24\n\n[8] Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models, Arxiv Preprint"}, {"review_id": "ZxeSvK0idP", "reviewer": "Reviewer_Sarg", "summary": "This paper introduces EMPO: a variant of GRPO rewarding models for minimizing response \"semantic entropy\". This is done first by clustering a set of responses $\\{s_1,...,s_K\\}$ into semantic clusters $\\{c_1,...,c_C\\}$ and then giving reward $r_i$ for $s_i$ as $p(c_{s_i})$ i.e. the probablity of the semantic cluster $c_{s_i}$. Experiments are conducted on math and free-form reasoning tasks.", "questions": "- Have you considered alternatives to final answer equivalence classes for reasoning? I think it would be very interesting to investigate the effect of different equivalence class schemes in more detail (this was briefly discussed in the final appendix section).", "limitations": "The authors have **not** included a limitations and broader impacts section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper proposes a smart extension of entropy minimization ideas to LLM reasoning\n- The paper is mostly clear and easy to follow\n- The paper evaluates a good range of model sizes (3B, 7B, 14B) over multiple reasoning datasets\n\n**Weaknesses:**\n\n- The implementation details for how semantic entropy is measured are not present in the main body of the paper. For math problems, only the final answer is used to do clustering, resulting in an objective very similar to self-consistency rewards. Clustering for free-form problems is done using an open-source instruction model (which is fine but should be made clear much earlier).\n- Because the final EMPO objective is so similar to self-consistency rewarding approaches, it would be nice to see a self-consistency rewarding baseline to compare to.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the smart idea, easy-to-follow writing and good evaluation range. We appreciate your positive review and address your concerns as follows.\n\n- W1: The implementation details for how semantic entropy should be made clear much earlier.\n\nThanks for your carefully reading and suggestions. It is our duty to clearly present the implementation details. We will reorganize and provide a clear illustration of semantic clustering at the beginning of the method part according to your suggestion.\n\n- W2: It would be nice to see a self-consistency rewarding baseline to compare to.\n\nThanks for mentioning the self-consistency rewarding baseline. According to your suggestion, we conduct additional experiments by training Qwen2.5-Math-1.5B on DeepMath-103K, and compare EMPO with 1) test-time majority voting and 2) self-consistency rewarding (SR). We report the accuracy average over 16 samplings, i.e., mean@16 of EMPO and self-consistency rewarding as the number of training steps increases.\n\n| Training step | 0   | 200  | 400  | 600  | 800  |\n| -------------- | ----- | ----- | ---- | ---- | ---- |\n| Self-consistency Rewarding   | 42.18  | 48.28  | 47.66 | 44.53  | 2.93 (collapse due to reward hacking) |\n| + Majority Voting (major@16)   | 56.21  | 58.48  | 57.82 | 52.22  | 2.52 |\n| EMPO   | 42.18  | 46.09 | 49.38  | 50.62 | 51.56 | \n| + Majority Voting (major@16)   | 56.12  | 56.98  | 61.50 | 59.86  | 63.23 |\n\nCompared to a self-consistency reward which is binary, EMPO's reward is \"soften\", offering better tolerance to reward noise. For example, assume the correct answer is \"B\". However, the LLM outputs the incorrect answer \"A\" 9 times and the correct answer \"B\" 7 times. In this scenario, self-consistency would yield an incorrect pseudo-label of \"A\" (based on the majority vote). The incorrect answer \"A\" would receive a reward of 1, while the correct answer \"B\" would receive a reward of 0, reinforcing an incorrect answer. In contrast, EMPO uses frequencies as rewards, assigning proximate rewards of 0.9 (for answer \"A\") and 0.7 (for answer \"B\"). Thus EMPO provides better tolerance to incorrect rewards, allowing the model to be less sensitive to the noisy model outputs. Noted that there exists several concurrent works utilized self-consistency rewarding [1-4], we primarily follow [1] to implement the self-rewarding with GRPO. We will add these results in our revision and further discuss the distinction between EMPO and [1-4].\n\n- Q1: Have you considered alternatives to final answer equivalence classes for reasoning? It would be very interesting to investigate the effect of different equivalence class schemes in more detail.\n\nYou raised a very interesting question! Currently, we use regular expressions or another language model to determine the pairwise equivalence between answers. Expanding these equivalence classes may be necessary for diverse tasks such as retrieval and long-text generation. 1) For tasks like retrieval, text embedding similarity can be used to define equivalence classes. 2) For long-text generation tasks, previous work [5] has utilized external models to first construct factual questions based on the generated content. The model is then prompted to answer these questions based on its generated text, and the semantic entropy calculated individually for each question is aggregated to determine the factual reliability of the entire passage. In this circumstance, two generations are equivalent if and only if all the factual questions yield the same answers. By expanding the equivalence class, EMPO will have the potential to be adapted for those long-text generation tasks that require guaranteed factual correctness.\n\nThanks for raising such an inspiring question. We will elaborate these future directions in our revision.\n\n- Limitation and Social Impact\n\nThanks for your kind reminder. **It is our duty to explicitly and well discuss the limitation and social impact.** This manuscript presents a new training algorithm that improves the reasoning capability by minimizing semantic entropy. By encouraging semantically coherent outputs, the model's reasoning capability can be boosted without any external supervision. There exists several noteworthy limitation of EMPO. The unsupervised nature of EMPO removes the dependency of external supervision at the cost of potential reward hacking risks. That is, the model may be highly overconfident on its prediction regardless correctness. Besides, as identified by [6], similar to other RL for reasoning methods, EMPO may exhibit undesirable behaviors that stem from pretraining corpus of the Base model, e.g., hallucination, bias and so on. Entropy minimization may not only increase accuracy by encouraging consistency, but also make the bias more severe. Thus, more studies are necessitated to mitigate such issues for large reasoning models. We will add these discussion to our revision. \n\n[1] Preference Optimization for Reasoning with Pseudo Feedback, ICLR'25 spotlight\n\n[2] Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models, Arxiv preprint\n\n[3] CREAM: Consistency Regularized Self-Rewarding Language Models, ICLR'25\n\n[4] Self-Consistency Preference Optimization, ICML'25\n\n[5] Detecting hallucinations in large language models using semantic entropy, Nature'24\n\n[6] Are Reasoning Models More Prone to Hallucination? Arxiv preprint"}, {"review_id": "lUARC8D8lQ", "reviewer": "Reviewer_shZN", "summary": "Reasoning models are growing in prominence. However, training LLMs for reasoning often requires substantial supervised data. It remains unclear how to elicit reasoning in LLMs in an unsupervised manner. Here, the authors develop a new method (EMPO) to encourage models towards better reasoning in an unsupervised manner.", "questions": "The authors primarily report accuracy. Is this averaged over multiple seeds? It would be helpful to see error bars, if possible. \n\nWhat are the error bars in Figure 3 showing? \n\nThe authors filter out by length: what happens if you then test the resulting EMPO-guided model on long prompts? We would want reasoning models to handle difficult tasks! I may have misinterpreted some of the prompt filtering.", "limitations": "Unfortunately, the authors did not include an explicit limitations section nor a negative societal impacts section. A limitations section is definitely needed. The authors are good about discussing the relative merits of their approach but could be more upfront on the current weaknesses of the method and experimental paradigm for testing thus far.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper was strongly motivated. The authors do a good job of motivating that they take on cases without a clear \"ground truth\" answer -- many real problems/queries are like that, and we want our reasoning systems to be robust to handling them! To my understanding, EPMO is an original method (though I'm not too familiar with methods for incentivizing reasoning so may have missed something). The overall idea is clever, intuitive, and well-explained: all strengths to the paper.  \n\nI also thought the authors had a good discussion on the relative role of pre-training vs. reasoning elicitation.\n\nWith that said, I found the experimental results quite challenging to work through. The large tables were not easy to interpret. I appreciated that the authors included the amount of supervising information provided (that was helpful for assessing the methods), but it would have been helpful for more discussion and work on the presentation of results. (more noted in the Questions below).\n\nIf the experiment section were conveyed more clearly, I think this could be a very strong paper.", "was_revised": true, "final_justification": "The authors did address my concerns around error bars and plan to add a limitations. I am happy with the substantial amount of work they put through during revisions. However, I worry that some of the changes may require a more substantive rewrite on clarity that may be beyond what is \"easy\" to do during revisions, hence, my Borderline Accept. I think the direction of the work is promising and hope the authors continue!", "rebuttal": "We thank the reviewer for recognizing the original, intuitive and well-explained method and good discussion of the relationship between reasoning-oriented post-training and pre-training. We appreciate your constructive and actionable suggestions and address your concerns as follows. It is our duty to clearly present our work and avoid any confusion.\n\n- Q1: The authors report accuracy. Is it averaged over multiple seeds? It would be helpful to see error bars if possible.\n\nThanks for raising concerns about error bars. Training LLMs with multiple seeds and reporting error bars can be extremely costly, thus many existing works do not have error bars [1-5]. For example, training a 7B LLMs on Numina-Math-20K for 3 epochs would cost more than 48 hours on 8*A100 GPUs. **While training large models with multiple seeds is indeed computationally prohibitive as noted, we share your concern about reproducibility.** We went to great lengths during the rebuttal period and **retrain** Qwen2.5-Math-7B on Numina-Math (1 epoch) for 3 independent runs (utilized all our available computational resources during the rebuttal). The randomness of results stem from different training dynamics (e.g., order of mini-batches, stochastic optimization) and also test-time random sampling (e.g., top-p, top-k). For each independent run, we report the average accuracy over 16 random sampling. The sampling parameters are temperature=0.6, top-p=0.95 and top-k=20. As shown below, the mean@16 accuracy averaged over 3 independent runs is relatively stable with small deviations across 3 independent runs.\n\n| Test set | 1st run   | 2nd run  | 3rd run  | average | standard deviation |\n| -------------- | ----- | ----- | ---- | ----- | ----- |\n| MATH | 79.57   | 74.32  | 75.90  | 76.60 | 2.20 |\n| MNIV | 46.07   | 29.18  | 31.80  | 35.68 | 7.42 |\n| OMNI | 40.53   | 38.37  | 39.26  | 39.39 | 0.89 |\n| AMC | 58.90   | 63.44  | 59.84  | 60.73 | 1.96 |\n| AIME | 18.96   | 24.16  | 22.71  | 21.94 | 2.19 |\n\nWe will further clarify where the randomness come from and the details of evaluation to ensure reproducability. **Besides, we promise to open all the source code, training recipe, model checkpoints, training logs and data once accepted.**\n\n- Q2: What are the error bars in Figure 3 showing?\n\nIn Figure 3, a Gaussian smoothing has been applied to the actual experimental curves using the Python Seaborn package. Due to the statistical fluctuations between mini-batches, the smoothed curves allow for a clearer presentation of the overall trends. Thus, the error bars are primarily for better visualization, which do not have significant practical meaning. We will further clarify this to avoid confusion in revision.\n\n- Q3: What happens if you then test the resulting EMPO-guided model on long prompts?\n\nThis is a very important question about the generalizability of EMPO to more challenging, longer prompts. We fully agree with your opinion that difficult tasks would be more deserving to be handled with reasoning models. However, **please kindly note that Qwen2.5-Math models have limited max position embeddings due to its original model design**. Due to the maximum positional encoding length limit, the sum of the input prompt and response length must remain within **4096** tokens. Consequently, we filter out excessively long prompts to allow the model to generate more tokens for generating long reasoning traces, which is a choice consistent with common practice [6].\n\nTo directly address your concerns, we conducted new experiments on OctoThinker-3B, which has a much longer maximum positional embeddings (131072). We used DeepMath-103K as the training set and set the maximum prompt length to a longer **4096 tokens (covering all problems)**. Under this setup, EMPO still demonstrates significant performance improvement, where the model's average response length exceeds 12K tokens. This demonstrates that the prompt length filtering in our main experiments was a practical choice due to model constraints, not a limitation of the EMPO framework.\n\n|  | Max prompt length   | Max generated tokens  | Accuracy  |\n| -------------- | ----- | ----- | ---- |\n| OctoThinker-3B-Base | 4*1024   | 12*1024  | 14.38  |\n| +EMPO | 4*1024   | 12*1024  | 31.41  |\n\n\n- Limitation and Social Impact\n\nThanks for your kind reminder. **It is our duty to explicitly and well discuss the limitation and social impact.** This manuscript presents a new training algorithm that improves the reasoning capability by minimizing semantic entropy. By encouraging semantically coherent outputs, the model's reasoning capability can be boosted without external supervision. There exists several noteworthy limitation of EMPO. The unsupervised nature of EMPO removes the dependency of external supervision at the cost of potential reward hacking risks. Such limitation widely exists in unsupervised learning literature [7], but still noteworthy. Besides, as identified by [8], same as other RL for reasoning methods, EMPO may exhibit undesirable behaviors that stem from pretraining corpus of the Base model, e.g., hallucination, bias and so on. Entropy minimization may not only increase accuracy by encouraging consistency, but also make the bias more severe. Thus, more studies are neccessiated to mitigate such issues for large reasoning models. We promise to further elaborate the limitations and social impact in our revision.\n\n- Formatting concerns of tables and Appendix G.\n\nThanks for your suggestion on formatting. We will 1) update mean@16 results 2) further explain how the metrics are calculated and where the randomness comes from in our experiments 3) add a explicit limitation and negative social impact section and 4) polish section G by replacing the python code with pesudo-code which is more suitable for acedamic writing. Besides, we promise to open all our code, model weights, training logs/recipes and data once accepted to make sure that the community can reproduce our results easily.\n\n[1]  rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking, ICLR'25 (142 citations)\n\n[2] Free Process Rewards without Process Labels, ICML'25 (55 citations)\n\n[3] Preference Optimization for Reasoning with Pseudo Feedback, ICLR'25 spotlight\n\n[4] Self-Consistency Preference Optimization, ICML'25 \n\n[5] Advancing LLM Reasoning Generalists with Preference Trees, ICLR'25 (139 citations)\n\n[6] DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning, Arxiv Preprint\n\n[7] The Entropy Enigma: Success and Failure of Entropy Minimization, ICML'24\n\n[8] Are Reasoning Models More Prone to Hallucination? Arxiv preprint"}]

Author rebuttal:

Meta-review: This paper introduces Entropy Minimized Policy Optimization (EMPO), an unsupervised approach for improving LLM reasoning. The main idea is to boost a model's reasoning skills without any labeled data by rewarding it for generating semantically consistent answers to unlabeled questions. The work is well-motivated, and the method itself is quite intuitive. Its core strength lies in its novelty and potential, offering a new direction for training models on tasks where supervision is hard to come by, and the experimental results are solid, showing that EMPO can match or even beat supervised methods. The initial reviews did bring up a few good points for improvements, such as more ablation studies on the entropy filtering mechanism, a clearer comparison to self-consistency baselines, and further discussion on how the method handles larger or more difficult datasets. The authors thoroughly addressed these concerns in their rebuttal with new experiments and clarifications, which strengthened the paper. I am happy to recommend acceptance.
Decision: Accept (spotlight)