Rectifying LLM Thought from Lens of Optimization
Authors: Junnan Liu
Hongwei Liu
Songyang Zhang
Kai Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Large Lanugae Model
Large Lanugae Model Reasoning
Reinforcement Learning with Verifiable Rewards
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose RePro to rectify LLM thought and thus enhance LLM reasoning performance.

Abstract: Recent advancements in large language models (LLMs) have been driven by their emergent reasoning capabilities, particularly through long chain-of-thought (CoT) prompting, which enables thorough exploration and deliberation. Despite these advances, long-CoT LLMs often exhibit suboptimal reasoning behaviors, such as overthinking and excessively protracted reasoning chains, which can impair performance. In this paper, we analyze reasoning processes through an optimization lens, framing CoT as a gradient descent procedure where each reasoning step constitutes an update toward problem resolution. Building on this perspective, we introduce RePro (**Re**ctifying **Pro**cess-level Reward), a novel approach to refine LLM reasoning during post-training. RePro defines a surrogate objective function to assess the optimization process underlying CoT, utilizing a dual scoring mechanism to quantify its intensity and stability. These scores are aggregated into a composite process-level reward, seamlessly integrated into reinforcement learning with verifiable rewards (RLVR) pipelines to optimize LLMs. Extensive experiments across multiple reinforcement learning algorithms and diverse LLMs, evaluated on benchmarks spanning mathematics, science, and coding, demonstrate that RePro consistently enhances reasoning performance and mitigates suboptimal reasoning behaviors.

Published as a conference paper at ICLR 2026
RECTIFYING LLM THOUGHT FROM LENS OF OPTI- MIZATION
Junnan Liu
Department of Data Science and AT, Faculty of Information Technology
Monash University
junnan. 1 Liumomaahh.ee .du
Hongwei Liu & Songyang Zhang' & Kai Chen
Shanghai Artificial Intelligence Laboratory
(1 (liuhongwei, zhangsongyang, beeprr/eeecceaaaeeea...b....
Code: ott.el/..t.ssss/..t...cc./cooocc
ABSTRACT
Recent advancements in large language models (LLMs) have been driven by their emergent reasoning capabilities, particularly through long chain-of-thouge (CoT) prompting, which enables thorough exploration and deliberation. Despite these advances, long-CoT LLMs often exhibit suboptimal reasoning behaviors, such as overthinking and excessively protracted reasoning chains, which can impair performance. In this paper. analyze reasoning processes through an optimization lens, framing CoT as gradient descent procedure where each reasoning step constitutes an update toward problem resolution. Building on this perspective, we introduce REPRO (Rectifying Process-level Reward), a novel approach to refine L.M reasoning during post-training. REPRO defines surrogate objective function to assess the optimization process underlying CoTT, utilizing a dual scoring mechanism to quantify its intensity and stability. These scores are aggregated into composite process-level reward, seamlessly integrated into reinforcement learning with verifiable rewards (RLVR) pipelines to optimize LLMs. Extensive experiments across multiple reinforcement learning algorithms and diverse LLMs, evaluated on benchmarks spanning mathematics, science, and coding, demonstrate that REPRO consistently enhances reasoning performance and mitigates suboptimal reasoning behaviors.
INTRODUCTION
Recent advancements in large language models (LLMs)) have been propelled by their emergent rea- soning capabilities, enabling them to tackle complex tasks (Huang Chang, 2023; Plaat et al., 2024; Ahn et al., 2024; Ke et al., 2025; Sun et al., 2025). These capabilities are pivotal in progressing toward artificial general intelligence (AGI) (Zhong et al., 2024). State-of-the-art LLMs, such as OpenAl's o-series (OpenAI, 2024a;b; 2025), DeepSeek-RI R1 (DeepSeek-AI al., 2025), Kimi-KI (Kimi-Tean et al., 2025), and Gemini-2.5-PP (Comanici et al., 2025), leverage long chain-of-thought (CoT) prompting to enhance reasoning. This approach facilitates comprehensive exploration and reflection, yielding robust reasoning processes (Chen et al., 2025a). Such improvements stem largely from reinforcement learning with verifiable rewards (RLVR) (Schulman et al., 2017; Shao al., 2024), which enables LLMs to autonomously explore reasoning steps based on a terminal reward, fostering self-improving models with scalable reasoning during inference (Snell et al., 2024).
Despite these advancements, long-CoT CoT LLMs often exhibit suboptimal reasoning behaviors (Chen al., 2025a). A significant issue is overthinking, where models generate excessive tokens or
Work done when Junnan's 's internship Shanghai Artificial Intelligence Laboratory. Corresponding authors.

---

Published as a conference paper at ICLR 2026
protracted reasoning paths that contribute minimally to problem resolution, incurring substantial computational costs (Chen et al., 2024; Wang et al., 2025c; Sui et al., 2025). For instance, in response to simple query like "What is the answer to plus 3?" (Chen et al., 2024), certain long-CoT LLMs produce reasoning chains exceeding thousands of tokens, increasing latency and resource demands, thus limiting applicability in time-sensitive domains (Sui al., 2025).
Drawing on prior work (Feng et al., 2023; Huang et al., 2025a), we analyze suboptimal reasoning through an optimization framework, conceptualizing CoT as a task-specific variant of gradient descent, where each reasoning step represents an optimization update (Liu et al., 2025a). In this paradigm, suboptimal reasoning manifests as oscillations around saddle points or local optima. hindering convergence to the optimal solution.
To address these challenges. we propose REPRO (Rectifying Process-level Reward), novel method to rectify LLM thought during post-training. REPRO formulates a surrogate objective function, J. monitor the optimization process of CoT, measuring the LLM's confidence in the ground truth via perplexity (Jelinek et al., 1977) over the ground-truth token sequence. For a reasoning
trajectory of N steps, we compute a sequence of objective values [Jo, JI,...,, JN]] and introduce dual scoring system to assess optimization intensity and stability. These scores are combined into a composite process-level reward (Lightman et al., 2024), integrated into standard post-training pipelines (DeepSeck-A) AI et al., 2025; Shao et al., 2024; Hu, 2025) enhance reasoning. REPRO is plug-and-play, compatible with prevalent reinforcement learning algorithms.
The efficacy of REPRO is substantiated by comprehensive empirical evaluation. We validate Re- PRO through extensive experiments using reinforcement learning algorithms like PPO (Schulman al., 2017), REINFORCE+ (Hu, 2025), REINFORCE++ Baseline (Hu, 2025), and GRPO (Shao et al., 2024), across LLMs of various families and scales, including base models, supervised fine- tuned variants, and native long-Col LLMs. Evaluations on benchmarks in mathematics, science, and coding demonstrate significant improvements in reasoning performance. Quantitative and qualitative analyses further confirm REPRO'S ss efficacy in optimizing reasoning behaviors. Our contributions are: We introduce REPRO, a plug-and-play method to rectify LLM reasoning in RLVR: @ We define a surrogate objective function to model reasoning as gradient descent, with dual scoring process-level
mechanism for optimization intensity and stability, and outline its integration reward; O Extensive experiments across reinforcement learning algorithms and LLMs show enhanced reasoning performance; ! Quantitative and qualitative analyses verify REPRO's 's ability to refine LLM reasoning behaviors.
2 PRELIMINARIES
Reinforcement Learning for LLM Reasoning. Proximal Policy Optimization (PPO) (Schulman et al., 2017) is the typical and effective policy gradient algorithm for LLM post-training (Ouyang et al., 2022; Hu et al., 2025). As an actor-critic method, PPO employs a policy model (actor) to optimize a reward function and value model (critic) to estimate the value of each state. PPO employs the clipped surrogate objective function to enhance training stability by constraining the magnitude of policy updates at each iteration with a clipping range e. Given the input data distribution P and policy model TT, the objective is formally defined as:
[(0) = E.g.PPTTTT
min (peAs, clip(e,, 1 1,,1
where Pt Th (T())4, T(5)) /Te (Tee)lg, T(so) is the importance sampling coefficient to reduce the gap between the current policy and the old policy. A, denotes the advantage estimate at time step f, which is computed using Generalized Advantage Estimation (GAE) (Schulman et al., 2016). GAE is derived from the temporal difference error, 6, = Tt + >V+1 = Vz, where ra is the reward at time step I, 7 is the discount factor, and V, is the value time step Then A, is calculated by the summation of the temporal difference error over a series of time steps as: A, xc 7 76044
Critic-Free RL Algorithms for LLM Reasoning. Despite the effectiveness of PPO, it experiences high computational costs due to the trainable value model. To address this challenge, a series of critic-free RI. algorithms have been proposed, substituting the value V2 with estimated reward
2

---

Published as a conference paper at ICLR 2026
Treating Optimizstion NE
Intensity optimization
Process-Lae Level Reward
Objective Score
Assess the stability of the optimization
Trajectory
Optimization Path
OETAIL
Recitfying Process Reward
Rollout
Update
Trajectories
Outcome Reward
Policy Loss
Group
Figure 1: Illustration of the REPRO framework. We incorporate rectifying process- reward into the RLVR training to enhance LLM reasoning. Initially, we conceptualize the reasoning trajectories generated by LLMs as an optimization process of the LLMs' internal state ($ 3. 3.2). We then propose two-fold score to evaluate the optimization process and utilize this score reward to rectify the LLM thought ($ 3.3 & $ 3.4).
baseline. These include ReMax (Li et al., 2024), RLOO (Ahmadian et al., 2024), GRPO (DeepSeek- AI et al., 2025; Shao al., 2024), and REINFORCE++ (Hu, 2025). Typically, these algorithms share the following objective function:
G 2
O(0)- E.
Σ
(ps clip(es,, l+e) A;) DKi
D [eel
where 7; = (TI,...,, Ta} (-1g) denotes a group of trajectories size G generated by the existing policy model TT. At' represents the normalized advantage using an estimated reward baseline time step for the i-th trajectory. DXL [ denotes the KL divergence penalty between the current policy and the reference policy Trets with 8 as the weighting factor for this penalty term.
REPRO: RECTIFYING LLM THOUGHT
In this section, we provide the details of the proposed REPRO and the illustration of REPRO is demonstrated in Figure 1.
3.1 PROBLEM FORMULATION
A typical LLM reasoning process involves question q randomly sampled from the question dis- tribution P(Q), denoted as q ~ P(Q), and an LLM parameterized by πi. For q. a long-CoT LLM generates step-by-step reasoning sequence Thinking (typically delimited by <think> and </think> tags in current reasoning LLMs), followed by conclusion Teonclusion- forming the trajectory:
Thinninng: Teouclusion)
Te(-)g).
Following prior work (Liu al., 2025a; Wang et al., 2025a). conceptualize the decoding of Thinking as optimization process over the LLM's internal states, iteratively increasing the likelihood of the correct answer. The objective function J (TT, q. T, a), where a is the ground-truth answer, is optimized as:
B.t t 0 ij. VoJ T(S)) a), l" arg max [( q, a,)),
where i an implicit learning rate, and V.J(TO (TO. q. T())), a) denotes the implicit gradient of J(To, 4, T(s)), a)) with respect 0, as the actual optimization process is complex and nontrivial.

---

Published as a conference paper at ICLR 2026
3.2 OBJECTIVE FUNCTION DEFINITION
Although the actual optimization process is complex and nontrivial, we can define a proxy metric to observe changes in the objective function from indirect perspective. Drawing inspiration from previous work (Tang et al., 2025; Zhou et al., 2025; Yu et al., 2025b), we find that the probability of the model generating the ground truth answer serves as effective proxy for the objective function J. Formally, we define the proxy objective function as follows:
I (To, q. T(S)): a)
log mo ((()).TTi)))
(5)
Intuitively, J quantifies the model's reasoning capability given certain context. As Tst updates the model's s internal states, the probability of producing the ground-truth answer increases, thereby increasing J.
Empirical Evidence. We pro-
vide empirical evidence supporting the effectiveness of J as proxy metric. Specifically, we prompt -1-----ill----weeeee
Probelm:
Curve of -J w.rt. token position
thnn that
LRM 1.5B) (DeepSeek-AI et al., 2025) with mathematical question sampled from AIME'24 generate multiple reasoning trajectories. We select four correct trajectories, computing 7 (to demonstrate, we show the negative value of J) at each position of the tra- jectory, and plot the curve as shown in Figure 2. From Figure 2, we observe that - gradually decreases as the reasoning trajectory length increases.
D0a Lox,
Find
of thinking process
Ground
A26
Figure 2: Empirical evidence supporting - J as proxy met- ric. The left panel presents the question and its corresponding answer, while the right panel plots - function of rea- soning trajectory tokens.
This indicates that J effectively serves proxy metric for monitoring and assessing the internal states of the LLM.
3.3 QUANTIFYING OPTIMIZATION PROCESS
Leveraging the proposed objective function J. we introduce a score S designed to evaluate the optimization process by tracking the dynamics J. For a given reasoning trajectory т, a sequence of J values, denoted as (J1,J2,... J[T[}. obtained, which represents the optimization process over the generation of т. An effective optimization process should fulfiil two key conditions: 1) the value of objective function exhibits a sufficient overall increase, indicating substantial progress to the optimization objective; 2) the increase is relatively smooth, with limited oscillation near local extrema, indicating efficient optimization. Building on these criteria, we propose dual quantitative score, evaluate the optimization process. This score comprises the Magnitude Score, Smagn measuring the intensity of the optimization process (i.e.. net improvement), and the Stability Score, Sata. assessing its stability, capturing the degree of oscillatory behavior the updates.
Magnitude Score. The magnitude score, Smagn. at position & (denoted as Smagn,(0)), quantifies the increase in J along the partial trajectory Tst- To address the disparities among J values corresponding to different q, a baseline Jz(g) is introduced, defined
Jv(q) 2 J (To, q, a),
(6)
httpe:://wgginggga...c////ttts/s hwwp://///g/.//////////.. _004
4

---

Published as a conference paper at ICLR 2026
which can be interpreted as the direct probabilistic prediction from TT. Subsequently, (magn,(t) is defined as:
Smagn.(1) - tanh (Ta, 9. T(S)), a) +1) + +1c (0,
where A q, ((E),, a) = J (To, q, T(S)), a) To(g)
Jo(g)
Intuitively, Smagn.(t) is normalized measure of the relative increase of I over the baseline Jo- Normalization of this relative decrease A(To, q, Tgt, a) through the tanh function ensures the score's range is restricted to (0, 1], thus mitigating the impact of extreme values. A higher Smagn((1) signifies greater increase in the objective function, concurrently indicating that the partial reasoning trajectory Tgr yields more substantial benefits to the reasoning process.
Stability Score.
As previously stated, the Saab quantifies the stability of the optimization process. Each step is expected to serve as an effective update, progressing towards increasing the objective function. For given sequence of objective values {Fi, J........................ - Trsal we evaluate Suab.()) by examining its correlation with the corresponding indices {1, 21,,,... [Tggl as follows:
Stthh(t) Eishisissimen sign (I. sign (i -)
1 + 2 [0.1].
(8)
(TsO) (|T<E))- -)
where sign is the sign function. Theoretically, we leverage Kendall's Tau Correlation Coeffi- cient (Kendall, 1938) measure the stability of the optimization process. If each step is an effective update, Saab approaches 1, whereas ineffective updates result in a score near O. In addition, con- sidering the influence of noise, smooth can also be introduced to smooth J, such as the common EMA (Exponential Moving Average) (Hunter, 1986) smoothing equation:
Tr. -=-=a a. Ji,-11 (1 -aaa) a) JT.:
where a is the smoothing factor.
Finally, combining Smagn and Sunb, we obtain the final score S by introducing a weight factor w E [0, 1] as follows:
== - -w) Smagn + w Sstab-
(10)
3.4 LEARNING WITH RECTIFYING PROCESS-LEVEL REWARD
We shall now discuss the integration of into the RL training for LLMs. A natural approach is to employ S as process reward in reinforcement learning training. Nonetheless, since current strong reasoning I.IMs generate lengthy reasoning trajectories, computing S at each token would incur prohibitive computational overhead. Moreover, token-level S calculation could introduce excessive noise, leading to futile computations of S values that adversely affect the optimization process.
Entropy-Based Selection Strategy. Recent studies (Cui al., 2025; Wang et al., 2025b; Cheng et al., 2025) have demonstrated that token entropy serves as an effective indicator within the trajectories of advanced reasoning LLMs. High-entropy tokens act as critical decision points, guiding the model toward diverse reasoning pathways. In this study, where reasoning trajectory generation is framed as an optimization process, high-entropy tokens may cause oscillations near extrema, yielding higher value than tokens with lower entropy. Therefore, to reduce computational overhead while providing more effective process rewards, we propose an entropy- based selection strategy. Specifically, we divide the thinking tokens within the reasoning trajectory into multiple segments (e.g., partitioned by two-line-break (n)n)) considering the thinking granularity of I.LMs:
Thhiniing + {C1, , cNt.
(11)
where N denotes the number of segments. We select the top-k segments {ē,, E2,.... Tx based on the entropy of the first token of each segment:
top-k (H (c), 7 (c2,(0)) ..... H (cx,())) + {τ,, ē2.... 2x}
(12)
The rationale for this strategy is that model uncertainty increases the conclusion of these segments, which, from an optimization perspective, indicates that suboptimal optimization processes are more likely to occur, phenomenon we seek rectify.
5

---

Published as a conference paper at ICLR 2026
Integrating S Into Reward. Given the selected segments {čz, C,...., ) we compute scores {S1, S2........ .ap....,,,, S for each segment by appending the ground truth answer at the end. The rectifying process-level reward ry is computed as:
[S] Sj F2 S
if > 1, if = 1,
(13)
which signifies the gain from introducing the partial trajectory from the end of C;-1 to 2G. fg rectifies the thinking process of LLMs by penalizing thinking processes associated with suboptimal optimization while encouraging those associated with optimal optimization. Subsequently, we also involve the normalization in critic-free RL algorithms on the rectifying process-level reward F; to mitigate the probabilistic prediction mismatch between different q and facilitate stable policy updates:
r'J (orm(f7yM7...l)
(14)
where (fza); denotes the specific group for the normalization of r and refer Appendix B.1 for more details. We separate the normalization of process-level reward from outcome reward to prevent interference with the correctness reward from noise signals. For a token T(t) where index (cg-L.de--D)) < S index (c,,())) and index((r) denotes the index of token x, the rectifying process-level advantage A, and overall advantage A, are defined A, ش A.
A + a A, where A is the verifiable outcome advantage to measure the correctness of the overall trajectory, computed from the correctness of reasoning trajectories, and a is the weight factor.
By synthesizing all propositions and definitions outlined above, the LLM parameters are updated using the computed advantage A, following the clipped surrogate objective specified in Equation (2).
Empirical Example For Understanding F. To illustrate the effectiveness of the rectifying process- level reward F, we present an empirical example. As shown in Example 3.1, we prompt the LRM to answer a question, compute for each reasoning step, and identify steps with low rectifying process- level rewards. We find that steps with low î typically involve self-doubt or redundant re re-examination, contributing little to the reasoning process and increasing inefficiency. Conversely, steps with high correspond to critical computational actions or preliminary conclusions that significantly advance reasoning. Thus, we propose that penalizing low-ř steps and incentivizing high-f steps can optimize the model's 's reasoning behavior.
Example 3.1: Empirical Example for Rectifying process-Leve Reward
QUESTION:
There exist real numbers r and y, both greater than 1, such that
log, (v/") = log, (zx 10.
Find ry.
SELECTED REASONING PROCESS WITH HIGH/LOW RECTIFYING PRUCESS-LEVEL REWARD:
(f = 0.143) Alright, so I have this problem here: There exist real numbers x and both greater than 1. ... (f = -0.217) Hmm, okay. Let's see. I remember that logarithms can be tricky, but maybe can use some logarithmic identities to simplify these equations. I think the change of base formula might be helpful here. The change of base formula says that log, maybe I can rewrite both equations using natural logarithms make them easier handle.
las lnb right? So
(i = -0.0533 That seems a bit cleaner. So, let me write those. ... (F.(f- -0.060) This trial and error method isn't 't working. Maybe should consider different approach. ... (f 0.085) Left side: 1(1 + 10e-1) R + 10 x 0.3679 Re 1 + 3.679 zs 4.679, which is greater than 3.2188. ... (F -0.113) Wait, that can't 't be. Wait, perhaps my approximation is off. ... Wait, perhaps my approach not efficient. (F = -0.06)) Maybe I can use linear approximation or to set up an equation. (F 0.053) Which the same equation as before. So, this equation satisfied when xy 25. So, zy 25 is the solution. ... 0.092) Therefore, despite the complexity of the original logarithumic equations, the product xy simplifies directly to 25.

---

Published as a conference paper at ICLR 2026
Table 1: Performance of REPRO on evaluation reasoning benchmarks. We report the average performance for 16 runs on AIME24 and AIME25, and runs others. We abbreviate LMB as LiveMathBench v202505, LCB as BinchodeBench v6, RF++ as REINFORCE++, and RF++ B as DEINFORCE++ Baseline. denotes the in-domain evaluation benchmark and & denotes the out-of-domain benchmark.
AIME24
ALME25 % Avg@ 16
MATH500 % Avg@4 4
LMB
GPQA-D & % Avg@
MBPP & % Avg@-1 1
LCB & % Avg
Methods
% Avg 16
Avg@11
CoonSee--I-Distil--POe
Original
10.5
32.7
61.8
14.9
PPO
34.3
24.4
86.9
T4.0
32.1
61.0
17.6
+RePRo
36.3
27.7
87.7
16.5
32.8
61.1
16.7
RF++
31.0
23.5
85.4
11.5
33.1
61.0
15.1
-REPRO
33.1
26.7
86.1
12.0
34.3
61.9
16.1
RF+ B
33.1
26.7
86.1
12.0
34.3
61.9
16.1
+REPRO
35.6
26.5
87.2
15.6
35.4
63.9
17.3
GRPO
32.9
25.3
86.0
TO.3
334.
62.3
15.2
+REPRO
36.0
26.5
87.1
14.3
37.0
65.4
18.4
Qwen3-1.7B
Original
46.3
36.1
93.0
18.3
39.3
66.9
30.6
GRPO
47.3
34.8
93.4
18.8
38.3
67.5
32.2
+REPRO
49.8
37.9
94.1
19.5
39.1
68.8
32.0
4
EXPERIMENTAL RESULTS AND ANALYSIS
4.1 SETUP
Evaluation Benchmarks. We evaluate all models across three domain benchmarks: 1) Mathematical Reasoning Benchmarks, which include AIME24, AIME25, and MATH500 (Hendrycks et al., 2021b), and LiveMatBBench (Liu et al., 2024); Scientific Reasoning Benchmarks, represented by GPQA- Diamond (Rein et al., 2023); 3) Code Reasoning Benchmarks, which comprise MBPP (Austin et al., 2021) and LiveCod<Bench (Jain et al., 2025).
Implementation Details. We conduct experiments on several prominent L.MMs, including DeepSeek- R1-Distill-QWenn-1.. (DeepSeek-Al al., 2025), distilled from DeepSeek-RI (DeepSeek-Al et al., 2025), Qwen3-1.7B (Yang et al., 2025), Qwen3-8B (Yang et al., 2025), S.nnyuan-I.BBInsttee cent, 2025), and MobileLLM-RRI-PP 1-950M (Liu et al., 2025c). The training corpus, proposed by Luo et al. (2025), comprises approximately 40,000 high-quality mathematical samples. More details of the model training are provided in Appendix B.4. For evaluation, we configured the sampling temperature to 0.6, top-p to 0.95, and top-k to 40. It worth mentioning that different LLMs have their recommended sampling parameters, and different sampling parameters may have certain impact performance. However, to ensure consistent evaluation pipeline, use same sampling parameters for all models and focus on the relative improvement in performance. Also, to minimize variance, report average performance relative to the size of each benchmark.
(Ten-
4.2 EFFECTIVENESS AND GENERALIZATION OF REPRO
Table illustrates the performance of REPRO on evaluation reasoning benchmarks. From the experimental results, we have the following findings.
REPRO Achieves Consistent Improvements Across RL Algorithms. As presented in Table I, RE- PRo consistently enhances the performance of various RL baselines, including PPO, REINFORCE++. REINFORCE+ Baseline, and GRPO. For instance, when applied to PPO on the DeepSeek-Ri. Distill-Qwen-Il 1.5BB backbone, REPRO improves AIME24 accuracy from 34.8 36.3 and MATH500 from 86.9 to 87.7. These results demonstrate that REPRO'S 's improvements are independent of the RI. algorithm, offering versatile, plug-and-pllay strategy enhance reasoning performance.
REPRO Generalizes to Out-of-Domin Benchmarks. Beyond in-domain reasoning tasks like AIME and MATH500, REPRO exhibits robust generalization out-of-domain benchmarks, such as
7

---

Published as a conference paper at ICLR 2026
Reasoning Tokens
PPO
GRPO
RF++ Base Line
5000
2000
7000
1000
6600
Buseling Line
PPO RePro
GRPO RePro
0000
RFae
Base Line RePro
300
3011
Training Step
Figure 3: Dynamics of the reasoning token cost during the training process of REPRO DeepSeek- RI-Distill- -eaill-Qwen--..BB.
45
0.7
Performance
GRPO
RePro
0.6
GRPO RePro
GRPO
PPO RePro
Backtracking
42-
RF++ Baseline RePro
0.4
41
0.3
88
8.2
38
0.1
37
0.0
0.8
9.2
8.4
9.6
8.8
1.8
100
200
300
400
560
W
Training Step
Figure 4: Ablation experiments of weight and REPRO weight a.
Figure 5: Dynamics of the backtracking pattern.
science reasoning benchmark GPQA-Diamond and code reasoning benchmarks including MBPP and LiveCodeBench. These findings underscore REPRO's ability to extend benefits beyond mathematical reasoning to diverse reasoning tasks like science reasoning, programming, and code generation tasks, highlighting its broad applicability.
REPRO Generalizes to Diverse LLMs. REPRO's effectiveness extends across different LLMs including DeepSeek-R1 ---t-t-------------- Sestill-Qeen-... 1.5B and Qwen3-1.7B. We also provide results of different architectures and sizes in Appendices C. and C.2. REPRO achieves consistent improvements across LLMs of different families and sizes. This scalability across model architectures and sizes indicates that REPRO serves general mechanism for enhancing reasoning capabilities, rather than an optimization specific to a particular backbone.
4.3 ABLATION STUDY
Impact of Magnitude Score & Stability Score. Magnitude Score and Stability Score are utilized to assess the intensity and stability of the optimization process, respectively, with the coefficient w balancing their contributions. To investigate their necessity, we analyze model performance across different values, as shown in Figure The dotted lines, segmented by dots, represent the average performance of models trained with varying values on four mathematical reasoning benchmarks. Across all w values, REPRO consistently outperforms the baseline, confirming the importance of both Magnitude Score and Stability Score. Notably, performance is slightly higher when w is lower, suggesting that the Magnitude Score, which measures optimization intensity, may play a more critical role in enhancing model performance.
Impact of the REPRO Weight a. To balance the outcome advantage and REPRO advantage, we introduce the coefficient set 0.1. In this part, evaluate the sensitivity of REPRO to variations in a. As depicted in Figure 4, the solid lines, segmented by dots, represent the average performance of models trained with different values across four mathematical reasoning benchmarks. The results
8

---

Published as a conference paper at ICLR 2026
PPO
GRPO
RF++ Baseline
PPO RePro
GRPO RePro
Baseline RePro
15800
12500
#Tokens
10000
IN
..33
.00
7500
5000
2500
AIME24
AIME25
MATHSGO
LNB
GPQA-D
HBPP
LCB
Figure Comparison of the inference reasoning token cost of SeepSeek-RR--Dssiil--eee------
Table 2: Ablation study of the number of selected segments k.
AIME24
AIME25 % Avg 16
MATH500
LMB
GPQA-D & % Avg@
MBPP % Avg604
LCB
% &vg 16
% Avg@4
4
% Avg604
ReepSee-RR-Diistill
Diseill-Qee--LSB
16
36.0
26.5
87.1
14.3
37.0
55.4
78.4
20
36.5
26.2
87.3
14.7
37.5
55.0
19.1
30
36.9
27.2
87.8
15.1
37.6
55.3
19.5
demonstrate that REPRO maintains relatively stable performance across various a values, indicating its robustness to changes balance coefficient.
Impact of the Number of Selected Segments k: The number of selected segments k is a key factor in balancing performance and training cost REPRo. shown in Table 2, while increasing is yields slight performance improvements, these gains are marginal. In practical applications, finding an optimal trade-off between training cost and performance is crucial. Additional experimental results and analysis are provided in Appendix C.4.
4.4 FURTHER ANALYSIS OF THE CHANGING THINKING BEHAVIORS
In this section, we will conduct quantitative and qualitative analyses of reasoning behaviors beyond reasoning performance to demonstrate how REPRO has improved the LLM thought.
REPRO Improves the Token Efficiency of Reasoning. We analyze the token cost of REPRO, examining its impact on reasoning efficiency. Figure 3 illustrates the dynamic changes in reasoning token cost during training with and without REPRO. The results show that REPRO effectively reduces token cost training progresses. Additionally, we compare REPRO's token cost against baselines in Figure Figure 6, demonstrating significant reductions in inference token cost across all benchmarks. These findings indicate that REPRO promotes more concise and effective reasoning trajectories, enhancing both efficiency and practical applicability.
REPRO Reduces Suboptimal Thinking Behaviors. To gain deeper insight into the impact of REPRO beyond mere token counts, we analyze changes in the thinking patterns of LLMs. Every 10 steps during the REPRO training process, we instruct the LLM to perform reasoning on the AIME24 benchmark and apply the prompt proposed by Gandhi et al. (2025) (detailed in Appendix B.5) for thinking pattern recognition. Figure 5 illustrates the evolving proportion of the backtracking pattern, which typically indicates ineffective and excessive reasoning. As training progresses, the prevalence of this suboptimal pattern significantly decreases compared with vanilla GRPO, highlighting REPRO's effectiveness in enhancing the reasoning behaviors and patterns of L.MMs.

---

Published as a conference paper at ICLR 2026
Case Study. We also present a qualitative analysis of the responses generated by REPRO, as illustrated in Case D.I, Case D.2, and Case D.3. Compared to responses from the LLM trained with a vanilla RL algorithm, REPRO significantly reduces inefficient and suboptimal backtracking and reasoning (highlighted in orange), resulting in a more linear and efficient thinking process. Furthermore, by mitigating oscillations around "saddle points", REPRO-trained models exhibit fewer errors (marked in red), enhancing overall reasoning accuracy.
5 RELATED WORK
Demystifying Reasoning Trajectories of LLMs. The advent of powerful reasoning L.Mss, enhanced by reasoning trajectories, has spurred extensive research to uncover the underlying mechanisms within these trajectories. Foundational studies (Yun et al., 2020a;b) have established that sufficiently universal approximators for continuous expressive Transformers (Vaswani al., 2017) can Subsequent analyses have explored their sequence-to-sequenee mappings over compact domains. computational power and expressive limitations (Dehghani et al., 2019; Bhattamishra et al., 2020; Yao et al., 2021; Hewitt et al., 2020; Weiss et al., 2021; Merrill et al., 2022; Chiang et al., 2023; Giannou et al., 2023; Liu et al., 2023). Recent research has shown that Transformers are capable of meta- learning optimization algorithms,, such as gradient descent, within their forward trajectories (Gatmiry et al., 2024; Huang et al., 2025b). The most related works (Dai et al., 2023; Liu et al., 2025a) treat the reasoning trajectories of LLMs as optimization processes for their parameters and internal states, providing a solid foundation for our work.
Promoting and Improving LLM Reasoning. RI. has emerged as a powerful paradigm for enhancing the reasoning capabilities of LLMs, with a notable approach being RLVR (OpenAI, 2024a;b; 2025; DeepSeek-Al et al., 2025; Kimi-Team et al., 2025; Team, 2025; Yang et al., 2025; Comanici et al., 2025; ByteDance Seed, 2025). Many of these methods leverage test-time scaling, a process where models engage in iterative self-improve eef--improvement by refining their internal thought processes, exploring diverse strategies, and executing self-correction, often guided by CoT prompting. The resulting models, often termed long-CoT LLMs, have shown remarkable performance improvements on complex tasks in domains like mathematics, science, and code. More recent work has focused on refining the RL algorithms themselves. For instance, methods such as Dr. GRPO (Liu et al., 2025d), VAPO (Yue et al., 2025), and DAPO (Yu et al., 2025a) introduce algorithmic adaptations, particularly in sampling strategies and advantage estimation, to further elevate the reasoning performance of LLMs. A significant limitation of long-CoT LLMs is their computational inefficiency, often resulting in "overthinking". characterized by the generation of redundant tokens or unnecessary reasoning steps that may lead to errors (Chen et al., 2024; Liu et al., 2024; Wang et al., 2025c). To address this issue, one research direction (Hou et al., 2025; Aggarwal & Welleck, 2025; Liu et al., 2025b; Wu et al., 2025b; Wang al., 2025a) focuses introducing new forms of regularization rewards based on length or information during training to reduce invalid token consumption. Another research approach (Y (Yang al., 2025; Fang et al., 2025; Wu et al., 2025a; Zhang et al., 2025) aims to learn adaptive policies that control the reasoning process by altering the reasoning pattern according to question difficulty or user instructions.
CONCLUSION
In this paper, we propose REPRO, novel framework designed refine the reasoning processes of LLMs from an optimization perspective. We conceptualize CoT reasoning as an optimization process and introduce two scores to evaluate its intensity and stability. These scores are integrated as process- reward into the training pipeline of RLVR. Extensive experiments across diverse reasoning benchmarks demonstrate the effectiveness of REPRO. Furthermore, we illustrate how REPRO enhances the reasoning behavior of L.LMs, improving their efficiency.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This study focuses solely general research tasks and poses risks to health, safety, personal security, or privacy. No human participants are involved, and no new datasets are released as part of this work. Furthermore, the research does not include potentially harmful insights, methods, or applications, nor does it raise concerns related to privacy, security, legal compliance, or research integrity. Consequently, we anticipate no ethical risks or conflicts of interest. We are committed to maintaining the highest standards of scientific integrity and adhering to ethical guidelines throughout all stages of the research process.
REPRODUCIBILITY STATEMENT
We provide comprehensive description of the proposed REPRO framework in $ 3. To ensure reproducibility, we detail implementation specifies, including datasets, model configurations, and additional information, in $ 4.1 and Appendix B. Key code implementations are included in the supplementary materials, with the complete code to be released publicly upon acceptance of the paper.
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Flaine Ya Le, Guillaume Leclere, James Park Lennon, Scott Lessans, Mario Lezcano- Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz. Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song. Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace. Xin Wang. Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojeiech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss- 120b & gpt-oss-20b model card. CoRR, abs/2508. 10925, 2025. C.3
Pranjal Aggarwal and Sean Welleck. LI: controlling how long A reasoning model thinks with reinforcement learning. CORR, abs/2503.04699 2025. 5, A,
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basies: Revisiting reinforce-style optimization for learning from human feedback in llms. In ACL (1), pp. 11248-12200 Association for Computational Linguistics, 2024. 2, B.1
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. EACL (Student Research Workshop), pp. 225-237. Association for Computational Linguistics, 2024. 1
Joshua Ainslie, James Lee-Thee Thorp, Michiel de Jong, Yury Zemlyanskiy. Federico Lebrón, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In EMNLP, pp. 4895-4901. Association for Computational Linguistics, 2023. C.1
11

---

Published as a conference paper at ICLR 2026
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry. Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR. abs/2108.07732, 2021. 4.1, B.3
Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of transformers to recognize formal languages. In EMNLP, pp. 7096-7116. Association for Computational Linguistics, 2020. 5
ByteDance Seed. Introduction to techniques used in seed1.6. https::/ httpss:/sseddbbyteeanccccom
en/seedl 6, 2025. 5
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. CoRR, abs/2503.0956m 2025a.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang. Zhaopeng Tu, Haitao Mi, and Dong Yu. Do NOT think that much for 2+3=? on the overthinking of ol-like lims. CoRR, abs/2412.21 2187, 2024. 1,5
Zhipeng Chen, Yingqian Min, Beichen Zhang. Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, Lei Fang, Zhongyuan Wang, and Ji-Rong Wen. An empirical study on eliciting and improving rl rl-like reasoning models. CoRR, abs/2503.04542 2025b. B.2
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. CoRR, abs/2506.14758, 2025. 3.4
David Chiang, Peter Cholak, and Anand Pillay. Tighter bounds on the expressivity of transformer encoders. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 5544-5562. PMLR, 2023. 5
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 10. 14168, 2021. C.I
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit S. Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan- Jiang Jiang, Krishna Haridasan, Ahmed Omran, Nikunj Saunshi, Dara Bahri, Gaurav Mishra, Fric Chu, Toby Boyd, Brad Hekman, Aaron Parisi, Chaoyi Zhang, Kornraphop Kawintiranon, Tania Bedrax-Weiss Oliver Wang, Ya Xu, Ollie Purkiss, Uri Mendlovic, Iai Deutel, Nam Nguyen, Adam Langley, Flip Korn, Lucia Rossazza, Alexandre Ramé, Sagar Waghmare, Helen Miller, Nathan Byrd, Ashrith Sheshan, Raia Hadsell Sangnie Bhardwaj. Pawel Janus, Tero Rissa, Dan Horgan, Sharon Silver, Ayzaan Wahid, Sergey Brin, Yves Raimond, Klemen Kloboves, Cindy Wang, Nitesh Bharadwaj Gundavarapu, Ilia Shumailov, Bo Wang, Mantas Pajarskas, Joe Heyward, Martin Nikoltchev, Maciej Kula, Hao Zhou, Zachary Garrett, Sushant Kafle, Sercan Arik, Ankita Goel, Mingyao Yang, Jiho Park, Koji Kojima, Parsa Mahmoudieh, Koray Kavukcuoglu, Grace Chen, Doug Fritz, Anton Bulyenov, Sudeshna Roy, Dimitris Paparas, Hadar Shemtov, Bo-Juen Chen, Robin Strudel. David Reitter, Aurko Roy. Andrey Vlasov, Changwan Ryu, Chas Leichner, Haichuan Yang, Zelda Marict, Denis Vnukov, Tim Sohn, Amy Stuart, Wei Liang, Minmin Chen, Praynaa Rawlani, Christy Koh, JID Co-Reyes, Guangda Lai, Praseem Banzal, Dimitrios Vytiniotis, Jieru Mei, and Mu Cai. Gemini 2.5: Pushing the frontier with advanced reasoning. multimodality. long context, and next generation agentic capabilities. CoRR, abs/2507.0622 2025. 1,5
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models. CoRR, 11//2555.22611 2025. 3.4
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why GPT learn in-context? language models secretly perform gradient descent as meta-optimizers. In ACL (Findings). pp. 4005-4019. Association for Computational Linguistics, 2023. 5
12

---

Published as a conference paper at ICLR 2026
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 933-941. PMLR, 2017. C.1
Dee,Seek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao. Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang. Minghui Tang, Meng Li, Miaojun Wang, Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning. CoRR, abs/2501. 2025. 1, 2, 3.2, 4.1, 5, A
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In ICLR. OpenReview.net, 2019. 5
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Thinkless: LLM learns when to think. CoRR, abs/2505.13379, 2025. 5
Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: A theoretical perspective. NeurlPS, 2023.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. CoRR, abs/2503.01307, 2025. 4.4, B.5
Bofei Gao, Feifan Song, Zhe Yang. Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni- math: A universal olympiad level mathematic benchmark for large language models. In ICLR. OpenReview.net, 2025. B.2
Khashayar Gatmiry, Nikunj Saunshi, Sashank Reddi, Stefanie Jegelka, and Sanjiv Kumar. Can looped transformers learn to implement multi-step gradient descent for in-context learning? In ICMI.. OpenReview.net, 2024. 5
Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. ICML, volume 202 of Proceedings of Machine Learning Research, pp. 11398-11445 1442. PMLR, 2023. 5
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR. OpenReview.net, 2021a. C.1
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Erie Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurlPS Datasets and Benchmarks, 2021b. 4.1, B.3, C.I
John Hewitt, Michael Hahn, Surya Ganguli, Percy Liang, and Christopher D. Manning. Rnns can generate bounded hierarchical languages with optimal memory. In EMNLP, pp. 1978-2010 Association for Computational Linguistics, 2020. 5
Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long thainhtf-thought of Ilms via reinforcement learning. CoRR, abs/2504.01296. 2025. 5, A,
13

---

Published as a conference paper at ICLR 2026
Jian Hu. REINFORCE++: A simple and efficient approach for aligning large language models. CoRR, abs/2501 1.0362,,,002,,... 2025. 2, B.I, B.4
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeue Shum. Open-reasoner--zeoo An open source approach to scaling up reinforcement learning on the base 2025. 2
model. CoRR,
Jianhao Huang, Zixuan Wang, and Jason D. Lee. Transformers learn to implement multi-step gradient descent with chain of thought. In ICLR. OpenReview.net, 2025a.
Jianhao Huang, Zixuan Wang, and Jason D. Lee. Transformers learn implement multi-step gradient descent with chain of thought. CoRR, abs/2502.21212, 2025b. 5
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In ACL (Findings), pp. 1049-1065. Association for Computational Linguistics, 2023. 1
Stuart Hunter. The exponentially weighted moving average. Qual. Technol, 18(4):203 200- 1986. 3.3
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In ICLR. OpenReview.net, 2025. 4.1, B.3, C.I
Frederick Jelinek, Robert Mercer, Lalit Bahl, and James Baker. Perplexity-a measure the difficulty of speech recognition tasks. Acoust. Soc. Am, 62((1):S63-SS5) S63, 1977.
Zixuan Jiang, Jiaqi Gu, Hanqing Zhu, and David Z. Pan. Pre-rmsnorm and pre-crmsnorm transformers: Equivalent and efficient pre-In transformers, In NeurIPS, 2023. C.1
Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, Caiming Xiong, and Shafiq Joty. A survey of frontiers in LLM reasoning: Inference scaling, learning to reason, and agentic systems. Trans. Mach. Learn. Res., 2025, 2025.
Maurice Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81 93, 1938. 3.3
Kimi-Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang. Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao. Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang. Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma. Shupeng Wei, Sihan Cao, Siying Huang. Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang. Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang. Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang. Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with 1lms. CoRR, abs/2501. 2025. 1,5
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, pp. 61 1-666. ACM, 2023. A, B.4
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi thi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In ICML. OpenReview.net, 2024. 2, B.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. ICLR. OpenReview.net, 2024. 1, A, B.3
14

---

Published as a conference paper at ICLR 2026
Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. In ICLR. OpenReview.net, 2023.
Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, and Kai Chen. Are your Ilms capable stable reasoning? CoRR, abs/2412.13147, 2024. 5, B.3
Junnan Liu, Hongwei Liu, Linchen Xiao, Shudong Liu, Taolin Zhang, Zihan Ma, Songyang Zhang, and Kai Chen. Deciphering trajectory-aide aided LLM reasoning: An optimization perspective. CoRR, abs/2505. 199155 2025a.
3.1,5
Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang, and Junxian He. Learn to reason efficiently with adaptive length-based based reward shaping. CoRR, abs/2505.15612, 2025b. 5, 3
Zechun Liu, Ernic Chang, Changsheng Zhao, Chia-Jung Chang, Wei Wen, Chen Lai, Rick Cao, Yuandong Tian, Raghuraman Krishnamoorthi, Yangyang Shi, and Vikas Chandra. Mobilellm-rl: Model card. https: /ttp::://huggtaaa http://////aagaaaccc.ccoc//////a mobii 1m- 2025c. 4.1, C.I
Zichen Liu, Changyu Chen, Wenjun Li. Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. CoRR, abs/2503.207833 2025d. 5, B.1
Michael Luo, Sijun Tan, Justin Wong. Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler:
Surpassing ol-preview with a 1.5b model by scaling rl. hattps:/pretty-eeee http:o/ww.....eeeeaaaaaeaaeaaaaa
Model goodel-be boydel-by-al --ll----ylylinggeee RL- (0w.0000000022660000ee0000006030 2025. Notion
Blog. 4.1, B.2
William Merrill, Ashish Sabharwal, and Noah A. Smith. Saturated transformers are constant-depth threshold circuits. Trans. Assoc. Comput. Linguistics, 10:843-856, 2022.
Meta. The Ilama 4 herd: The beginning of a new era of natively multimodal innovation. https: ww..eta.com/blllt.. htt.eet...obooo///eeeeeeee /llaaoog//llaaaaaa/ 4 4-multt -tddmatmmmaa--tnneeeeeeeeeeeeee 04. C.I
2025. Accessed: 2025-
OpenAL. Learning to reason with llms. httpss///oeea..coww//n/////eaeet
reason-with- wwwwwwhhhhmssss.. l1ms/. 2024a. Accessed: 2024-09. 1,5
OpenAl. Introducing openai o3 and o4-mini. http::////peaaaa.oaooo/////tttta o o3-and-o4-minh 2024b. Accessed: 2024-12. 1,5
OpenAl. Gpt-5 and the new of work. httpss///ppeaalccoooo///x////ses o of-work/, 2025. Accessed: 2025-08. 1,5
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022. 2
Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Bäck. Reasoning with large language models, survey. CoRR, a55/2407.11511, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level moogle-proof q&a benchmark. CoRR, abs/2311. 1. 12022 2023. 4.1, B.3
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In ICLR (Poster), 2016.
15

---

Published as a conference paper at ICLR 2026
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1107.066 00007000447 2017. 1,2, B.4
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang. Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300,, 2024. 1, 2, B.1, B.4
Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, (bss/1911002111 2019. A
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. In EuroSys, pp. 1279-1297. ACM, 2025. B.4
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, 03324, 2024. 1
Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputingg 568: 168:127063, 2024. C.I
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Ben Hu. Stop overthinking: survey on efficient reasoning for large language models. CoRR, abs/2503. 1112503..4119 64119, 2025.
Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, Yue Wu, Wenhai Wang. Junsong Chen, Zhangyue Yin, Xiaozhe Ren, Jie Fu, Junxian He, Wu Yuan, Qi Liu, Xihui Liu, Yu Li, Hao Dong, Yu Cheng, Ming Zhang, Pheng-Ann Heng, Jifeng Dai, Ping Luo, Jingdong Wang. Ji-Rong Wen, Xipeng Qiu, Yike Guo, Hui Xiong, Qun Liu, and Zhenguo Li. A survey of reasoning with foundation models: Concepts, methodologies, and outlook. ACM Compul. Surv. (7(11)2788::1227788
2025.
Yunhao Tang, Sid Wang, and Rémi Munos. Learning to chain-of taain-of-thought with jensen's 's evidence lower bound. CoRR, abs/2503.11911 2025. 3.2
Qwen Team. Qwq-32b: httt::::///eemmmgiih.b.....
Embracing the power of reinforcement learning, March 2025. URL
32b/.
Tencent. Hunyuan-L.8b. httpps///sttuaooooo/////ttttaaaa 2025. 4.1, C.1
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pp. 5998-6008, 2017. 5
Jingyao Wang, Wenwen Qiang, Zeen Song, Changwen Zheng, and Hui Xiong. Learning to think: information-thhorete reinforcement fine-tuning for llms. CoRR, abs/2505. 10425, 2025a. 3.1,
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In ACL (1), pp. 9426 9439. Association for Computational Linguistics, 2024. A
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. CoRR, abs/2506.01939 2025b. 3.4
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang. Zhaopeng Tu, Haitao Mi, and Dong Yu. Thoughts are all over the place: On the underthinking of ol-like llms. CoRR, abs/2501.18885. 2025c. 1,5
Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 10800110900. PMLR, 2021.
Siye Wu, Jian Xie, Yikai Zhang, Aili Chen, Kai Zhang, Yu Su, and Yanghua Xiao. ARM: adaptive reasoning model. CoRR, abs/2505. 20255,, 2025a. 5
16

---

Published as a conference paper at ICLR 2026
Xingyu Wu, Yuchen Yan, Shangke Lyu, Linjuan Wu, Yiwen Qiu, Yongliang Shen, Weiming Lu, Jian Shao, Jun Xiao, and Yueting Zhuang. LAPO: internalizing reasoning efficiency via length-adaptive policy optimization. CoRR, abs/2507.15758 2025b. 5
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jian Yang, Jiaxi Yang, Jingren Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Kegin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, abs/2505 00s50555500988, 2025. 4.1, 5, B.5, C.I, C.2, C.3
Shunyu Yao, Binghui Peng. Christos H. Papadimitriou, and Karthik Narasimhan, Self-attention net- works can process bounded hierarchical languages. ACL/EJCNLP, pp. 3770-3785. Association for Computational Linguistics, 2021. 5
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue. Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang. Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma. Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: an open-source LLM reinforcement learning system at scale. CoRR. abs/2503. 14476, 2025a. 5, B.1
Tianyu Yu, Bo Ji. Shouli Wang. Shu Yao, Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, Maosong Sun, and Tat-Seng Chua. RLPR: extrapolating RLVR to general domains without verifiers. CoRR, abs/2506. 01125006.1224,,... 2025b. 3.2
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Cheng-Xiang Wang. Tiantian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma. Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang. Yonghui Wu, and Lin Yan. VAPO: efficient and reliable reinforcement learning for advanced reasoning tasks. CoRR, abs/2504.05118 18, 2025. 5, B.I
Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Are transformers universal approximators of to-euuueeeee functions? In ICLR. OpenRe-
view.net, 2020a.
Chulhee Yun, Yin- Min-een Chang, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. O(n) connections are expressive enough: Universal approximability of sparse transformers. In NeurlPS, 2020b. 5
Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng. and Juanzi Li. Adaptthink: Reasoning models can learn when to think. CoRR, abs/2505. 111/1/755511177, 13417, 2025. 5,
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization. CoRR, abs/2507. 18071, 2025. B.1
Lianmin Zheng. Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In NeurIPS, 2024. A
Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, Chao Cao, Hanqi Jiang, Hanxu Chen, Yiwei Li, Junhao Chen, Huawen Hu, Yihen Liu, Huaqin Zhao, Shaochen Xu, Haixing Dai, Lin Zhao, Ruidong Zhang, Wei Zhao, Zhenyuan Yang, Jingyuan Chen, Peilong Wang, Wei Ruan, Hui Wang, Huan Zhao, Jing Zhang, Yiming Ren, Shihuan Qin, Tong Chen, Jiaxi Li, Arif Hassan Zidan, Afrar Jahin, Minheng Chen, Sichen Xia, Jason Holmes, Yan Zhuang, Jiaqi Wang. Bochen Xu, Weiran Xia, Jichao Yu, Kaibo Tang, Yaxuan Yang, Bolun Sun, Tao Yang, Guoyu Lu, Xianqiao Wang, Lilong Chai, He Li, Jin Lu, Lichao Sun, Xin Zhang, Bao Ge, Xintao Hu, Lian Zhang, Hua Zhou, Lu Zhang, Shu Zhang,
17

---

Published as a conference paper at ICLR 2026
Ninghao Liu, Bei Jiang, Linglong Kong, Zhen Xiang, Yudan Ren, Jum Liu, Xi Jiang, Yu Bao, Wei Zhang, Xiang Li, Gang Li, Wei Liu, Dinggang Shen, Andrea Sikora, Xiaoming Zhai. Dajiang Zhu, and Tianming Liu. Evaluation of openai ol: Opportunities and challenges of AGI. CoRR, abs/2409. 1184409.18888 18486, 2024.
Xiangxin Zhou, Zichen Liu, Anya Sims, Haoman Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. CoRR, abs/2505.2144 2025. 3.2
18

---

Published as a conference paper at ICLR 2026
Appendix
Table of Contents
Discussions
20
B More Implementation Details
20
B.1 Implementation of Normalization Rectifying Process Reward in Critic-Free Free RL
20
B.2 Training Data
21
B.3 Evaluation Benchmarks
21
B.4 Training Parameters
22
B.5 Prompt for Thinking Pattern Recognition
24
Additional Experimental Results
24
C.I REPRO LLMs of Diverse Families
24
C.2 REPRO on Larger LLMs
26
C33 REPRO for -eto-RLVR
26
C,4 Ablation of the Number of Selected Segments
27
C.5 Entropy Dynamics of REPRO
28
Cases
28
E Limitations
42
F LLM Usage
42
19

---

Published as a conference paper at ICLR 2026
A DISCUSSIONS
Computational Efficiency. While REPRO necessitates additional computation due to the entropy- based selection strategy and reward calculation, these forward processes exhibit significant prefix overlap. Modern LLM inference engines like vLLM (Kwon et al., 2023) and SGLang (Zheng et al., 2024) expedite forward computation by caching key-value pairs (KV cache) (Shazeer, 2019). In training process, we utilize the inference engines supported by the training framework to accelerate the computation S.
Comparison with Process Reward Models. Process reward models (Lightman et al., 2024; Wang al., 2024) have been proposed to offer process-level granularity in supervising the training of reasoning LLMs. However, these methods struggle to generate effective supervision signals due to the complex dependencies among different processes and the final answers (DeepSeek-Al et al., 2025). In contrast, the method proposed in this paper, REPRO, neither relies on additional models nor provides absolute supervision signals of correctness or incorrectness. Instead, it evaluates the contribution of each process to reasoning and offers relative advantages and disadvantages through group normalization, significantly reducing noise in the signals provided for training.
Comparison with Efficient Reasoning Methods. Recent studies on efficient LLM reasoning (Aggarwal & Welleck, 2025; Hou et al., 2025; Liu et al., 2025b) emphasize the inclusion of penalty coefficient for response length during the post-training process of LLMs, which rewards shorter, correct trajectories and penalizes longer ones. These methods only perform coarse-grained length- based evaluation of the model's trajectories, which can easily lead to incorrect penalties for necessary and correct trajectories, thereby having a negative impact on the model's performance (Liu et al., 2025b). Alternatively, our method provides more nuanced process reward, enhancing the LLMs' capability and efficiency by refining its reasoning patterns with finer granularity. As shown in Table 3, REPRO significantly outperforms salient baselines focusing reasoning efficiency. Compared to the Vanilla GRPO baseline, REPRO achieves superior performance across all benchmarks (e.g.. +3.1% AIME24) while reducing token consumption by approximately 50% (12,367 to 6,158 tokens), In contrast to seficiency-focusse methods like ThinkPrune and Laser, which suffer significant performance drops achieve lower latency, REPRO successfully decouples efficiency from accuracy degradation. For instance, while LI-Max matches our token efficiency, it severely lags in reasoning capability (26.7% vs. 36.0% on AIME24), highlighting REPRO's ability to condense reasoning without information loss. Similarly, REPRO surpasses AdaThink by clear margin of 4.3% on AIME24 while requiring 28% fewer tokens, demonstrating more effective internal optimization than external stopping criteria.
Table 3: Comparison bewteen REPRO and ThinkPrune (Hou et al., 2025), L1 (Aggarwal & Welleck, 2025), Laser (Liu et al., 2025b), and AdaThink (Zhang et al., 2025).
AIME24
AIME25
MATH500 % Avg@ f
GPQA-D
Avg #Tokens
Methods
Avg@16
Avg@161
Avg@41
Original
30.6
24.8
84.4
32.7
10,089
Vanilla GRPO
32.9
25.3
86.0
34.5
12,367
ThinkPrune-4k
31.7
20.0
84,5
33.2
8,376
1.1-Max
26.7
17.9
85.3
34.1
6,249
Laser-D-L4096
27.1
17.5
85.0
34.2
5,713
Laser-DE-L4096
27.1
22.5
84.7
32.5
5,862
AdaThinksn.os
31.7
25.9
82,5
34.1
8,549
REPRO
36.0
26.5
87.1
37.0
6,558
B MORE IMPLEMENTATION DETAILS
B.1 IMPLEMENTATION OF NORMALIZATION RECTIPYING PROCESS REWARD IN CRITIC-FREE RI.
In this section, we present the implementation details for normalizing rectifying process rewards in several representative critic-free RI. algorithms.
20

---

Published as a conference paper at ICLR 2026
GRPO and Its Variants. For GRPO (Shao al., 2024), normalization is applied across all segments within the trajectory group G for each question 9, as follows:
ド mean (frmm)isssa std (fft.m hmttk.
(15)
where fj, denotes the normalized reward for the j-th segment of the i-th trajectory in the group G. This normalization also applies to the variants GRPO, such as Dr.RRPO (Liu et al., 2025d), VAPO (Yue al., 2025), DAPO (Yu al., 2025a), and GSPO (Zheng al., 2025).
REINFORCE++. In contrast to GRPO, REINFORCE+ (Hu, 2025) normalizes the reward for each segment of each trajectory i across the full batch B:
T mean (frrmsstsssssmssss std (f (
(16)
considering all segments in all trajectories within the batch B.
RLOO. Similar to GRPO, RLOO (Ahmadian et al., 2024) performs normalization within the group G for each question q follows:
f,.--
(17)
K:G 1)
[E(A] m
ReMax. In ReMax (Li et al., 2024), we normalize the rectifying process reward by the mean of all rectifying process rewards from the trajectory generated by greedy decoding:
(18)
[E[K]
where in denotes the rectifying process rewards of the trajectory generated by greedy decoding.
B.2
TRAINING DATA
We utilize DeepScaleR- Prreeiele-- -aatsstt proposed in Luo et al. (2025) for all model training. The dataset consists of approximately 40,000 unique mathematics problem-ans answerr pairs compiled from:
American Invitational Mathematics Examination problems (1984-2023).
American Mathematics Competition problems (before 2023).
Omni-MATH dataset (Gao al., 2025).
Still dataset (Chen et al., 2025b).
B.3
EVALUATION BENCHMARKS
The following details describe our evaluation benchmarks:
AIME24, This dataset consists of 30 challenging problems from the 2024 American Invitational
Mathematics Examination.
AIME25. This dataset consists of 30 challenging problems from the 2025 American Invitational Mathematics Examination.
MATH500. The original MATH dataset (Hendrycks et al., 2021b) contains 12,500 problems from American high school mathematics competitions. MATH500 (Lightman et al., 2024), widely used subset of its test split, includes only Level 5 questions, which we adopt in this paper.
LiveMathBench. LiveMatBBench (Liu et al., 2024) is continuously updated benchmark of challenging mathematical problems. We use the 202505 hard split, which contains 100 high- quality English questions.
21

---

Published as a conference paper at ICLR 2026
The Graduate-Level Google-Proof Q&A Benchmark (GPQA) (Rein al., 2023) is multiple-choice science question-anse -aswwerinn dataset designed to be resistant to web search. We evaluate on its diamond subset, which comprises 198 questions.
GPQA.
MBPP. The Mostly Basic Programming Problems (MBPP) dataset (Austin et al., 2021) evaluates programming models on elementary Python tasks. It was created via crowdsoureing, with workers generating problems and solutions under specified guidelines. Problem statements were later refined to remove ambiguity, and selected items underwent manual review and editing to ensure clarity and accuracy of test cases.
LiveCodeBench. LiveCodeBench (Jain et al., 2025) is designed to provide comprehensive and contamination- free evaluation the coding abilities of large language models. incorporates problems from LeetCode, AtCoder, and Codeforces.
B.4 TRAINING PARAMETERS
We set the hyperparameters w = 0.5 and = 0.1, selecting the top-10 segments for each reasoning trajectory. Training utilized the veRL (Sheng et al., 2025) and vLLM (Kwon al., 2023) frameworks. Table 4, Table 5, and Table present the training parameters for PPO (Schulman et al., 2017), GRPO (Shao al., 2024), and REINFORCE+ baselines (Hu, 2025), respectively.
Table 4: Training Parameters of PPO.
Parameters
Values
Batch Size
256
Number of Rollout Per Question
8
Rollout Temperature
1.0
Rollout Top-p
1.0
Maximum Number of Generation Tokens
16384
Learning Rate
le-6
KL Loss Coefficient
0.001
Emin 6max
0.2
0.28
X
1.0
1.0
Gradient Clipping
1.0
Number of Training Steps
500
Table 5: Training Parameters of GRPO and REINFORCE++ Baseline.
Parameters
Values
Batch Size
256
Number of Rollout Per Question
8
Rollout Temperature
1.0
Rollout Top-p
1.0
Maximum Number of Generation Tokens
16384
Learning Rate
le-6
KL Loss Coefficient
0.001
Emin
0.2
6max
0.28
Gradient Clipping
1.0
Number of Training Steps
500
22

---

Published as a conference paper at ICLR 2026
Table 6: Training Parameters of REINFORCE++.
Parameters
Values
Batch Size
256
Number of Rollout Per Question
1
Rollout Temperature
1.0
Rollout Top-p
1.0
Maximum Number of Generation Tokens
16384
Learning Rate
1e-6
KL Loss Coefficient
0.001
6min
0.2
6max
0.28
Gradient Clipping
1.0
Number of Training Steps
500
23

---

Published as a conference paper at ICLR 2026
B.5 PROMPT FOR THINKING PATTERN RECOGNITION
Prompt B.1 illustrates the prompt proposed in Gandhi et al. (2025) for recognizing beneficial thinking patterns in the reasoning process. this paper, we utilize Dwen3-235B-A22B 2 (Yang et al., 2025) to perform the recognition
Prompt B.1: Prompt for Thinking Pattern Recognition
Below is a -frasoii-- generated by a Language Model when attempting to solve math problem. Evaluate this recasonn-n to determine whether it demonstrates beneficial problem-solving behaviors that deviate from typical linear, monotonic reasoning patterns commonly observed in language models. <start.of_reasoning {input}
<endLof ser.offrresooinng.
Specifically, actively identify and emphasize beneficial behaviors such as:
Explicitly revising approaches upon identifying errors dead ends (e.g.. "This approach won't work because...).
Backtracking:
Verification: Systematically checking intermediate results or reasoning steps (e.g., "Let's verify this result by...").
Subgoal Setting: Breaking down complex problems into smaller, manageable steps (e.g.. "To solve this, we first need to...").
Enumeration: Solving problems by exhaustively considering multiple cases or possibili-
ties.
Additionally, remain attentive to and encourage the identification of other beneficial behaviors not explicitly listed here, such as creative analogies, abstraction to simpler cases, or insightful generalizations.
Important: Clearly specify each beneficial behavior you identify. Provide explicit examples from the reasoning chain. If no beneficial behaviors observed, explicitly return empty list.
Provide your evaluation clearly, formatted as follows:
C ADDITIONAL EXPERIMENTAL RESULTS
C.I REPRO ON LLMS DIVERSE FAMILIES
In this section, to further verify the universality and generalization ability of REPRO, we conduct experiments on LLMs of different families.
We include three LLMs in our experiments: Qwen3-1.7B (Yang al., 2025), Hunyuan-1.8B Instruct (Tencent, 2025), and MobileLLM-RR 1--000 (Liu et al., 2025c). Qwen3-1.7BB part of the dense LLM with 28 layers and 32k context length, incorporat- ing Grouped Query Attention (Ainslic et al., 2023), SwiGLU (Dauphin et al., 2017), Rotary Positional Embeddings (Su al., 2024), and RMSNorm (Jiang al., 2023). Hunyuan- In from the Hunyuan series, is 32-layer transformer-based dense LLM that also employs Grouped Query Attention (Ainslie et al., 2023) and supports 256K context window, maintaining stable performance from the MobileLLM series, is efficient reasoning model based on the Llama4 (Meta, 2025) architecture. Pre-trained on approximately 2T high-quality tokens and with fewer than 5T total training tokens, MobileLLM-RR L--eeLLMM-11-5000 R1-9550 achieves performance comparable or superior to Qwen3-0.66
LLMs.
Qwen3 series, is a transformer-base
on long-text tasks. MobileLLM-R1-99000
which was trained on 36T tokens, across benchmarks such MATH (Hendrycks et al., 2021b), GSM8K (Cobbe et al., 2021). MMLU (Hendrycks al., 2021a) 1a), and LiveCodeBench (Jain et al., 2025). We utilize the same hyperparameters as shown in Table 5.
Performance. As shown in Table 7, REPRO consistently enhances performance across various LLM families. For Qwen3-1.7B, integrating REPRO with reinforcement learning algorithms such as PPO, REINFORCE+ Baseline (RF++B), and GRPO yields substantial improvements across nearly
24

---

Published as a conference paper at ICLR 2026
Table Performance of REPRO on LLMs of diverse families. benchmark and denots the out-of- benchmark.
denotes the in-domain evaluation
AIME24
AIME25
MATH500
LMB
GPQA-D
MBPP & % Avg@4 1
Methods
% Avg@11 161
% Avg@ 161
% Avg@4
Avg@41
% Avg@4
% Avg@4
Qwen3-1.7B
Original
46.8
36.1
91.5
18.8
39.5
66.9
30.6
PPO
45.2
36.5
92.1
20.5
40.2
68.2
31.3
+REPRo
49.0
92.4
18.8
69.1
31.5
Ri++ B
46.5
34.4
91.5
18.5
38.3
66.1
31.1
+REPRO
49.8
39.0
92.7
22.0
39.8
71.7
32.4
GRPO
47.3
34.8
93.4
18.8
38.3
67.5
30.2
+REPRo
49.8
37.9
94.1
19.5
39.1
68.8
32.0
Hunyuan-1.88l
Original
38.8
32.8
81.3
15.0
38.0
73.1
24.5
PPO
42.1
33.3
42.1
75.8
25.9
+REPRO
43.5
32.1
84.3
20.5
42.7
76.2
26.3
RF+ B
42.5
33.3
85.5
17.5
42.4
76.3
26.5
+REPRO
43.1
77.0
26.2
GRPO
43.3
32.7
85.6
16.0
43.6
75.9
27.1
+REPRO
44.6
33.5
84.2
17.5
44.8
77.5
27.7
MobileLLM-RR-9555
Original
15.8
18.1
76.4
10.5
19.2
58.2
18.0
PPO
23.2
22.5
81.4
14.5
22.6
63.6
19.8
+REPRO
24.2
23.1
83.5
15.8
24.1
65.1
21.4
all benchmarks. Specifically, PPO+REPRO improves AIME24 performance from 45.2 to 49.0 and AIME25 from 36.5 37.5, while RF++B+REPRO further boosts AIME25 39.0, demonstrating its effectiveness on challenging mathematical reasoning tasks. Comparable gains, up to +1.5 points, are observed on MBPP and LiveCodeBench. For Hunyuan- (.uuan-1.8B... 1.88-- improvements; for instance, GRPO+REPRO enhances AIME24 from 43.3 3 to 44.6 and AIME25 from 32.7 33.5, while RF++B+REPRO increases AIME24 from 42.5 to 44.3 and MBPP from 76.3 to 77.0. Similarly. for the efficienccy-rrinnted MobileLL.M-R 1A0MM-R1-55000 -950M, REPRO provides consistent benefits, with PPO+REPRO improving AIME24 from 23.2 to 24.2 and AIME25 from 22.5 to 23.1. On broader reasoning tasks like MATH500 and from +1.0 to + +1.6 points. These results confirm that REPRO is effective not only for larger-scale dense models but also generalizes compact, efficiency-optimized architectures.
REPRO also delivers
REPRO achieves gains ranging
Token Efficiency. We evaluate the inference token efficiency of REPRO across LLMs from different families. As shown in Figures 7 and 8, REPRO consistently outperforms baseline methods in terms of token efficiency. For instance, PPO+REPRO requires fewer tokens than standalone PPO while achieving higher performance, demonstrating REPRO's ability to enhance reasoning efficiency across diverse LLM architectures.
PPO
GRPO
RF+ Beseline
RePro
FSS RF+ Baseline RePro
20000
15000
#Tokens
5000
ATMEZA
AIME26
MATHEO0
GPQA-D
kopp
LCB
Figure Comparison of the inference reasoning token cost REPRO and basclines on Qwen3-1.7B.
25

---

Published as a conference paper at ICLR 2026
RF++ Baseline
PPO RePro
GRPO RePro
Rf++
Daseline RePro
15000
#Tokens
10000
5000
AIMEZA
AINE26
MATH600
LMB
GPQ4-D
NoPp
LcB
Figure 8: Comparison the inference reasoning token cost of REPRO and baselines on Hunyuan- 1.8t-Instrue..
C.2
REPRO ON LARGER LLMs
PPO
K
15900
#Tokens
10000
AIME24
AIME25
MATHS0D
LMB
GPQA-D
HBPP
LCB
Figure 9: Comparison of the inference reasoning token cost of REPRO and baselines on Qwen3-8B. In this section, to further verify the effectiveness of REPRO on LLMs of larger scale, we conduct experiments LLMs of larger scale.
LLMs. We include Qwen3-8B (Yang et al., 2025) in our experiments. Qwen3-8B, part of the Qwen3 series, is dense LLM with 36 layers and 128k context length. We utilize the hyperparameters specified in Table 5 for training.
Performance. As shown in Table 8, REPRO consistently outperforms baseline methods when applied to Qwen3-8B. For example, GRPO+REPRO enhances performance on AIME24 from 75.6 to 76.1 and on AIME25 from 67.9 to 68.5, demonstrating improved mathematical reasoning capabilities. Significant gains are also observed in science and code reasoning tasks: GPQA-Diame improves from 59.5 60.4, MBPP from 68.8 72.1, and LiveCodeBench from 52.2 53.4. These results confirm that REPRO delivers consistent performance improvements across mathematical, scientific, and coding reasoning tasks, even when applied to larger-scale models like Qwen3-8B.
Token Efficiency, As shown in Figure 9, REPRO consistently outperforms vanilla GRPO in terms of token efficiency when applied to Qwen3-8B. For instance, GRPO+REPRO requires fewer tokens than standalone GRPO while achieving higher performance, demonstrating REPRO's 's ability to enhance reasoning efficiency across larger-scale models.
C.3 REPRO FOR ZERO-RLVR
Previous discussions and experiments related to REPRO mostly based LLMs with deep thinking capabilities. In this section, we also conduct relevant experiments on Rero-RLVR trained starting from base L.Mss.
26

---

Published as a conference paper at ICLR 2026
Table 8: Performance of REPRO on evaluation reasoning benchmarks with Qwen3-8B. in-domain evaluation benchmark and denots the out-of-domain benchmark.
denotes the
AIME24
AIME25
MATH500
LMB Avg@4
GPQA-D Avg@4
MBPP & % Avg@4 t
LCB
Methods
Avg@
Avg@16
% Avg@4
Avg@4
@wen3-8B
Original
75.2
66.5
96.8
35.3
58.7
70.0
49.8
GRPO
75.6
67.9
97.3
35.1
59.3
68.8
32.2
+REPRo
76.1
68.5
97.2
35.8
60.4
72.1
53.4
Table 9:
Performance of REPRO on evaluation reasoning benchmarks with Owen3-4B-Ban denotes the in-domain evaluation benchmark and & denots the out-of-domain benchmark.
AIME24
AIME25
MATH100
LMB
GPQA-D Avg@11 f
MBPP & % Avg@0 1 t
Methods
Avg 16
Avg@ 16
Avg@4 f
Avg@-1 f
Avg@4
Owen3-4B-BBse
Original GRPO
11.5
7.9
68.3
7.5
16.7
25.0
15.9
23.3
19.1
83.6
12.5
39.7
58.7
15.3
+REPRO
21.0
16.7
83.0
14.5
40.5
59.3
17.3
LLMs. To be more specific, we train Qwen3-4B-Bal Base model with GRPO and GRPO REPRO, we utilize the same hyperparameters as shown Table 5.
Performance. As presented in Table 9, REPRO and GRPO achieve comparable performance on both in- domain and out-of-domain benchmarks. In AIME24, GRPO slightly outperforms REPRO, but REPRO surpasses GRPO in AIME25. Both methods perform comparably in MATH500, with REPRO showing marginal advantage. REPRO demonstrates stronger performance in LiveMath- Bench and the Original method. In MBPP and LiveCodeBench, RE- PRO and GRPO perform closely, with REPRO slightly leading in LiveCodeBench. Overall, REPRO and GRPO show competitive results, with REPRO displaying slight edge in certain out-of-domain tasks.
5000
G27g
noho
TTokens
significantly outperforming
Training
Step
Figure 10: Token growth of REPRO and GRPO on base models.
Analysis of Token Cost Growth. Notably, Figure 10 illustrates that the token cost growth for REPRO relatively modest compared to baselines. This suggests that REPRO promotes more efficient reasoning patterns, reducing suboptimal thinking behaviors. Consequently, REPRO emerges as a promising approach for training low-cost reasoning LLMs, such as the Instrutt series (Yang et al., 2025) and GPT-OSS-Low (Agarwal et al., 2025).
C.4 ABLATION OF THE NUMBER OF SELECTED SEGMENTS
The number of selected segments & is a critical hyperparameter in REPRO. A larger k provides more precise process-level supervision, potentially improving performance, but it also increases computational overhead. We conducted experiments on DeepSeek-RI-Distil---eeee -iii-istilll--ee--..55 to evaluate performance with & (5, 10, 20, 30) (in this paper, we set 10). The results, shown in Table 10, indicate that increasing & slightly enhances performance. Specifically, the model achieves consistent improvements across most benchmarks to 36.9 at & 30, and MATH500 rising from 86.3 to 87.8. Similar upward trends are observed on -PaA-DDamond, MBPP, and LiveCodeBench. These findings suggest that larger number of selected segments indeed provides more reliable supervision signals, leading to better reasoning and problem- solving abilities. However, the gains become marginal beyond 20, indicating diminishing returns relative to the additional computational cost. Consequently, we adopt 10 as balanced choice that achieves strong performance while maintaining training efficiency. However, given the additional training overhead, striking balance between computational cost and performance gains is essential.
h grows, with AIME24 improving from 35.7 h:
27

---

Published as a conference paper at ICLR 2026
Table 10: Ablation study of the number of selected segments N.
AIMEZ4 A Avg@16
AIME25 A % 1vg@16
MATH500
LMB
GPQA-D + Avg@1
MBPP & Avg@1
LCB
N
Avg@11
Avg@1 f
Avg@11
Deep Beeeee-R---issiee -i---iiiiDll--enne 1.e
35.7
25.4
86.3
14.6
36.6
18.0
20
36.5
26.2
87.3
14.7
37.5
55.0
19.1
30
36.9
27.2
87.8
15.1
37.6
55.3
19.5
Entropy
GRPO
RoPro
80
75
100
125
175
209
Training Step
Figure 11: Entropy dynamics of RePro and vanilla GRPO DeepSeek-Rl--Disill--eenn
C.5 ENTROPY DYNAMICS OF REPRO
In Figure 11, we compare the entropy dynamics of GRPO REPRO (red dashed line) with standard GRPO (green solid line). Vanilla GRPO exhibits rapid and continuous decay in entropy -oropiy-dropping from ~0.275 to ~0.255, indicating a tendency toward faster policy collapse, REPRO successfully sustains significantly higher entropy level, remaining stable around 0.28 for the first 100 steps and finishing above 0.27. This empirical evidence confirms that REPRO effectively mitigates the rapid loss of diversity in the policy, allowing the model to sustain meaningful exploration for longer duration and thereby avoiding the premature plateauing often observed in standard RL training settings.
D CASES
Case D.1: :: Case in Math Reasoning
QUESTION
Jen enters lottery by picking 4 distinct numbers from S (,,2,3, 9, 10). numbers are randomly chosen from S. She wins a prize if at least two of her numbers were 2 of the randomly chosen numbers, and wins the grand prize if all four of her numbers were the randomly chosen numbers. The probability of her winning the grand prize given that she won prize is where m and are relatively prime positive integers. Find m n.
ANSWER
116
BASELINE RESPONSE
Okay, so Jen is playing this lottery game where she picks 4 distinct numbers from the set S, which is 1,2.3....9,.00 Then, 4 numbers are randomly chosen from S. She wins a prize if at least two of her numbers match of the randomly chosen numbers, and she wins the grand prize all four of her numbers are ones chosen. The question asking given that she won a prize. They want the answer in the form m/n where m and n are coprime, and then find m n.
probability that she wins grand prize
28

---

Published as a conference paper at ICLR 2026
Wait, maybe made a mistake in computing P(Prize). Let me check again.
Wait, the number of ways to have at least two matches. Alternatively, maybe can compute it directly.
Let me check.
Number of ways for exactly 2 matches: C(422)C(6..). Because choose 2 numbers from her 4, and numbers from the remaining 6. Then exactly matches: ((,.)))(( Exactly matches: C(4,4)"C(600)=1. Therefore, total number of ways for at least 2 matches is [C(4.2))C(6.2) + C(4,3)*C(6,1) C(4,4)+C(660))
Let me compute that:
C(4.2)*C(6,2) * =
C(4.3)*C(611) 24
C(4.4)*((6.0) = 1=1
Total: 90 + 24 1 115. Which matches the previous result. Therefore. P(Prize) is 115/210 = 23/42. So that's correct. Then P(Grand Prize) is 1/210. Therefore, the conditional probability is (1/210)((3/42)
((1210))(42/23) 42/(210*23)= 1/(5*23)= = 1/115. Therefore, m/n 1/115, so m + n is + 115= 116. Wait, but that seems low. Is this correct? Wait, 1/115? Let me think again.
Wait, but let me think about it in another way. Suppose we have her numbers fixed. How many lottery draws result in her winning prize? 115, How many of those result in her winning the grand prize? 1. So, the conditional probability is 1/115. So, that would be the answer? But maybe made mistake in computing P(Prize). Wait, let me check once again.
Alternatively, maybe need to compute the conditional probability differently. Wait, no. The conditional probability is the number of favorable outcomes (grand prize) divided by the number of outcomes where she won prize. Which is 1/115. So, if that's the case, then n is 1 + 115 116. However, I need to make sure that this is correct.
Wait, but let me think another way. Suppose we think of the problem as: given that there at least two matches, what is the prohability that all four are matches. So, in other words, given that there are at least two matches, what's the chance that it's exactly four matches. Which is equivalent to the probability of exactly four matches divided by the probability of at least matches. Which exactly what calculated.
Alternatively, maybe can compute the conditional probability as follows: Given that there are at least two matches, what is the probability that all four are matches. So, in terms of combinations, the number of ways to have exactly four matches is 1. The number of ways to have at least two matches is 115. Therefore, the probability is 1/1 So, that seems be correct.
But wait, let me think again. Let me think of the possible overlaps. For example, if she has numbers, and the lottery has 4 numbers. The number of ways that they share exactly k numbers is C(4, k)*C(6, k-k). So, for k=2, 3, 4. Which is exactly what did. So, for k=2, For k=4, C(4.4 (44,))((660)=1111))
For k=3, So total 115. So, the number of ways for exactly matches is 1. Therefore. the conditional probability is 1/115. Therefore, answer is 1/115. Then m + n 1 + 115 = 116. So, is that the answer?
(443))*((61)=4446-22)
But wait, I feel like 1/1 is quite small. Let me check with an example. Let's say, for instance, if there are more numbers, but let me think of a simpler case. Suppose the lottery was choosing 2 numbers from 4, and she picks numbers. Then, the probability of winning the grand prize given that she won prize. Let's compute that. Total combinations: C(4.2)=6. Probability of grand prize 1/6. Probability of winning a prize is the same as winning the grand prize, since to win a prize, you need at least two matches. Wait, in this casc, since she picks two numbers, and the lottery picks two numbers. So, if she wins a prize, it's exactly the grand prize. Therefore, the conditional probability is 1. Which 1/1, which is correct. But we use the same formula, P(Grand Prize)/(Priize) = (1/6)(1/6)= 1. Which correct. So that works.
Another example: suppose the lottery is choosing 3 numbers from 4, and she picks 2 numbers. Wait, mayhe not. Let me think of another case where the numbers are different. Let me think of the original problem again.
Alternatively, maybe there is mistake in assuming that number of ways for at least two matches is 115. Wait, let me check the numbers again. The total number of possible lottery draws is 210. The number of ways with exactly 0 matches is C(6,4)=15. Exactly match is C(4,
So, exactly 0 or 1 is 1.5 80= 95. Therefore, exactly 2,3,4 matches is 210 95 115. Which is correct. Therefore, the number of ways for least two matches 115. So, she has 115 ways to win a prize, and only of those is the grand prize. Therefore, the conditional probability is 1/115. Therefore, the answer is 1 + 115 116.
But let me check with another approach. Suppose we think of the conditional probability as follows: Given that there are at least two matches, what is the probability that all four matches. So, the probability equal to the number of ways to have exactly four matches divided by number of ways to have at least two matches. Which is 1/ 115, So, same answer.
29

---

Published as a conference paper at ICLR 2026
Alternatively, maybe there's a different way to compute the conditional probability. Let me think. Suppose we think of the problem as: given that she has at least two matches, what is the probability that she has all four matches. So, maybe using hypergeometric distribution? Alternatively, think of the conditional probability as follows. Let me think of the possible overlaps. So, given that there are at least two matches, the probability that all four are matches. So, the conditional probability is equal to the probability of all four matches divided by the probability of at least two matches, which is again 1/210 divided by I 115/210 = 1/115. So same result.
Alternatively, maybe there's an error in the problem interpretation. Let me check the problem again. "Jen enters a lottery by picking 4 distinct numbers from S=1.2.3....., 90. 4 numbers randomly chosen from She wins a prize if least two her numbers were 2 of the randomly chosen numbers, and wins the grand prize if all four of her numbers were the randomly chosen numbers."
Wait, the wording says "at least two of her numbers were 2 of the randomly chosen numbers". Wait, that wording is a bit confusing. Wait, does it mean that she has at least two numbers matching? Or does it mean something elsc?
Wait, the original problem says: "She wins prize if at least two of her numbers were 2 of the randomly chosen numbers, and wins the grand prize all four of her numbers were the randomly chosen numbers." Wait, maybe there's is a translation issue or wording issue. Let me parse that again. "at least two of her numbers were 2 of the randomly chosen numbers".. Wait, that seems like possible misphrase. Maybe it's supposed be "at least two her numbers were among the randomly chosen numbers", i..., she has at least two matches. Because if you have "at least two of her numbers were 2 of the randomly chosen numbers". that might be confusing. For example, if she has 4 numbers, and the lottery has 4 numbers, then "at least two of her numbers were 2 of the randomly chosen numbers"- but that seems like it's saying that two of her numbers are equal to two of the randomly chosen numbers? Which is the same as saying that there are at least two matches. So, maybe that's 's just a confusing way of phrasing "at least two matches". Therefore, the original interpretation is correct.
Therefore, the answer would be 1/115, so + == 116. However, I have feeling that maybe I made a mistake here. Let me check with actual numbers.
Suppose that Jen has numbers 1,2,3,4. What is the probability that the lottery has at least two these numbers?
Total number of possible lottery draws: 210. Number of favorable: 115, So, if we pick random lottery draw, there's ' 1 115/210 chance it has at least two numbers in common with hers. Now, given that, the chance that it's all four numbers is 1/1 15. So, for example, if there are 115 such cases, only one of them the grand prize. So, that seems correct. But maybe there's ' an error in the calculation of the number of ways for at least two matches. Wait, let me think again. If she has 4 numbers, and the lottery draws 4 numbers. The number of ways they share exactly numbers is C(4, k)* * C(6, 4 k). Wait, but wait, the total number of numbers not in her set is 10 4 = 6. Therefore. we want exactly matches, we need to choose k numbers from her 4, and 4- numbers from the remaining 6. Therefore, the formula is correct. Therefore, for k=2; C(4,))(((622==6* 15=90. For k=3: C(6, (006611)4466222 6=24. For k=4; ((4,4))((6,0)=1111)) Total: 90 + 24 1 = 115. Which is correct. Therefore, the numbers are correct.
...
Same result. Therefore, I think that is correct.
But wait, let me check with actual numbers. Suppose the lottery draws numbers that have exactly two matches with Jen's 's numbers. How many such cases are there? 90. Similarly, exactly three matches: 24, exactly four: 1. So, if you are told that there is at least two matches, then the probability that it's exactly four is 1/(90 +
1) = 1/115. So, that seems correct.
Therefore, think that 1/115 is correct, so the answer 1 115 = 116. However, to make sure, let me check if the question says "at least two of her numhers were 2 of the randomly chosen numbers" -mayee there's 's misinterpretation here. Mayhe it's supposed to be that she has exactly two numbers matching? But the problem says "at least two of her numbers were 2 of the randomly chosen commeers"wwlin confusing. Wait, maybe it's supposed to be "at least two of her numbers were among the randomly chosen numbers". --- which would be the standard "at least two matches". Since the original wording is a bit awkward. If that's 's the case, then our calculation is correct.
Alternatively, if the problem had said "exactly two of her numbers were among the randomly chosen numbers", ,, then the probability would be different. But since says "at least two", then our calculation correct.
Alternatively, let me check the problem statement again:
"She wins prize if at least two of her numbers were 2 of the randomly chosen numbers, and wins the grand prize if all four of her numbers were the randomly chosen numbers."
Wait, the wording is strange. Maybe it's supposed to say "at least two of her numbers were among the randomly chosen numbers". Because "were 2 of the randomly chosen numbers" is awkward. Maybe it's a translation error or a typo. If we take it as "at least two of her numbers were among the randomly
30

---

Published as a conference paper at ICLR 2026
chosen numbers", then our calculation correct. Otherwise, if it's something else, answer would be different.
But given the context of lotteries, it's standard that winning prize would be having least two numbers matching. So, I think that the original problem is just using awkward wording, and the correct interpretation that wins prize she has at least two matches. Therefore, our calculation correct.
Therefore, I think the answer is 1/115, so m + = 116. However, to check once more, let me think of the total number of possible cases where she has at least two matches. 115. The number of those cases where she has all four matches is So, the probability 1/115. Therefore, the answer 116.
wait, just thought of something. Wait, the grand prize considered because if you have all four numbers, then you have at least two. Therefore, the conditional probability is indeed the number of grand prize cases divided by the number of prize cases. Which is 1/115. Alternatively, maybe the problem is from official souree, and maybe the answer is different? Wait, but since can't check that, need to rely my calculations.
subset
prize? Yes.
Alternatielly, maybe made a mistake in calculating the number of ways for exactly two matches? Wait, let me check that again. For exactly two matches: choose 2 numbers from her 4, and 2 numbers from the remaining 6. So, C(4.2)*C(6,2) Since there are numbers in her set, and the lottery has numbers. So, if exactly So, yes, that correct. her 4, and from the remaining ((6.)) Exactly four matches: So, total 90 + 24 + 1 = 115. So, that seems correct.
in her then the other must be from the remaining numbers. "C(6)))=661
Similarly, for exactly three matches: choose from
Same result. Therefore, I think it's safe go with 1/115. Therefore, the answer is 116.
Final Answer 116
</think>
To solve this problem, we are asked to compute the conditional probability that Jen wins the grand prize, given that she wins prize in lottery.
())(}
6
= 6 15 90
***xxatlyy 3 matches**; Choose of her numbers and 1 of the remaining 6 numbers:
())
24
3
.**Exxctly 4 = Choose all 4 of her numbers and none from the remaining 6:
4
(
1
So, the total number of favorable outcomes for winning a prize
90 24 1 = 115
Thus, prohahility winning prize
115 P(Prize)
210
Step Compute the Conditional Probability
Now, using the formula for conditional probability:
P(Grand Prize)
1
P(Grand Prize Prize)
210
P(Prize)
115
115
210
Step Final Answer
We are told the probability is of the form m, where m and n are relatively prime positive integers. Here, m = 1, n = 115, and they are coprime. Therefore, the final answer is:
115
116
Total Tokens: 8767
31

---

Published as a conference paper at ICLR 2026
REPRO RESPONSE
Okay, so I have this problem here about Jen entering lottery. She picks 4 distinct numbers from set S which is 1,2, 3, ..., 9, 10. Then, numbers randomly chosen from S. She wins a prize if least two her numbers are among the chosen ones, and she wins the grand prize all four of her numbers are the chosen ones. need to find the probability that she wins the grand prize given that she won a prize. The answer should be in the form m/n where m and n are coprime, and then find m n. Hmm, okay. So. this is conditional probability problem. The formula for conditional probability is P(A[B)= P A B)/ (B). this case, event A her winning the grand prize, and event is her winning a prize. So, we need Pigrand prize - prize) = P(grand prize) P(prize).
So, let me first figure out what P(grand prize) is. That would be the probability that all four numbers she picked are the ones that were randomly chosen. Since she's 's's picking 4 numbers out of 10, the total number of possible combinations C(10,4). The number of favorable outcomes for grand prize is I (since there's 's only one set 4 numbers that would match exactly). So, P(grand prize) 1 C(10,4). Now, P(prize) is the probability that at least two of her numbers are among the four chosen. Hmm, okay. So, this is the probability that in her 4 numbers, 2, 3, or 4 are selected. So, I need to calculate the number of ways she can have exactly 2. exactly 3, or exactly 4 matching numbers with the randomly chosen set. Then, divide that by the total number of possible lottery combinations, which is C(10,4). So, let's compute the number of favorable outcomes for prize. That would be:
Number of ways to choose exactly 2 correct numbers: C(4,2) C(6,2). Because she needs to choose 2 correct numbers from her 4, and the remaining 2 numbers from the remaining 6 (since 10 40-4: 40 6). Wait, is that right? Let think. If she has 4 numbers, and of them are correct, then the other must be from the remaining numbers. So, yes, C(4.2) C(6.2).
Similarly, for exactly 3 correct numbers: C(4.3) C(6,1). Because she needs to choose correct numbers from her 4, and the last number from the remaining 6.
And for exactly 4 correct numbers: C(4,4) a C(6,0) I, since she has to pick all 4 numbers correctly. So, total number of favorable outcomes for prize is: C(4.2))C(6.2) C(4.3)*C(6.1) + C(4,4)
Let me compute these values.
First, C(4.2) is 6. C(6.2) is 15, 6*15=90.
Next, C(4.3)=4, 1(6,1)=6, so 4"6=24. Then, C(4,4)=1, C(6,0)=1, so 1*1=1.
So total favorable outcomes for prize: 90 + 24 115.
Therefore, P(prize) is 115 / C(10,4). Ieet me compute C(10,4): 10! / (4!6!) = (10*9=8*7)
210. So, 115 210. We can reduce this fraction: 115 divides by 5, 210 divides by 5, so 23/42. So, P(prize) 23/42.
Wait, hold on, 1 thought P(grand prize) was 1 210, but that seems too small. Wait, no, Pigrand prize) 1/C /(104) = 1 210. So, P(grand prize)= = 1/210.
But then, P(grand prize- prize) is (1/210) / (23/42). So, that is (1/210) * (42/23) = (42)(210*23). Simplify 42/210: that's 's 1/5. So, (1/5) 23 1/(5*23) 1/115.
So, 1/1 So, 1. So, m + n is + 115 116. Wait, but that seems a bit counterintuitive. Because the probability of winning the grand prize given that you won prize is low, but intuitively, if you won prize, which is at least two numbers correct, you have better chance of having all four correct. So, why it 1/1 That seems low.
1, is 115. Are they coprime? Yes, 1 and 115 share no common divisors other than
Wait, let me think again.
Wait, P(grand prize) is 1/210. P(prize) is 115/210.
So. 1/210 divided by 115/210 is 1/115. So, that's correct.
So, 1/115 is the correct probability,
+ n= +15 116.
Wait, 1/115 is the probability, so m is 1, n 115, which are coprime, is 116.
Hmm, okay. So, maybe that correct. think did it correctly.
Alternatively, perhaps can think of it as the probability that her numbers are all correct given that at least two are correct.
So, the probability that all four correct is 1/ /C100) 1/210.
The probability that at least two are correct is 115 /100 Therefore, the conditional probability is 1/115. Yes, that seems correct.
Therefore, + is 1 115 = 116.
Final Answer 116
</think>
Jen enters lottery by picking distinct numbers from the set S = (1.2,3..... 10). Four numbers are randomly chosen from S. She wins a prize if least two of her numbers are among the chosen
32

---

Published as a conference paper at ICLR 2026
ones, she wins the grand prize all four her numbers are chosen ones. need to find probability that she wins the grand prize given that she won a prize.
First, we calculate the total number of ways to choose 4 numbers from 10, which is (4)) 210.
Next, we determine the number of favorable outcomes for winning prize (at least two correct numbers). This includes: Exactly correct numbers: (2)
Exactly correct numbers:
()) (9) 4 6 24 Exactly 4 correct numbers: (3) (a)
Adding these, total number favorable outcomes for prize 90 + 115. Thus, probability of winning a prize is
115.
210'
The probability winning the grand prize 210
The conditional probability of winning the grand prize given that she won prize is:
115
115
210
Thus, probability TIS where m n 115. Therefore, m n = 116.
116
Total Tokens: 2267
Case D.2: Case in Science Reasoning
QUESTION
Among the following exoplanets, which one has the highest density?
a) An Earth-
marth--mass mass and Earth-radius planet.
b) A planet with 2 Earth masses and density of approximately 5.5 g/cm3.
c) A planet with the same composition as Earth but times more massive than Earth.
d) A planet with the same composition Earth but half the mass of Earth.
A) b
B) d
C) c
a
ANSWER
C
BASELINE RESPONSE
Okay, let's try to figure out which exoplanet has the highest density. The question gives four options, and I need to compare their densities. Let me start by recalling what density means. Density is mass divided by volume. So, for each option, need to determine their mass and volume, then calculate density.
First, let's at option a: Farth-mass and Farth-radius planet. Wait, Farth's density is about g/em², right? Because Earth's 's mass is around 5.97x kg, and its radius is about 6,371 km. So this planet has the same mass and radius as Earth, its density should be the same as Earth's, s, which is 5.5 g/em". . So option a's density 5.5 g/em² Option says a planet with 2 Earth masses and density of approximately 5.5 g/ sm/mm" Wait, density given here. So regardless of the mass, the density 5.5 g/em², then that's 's straightforward. But wait, density is mass over volume. If the density is the same as Earth's, 's,, but the mass is double, then the volume must be larger. Because density mass/volume. If density same, then volume would be proportional to mass. So for example, if Earth's 's same density, then volume would be 2V. But that's not possible unless the radius larger. Wait, but the problem states that the density is 5.5 g/ cm.. So option b's density is given as 5.5, which is same as Earth's ss. So option b's density is 5.5, same as But then why is option b different? Wait, maybe there's is trick here. Wait, option b says "a planet with Earth masses and density of approximately 5.5 g/ 3r So the density already given as 5.5. So option b's density is 5.5. So same as option a. But then why is option given as option? Wait, mayhe need check other options. Option e: A planet with same composition as Farth but 5 times more massive. So same composition implies same density? Wait, no. Wait, if the composition is same, then density depends the mass and radius. But if the composition is same, then the density would be same Earth's only if the planet similar size. But if the mass 5 times Earth's, s. then the radius would be larger. Wait, because
is M and volume V, then density M/V. If the
planet has 2M
33

---

Published as a conference paper at ICLR 2026
a planet with same composition, the density would be similar. But the mass increases, the radius increases, but the density might not change. Wait, but density is mass over volume. So if the planet is made of same material (same density), ,, then the density would be same. Wait, but that's only if the planet is in hydrostatic equilibrium and same composition. Wait, maybe not. Let me think. If planet has same composition as Earth, but is times more massive, then its radius would be larger than Earth's s. Because the radius of planet depends on the mass. For example, if you have a planet with same composition, the radius scales with mass to the power of 1/3. So, if mass increases by 5 times, radius increases by 51/3 Ry 1.71 times. So the volume would be (radius)". so times mass, but volume increases by (52/2)a =5. So the volume increases by 5 times. Therefore, density is mass/volume. So original density is 5.5 g/em². If mass is 5M and volume is 5V. then density remains 5.5. So same as Earth. Wait, but that would mean that option e has same density Earth. But wait, that's only if the planet is made of same material and the density remains same. However, if the planet is more massive, maybe it's compressed more? Wait, but Farth's 's density already about 5.5. the planet is times more massive, but same composition, then it would be a larger planet, but the density might not change. Wait, but mayhe the density would be higher? Because if the planet is more massive, but same composition, perhaps the gravity higher. leading to higher compression? Wait, but that's not necessarily true. For example. Earth's 's density is 5.5 g/ cmn". you have a larger planet with same composition, the density would be similar. Wait, but maybe not, For example, if the planet is so massive that it's a gas giant, but in this case, the composition is same as Earth, so it's a rocky planet. So, if the planet is more massive, but same composition, then it would have larger radius, but the density would be similar. Wait, but if the radius increases, then the volume increases, and since the mass increases, but the density remains same. Wait, but that's only if the density is same. But the question says "same composition as Earth". So the composition is same, then the density would be same. But maybe that's not the case. Wait, perhaps I'm mixing things up. Let me think again. Suppose Earth's density is 5.5 g/ /cmm". If you have a planet with same composition (so same average density) but 5 times the mass, then the radius would be larger. So the density remains 5.5. So option c's density would be same as Earth's. ss. So same as option a and But then why is option given? Wait, but maybe the problem that if the planet is more massive, but same composition, the density might not be same, Wait, but how? Because density is mass divided by volume. If the composition is same, then the density is same. Wait, but that would require that the planet's radius is scaled appropriately. For example, if the planet is 5 times more massive, but same density. then the radius would be (5) 1/3 times larger. So the volume would be 5 times larger. Then, mass is 5M, volume is 5V. So density is same. Therefore, the density of option e would be same as Earth's. s. So same as option and b. But then, how does that fit with the options?
Wait, but maybe the problem is that the planet is more massive, but same composition, then the density would be higher? Wait, no. Because if the composition is same, then the density is same. Unless the planet is more compressed. But for a rocky planet, the density is determined by the material. So if it's same composition, same density. So then, option would have same density as Earth. So then, how does that affect the answer?
Wait, let me check again. Let's take each option:
a) Farth mass and Earth radius. So density is mass/volume. Earth's 's density is 5.5 g/em², 9 So this is 5.5. h) 2 Earth masses, density 5.5. So density is given 5.5. So same <) Same composition as Earth, times more massive. So same composition implics same density? Wait, no. Wait, same composition would mean same average density? Or same material but different mass? Wait, if the composition is same. then the density depends on the size. Wait, for example, if you have planet made of the same material Earth but larger, then the density would be same as Earth's. 's. Because density is mass/volume. If the planct is larger, but same material, then the density would be same. Wait, but how? For example, if you have a cube of the same material, increasing the size doesn't 't change the density. So if the planct is same composition (same material), then regardless of the size, the density would be same. So then, option c's density is same as Earth's. ss So same as and b. So then, why is option given? But then, option d is same composition but half the mass. So same density. So all options have same density? ? But that can't 't be. The question is asking which has the highest density. So perhaps I' 'm misunderstanding something here.
Wait, maybe the problem is that when the planet is more massive, but same composition, the density might not be same. Wait, for example, if the planet is a gas giant, but that's not the case here The question says same composition Earth, so rocky. So for a rocky planet, the density is determined hy the material. So if the composition is same, then density is same. So then, the answer would be all options have same density. But that's 's not possible. So mayhe I'm missing something. Wait, perhaps the key is that option b says "a planet with 2 Earth masses and density of approximately 5.5 gf cmr". " But if the planet has a higher mass, but same density, then the volume must be larger. However, the planet has a higher mass, but same density, then the radius would be larger. But the density is given as 5.5, same as Earth. So that's same as a. But then, option e says same composition
34

---

Published as a conference paper at ICLR 2026
as Earth but 5 times more massive. If same composition, then the density would be same, So same as Earth. So then, all options have same density. But the question is asking which has the highest density. So perhaps there's s mistake here. Or maybe I'm misunderstandinn the options.
Wait, let me think again. Maybe the problem is that when the mass increases, the radius increases, but the density may not change. However, for a planet with same composition, if it's more massive, then the density remains same. So all options a, b, c, d have same density, But that's not possible. Wait, but the answer options are given. Let me check the options again.
The options are:
Farth mass and Earth radius. So density is 5.5. b) Earth masses, density 5.5. So same as a. c) same composition as Earth, 5 times more massive. So same density? So 5.5. d) same composition Earth, half the mass. So same density. So 5.5. So all have same density. But that's not possible. So there must be mistake in my reasoning. Alternatively, maybe the composition is same, but the density isn't 't' same. Wait, perhaps the question trying to trick us. For example, if the planet has same composition as Earth but is more massive, then the density could be higher? Wait, but that's not the case. For example, Earth's density is 5.5 g/cm². If you have larger planet with same composition, the density would be same. So for example, a planet with twice the mass would have a larger radius, but same density. So the density remains same. So in that case, option would have same density as Earth. So then, all options have same density. But that's not possible. So why are these options given?
Wait, but maybe the problem is that for option c. the planet is 5 times more massive than Earth but same composition. However, if the planet is more massive, it might have higher density due to compression? But that's 's only if the planet is gas giant. For a rocky planet, the density determined by the material, not by the mass. Unless the planet is so massive that it's different state of matter. But Earth's 's composition is rocky, so if you have a planet with same composition but 5 times more massive, it would be larger planet, but the density would be same. So, same as Earth's ss So then, all options have same density. But that can't be. The question must have answer. So perhaps made mistake in considering option Let me check option again. Option b says planet with 2 Earth masses and a density of approximately 5.5 gf /cm". But if the planet has higher mass, but same density, then the volume must be larger. However, if the planct has a higher mass, but same density, then the radius is larger. But the density is given 5.5. So that's same Earth. But then, what about the other options?
Wait, maybe the question is about the density of the planet, not the average density. For example, if the planet is more massive, but same composition, maybe the density is higher because it's compressed? But for rocky planets, the density is determined by the material. For example, if you have planet made of the same material as Earth but larger, the density is the same, So, I think the answer must be that all options have same density. But since that's not an option, must have made mistake. Wait, let me look at the options again. The options are: a) Earth mass and Earth radius. So density is 5.5.
b) 2 Earth masses, density 5.5. So same
Same composition as Earth, 5 times more massive. So same density as Earth. So 5.5. d) Same composition as Earth, half the mass. So same density as Earth. So 5.5. So all options have same density. But the answer choices are A) b. B) d, C) c. D) a. So none of them are correct. But this can't 't be. Therefore, must have misunderstood the question. Wait, perhaps the question is not about average density, but about the density of the material. For example, the planet is more massive, but same composition, the density the material is same, But the planet is more massive, perhaps it's different type of planet. Wait, but the question says same composition as Earth. So the material is same. So the density is same. ...
Wait, but what if the planet is not in hydrostatic equilibrium? Like, it's small planet, but I think the
question is about planets in equilibrium.
Alternatively, perhaps the answer is that option b has the same density as a, but option e has higher density. Wait, how? Let me think. If the planct is more massive, but same composition, then the density is same. So why would option have higher density? Maybe I'm missing something.
Wait, maybe the key that option с 5 times more massive than Earth, but same composition. So the density would be higher. Wait, but how? Let me think about the formula. Density is mass divided by volume. If the planet is 5 times more massive, but same composition, then the volume would be larger. So the density remains same. For example, if Earth's 's mass is M. volume V. density D = M/V. For planet with mass 5M and same composition, volume would be 5V (since volume scales with mass for same density). So density is 5M/5V) M/V = D. So same as Earth.
So all options have same density. But that's 's not possible. So perhaps the answer is that all have same density, but since that's 's not an option, must have made a mistake.
35

---

Published as a conference paper at ICLR 2026
Wait, perhaps the problem that option is not the same composition. Wait, no, the question says "same composition as Earth". So I'm stuck.
Alternatively, maybe the answer is option b. Because option b's density is given as 5.5, which is same as Earth's. s. But option c's density is higher. Wait, how? Let me think again. If the planct has same composition as Earth, but 5 times more massive, then the radius larger. But the density mass divided by volume. Let's calculate. Let me use the formula for density of a planct. For a planet with same composition as Earth, the density is constant. So if the mass increases, the radius increases, but the density remains the same. So no change. Therefore, all options have same density.
But the answer choices don't 't have that. So perhaps the correct answer is that all have same density, hut since that's s not an option, must have misunderstood the question.
Wait, maybe the question is asking which one has the highest density, and the answer is that option has the same density as Earth, but option has higher density, But why? Maybe because if the planet is more massive, the density increases. But how?
Wait, maybe the question not about average density, but about the density of the material, For example, the planct is more massive, it might be compressed, so the material is denser. But for a rocky planct, the density is determined by the material, not by the mass. So even if the planet is more massive, the material density is same.
At this point, I'm confused. Maybe should look for another approach. Let's consider that the highest density would be the one with the highest mass and smallest volume. So, for example, if planet higher mass but smaller volume, it would have higher density.
So. let's analyze each option:
Earth mass and Earth radius - density 5.5 g/cm²
h) 2 Earth masses, density 5.5 g/em² So same
<) same composition as Earth, 5 times more massive. So, same composition, the radius would be larger. Let's calculate the radius. For planet with same composition, the radius scales with mass 1 1/3). So for times mass, radius is 5(1/3) = 1.71 times Earth's 's radius. So volume is (1.71)* ~ 5 times Earth's volume. So density is mass/volume = 5M/ / 5V = M/V = 5.5 g/cm². Same before. So same density.
d) same composition as Earth, half the mass. Radius is (0.5) / as 0.79 times Earth's 's radius. Volume is (0.79)* 0.5 times Earth's volume. Density is 0.5M/ 0.5V M/V = 5.5 g/em", Same as before. So all have same density. But the options don't 't include that. Therefore, must have misunderstood the question.
Wait, perhaps the question is not about the density of the planet, but about the density of the material. For example, if the planet is more massive, it might he denser. But that's 's not the case. Altematively, mayhe the question has typo. For example, option says "same composition as Earth but times more massive than Earth." the composition same, then the density is same. But if the composition is different, like if it's a gas giant, but the question says same composition as Earth. Alternatively, maybe option e is not same composition, but the question says it is.
At this point, I think the correct answer must be that all have same density, but since that's not an option, perhaps the intended answer option b, which has the same density Earth. But I'm not sure. Alternatively, maybe the question is trying to trick us into thinking that option e has higher density, Wait, another approach: the highest density would be the one with the highest mass and smallest volume. Let's assume that the planets in options b, d are not all the same composition. Wait, option Earth mass and Earth radius -> density 5.5. Option 2 Earth masses, density 5.5. So volume 2M / 2M/ 5.5. Earth's 's volume M/ / 5.5. So option b's volume is twice Earth's volume. So radius is larger.
Option e: same composition as Earth, 5 times more massive. So if same composition, the density is same. So same as Earth's.
Option d: same composition as Earth, half the mass. Same density as Earth.
So all options have same density. So the answer is that all have same density, but since that's not an option, I must have made a mistake.
Wait, the answer choices are A) b. B) d. c, D) So if all have same density, then the answer is not among them. But that's not possible.
Wait, perhaps the question is asking which has the highest density, and the answer that option b has the same density as Earth, but option e has higher density. But how? Let me think again. Maybe the composition is different. For example, if the planet is more massive, it might be different composition. But the question says option has same composition Earth.
At this point, I'm stuck. Given the options, and assuming that all have same density, but the answer must he one of them, perhaps the intended answer is option b, which has the same density as Earth, hut the question might have intended that option e has higher density. Or perhaps the answer is option c. Alternatively, maybe I made a mistake in assuming that same composition implies same density. Let me think: if planet has the same composition as Earth, but is more massive, then the density might be
36

---

Published as a conference paper at ICLR 2026
higher because gravitational compression. For example, you have planet with same composition but higher mass, would be more compressed, leading to higher density. But for rocky planets, this might not be the case. For example, Earth's 's density is about 5.5 g/em", . If you have a larger planet with same composition, the density would be same. But for gas giant, the density is lower.
Wait, but for rocky planet, it's more massive, the gravity stronger, the material is compressed more. So the density would be higher. For example, you have a planet with times Earth's mass, but same composition, it would he more compressed, leading to higher density. So the density would be higher than Earth's s.
But how calculate that? Let's assume that the planet's density increases with mass due to compression. But how?
This is getting too complicated. Maybe the intended answer is option c, because if the planet is more massive, and same composition. the density is higher. So the answer is C) c. Alternatively, the answer is option b, which has the same density as Earth. But I'm not sure. Given the options, and the fact that option has times the mass but same composition, and if we assume that the density increases with mass (which might not be true, but perhaps the question assumes that). then option c would have higher density. So the answer is C) c.
But I'm not confident. However, given that option b has the same density as Earth, and others have different, think the intended answer is C) c.
But I'm not sure. Alternatively, maybe the answer option d. But no. Alternatively, the answer is option a. but that's same as others.
I think need to make a decision. Given that option is 5 times more massive than Earth and same composition, and assuming that higher mass leads to higher density (even though it's not accurate), the answer So I'll go with that. 3/think>
To determine which exoplanet has the *"ighestt density**, we need to analyze cach option based on the relationship between ma **olume and density (density = mass/volume).
Step-by-step Analysis:
All four options have **the same density of 5.5 g/ 3/ based on the assumption that **same composition ** implies **same density** However, the question asks for the *highest*** density, and **nne of the options have higher density than the others.
Key Insight:
But, **option b** explicitly states that the **density is 5.5 g/ 3=s, which is **equal to Earth's density and **optio c* and ** are #simplicitly*** the same as Earth's 's density due to same composition.
Since all options have **equal de the question might be testing the understanding that density is mass/volume** and **igheer mass with the same composition ** lcads to **larger v * **maintaining the same d
3 However, the **only explicit statement** of density is **option b which directly states ##5.5 g/cm" Se*, Therefore, **option b** is the correct answer.
Final Answer: options have the same density of 5.5 g/cm².
//mm.
ANSWER: A
Total Tokens: 7722
REPRO RESPONSE
Okay, so I've got this question about exoplanets and their densities. Hmm, I'm rusty on this, but let me think through it.
The question is asking which exoplanet has the highest density. The options are to d, each describing different scenario. The choices are A to D. which correspond to each option.
Wait, remember that density mass divided by volume. So the density (rho) is m/V). Since planets are spheres, their volume (4/3)rr² So planet is larger in radius, even if it's less dense, its mass might make up for it. Looking at the options:
Option a: Earth-mass and Earth-radius. So if a planet has the mass and radius Earth, its density would the same as Earth's. s. Because if m and r are same, then tho m/( (4/3 π r²)) which is same as Earth's ss.
Option b; 2 Earth masses and density of about 5.5 g/em". Wait, wait. Wait, if planct has density of 5.5 g/em², S then what's its mass? Let me think about Earth's density. Earth average density is about 5.5 g/em²? Or wait, no. think Earth's 's density is about 5.5 g/em²? Or is that another planet? Wait, Earth's density is about 5.51 g/em². So option
Earth masses and density of 5.5 g/em², that would mean that the volume would be smaller than Earth's. s. Because density is mass/volume. So for a mass of 2 Earth, if volume is smaller, then density is higher. Let me compute. Earth's 's radius is R. Volume is
37

---

Published as a conference paper at ICLR 2026
(4/3)T R Mass is M. Density is M/( (4/3)r R²) Option b's mass is 2M, and density is 5.5 g/em² So 5.5 (2M) V. But V is (4/3 r² ), where is the radius of option b's planet. Let me solve for Wait, if the density is 5.5 g/em', and mass is Mearth, then the volume would be (2 M) (5.5 g/em²) Wait, M Earth's 's mass, so 2 M/ / (5.5) = V.
But V is (4/3 æ r*). So for option b. V is 2M/(5.5) cm3 (since density is mass/volume, volume mass/density). So V (2 Mearth)) (5.5) cm Earth's volume is (4/3 π R²), where R is Earth's radius. So the ratio of volumes is (2 Magrth)
(4/3 R²) )= / (5.5 Earth's volume). Wait, but wait, Marrth (4/3 R # Earth's 'ss density). Oh wait, Earth's 's density about 5.5 g/em² S So
Mearth = (4/3 v R³) 5.5 g/cm" So option b's volume would be 2 Meareh / 5.5. Let me substitute
Mearth-
Mearth = (4/3 R²y 5.5 g/em".
So option b's volume V (2 * (4/3 RD 5.5 /.55
Simplify: 2 ** (4/3 x R³) *
24/33 RR) cm².
Which is 2 Earth's volume. So the volume of option b's planct is twice Farth's. So since its mass is 2 Earth masses, then its density is (2 Mearth) / (2 Vearty)) )= Mearth / Vearth Farth's 's density. Wait, that can't 't be right. Because Earth's density is 5.5 g/em², so if option b's planct has density 5.5, but its mass is Earth, and volume is twice Earth's volume, then (2 Mearts)) / (2 Vearta) Mearth / Venrth 5.5 g/cm². That makes sense. So option b is Earth's 's density.
Wait, but the question says option b's density is 5.5 g/cm* So the planet in option b would have the same density as Earth, but larger radius. Because if the volume twiec that of Earth, but mass is twice, then density would be same as Earth's. s. So option b's density is same as Earth's.
Option c: same composition as Earth, times more mass than Earth. So Earth's density is about 5.5 g/em", then planet with 5 times Earth's 's mass would have a density of times Earth's 's density, right? Because density is mass/volume. Wait, but the composition is same, same density? No, wait, same composition would mean same average density. So if Earth's 's average density 5.5 g/em", then a planet made of same average composition hut times Earth's mass would have a much larger volume. Wait, but if same composition, same density. So a planet with same composition would have same average density as Earth. So option c's planet's density is 5 * 5.5 27.5 g/em".
Option d: same composition as Earth, half the mass of Earth. So density is (0.5 /earth) V. But since same composition, V Earth's 's volume. So density 0.5 Earth's 's density 2.75 g/em².
Wait, but wait. Let mc think about each option's 's density; Option a: same as Earth, so density is Earth's 's density, 5.5 g/em².
Option b: same as Earth, density 5.5 g/em².
Option 5 times Earth's 's mass, same composition (so same average density), so S * 5.5 27.5 g/em". Option half mass, same composition (so same average density as Earth), so density is 0.5 5.5 = 2.75 g/em².
Wait, but wait. Option b says the planet has a density of approximately 5.5 g/em² So its density is same Earth's. s. So and both have same density Earth. So why would the question have them as options? Mayhe I'm missing something. Wait, the question is which has the highest density. So, among the options, option e has 27.5, which the highest. So option e would have higher density. Wait, but let me recheck each option.
Option same as Earth, density is 5.5.
Option same as Earth, but density given as 5.5. So same as a. Option c: 5 times Earth's 's mass, same composition, so density is 5 5.5 27.5. Option d: half mass, same composition, density is 2.75.
So the highest is option c, which would be the correct answer. Looking at the options, which one is option c?
The choices are labeled as a, b, c, d. The question is, which exoplanet has highest density.
The options for the answers are A) b, B) d, C) D) a. So the correct answer is option c, which is choice C. Wait, but wait. Did make mistake in option b's density?
Wait, the question says option b has a density of approximately 5.5 g/cm². Which is same as Earth's. So the planet option is Farth-like, same as Earth, so same density.
So the exoplanet with highest density is option which has 5 times Earth's 's mass. So answer is C. e/think.
To determine which exoplanet has the highest density, we analyze the given options based their mass, radius, and density. Density calculated as p = m. where m is mass and is volume. For spherical planets, V _ 3 Since the volume depends the radius cubed, a larger radius significantly increases the volume, which can reduce density if mass remains constant. Analysis of Each Option:
38

---

Published as a conference paper at ICLR 2026
1. Option a: An Earth Earth-mass and Earth-rading planet. If the planct has the same mass and radius as Earth, its density is identical to Earth's density, which is approximately 5.5 g/cm².
2. Option b: A planet with Earth masses and density of approximately 5.5 g/cm". Using the formula
p the volume V can be calculated V p For a planet with Earth masses and density of 5.5 3/5g//mm, the volume is V 5.5 cm Since Earth's volume Viatth $ T the ratio volumes is V = 2. Thus, the planet in option b has the same density Earth, 5.5 g/em', with larger radius. Vean
3. Option A planet with the same composition as Earth but 5 times more massive than Earth. If the planet has the same composition, has the same average density Earth, 5.5 g/em". However, its mass is times Earth's mass, so volume is times larger, and its density remains 5.5 g/emm.
4. Option d: A planet with the same composition as Earth but half the mass of Earth. Since the composition is the same, the density is also the same as Earth's ss 5.5 g/em². However, with half the mass, its volume is half of Earth's 's and its density remains 5.5 g/em².
Conclusion:
The exoplanet with the highest density is one described in option has 5 times Earth's mass but the same composition, resulting in a density 5.5 g/em". This higher than the densities of the other planets in options a, b, and d.
ANSWER: C
Total Tokens: 2450
Case D.3: Case in Code Reasoning
QUESTION
You are an expert Python programmer, and here is your task:
Write python function to check whether the given two integers have opposite sign or Your code should pass these tests:
assert opposite
Signs(1.-2) opposite_Sinns(L..
True
assert opposite Signs(3,2) == False
assert opposite Signs(-10:, 10,- = False
assert opposite_ Signs(-2,2) 2-,2) = True
BASELINE RESPONSE
Okay, need to write a Python function called opposite_Signs that takes two numbers as arguments and returns True if they have opposite signs, otherwise False. Let's think about how to approach this. First, the function needs to check if one number is positive and the other is negative. So, for example, I and would return True because so False. Similarly, Hmm, how to check the signs. Well, the product of two numbers with opposite signs is negative. So if multiply the two numbers and check if the result is less than zero, that would indicate opposite signs. Wait, but what if one of them is zero? Wait, the prohlem says "opposite signs" but zero isn' positive negative. So if either numher zero, then they don't 't have opposite signs. For example, if one is zero and the other is positive, then they aren't 't opposite signs. So the function should return False in such cases.
is positive and the other is negative. But 3 and 2 are both positive, and -10 are both negative, so False. And -2 and are opposite signs, so True.
Wait, but the problem's 's test cases don't include zero. Let me check the test cases again. The given tests are (1.-2) - True, (3,2) - False, (-10.-10) False, (-2,2) True. So the function needs return True only when one is positive and other negative. So, how to handle zero?
But the problem might assume that the inputs are non-zcro. Wait, but the function's 's parameters are numbers. Let me think. Suppose one of the numbers is zero. For example, opposite_Signs(O. 5,)) would return False because zero isn' 't't positive or negative. Similarly, opposite. Signs(0,0) would also return False. But the problem's test cases don't 't cover that, However, the problem's tests don't include such cases, so maybe the function supposed to handle those as well.
But the problem says "opposite signs"-on -so if either number is zero, then they can't tnn't have opposite signs. So the function should return False in those cases.
So how to check if the numbers have opposite signs. The product approach: if b 0, then they have opposite signs. But if either or is zero, then b zero, which not less than zero. So that would correctly return False
So the function can he written as: return (a h) < O. But wait, what if one of them is zero? For example, ==0 and 5=5. Then a is 0, which is not less than zero. So the function returns False, which is correct. But what if one of them is zero and the other is negative? Like a=0, b Then product is zero, so returns False. Which correct because zero doesn't have sign.
39

---

Published as a conference paper at ICLR 2026
So the function can be as simple as:
def opposite > S_siissignss return
(a, b)
Wait, but let's test sample cases.
Test case I: opposite_Signet 22) + =
True. Correct.
Test case 2: Srrosite_Sinn(333 + 6 > 0 + returns False. Correct.
Test case 3: opposite. -innnSiiinn-000000 + 100 > 0 - returns Falsc. Correct.
Test 4: opposite
opposite_Signs(2222 Signs(-2.2) - <4<0
True. Correct.
So test cases pass with this approach. So the function can written this way.
But wait, what about when one of the numbers is zero? For example, opposite Signs(0,5) - 0 $5=0 returns False. Which is correct.
But what if one is and the other is negative? Same result. So the function handles that.
So this approach seems cover all cases.
code is straightforward.
problem expects to handle cases where numbers are zero. But the problem's 's tests include those, but the function's 's code should handle them.
But maybe
So the code is simple. So the function is just returning whether the product of and b is negative. So the code would be:
def o return
b)
But wait, what about if or b is zero? Well, discussed, that's handled correctly.
But what if the numbers are floating points? For example, 0.0 and 5.0. The product is zero, returns False. Which correct.
So this approach works for integers and floats. So the code is simple. Let me check the test cases again.
Yes, all test pass. So that's the solution. </think>
To solve the problem of determining whether two numbers have opposite signs, we can use a simple and efficient approach based on the mathematical property of multiplication:
i numbers have opposite signs, their product is negative.
? they have the sign, their product is positive zero (if either number zero). Solution
We can implement the function 'opposite Signs(a, b)* as follows:
" 'pythhn
def o Signs (a, b)
return *
Explanation
-Multipliaaiion Rule:
"a*b;0 + the numbers have opposite signs.
=aa=b==0' -> at least one number is zero (no sign, so not opposite).
-aa*b&0' both numbers are either positive or negative (same sign).
-Thi approach automatically handles all edge cases:
Zero values (c.g., 'opposite 'opposite_Sinns(" Signs(0. 5)" 'False")
Floating--Pon point numbers (c.g., 'opposite_ SiinsS-ggnS..--- 00.5, 2.0)' + 'True')
Large integers (c.g., Syposite_Signs(10000 -1)* + 'True")
Test Cases
'python assert Oopositel_Siins (1, -2) -- True
assert oppositel__iinss (3, 2) - False
assert Sopositei_Signs (-10, -10) False
(-10,
assert oppositel (-2, 2) True
All test cases pass successfully using the above function.
Final Answer
40

---

Published as a conference paper at ICLR 2026
'python def Appositel_Signe b) return ...
Total Tokens: 2457
REPRO RESPONSE
To solve this problem, I need determine the conditions under which two numbers have opposite signs. The function Apposite_Signs should return 'True' if one number is positive and the other is negative, and 'False' otherwise.
First, I'll consider the definition of opposite signs. A positive number and negative number have opposite signs. This means should be greater than and the other less than zero. I'll start by checking the sign each number. If one number is positive and other is negative, function should return 'True'.
Next, I'll examine the cases where the signs are the same. If both numbers are positive both are negative, the function should return 'False'.. I'll also need to handle cases where the numbers zero. one number is and the other is positive or negative, the signs are not opposite, so the function should return 'False'. Testing the provided examples:
'oppssite Signs( I, -2) should return "True' because is positive and -2 is negative. Popposite_Signssee 2)' should return 'False' because both positive.
"opposite "opposite_Sinns (00ass(-0----0 100.-0000000 -10)' should return 'False' because both are negative.
"pposstee oppositee_ignn((( 2, 2)' should return "Truc' because -2 is negative and is positive.
By implementing these checks, the function should correctly determine whether the two numbers have opposite signs.
</think:
To solve this problem, we need to determine whether two given numbers have opposite signs. positive numher and a negative number have opposite signs, while both positive or both negative numbers do not. Additionally, zero is neither positive nor negative, so if one number is zero and the other is non-zero, their signs
opposite.
Approach
1. Check the Sign of Each Number:
Determine if cach number is positive, negative, or zero.
2. Evaluate Opposite Signs:
If number positive and the other is negative, return "True'.
both numbers are positive or are negative, return "aasee.
If one of the numbers is zero, return 'False' since zero does not have sign.
Solution Code
'python
def opposite \___e__________ Loeee__SSggg Signs (a, b):
signl = a 0 sign2 b 0 return not (si (signl sign2)
Explanation
Check the Sign of Each Number:
'sigal 0' checks if the first number 'a' is positive.
'sign2
checks if second number is positive.
Evvllate Opposite Signs:
The function returns 'not (sign1 = sign2),, which means if 'sign'' and 'sign2" are not equal, the numbers have opposite signs. If they equal, the numbers have the same sign. This approach efficiently determines whether two numbers have opposite signs with simple logical
check.
Total Tokens: 654
41

---

Published as a conference paper at ICLR 2026
E LIMITATIONS
In this paper, we introduce REPRO, a framework designed to enhance the reasoning capabilities of LLMs. While REPRO demonstrates promising results, several challenges remain for future investigation. First, the issue of scaling persists. Although we validate REPRO's effectiveness on LLMs with up 8B parameters, its scalability to larger LLMs remains an open question. Due to computational resource and time constraints, experiments on larger-scale models were not feasible, but we aim to address this future work. Second, the scope of problem types is limited. While REPRO performs well on verifiable objective reasoning benchmarks in mathematics, science, and coding, its efficacy on subjective or difficult-to- to-verify problems is unexplored. This question, closely tied to the development of verification systems for such tasks, merits further consideration.
F
LLM USAGE
In this study, the usage of LLMs is limited to the final stages, specifically for refining and proofreading the manuscript. LLMs are employed solely to enhance the clarity, logical consistency, and linguistic accuracy of the text, ensuring that research ideas communicated clearly and professionally. Importantly, LLMs played no role in the core foundational aspects this research, including the development of the research methodology, the design of the experimental framework, the analysis and interpretation of results. We take full responsibility for all content in this paper and explicitly acknowledge accountability for every aspect of the manuscript.
42


Reviews:
[{"review_id": "tGQ9bM9Gum", "reviewer": "Reviewer_gz9P", "summary": "This work aims to address the \"overthinking\" and inefficiency found in the long CoT reasoning of LLM. The authors innovatively frame the CoT reasoning process as a form of gradient descent optimization. Based on this, the paper proposes REPRO, a framework that defines a surrogate objective function based on the perplexity of the ground-truth answer to evaluate the \"magnitude\" and \"stability\" of the reasoning process. These two scores are combined into a process-level reward and integrated into the RLVR training loop to rectify the model's reasoning behavior. Experiments show this method improves performance on several benchmarks while increasing token efficiency and reducing invalid reasoning steps.", "questions": "1. See weaknesses 1.\n\n2. Given all the extra computations (entropy, surrogate objective function), what is the actual total increase in training overhead for REPRO compared to a baseline RL algorithm?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper tackles a critical problem in RLVR, i.e., how to optimize the reasoning process itself, rather than just rewarding the final answer, which is key to solving overthinking.\n\n2. The empirical results are the paper's greatest strength, particularly in efficiency gains. The data clearly shows that REPRO not only boosts task accuracy but also significantly reduces the number of tokens required for reasoning, directly addressing its stated goal.\n\n3. The method demonstrates good generality, proving to be a plug-and-play component that can be combined with various major RL algorithms like PPO, GRPO, and etc.", "weaknesses": "1. The analogy of CoT as gradient descent is illustrative but lacks rigorous support. Using the log-prob of the correct answer as the surrogate objective essentially assumes that a better reasoning path is one that converges to the answer \"sooner and smoother.\" This design may excessively penalize exploration, trial-and-error, and backtracking (which might be necessary for hard problems, but would look like oscillation or low magnitude under this metric). I am concerned it rewards simple, direct paths rather than truly rectifying complex reasoning.\n\n2. This work involves several new and critical hyperparameters to tune. The ablation studies suggest that performance is highly sensitive to the values of these hyperparameters, which significantly increases the difficulty of reproducing the results and applying the method in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jWPpxQZkKK", "reviewer": "Reviewer_Q85p", "summary": "REPRO introduces a modification to the LLM RL objective by augmenting the advantage function with a surrogate objective that quantifies the posterior of (ground truth answer | current reasoning chain). The idea is that by quantifying the magnitude and stability of this posterior and encouraging confident and stable reasoning path, we can improve model performance while reducing thinking length. Authors performed experiment by adding the REPRO objective on top of PPO,RF++ and GRPO for DeepSeek-R1-Distill-Qwen-1.5B and Qwuen3-1.7B, and demonstrated consistent performance gain on math and code benchmarks.", "questions": "It is interesting that REPRO is uniformly better than GRPO across different values of alpha. I'm quite curious to see what happens if we just train on the model confidence and not on the ground truth reward.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The biggest strength of the paper is in its simplicity. The idea of reward shaping with posterior of ground truth given intermediate reasoning is simple and elegant and appears effective.\n\nIt requires no additional rollouts during RL process and therefore can be readily adapted to existing approaches without much overhead.\n\nThe paper is generally well written and easy to follow.", "weaknesses": "The biggest drawback is that it is not clear from the manuscript what is the fundamental mechanism that made the difference in the proposed algorithm. In particular, since the objective of P(answer | thinking) can be potentially by artificially inflated by increasing model confidence across the board (collapsing entropy). It would be interesting to see and control for entropy collapse during training to see how much the conditioning actually contributes to model performance improvement, other than collapsing model entropy. Additionally, would naive length penalty also achieve the same result? Is the length reduction leading to increasing confidence? These important counterfactual questions are mostly unaddressed in the current work.\n\nOne minor thing is that the reward shaping is very similar in spirit to PPO's critic model, except the critic here is hardcoded and not learned. In my opinion, this is a potentially more interesting question to address, which is what if the critic is trained using this surrogate objective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jUK7wTIDba", "reviewer": "Reviewer_qdvU", "summary": "This paper’s main focus is on how to obtain a process reward *without* introducing anything akin to a PRM.  \nThe proposed method centers on two components: the Magnitude Score, which captures the change in the model’s confidence brought about by the reasoning trajectory, and the Stability Score, measured with Kendall’s τ correlation coefficient.  \nIn small-scale experiments these two scores produce observable improvements.", "questions": "1. In the process of computing the advantage (accumulation), what does k represent? If it is consistent with the meaning of k mentioned earlier, should this formula still be expressed this way?\n2. In the “Entropy-Based Selection Strategy”， why is “\\n\\n” chosen as the delimiter? Are there better ways to split the sequence? And why use the entropy of the *first* token of a segment—can it really characterize the whole segment?\n3. The fact that larger k yields better results seems to imply that the selection scheme itself is flawed. Have you run an ablation exp that selects all the segments?\n4. As can be seen, the proxy objective function J changes sharply during the first half of the reasoning chain and then flattens out in the second half. How should this be interpreted? Does it indicate insufficient representational capacity? Is there a need for additional corrective terms?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is written in a fluent and highly readable style.  \n2. The experimental content is rich and concrete, and the ablation studies are comprehensive.  \n3. The idea is clear, and the algorithmic design is relatively concise.", "weaknesses": "1. Please refer to the \"Questions\" section for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n2RLzXdk5R", "reviewer": "Reviewer_zYTx", "summary": "This paper presents REPRO, a framework for improving LLM reasoning by viewing chain-of-thought (CoT) as an optimization process. The authors define a surrogate objective function to model reasoning progress, introduce dual metrics—magnitude and stability—to assess optimization quality, and integrate them as process-level rewards within RL post-training. Experiments across several RL algorithms (PPO, GRPO, REINFORCE++) and reasoning benchmarks in math, science, and coding show consistent accuracy improvements and reduced reasoning inefficiencies.", "questions": "1.\tCould the process-level reward interact negatively with other RLVR objectives, especially in long-horizon reasoning?\n2.\tHave the authors considered token-level or adaptive segment granularity for more fine-grained control?\n3.\tHow sensitive is REPRO to inaccuracies in the surrogate objective when applied to open-ended reasoning without clear ground truth (e.g., creative tasks)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed method itself is coherent and easy to follow.  The authors conduct thorough experiments on several benchmarks to validate the effectiveness of the proposed method.", "weaknesses": "While the proposed framework is conceptually appealing, several limitations temper its overall contribution.\n\n**(1) Limited model scale:** Most experiments are conducted on small- to medium-sized models (e.g., ≤8B parameters), which constrains the generality of the findings. Modern reasoning research increasingly focuses on 30B–70B or larger models, where optimization dynamics, token efficiency, and stability behavior differ substantially. It remains unclear whether REPRO’s process-level rewards would scale effectively or remain computationally practical in such large models.\n\n**(2) Theoretical abstraction:** The analogy between chain-of-thought reasoning and gradient descent optimization, though intuitively appealing, is primarily heuristic and lacks rigorous formal grounding. The paper would benefit from a stronger mathematical justification or empirical validation of this analogy.\n\n**(3) Simplified proxy objective:** Using ground-truth log-likelihood as a surrogate for reasoning quality oversimplifies complex internal cognitive states and may not generalize to open-ended or ambiguous reasoning tasks.\n\n**(4) Modest performance gains:** Despite consistent improvements, the absolute accuracy increases are relatively small, raising questions about practical significance versus added complexity. Overall, the approach is promising but under-evaluated at scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)