Test-Time Scaling with Reflective Generative Model
Authors: Zixiao Wang
Yuxin Wang
Xiaorui Wang
Mengting Xing
Jie Gao
Jianjun Xu
Guangcan Liu
Chenhui Jin
zhuo wang
Shengzhuo zhang
Hongtao Xie
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: We introduce a new Reflective Generative Model (RGM), which obtains OpenAI o3-mini's performance via a novel Reflective Generative Form. This form focuses on high-quality reasoning trajectory selection and contains two novelties: 1) A unified interface for policy and process reward model: we share the backbone network and use task-specific heads for reasoning trajectory predicting and scoring respectively, introducing only 50M extra parameters for trajectory scoring. 2) Eliminating the reliance on process-level annotation: we provide a self-supervised process reward model (SPRM), which can directly learn the high-quality reasoning trajectory selection from the outcome reward. Equipped with the reflective generative form, RGM is naturally suitable for test-time scaling based on the controllable thinking length. Experiments show that our RGM, equipped with only 50M additional parameters in SPRM, outperforms policy models with 72B extra reward models, thereby enabling 32B model to outperform OpenAI o3-mini on AIME24 (84.2 vs. 79.6) and HMMT25 (53.1 vs. 53.0).
Code is available at https://github.com/MetaStone-AI/XBai-o4.

Published as a conference paper at ICLR 2026
TEST-TIME SCALING WITH REFLECTIVE GENERATIVE MODEL
Zixiao Wang'* Yuxin Wang2*
Xiaorui Wang
Mengting Xing²
Jie Gao² Jianjun Xu'
Guangcan Liu²
Chenhui Jin
Zhuo Wang' Shengzhuo Zhang*
Hongtao Xie
University of Science and Technology of China
MetaStooe Technology
wzx990ma ust wuw..ceuuu........omo..o...
wangyx58 ednnyyx, edu. htxie@ustc.edu.co
ABSTRACT
We introduce a new Reflective Generative Model (RGM), which obtains OpenAl o3-mini's performance via a novel Reflective Generative Form. This form fo- cuses on high-quality reasoning trajectory selection and contains two novelties: 1) A unified interface for policy and process reward model: we share the back- bone network and use task-specific heads for reasoning trajectory predicting and scoring respectively, introducing only 50M extra parameters for trajectory ing. 2) Eliminating the reliance on process-level annotation: we provide a self-supervised process reward model (SPRM), which directly learn the high- quality reasoning trajectory selection from the outcome reward. Equipped with the reflective generative form, RGM is naturally suitable for test-time scaling based on the controllable thinking length. Experiments show that our RGM, equipped with only 50M additional parameters SPRM, outperforms policy models with 72B extra reward models, thereby enabling 32B model to outperform OpenAl o3- mini on AIME24 (84.2 vs. 79.6) and HMMT25 (53.1 vs. 53.0). Code available at httts::
INTRODUCTION
Over the past two years, the field of Large Language Models (LLMs) has experienced rapid advance- ments, marked by the emergence of increasingly sophisticated models. Notable developments in- clude OpenAl's GPT-4, Google's Gemini, Meta's LLaMA series, Alibaba's Qwen, and DeepSeek's 's RI, which have collectively pushed the boundaries of natural language understanding and gener- ation. This progress is attributed to innovations in model architectures and training techniques, enabling LLMs to process and generate content across various formats.
Recent analyses suggest that OpenAl's o3 model achieves its advanced reasoning and coding capa- bilities through Test-Time Scaling (TTS) techniques such massive sampling, candidate scoring, and search over multiple reasoning paths (Labs) 2025 Zeff 2024). For instance, during ARC-AGI and competitive coding evaluations, o3 was shown to generate up to 1024 candidate samples for each query (Chollet] 20244; OpenAl 2025). These inference-tine time strategies mark a significant shift from traditional one-pass models, enabling 03 to adapt dynamically to novel tasks and achieve near- human performance in reasoning benchmarks.
TTS approaches can be categorized into types: internal TTS and external TTS. Internal TTS (also called sequential TTS in Zeng et al. (2025)) strategies use CoT for longer thinking processes (Guo al. 2025 OpenAl 2024), which benefits from Long-CoT Supervised Fine-TTuiig and inforcement learning. Recent internal TTS methods (Guo et al. 2025) mainly suffer from the false positive reasoning process, as the outcome reward will misclassity the correct answer with incorrect reasoning during the training stage. External TTS (also called parallel TTS in Zeng et al. (2025) proposed for selecting the correct reasoning process. Prominent external TTS algorithms include
"Equal contribution. "orresponning author.

---

Published as a conference paper at ICLR 2026
Separate training Stage
Unified training Stage
Data Final answer
Policy Model
Data Process label
LLM based
Palicy Model
answwrr
Rollout,
Score,
Rollout,
Policy
LLM based
Policy
Rollout,
Rollout,
Score,
Two Inference with Large reward models
Unified inference with Light reward heads
(a) Previous External TTS framework
(b) Our RGM form
Figure 1: Comparison between the previous external TTS framework (a) and our RGM (b).
Best-of-N sampling, Beam Search, and Diverse Verifier Tree Search, using the reward model as the verifier to select high-quality reasoning trajectories. Researchers (Lightman et al., 2023) have shown that the Process Reward Model (PRM) is more effective performance boosting compared with the Outcome Reward Model (ORM). However, Wang et al. (2023); Guan et al. (2025) point out that training a high-quality PRM remains costly, primarily due to the lack of accurate process-level annotations. Moreover, during the inference stage, introducing additional LLM-based PRM intro- duces significant extra parameters and computational overhead, which severely limits the practical deployment of external TTS.
This paper focuses on external TTS and proposes a new Reflective Generative Form for high-quality reasoning trajectory selection. Specially, the proposed new form shares the backbone of the policy model and process reward model, providing a more efficient scoring process with little parameter and computational overhead. Besides, Self-supervised Process Reward Mode(SPRM) is intro- duced for self-supervised training to eliminate the reliance on process-level supervision. Based on the Reflective Generative Form, the proposed RGM can improve the performance by increasing the controllable thinking length during inference. Compared with the existing external TTS framework with LLM based PRM, our proposed RGM introduces a unified form for both training and infer- ence, achieving overhead (Fig. D. Experiment results show that RGM achieves comparable performance to Ope- nAI ol-mini and 03-mini with 7B and 32B parameters. And our SPRM with only 50M parameters outperforms existing 72B level reward models.
streamlined pipeline with significantly reduced computational and parameter
In summary, the main contributions of this paper are as follows:
We provide a new Reflective Generative Form for high-quality reasoning trajectory se- lection, which enables single network achieve both reasoning trajectory prediction and selection with Zero process-level annotation.
We provide both qualitative and quantitative analysis for the aha moment and generaliza- tion capability of the proposed new form. These exhaustive discussions will effectively benefit the community for future research.
RGM outperforms existing 72B reward models with only 50M SPRM, and achieves com- parable performance to OpenAl o3-mini with only 32B policy models.
2 RELATED WORKS
Test-Time Scaling. Test- Time Scaling (TTS) is technique that leverages additional computa- tional resources inference time to tackle challenging problems. TTS can be divided into two categories: internal TTS and external TTS. Internal TTS introduces the long Chain-of-Thouu (CoT) to generate answers based on the detailed reasoning process. OpenAl ol (Jacch et al. 2024) and DeepSeek R R (Goo et al. 2025) introduce thinking process to plan the solution and guide the final answer. Jin et al. (2024); Yeo et al. (2025) have shown that long CoT can help models correct mistakes by themselves ves and decompose complex problems more effectively. However, Chen et al. (2024b a) have highlighted the risk of overthinking, where excessively long reasoning trajectories may lead performance degradation. On the other hand, external TTS scales up inference through
2

---

Published as a conference paper at ICLR 2026
search-based strategies and auxiliary reward models. A common approach is the Best-of-N strategy (Lightman et al. 2023 Brown et al., 2024] Wang et al. 2023). Fine- line-graind step level searching
methods have also been explored, such Beam Search Liu et al. 2025 Snell et al. 2024 Diverse Verifier Tree Search (Beeching et al.) and Monte Carlo Tree Search (MCTS) (Zhang et al. 2024
Guan et al. 2025 Luo et al.) 2024). These methods search the step level and utilize Process Reward Models (PRMs) to guide the reasoning trajectory step-by-step. Beyond search strategies,
recent work emphasizes that the quality of the reward model a crucial factor in external TTS (Guan
et al. 2025
Process Reward Model. Process Reward Models (PRMs) focus on evaluating LLMs at the step level. Lightman al. (003)) unveil that this fine-grained guidance can lead to better TTS per- formance compared with the global-level Outcome Reward Model (ORM). However, accurately identifying logical errors in LLM outputs remains challenging, and PRMs require high-quality task- specific annotated data for training. To this end, recent works Wang et al. (2023) leverage Monte Carlo estimation automatically assign step-level scores using only the final answers as supervi- sion. Zhang et al. (2024); Guan et al. al.] (2025) iteratively synthesizes data by MCTS and fine-tuning both LLMS and PRMs, improving performance across both models. [Tan et al. (2025) follow the L.LM-as-ajjudee method and introduce a new LLM to annotate the reward of each step. Nonethe- less, less, Zhang et al. (2025) point out that labels generated by Monte Carlo estimation can be noisy, as incorrect reasoning processes may still yield correct final answers. They further propose hybrid approach that combines both Monte Carlo estimation with the LLM-as-a-judge.
3 PROBLEM FORMULATION
This paper aims to find a high-quality reasoning trajectory more efficiently at inference time based on TTS. We first summarize the general inference forms for standard LLMs (policy models) and existing TTS methods, and then formally define our proposed Reflective Generative Form.
1) LLMs without TTS. The model directly generates an answer based on the input query Q. This basic inference form can be formulated as:
answer = LL (Q).
(1)
TTS based methods can be categorized into two types: sequential scaling based internal TTS and parallel scaling based external TTS.
2) Internal TTS. The internal TTS first generates a reasoning trajectory by Long-CoT using L.L.Mainkk and then predicts the final answer with this trajectory using T.T.Manwwer, which can be expressed as:
answer LLM (atnsseeeeLLLaaaii ((LMMinn((uuur))))
To be specific, recent methods (e.g. DeepSeck R1 (I et al. 2025) use the same policy model for both L.L.Maiink and I.T.Manswer-
3) External TTS. Firstly, the Long-CoT generation is extended by generating multiple reasoning trajectories and answers in parallel. Then, reward model (e.g. PRM) used score and select the best result (Lightman et al. 2023 Liu et al. 2025). This inference form can be described as:
answer = arg max LLMpRAS .E[1.A]
(L.LManwwer MLLLManwweeLLLMMMMC (LLManinking ( (query))ls).
(3)
where [+], denotes the i-th candidate among k parallel generations.
Though existing external TTS methods have been proven to obtain considerable performance en- hancement, they still encounter several problems: (1) Extra Computation: PRM contains individual parameters from the policy model (LLMuink and LLM which introduces additional huge computation. (2) Expensive Annotation: It is difficult to obtain the large-scale reasoning trajectory annotations for PRM training.
3

---

Published as a conference paper at ICLR 2026
SSR
Score Score,
"cthinko-Let
GRPO
SPRM Head
Policy Head
SPRM Head
Score
layer
Linear
N-1 layer
Dropout
Policy Model
Linear
Input tokens
Question/Answerr takens
<think>/</think> tokens
Think process tokens
Step- tokens
(a) Training framework
(b) Inference framework
Think,
S
5
SPRM Head
Policy
Policy Model
Model
Think
Sk
Figure 2: The training and inference framework of Reflective Generative Models.
Reflective Generative Form. To address the extra computation and expensive annotation issues, we propose new Reflective Generative Form focusing on the efficient and label-free reasoning trajectory selection. The proposed Reflective Generative Form shown in follows.
answer LLM answer
arg max LLMSPRM
(LLMiiinkngg (query)l:)
(4)
(E[1,]] share backbone
share backbone
Firstly, share the backbone of the policy model and PRM in single network, which enables reasoning trajectory generation and scoring unified interface for parallel prediction. The score measures the quality of each reasoning trajectory, and the trajectory with higher score is selected as the high-quality candidate TTS. This unified interface is proved to be effective for parame- ter reduction in experiments. Secondly, introduce novel Self-supervised Process Reward Model (SPRM) to eliminate the reliance on process-level annotation, which can be optimized with only outcome-level annotation in self-supervised manner. In particular, only implement the SPRM for the LLinkk selection, which can further improve the inference efficiency during the real implementation.
APPROACH
4.1 UNIFTED INTERFACE IN REFLECTIVE GENERATIVE FORM
Our proposed Reflective Generative Form establishes a unified interface for the policy model and the PRM. For the policy model, we employ reasoning LLMs that contain the thinking process in response, delineated by the "<think>' and '</think>" tokens. For the PRM, we introduce Self-supervised Process Reward Model (SPRM), which shares the same backbone the policy model but incorporates an additional lightweight SPRM head. The SPRM head implemented by binary classifier consisting of two linear layers and dropout layer: Linear(c, 2c) - ReLU - Dropout(0.5) + Linear(2c, 1), where c is the channel of the input hidden states. An overview of the joint framework is illustrated in Fig. a).
4

---

Published as a conference paper at ICLR 2026
Within this unified form, the policy model first generates multiple thinking processes as the reason- ing trajectories. Subsequently, the SPRM evaluates each thinking process for reasoning trajectory selection. The evaluation procedure contains two steps:
1. Step Segmentation. We segment each reasoning trajectory using tokens that are already sup- ported by the policy model's tokenizer, eliminating the need to introduce additional step-specific to- kens or fine-tune the LLM for step-format outputs. Specifically, we treat tokens containing '. An\nn as step-tokens and split the trajectory accordingly. Additionally, any sequence of consecutive step-tokens and ignore the step-token appearing at the beginning of the trajectory, does not contain valuable information.
retain only the first token in
2. Trajectory Score Prediction. After using step-tokens to mark the end of individual reasoning steps, we evaluate each step based on the representation of the corresponding step-token. Since the representation in the last layer mainly captures the logits prediction for a single token, we use the hidden representations from the second-to-lass layer of the policy model to provide richer contextual information of the entire step. These representations are then fed into the SPRM head to predict process scores for each step. When calculating the final score, [ightman et al. (2233) proposes to use the product of process scores. However, this results in lower final scores for longer reasoning trajectories. Thus, we further use the geometric mean the process scores to eliminate the influence step numbers.
*
NN II
N
Sunal II Score,
SPRM (froken.)
(5)
where N denotes the total number of steps, and fuken.. is the representation of the n-th step-token obtained from the policy model. Score, the SPRM's process score for n-th step.
Through this unified interface, single network can generate reasoning trajectories and score them in parallel, enabling joint training an end-to-end manner. This design facilitates a straightforward and efficient training pipeline for on-policy PRM learning, where both the policy model and the SPRM continuously refine their parameters from shared experiences, thereby improving the overall quality of the generated trajectories.
4.2 OPTIMIZATION OF REFLECTIVE GENERATIVE FORM
During optimization, we train the policy model and the SPRM head simultaneously. For the pol- icy model, we adopt Group Relative Policy Optimization (GRPO) following (Shoo al. (2024) To optimize the SPRM head, we propose a Self-supervised Process Reward Loss (SPR Loss), which enables learning process discrimination ability only from outcome reward (e.g. final answer correct- ness). The SPR Loss is formulated as follows,
N Σ Ily - BCELoss(Scoree,, vn), where 3ie [[Score, > 0.5), (6) 1=
LSPR
N
where I is the indicator function, denotes the step-tokens, Score, is SPRM's process score step n. yr the pseudo label from SPRM on step n, and y denotes whether the final answer from the policy model is correct. Since correct final answer may include incorrect intermediate steps and vice versa (Lightman et al. 2023). we optimize the process score based on both final answer correctness and the pseudo label from SPRM. Specifically, label is consistent with the final answer's correctness. This dynamic filtering allows the model to avoid noisy samples and focus on the most representative steps of correct and incorrect solutions. Thus, by enlarging the score gap between correct and incorrect steps, SPRM can progressively learn the process evaluation ability with only final annotations.
only update the steps when the pseudo
4.3 INFERENCE WITH REFLECTIVE GENERATIVE FORM
In the inference stage, our Reflective Generative Form is naturally suitable for TTS where the SPRM can provide guidance for selecting the high-quality reasoning trajectory from the policy model. The total inference process divides into three steps(shown in Fig. b)): (I) For the given ques- tion, the policy model first samples thinking processes as the candidate reasoning trajectories: think,, thinks,.-- thinks. (2) The SPRM evaluates the steps in each process and obtains the final
5

---

Published as a conference paper at ICLR 2026
score by the geometric mean of corresponding process scores: S1, S2. ..... Sg. (3) The reasoning trajectory with the highest final score is chosen and guides the policy model to answer the question (
answer Lns
(think;=), (tsiikkkttthhnk where i = argmar(Si, S S2..
4.4 DISCUSSION WITH OTHER METHODS
Comparison with other PRMS. Recent works (Rafailov et al.; Chen et al. 2025 Zhong et al.
2024 Yuan et al. 2024) can also generate process reward with only final answers. However, Rafailov et al. Zhong et al. (2024) require the reference model in Reinforcement Learning to help calculate the reward, which is mainly used for improving the training efficiency of the policy model. Chen cct al. (2025); Yuan et al. (2024) require training additional LL--ased reward models (e.g.. Llama- 3-70B-Instruce in Chen et al. (2025) and ImplicitPRM-Si in Yuan et al. (2024). Overall, these methods still depend external LLMs the reward model, thus assigning the process evaluation
capability to an additional model. In comparison, our method unifies the process reward model and the policy model within a single LLM, thereby integrating both reasoning and evaluation capabilities into a single model.
Comparison with other External TTS methods. Recent works Toshniwal et al. (2025); Qi et al. (2025) also focus External TTS. Toshniwal et al. (2025) integrates both response generation and evaluation into a single LLM by feeding the model's 's own responses back to itself through manually designed prompts. Qi et al. (2025) trains an additional reward model to jointly evaluate multiple sampled trajectories. However, these approaches still fail to fully unify the generation and evaluation processes. In Toshniwal al. (2025), it requires manually designed prompts. Besides, its generation and evaluation process are decoupled: the model must be queried multiple times to compare candidate responses, and its evaluation requires an autoregressive reasoning process, which introduces additional computational cost. Meanwhile, Qi et al. (2025) still requires training an additional LLM for evaluation. In contrast to them, our RGM unifies generation and evaluation in both model architecture and inference pipeline, requiring neither additional models nor extra forward passes.
EXPERIMENT
5.1 BASELINE & DATASET
We conduct experiments the baseline models with different sizes and architectures, including -i-icck----issil-----ee
(Guo et al. 2025 QWQ-32B (Temm 2025). Qwen3-32B Team 2025). and GPT-OSS-22 20B. For all models, add the SPRM head into the second-to-last layer while keeping the remaining architecture unchanged. Our training dataset is sampled from multiple publicly available math-related sources, including NuminaMath (Li et al. 2024).
DeepScaleR- -. Preview (Luo et al., 2025).
-MathR-MMtt-220-- DeepScaleR (Luo et al., (025). LIMR (Li et al., 2025 and OREAL-RL (Lyu et al. 2025). During data cleaning, first, we use difficulty classi- lication model trained on the MATH dataset to filter out easy data. Then, we use the pass rate to sample valuable data, following Li al. (2025). We finally sampled 40k training examples. In the training stage, the models are trained on 64 H200 GPUs with batch size of 128 and response length of 32k. We train the models using GRPO and our proposed SPR loss for 80 iterations (140 steps for QwQ-32B as explained in Sec5.4). In the inference stage, we use the sampling temperature of 1.0 for GPT-OSS-200 and 0.6 for other models. The output length is set to 38k for mathematical tasks and 32k for other tasks. We denote RGM, for models reasoning with i candidates in Eq 7
We evaluate our models on challenging mathematical benchmarks: AIME2024/2025 (AIME)
2025). BRUMO25 (Balunović et al. 2025), ) and HMMT25 (Balunović et al. 2025). To verify the bustness and generalization on other general tasks, we further introduce an extra of distribbbt
benchmark: BivecodeBencc(240055 (20000-22200000011 (Jain et al. 2024) (for coding capability evaluation).
Following recent works Zou et al. (202);; Choudhury (205), we use the results of BoN to test the capability of PRMs. We adopt Pass @ the evaluation metric. For each problem, the model

---

Published as a conference paper at ICLR 2026
Mathematical
Out-of- Out-of-Distriuutin
Model
TTS
AIME24
AIME25
BRUMO25
HMM125
LiveCodeBench
Open-Source Models
s1-32B
56.7
50.0
RI-Distill-Qeen-32B
72.6
33.3
GLM-Z1-32B-0414
59.1
DeepSeek-R1-61IB
79.8
70.0
44.4
65.9
Clored-Source Models
Claude-3.5-Sonnet
16.0
7.4
37.2
GPT-40-0553
9.3
11.6
32.9
OpenAl ol-mini
63.6
50.7
53.8
OpenAl o1-1217
79.2
63.4
OpenAl 03-mini(med)
79.6
74.8
80.0
53.0
67.4
30.0
37.4
19.3
22.9
+GRPO
29.9
38.9
18.1
22.4
+RM-72B
50.7
33.2
41.4
16.7
24.2
+PRM-72B
52.5
34.6
42.0
18.2
22.4
+5GM:-5M
53.1(+0.6)
35.7(+1.1)
43.2(+1.2)
21.3(+3.3)
26.6(+2.4)
RI-Distill RI-Distill-Qwen-TB
39.2
24.1
37.6
+GRPO
41,2
25.0
39.4
+RM-72R
56.5
43.8
55.8
27.5
41.7
+PRM-72B
60.1
47.3
29.9
42.8
+RGM,-26M
66.3(+6.2)
48.3(+100)
56.9(+1.1)
33.4(+3.5)
44.1(+1.3)
QwQQ32B
79.5
69.5
47.5
63.4
+GRPO
70.5
47.1
63.4
+RM-72B
82.9
71.7
46.4
62.9
+PRM--2B
72.3
51.7
63.0
+5GM:-54M
84.2(+0.9)
73.4(+1.1)
78.1(+1.6)
53.1(+1.4)
64.0(+11.)
Qwen3-32B
81.4
72.9
78.0
51.9
64.1
+GRPO
81.7
73.3
78.3
52.1
64.6
+RM-72B
80.5
74.4
79.2
52.3
63.3
+PRM-72B
82.9
77.1
56.7
64.9
+RGM,-54M
83.8(+0.9)
77.6(+0.5)
85.8(42.3)
56.3(-0.4)
66.2(+11.3)
(eT-)
80.0
72.1
71.2
55.5
67.9
+GRPO
78.1
73.9
71.7
55.8
69.8
+RM-72B
80.8
77.7
71.8
60.4
68.4
+PRM-72B
82.2
78.1
73.1
58.9
69.5
+1GM
81.9(-0.3)
79.1(+1.0)
73.3(+0.2)
63.4(+3.0)
69.9(+0.4)
Table 1: Comparison of our RGM and other models. The best results are shown in bold. The values in parentheses indicate the performance gain over other TTS methods. Lines start with "RGMg- XM" denotes our method and X denotes the number of parameters (in millions) of our SPRM.
generates only final answer, and the Pass score is computed the proportion of correctly solved problems. To improve the stability of the results, we repeat the evaluation times and report the average accuracy the final score.
5.2 MAIN RESULTS
Table summarizes the performance of proposed RGM across four representative benchmarks. The baseline the models trained with only GRPO on our training set. To evaluate the effectiveness of RGM, we compare it with widely adopted large-scale reward models: Swen2.5-Math-PP.
72B (an outcome reward model trained on 600k math problems) (Yang et al. 2024) and Qwen2.55 5- Math-PRM-72B (a process reward model trained on 500k math problems) (Zhang al., 2025).
These reward models are also applied to the baseline models with GRPO training. When testing with TTS, we sample candidates for each problem and predict the final answer based on the best candidate using the reward model. As listed in Table [1 the baseline with GRPO has similar results to the basic policy models. This is reasonable as we only trained the policy models with a few iterations. After using RGM, method consistently surpasses the baseline and the original models by a significant margin, especially on mathematical tasks. This verifies that the main improvement of RGM comes from our proposed SPRM. When comparing with other reward models, despite

---

Published as a conference paper at ICLR 2026
1.58 AIME24
1.5B AIME25
1.5B LivennieBench
AIME24
7B AIME25
LivessdeBesch
Operr Mal sadeeitlel
Opeer martc tag
Ovey UCS Mara 2il
Deendd Januet dan
Sweet Sunaet eeniie
Castibers
Contibans
Contuabiee
Candidore
Figure 3: Evaluation of varying numbers of candidate reasoning trajectories.
having only million-level parameters, our RGM achieves comparable or even better performance than billion-level reward models. Specifically, our RGM outperforms other 72B reward models on
AIME25, BRUMO25, and LiveCodeBench for all models. On AIME24 and HMMT25, our RGM still surpasses other 72B reward models on most architectures. Finally, we also compare RGM with several advanced spen-source models (DeepSeek-P) -ieess-R---sstt---weeeeeee
GLM-Z1 11211-32--4111 (GLM
et al. 2024), s1-32B (Muennighoff et al. 2025 and PeepSeek-R1-6711 (Guo et al. 2025) and
closed- sloued-source models (Claude-3.5-Sonne 1022, 4PT-4-0522, OpenAl ol o1-mini, OpenAl of-11117,
and OpenAT o3-mini-medium). Our RGM also surpasses all these advanced L.MMs, Especially on with RGM achieves performance comparable to OpenAl ol-mini, and QwQ-32B with RGM achieves performance comparable to OpenAl o3-mini. These demonstrates the ability of our RGM to improve the performance upper bound of advanced models.
mathematical tasks, R1 R1-Distill-Qwen-
In summary, the results demonstrate that: 1) With substantially fewer parameters and less train- ing data, the proposed SPRM achieves superior performance. This highlights the effectiveness of RGM in enhancing reasoning ability without the need for additional large-scale reward models. 2) Although SPRM is trained only on mathematical data, the improvements in reasoning capability generalize to other domains (e.g., Qwen3-32B improves from 64.6 66.2 on LiveCodeBench), Ap- pendix A further demonstrates its generalization ability on Chinese tasks. These results confirm that the reasoning gains from RGM are not limited to mathematical tasks. Instead, they generalize robustly to other domains, highlighting RGM's strong transferability.
5.3 ABLATION STUDY
The candidate number in SPRM. To examine the effect of the number of candidate reasoning trajectories h. we report the results for different i in Fig. 3 The results of Qwen2.5-Math- 72B and Dwen2.5-MMath-RRM-72
are also listed. It is shown that a larger results in higher performance. Besides, across different & and model sizes, SPRM consistently outperforms other large scale reward models, indicating its strong ability to distinguish between high and low quality reasoning trajectories. The detailed examples of SPRM are shown in Appendix E
Effectiveness of self-supervised optimization. We evaluate the effectiveness of SPRLoss in Ta- ble Compared with using the final answer correctness as process-level supervision for PRM training. our proposed self-supervised optimization method achieves larger performance gains on both 1.5B and 7B models. Furthermore, Fig. shows the prediction score gap between correct and incorrect solutions. Compared to the BCELoss. SPRLoss demonstrates stronger discriminative ca- pability with a larger score gap. This indicates that treating final answer correctness as process-leve labels introduces substantial label noise, which harms the optimization. In contrast, SPRLoss lever- ages self-supervised signals to reduce the impact of noisy supervision, leading to stable and accurate training.
The calculation of final score. In Eq. 5 we compute the final score using the geometric mean of the process scores. In Table we compare the geometric mean with two alternatives: the arithmetic mean and the direct product of scores. The results show that the geometric mean achieves the best performance, and there is a large gap between the product and the geometric mean of process scores. This is expected, as for long reasoning trajectories, the score tends to diminish as the number of steps increases, causing the final score to be overly sensitive to the response length. In contrast, the geometric mean alleviates this bias by reducing the impact of trajectory length.
8

---

Published as a conference paper at ICLR 2026
Model
Loss
AIME24
LiveCodeBench
BCELoss
56.7
27.9
DeepSealeR:1.BB-RGM..
SPRLoss
57.9
28.1
BCELoss
69.1
43.9
RI-Dissill-Qwann77BRRGMA
SPRLoss
70.2
44.4
Table 2: Evaluation on SPRLoss.
DeepScaleR-I.5B R.M.
RI-Distill Qwen-
0.4
Scate g04
0.25
0.35
0.30
SPRLoss
SPRLoss BCELoss
0.0
BCELoss
0.00
10
20
60
70
25
30
40
50
60
70
Training stogs
Training steps
Figure 4: The prediction score gap between correct and incorrect solutions. The blue curve shows the SPRLoss. The red curve shows the BCELoss.
Method
AIME24
AIME25
BRUMO25
HMMT25
Product
44.2
31.1
40.0
17.9
Arithmetic mean
52.9
35.2
43.3
21.1
Geometric mean
53.1
35.7
43.2
21.5
Table 3: Evaluation on the calculation of final score on BeepScaleR-1.B-RRGMM.
5.4 AНA MOMENT OF RGMS
Guo et al. 2025): ; Hu et al. (2025) propose that the "aha moment" enables the model to perform self- correction and self-reflectio In RGM, as propose SPRM head to evaluate itself, define an "aha moment" as the step at which the SPRM starts to discriminate the correct and incorrect reasoning trajectories. In Fig. 5 present curves of the final evaluation scores from RGM for correct and incorrect reasoning trajectories. At the initial phase of training, due to the prediction bias of the initial model, the scores for both correct and incorrect samples increase rapidly, indicating that the SPRM is dominated by the correct samples. Besides, the pseudo labels of the initial model also contain noisy interference for optimization. However, we observe step which the optimization behaviors for correct and incorrect reasoning trajectories begin to diverge, indicating that the SPRM the "aha moment". Formally, the "aha moment"
starts acquire the ability to discriminate defined the first training step at which the slope of the curve for correct trajectories becomes positive while that for incorrect trajectories becomes negative.
This observation indicates that, under our unified training framework and self-supervised loss, the bias issue can be mitigated. After this "aha moment", RGM iteratively learns to evaluate itself with SPRLoss, resulting in clear distinction between correct and incorrect reasoning trajectories and thereby enabling effective external TTS. The typical "aha moment" case study can be found in Appendix Specifically, since the "aha moment" for QwQ-32B occurs around 60 steps, we further train the model for additional 60 steps to ensure model convergence.
5.5 PROCESS-LEVEL EVALUATION OF RGMS
To evaluate whether the improvements of RGMs come from process-Level rewards, we conduct an additional experiment in which only the score of the final step token is used (treating SPRM as an outcome reward model). As shown in Table 4 incorporating step-level tokens within the reasoning trajectory leads to notable improvements final performance (4.6/0.7/0.7/2.7700

---

Published as a conference paper at ICLR 2026
DeepScaleR 1.50 RGM
RI Disill Qwen 7D RGM
QwQ 32B RGM
Qwen--22--RGM
GPT-00S-200--RGA
Munece
Toining sipps
Truiiing strps
Thisiing exp
sayo
Triining siogs
Figure 5: The training process of SPRM. The blue and red curves denote the final score on correct and incorrect reasoning trajectories. The green dashed line indicates the "aha moment".
on ANNEZHANNESSRRROOO///ASSSSSCCCCC This highlights the importance of process-level rewards in guiding reasoning.
Method
AIME24
AIME25
BRUMO25
HMMT25 LiveCodeBench
Outcome-level
48.5
35.0
42.5
18.8
25.7
Process-level
53.1
35.7
43.2
21.5
26.6
Table 4: Performance of DeepScaleR-1.5B-RGGM.
Furthermore, we introduce a process-level TTS method based on Monte Carlo Tree Search (MCTS) for evaluation. MCTS requires the reward model to select intermediate steps during the reasoning process, rather than only evaluating the whole reasoning trajectory. The inference settings for this experiment are provided in Appendix C In Table increasing the maximum number of searching tokens to means more intermediate steps are selected by our SPRM. And the performance consis- tently improves from 43.8 52.8 with increasing searching tokens. These results demonstrate the effectiveness of SPRM in providing high-quality process-level guidance.
Searching Tokens(k)
40
80
120
160
Accuracy( %)
43.8
48.8
50.0
51.7
52.8
Table 5: Performance of DeepScaleR-I.SB-RR
with MCTS on AIME24. A larger searching token number indicate more process steps are selected by our SPRM.
However, the performance of MCTS remains below that of the Best-of-N strategy reported in Ta- ble []] We attribute this gap primarily to the computational overhead of tree-based search, which leads to incomplete exploration under our experimental settings. Specifically, long reasoning trajec- tories in challenging tasks correspond to deep search levels in MCTS. This results in very large search space, as the total number of explored nodes grows rapidly with increasing depth. Conse- quently, the computational overhead of MCTS becomes extremely high. Under limited compute budget, stopping the search at an early stage not only restricts the search space but also risks miss- ing errors that appear in later steps of the reasoning trajectory, resulting in degraded performance. Nevertheless, the observed gains over the baseline confirm the capability of our RGMs to identify and search for better reasoning processes.
6 CONCLUSION
In this paper, we propose novel Reflective Generative Form, which enables single LLM to both generate and select high-quality reasoning trajectories for Test-Time Scaling (TTS). Based on this form, we present the reflective generative model (RGM). Specifically, we design unified interface that integrates the policy model and process reward model (PRM) within a single network, resulting in low parameter overhead and efficient TTS inference. A self-supervised process reward model (SPRM) is proposed to learn process-level evaluation with only final answer annotations. Exper- iments show our QwQ-32B equipped with our RGM reaches comparable performance to OpenAl o3-mini and our SPRM with million level parameters outperforms billion level reward models across most tasks and models.
10

---

Published as a conference paper at ICLR 2026
Acknowledgment. This work is supported by the National Nature Science Foundation of China (62425114, 62121002, U23B2028), and the Fundamental and Interdisciplinary Disciplines Break- through Plan of the Ministry of Education of China (YYB2025XDXM103)
REFERENCES
AIME AIME problems and solutions, 2025. /tthh//tps////ttoooooooooo/o///
wiki/ / www.oooooo.eaaaccoo.....oo...aa.
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating Ilms uncontaminated math competitions. arXiv preprint arXiv:2505.2528/ 2025.
Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with open models.
URL http:///www...aaaac/////////////
test time- compute
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21788 2024.
Hongzhan Chen, Tao Yang. Shiping Gao, Ruijun Chen, Xiaojun Quan, Hongtao Tian, and Ting Yao. Diseriminative policy optimization for token-level reward models. arXiv preprint arXiv:2505.23363 2025.
Qiguang Chen, Libo Qin, Jiaqi Wang, Jingxuan Zhou, and Wanxiang Che. Unlocking the capa- bilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thown tfhtoughh. Advances in Neural Information Processing Systems,
2024a.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. arXiv preprint arXiv:2412.211882 2024b.
François Chollet.
Openai 03 breakthrough high score arc-agi-pub, 2024.
URL
https:
http://wwwwoooooooeoeeeeoeee//rs 20rew
Accessed: 2024-12-20.
Sanjiban Choudhury. Process reward models for Ilm agents: Practical framework and directions.
arXiv preprint 10XX::2502210325.. 2025.
Team GLM, Aohan Zeng. Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai. Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui. Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlic Huang, Peng Zhang, Qinkai Zheng, Rui Lu. Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang. Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm- 130b to glm-4 all tools, 2024.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. math:-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint PrXiv:2501.04519 2025.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12941 2025.
Zhiyuan Hu, Yibo Wang. Hanze Dong, Yuhui Xu, Amrita Saha, Caiming Xiong, Bryan Hooi, and Junnan Li. Beyond aha!': Toward systematic meta-abilities alignment in large reasoning models. arXiv preprint 2505 2055.00544 2025.
11

---

Published as a conference paper at ICLR 2026
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese eval- uation suite for foundation models. Advances in Neural Information Processing Systems, 36: 62991-63010, 2023.
Aaron Jaech. Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.cm 079777 2024.
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng. Yongfeng Zhang, and Mengnan Du The impact of reasoning step length on large language models. arXiv preprint arXiv:2401.044 04995, 2024.
Adaline Labs. Inside reasoning models openai o3 and deepseek rl, 2025. URL https:// babs.adalline.ai///nneee rew.iisidddrreeiinnnnn...e reasoning model openai-onnaa Accessed: 2025-
02-18.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository. 13:9, 2024.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. arXiv:2502. 1886, 2025.
Limr: Less is more for rl scaling, arXiv preprint
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can lb Ilm surpass 405b 11m? rethinking compute-optimal test-time scaling. arXiv preprint arXiv:2502.06703, 2025.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint 24X.v:2406.06592, 2024.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol- preview with 1.5b model by httpp//////eettttoooooooooooooa.
RL- RR-1188000224446055ee33333333332
2025. Notion Blog.
Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprini arXiv:2502.06781, 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501. 19393, 2025.
OpenAI. Learning to reason with llms, 2024. URL httpp:://ppeaaa..ommnnbae learning-ter to- reason with 1 llms/]
Accessed: 2025-04-13.
OpenAl. Openai o3-mini evaluation, 2025. URL. httppess//geensss.coo/////e////p
03 33-mini/ Accessed: 2025-01-31.
Jianing Qi, Xi Ye, Hao Tang, Zhigang Zhu, and Eunsol Choi. Learning to reason across parallel samples for Ilm reasoning. arXiv preprint arXiv:2506.090// 09014, 2025.
12

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to qi Your language model is secretly a q- 2024b. URL https://arxiv. org/abs/2404.12338.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03. 00t.0020000300,, 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXivv2408..3331, 2024.
Xiaoyu Tan, Tianchu Yao, Chao Qu, Bin Li, Minghao Yang, Dakuan Lu, Haozhe Wang, Xihe Qiu, Wei Chu, Yinghui Xu, et al. Aurora: Automated training framework of universal process reward models via ensemble prompting and reverse verification. arXiv preprint arXiv:2502. 2025.
Qwen Team. Qwq 32b: Embracing the power of reinforcement learning, March 2025. URL
Shubham Toshniwal, Ivan Sorokin, Aleksander Ficek, Ivan Moshkov, and Igor Gitman. Genselect:
A generative approach to best-of-n. arXiv preprini arXiv:2507. / 17797, 2025.
Peiyi Wang. Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifci Li, Deli Chen, Yu Wu, and Zhifang Sui. sheh-shepperd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint XXX1:23122.09333 2023.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. 2024.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain- of-thought reasoning Ilms, arXiv preprint arXiv:2502.03373, 2025.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprins arXhv:2412.01981, 2024.
Maxwell Zeff. Openai's o3 suggests ai models are scaling in new ways but so are the costs, 2024. URL http:://eeeeeccccccccco/////////
the htg.://sse.ccuuteee.....eeeeeaaa 2bc.haaaaaaann.....ctaaaaaeeeed.
Accessed: 2024-12-23.
2224122::2024-12222
Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Yunhua Zhou, and Xipeng Qiu. Revisiting the test-time scaling of ol-like models: Do they truly possess test-time scaling capabilities? arXiv preprint arXiv:2502. 2025.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 77764735-64772 2024.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.0730) 2025.
Han Zhong, Zikang Shan, Guhao Feng. Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprint arXiv:2404.1888 18922, 2024.
Jiaru Zou, Ling Yang, Jingwen Gu, Jiahao Qiu, Ke Shen, Jingrui He, and Mengdi Wang. Reasonflux- prm: Trajectory-awant prms for long chain-of-thoughe reasoning in 1lm.. arXiv preprint TrXiv:2506.1880 2025.
13

---

Published as a conference paper at ICLR 2026
A EXTEND ON CHINESE TASKS
We adopt C-Eval (Huang et al. 2023 to evaluate the Chinese QA capability of our RGM. During inference, all tokens containing "An" treated as step-tokens. As shown in Table 6 although RGM is trained on English data, it consistently improves Chinese QA performance across all model architectures. These results suggest that RGM provides stable gains without causing performance degradation on unseen tasks, thereby demonstrating its applicability to more general scenarios.
Model
DeepScaleR- RI-Distill 11-Distill-wwen-77
QwQ-32B
Qwen3-32B
GPT-OSS GPT-OSS-20B(med))
Baseline 39.9
56.8
89.4
89.0
67.8
+RGMs
43.9
62.5
89.6
89.5
68.1
Table Performance RGM on C-Eval.
B ADDITIONAL ABLATION STUDIES
The threshold in SPRLoss. In Eq. 6 adopt hard threshold of 0.5 to obtain pseudo labels. Here, we further compare two dynamic alternatives that use the sample-wise mean and median as the threshold. The results reported in Table 7 All three thresholding strategies achieve comparable performance, while the hard threshold performs slightly better. We attribute this to the fact that fixed hard threshold provides sample-independen and absolute criterion for pseudo- label partitioning, enabling the predicted scores to be comparable across different samples.
Method
AIME24
AIME25
BRUMO25
HMMT25
Mean 50.0
35.0
45.0
21.1
Median
51.7
35.0
42.5
20.0
Hard
53.1
35.7
43.2
21.5
Table 7: Evaluation on the threshold in SPRLoss on DeepScaleR-1.5B-RGGN.
Independent reward model. In Table we train an external outcome reward model and an RGM without parameter sharing, as shown in Table 8 Compared with the outcome reward model, our RGM has smaller total parameters (share backbone), simpler training pipeline (one stage training), and higher performance.
As for the independent RGM, the results indicate that the hhared-parameter and external models achieve comparable performance improvement compared with the baseline (around 1% accuracy difference). This slight bias is reasonable as we combine tasks in a single model. Since the inde- pendent reward model also leads huge training (training 2 LLMs) and parameter cost(2X parameter cost than RGM), This does not affect our main conclusion that our method offers a lightweight and effective approach for External TTS.
Method
Extra Param
AIME24
AIME25
BRUMO25
HMMT25
Deepscaler-1.51
43.1
30.0
37.4
19.3
+Independent ORM
1.5B
51.7
33.3
41.7
18.9
+ndeependent RGMs
1.5B
54.7
36.7
43,8
21.9
+RGMg
5M
53.1
35.7
43.2
21.5
Table 8: Evaluation on independent reward models on BeepScaleR-1.5B-RGG.
The position of SPRM. We add the SPRM to the second-to- to-lat layer of the policy model. To further examine the effect of its placement, Table 9 compares it with attaching the SPRM to the final layer. The results showed that using the second-to-Be -a-sst layer can obtain better performance. This behavior is expected. First, the final layer must directly compute similarity with the policy
14

---

Published as a conference paper at ICLR 2026
classifier kernels for prediction, which limits its ability retain contextual information. Second, ing a single feature representation for two different classifiers introduces task interference, whereas intermediate-layee features more suitable for auxiliary objectives.
Method
ALME24
AIME25
BRUMO25
HMMT25
Last layer
48.8
34.6
41.3
18.8
Second-to-las layer
53.1
35.7
43.2
21.5
Table 9: Evaluation on the position of SPRM on
Comparison with majority. Table I0 further shows the comparison with majority voting. We can see that our SPRM obtains better performance. Besides, our RGM is more flexible. For some general scenarios (e.g., code generation), simple majority voting may be applicable.
Method
AIME24
AIME25
BRUMO25
HMMT25
Majority RGM
50.4
34.0
41.7
20.3
53.1
35.7
43.2
21.5
Table 10: Comparison with majority voting on DeepScaleR-1.5B-RRGM.
The model design. In this paper, we implement SPRM using two linear layers. Table I further analyzes different SPRM architectures: single linear layer and two-layer model with additional gating mechanism. The results indicate that a single linear layer is insufficient for reliable evaluation, whereas expressive architecture yields better performance.
Method
AIME24
AIME25
BRUMO25
HMMT25
Linear
48.5
35.5
38.9
20.0
Linear*2 Gate
53.3
35.6
46.7
22.3
Linear*2
53.1
35.7
43.2
21.5
Table 11: Evaluation with different architectures of SPRM DeepScaleR-1.5B-RGGG
Comparison with ImplicitPRM. The ImplicitPRM Yuaan et al. 2024 also does not require pro- cess annotations. However, it still depends on least one additional reward model (the externally trained 8B ORM) during prediction. In Table T2 we compare our RGM with ImplicitPRM-8B The results show that our RGM outperforms it while using fewer extra parameters. Moreover, similar to previous PRMs, the ImplicitPRM requires a two-stage training and inference pipeline, while our RGM unifies the process reward model and the policy model within a single LLM, enabling efficient test time scaling.
Method
Extra Param
AIME24
AIME25
BRUMO25
HMMT25
Deepscaler-1.5B
43.1
30.0
37.4
19.3
+ImpliciuPRM
8B
52.5
35.5
40.8
20.0
+RGMg
5M
53.1
35.7
43.2
21.5
Table 12: Comparison with ImplicitPRM on 1.555aalee--.BBRGGM......
C DETAILS FOR MCTS
In the expanding stage, expand 4 children for the selected node. Since the complete reasoning trajectory is very long in challenging benchmarks, we generate 1024 tokens as 1 step in each node reduce the complexity of MCTS. Instead of performing full simulations to the end of reasoning
15

---

Published as a conference paper at ICLR 2026
trajectory, directly use SPRM to estimate the value of current node during the searching stage. This enables a more efficient evaluation at each step and is more suitable for evaluating the pro- cess reward of our SPRM. To balance the computation cost, set the maximum number of total searching tokens during the MCTS process from 0 (without MCTS) to 160k for each question.
D DISCUSSION
Dependency of SPRM on Policy Models. Compared with existing pretrained reward models, proposed SPRM requires training on the target policy models, which introduces an additional training process when adapting to new policy models. However, we argue that such a design is in fact necessary for building reliable reward models. Traditional pretrained reward models can indeed be directly applied to unseen policy models, but they inevitably face the risk of out-of- distribution mismatch when the output patterns of the target model deviate from those observed in the reward model's training data. For example, applying R.M-72B on Qwen3-32B leads to a perfor- mance drop of -1.2 AIME24; applying PRM-72B on QwQ-32B results in 1 performance drop on HMMT25, To obtain better results, they still need to be finetumed to align the target model, which results in huge cost for training these large-scale LLMs. In contrast, our SPRM can be trained together when optimizing the policy models, which actually results in no additional training stages. Furthermore, SPRM adopts a lightweight architecture that converges within approximately 100 iterations. Thus, the training cost of SPRM is substantially lower than the optimization of the policy model itself, making both practical and efficient.
Transfer of SPRM. Since different LLMs have different hidden-state dimensions, SPRM cannot be directly applied to new model. To address this issue, we add a linear projection layer before the SPRM head to align the feature dimensions, and we only finetune the lightweight SPRM module. In Table[13] we transfer an SPRM trained on Qwen3-32B to DeepScaleR-1 show that, without requiring any training of the policy model, the SPRM learned from one model can be effectively transferred to another.
The results
Method
AIME24
AIME25
BRUMO25
HMMT25
Deepscaler-1.55 1.5B +SPRM
43.1
30.0
37.4
19.3
53.1
35.7
43.2
21.5
+SPRM from328
52.2
37.1
45.5
21.1
Table 13: Evaluation on transferring SPRM from Qwen3-32B to DeepScaleR-1.5B.
Limitation. First, despite the promising performance, our method requires the policy model to first generate complete reasoning trajectories, which can introduce latency. This is common problem for External TTS methods. However, unlike other External TTS methods that typically set from 64 to 512, we only set k=8 in most experiments, which makes easy to obtaín results through batch-parallel inference and thereby ensures latency comparable to the original model. Second, for tasks that do not require reasoning or require very little reasoning (e.g., real-world fact-based QA or simple questions), our RGM is less effective due the lack of reasoning rajectories and step tokens. However, in these cases, we believe the main bottleneck is the LLM's knowledge rather than TTS.
E
CASE STUDY
Case study of aha moment. We show an example in Fig. 6 use RGM before and after the aha moment for scoring. In this case, the model mistakenly confuses Ey and 1, resulting in an incorrect solution. Our model fails to recognize the error before the aha moment, while the model after the aha moment can correctly discriminate it.
We fix the reasoning trajectory and
Case study of SPRM Fig. 7 shows the visualization of step-wise evaluation scores from SPRM. It can be observed that SPRM effectively identifies low-quality processes generated by the policy model, including logical error (e.g. the misunderstanding of b in step 58 of example 1) and calcula- tion error (e.g., the incorrect computation 9 x 21 7 193 in step 32 of example 2). SPRM assigns
16

---

Published as a conference paper at ICLR 2026
Example Aha Moment in SPRM
Question: Let the sequence of rationals *2. be defined such that xj ff and [Mee ( 1] 42025 can be expressed for relatively prime positive inlegers and Find the remainder when m +n is divided by 1000.
Reasoning Trajectories:
Step:
Altematively, perhaps make substitution. Let define μı 1/5,, Then, 1/ex-1 =1/eg + (8/9)/e²2
Multiply both sides by r:::
6, -& (8/9)
Thus,
6 + Au: 8/9
Bo, not quite.
Score: 0.52 (before Aha Moment) -> 0.12 (after Aha Moment). Post-description: Encounter calculation error, model confuses Et and #x-1 during the simplifi cation.
Figure 6: Comparison of SPRM's predictions before and after the aha moment. Only key steps are listed. The error steps marked in red.
low scores to these low-quality steps. Since SPRM only outputs process scores, we additionally provide post-descriptions within the dashed boxes for better clarity.
F LLM USAGE STATEMENT
We declare that the LLM was only used seneral-purposss writing assistant improve the gram- mar of the manuscript. The LLM did not contribute to the research idcation, methodology design, experimental execution, data analysis, result interpretation.
17

---

Published as a conference paper at ICLR 2026
Example
Question: Among the 500 residents of Almeville, there are 195 who own ciamond ring, 367 who own set of golf and 562 who
parden quade. addition, residents who own exactly two these things. and 254 residents who own exactly three these things. Find the number of residents of Aimeville who own all four of these things.
the residents bag candy hearts.
Reasoning Ieajectories:
Step &
However, this have four diamond rings clubs (G), garden and candy bearts
Scone: 1.00 Defime variables.
Slep 58:
The number of residents who exactly two things b 437. This includes those who exactly two things among
own evactly thing D, exactly thing
Score: 0144, Post-description Encounier logical enror. As everyone owns should noi include those who exactly things among
Final score: 0.00
Example2
Question: Find the sum of all integer bases for which 17, is divisor of 97,.
Reasoning Trajectories:
Step 30:
Therefore, possible bases and 49.
Score: 1.00. Post description: The model successfully solved the question and the correct 21449-711.
Step
Let me check 21:
Compute (17a):
Compute (97m) + +7 186+7 = 193)
no, 28/7-196, assumption was wrong.
divides
Walt, perhaps my
Score: 0.04. Post Post-deseription Encounter caleulation eroor. x +7 should be 196 rather than 193. model discard
Hinal score: DLB
Figure 7: SPRM's predictions on reasoning trajectories. Only key steps are listed. Correct and error steps marked in green and red.
18


Reviews:
[{"review_id": "hNl6zAZG4H", "reviewer": "Reviewer_WqcA", "summary": "The paper introduces a Reflective Generative Model that unifies a policy model and a process reward model through a shared backbone. It proposes a filtering mechanism for learning a process reward module (SPRM) based on outcome rewards, aiming to eliminate the need for process-level annotation", "questions": "- What is the reasoning behind using the same backbone for the reward model and policy? Could this introduce bias or reward hacking effects?\n- What is the multi-agent data cleaning framework used to obtain high-quality samples for the dataset?\n- Why is SPRM described as a self-supervised method, given that it relies on filtered outcome correctness?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "Strong empirical validation: The experiments are extensive and include multiple benchmarks and LLMs.", "weaknesses": "- SPRM is a filtering mechanism, not self-supervised: The model filters step-level data via a binary weight that retains only steps consistent with the outcome.\n- Limited novelty: The shared backbone between policy and reward model is an engineering optimization rather than a conceptual advance in test-time scaling. Moreover, the paper does not study whether shared parameters introduce bias.\n- Terminology and clarity issues: The formulation of LLMs lacks rigor  (e.g., “basic LLM”)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l0Pk8H5gBW", "reviewer": "Reviewer_jVpd", "summary": "The paper introduces reflective generative models for test-time scaling, which use a shared network for policy and process reward models. This dramatically decreases the number of extra parameters to 50M. Additionally, the paper introduces a self-supervised loss (SPR Loss), which directly learns the quality of the reasoning trajectory from the outcome reward. The core idea is to have the same model both generate reasoning trajectories and score them with minimal extra parameters. The authors conducted a wide range of experiments across baseline models and demonstrated high performance (OpenAI o3-mini level), outperforming billion-scale reward models.", "questions": "1. The final trajectory score is defined as the geometric mean of the single-step scores (Eq. 5). It might be helpful to add a sentence on the motivation/justification for this choice. I assume the geometric mean is chosen to push all step scores to be decent, since a single low score can substantially reduce the overall score. It might be interesting to add what scenarios this choice of final score does not yet capture. For example, right now, the final score treats each single step score equally and independently. In the case where the reasoning trajectory later corrects for its earlier mistakes, the earlier bad step might undesirably penalize the entire trajectory. There is an interesting question of how to incorporate the trajectory structure into the score. Have you tried any alternative form? \n2. Minor type: Eq. 6 (line 245), score_i and y_i should be score_n, y_n.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Originality: The paper proposes a highly original idea of using the same backbone for policy and reward models. This idea is a novel and exciting extension of prior reward models, which are typically large and separately trained. This idea opens up a lot of exciting directions for enabling richer interactions between reasoning trajectory generation and evaluation. \n- Quality: The proposed framework is clearly defined and well-motivated. The experimental evaluation is comprehensive, including multiple models and benchmarks. \n- Clarity: The text and figures are clear and easy to understand. \n- Significance: The paper makes a highly significant contribution to test-time scaling.", "weaknesses": "1. The design of the self-supervised process reward loss (SPR loss) could benefit from additional motivation and clarification. Specifically, the binary weight w_n only includes a step in the loss when the predicted step score aligns with the final outcome. Why choose a hard threshold (0.5) vs other alternatives? Could such a hard cutoff potentially discard a large fraction of training samples, particularly early in training? And could this selective inclusion behavior relate to later observations, such as the “aha” moment? Also, a minor point: y_n is the correctness of the final answer, which shouldn’t depend on n? Why use a subscript (which could be misleading)? \n2. The paper’s discussion of the “aha moment” (Sec. 5.4; Fig. 5) is vague and under-analyzed. The authors highlight a green dashed line to indicate where correct and incorrect trajectory scores begin to diverge, yet provide no quantitative criterion for identifying this point. Visually, the gap between curves appears to increase gradually rather than showing a discrete transition. If there is indeed a transition, could it be simply explained by the use of a hard 0.5 threshold in the SPR loss? The authors could add more quantitative analysis on the representation or gradient through learning if this “aha moment” is indeed an important finding.  \n3. The paper claims that SPRM generalizes across domains, but this claim is only supported by results on LiveCodeBench. Given that mathematics and coding reasoning tasks share very similar structures, this claim of generalization currently lacks sufficient evidence. This claim could be substantially strengthened by either including evaluation on more diverse tasks or adding a discussion on what types of domains the current approach is expected to generalize well to and where its limitations might lie. For example, the segmentation choice of using ‘\\n’ might not be as suitable for tasks involving natural language?\n4. The explanation for why MCTS underperforms relative to Best-of-N (BoN) is speculative and unsupported by evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w9Doze0pal", "reviewer": "Reviewer_sWYC", "summary": "This work aims to improve the test-time performance of policy models. Specifically, the focus is on allowing the policy to generate multiple candidates for a given query and then using a process reward model to select the best candidate from the pool. The key innovation of this paper is sharing the backbone network between the policy and the process reward model to reduce parameter overhead. Additionally, the authors propose a method that leverages the consistency between final answer correctness and the scores generated by the process reward model, in order to mitigate the negative effects caused by false positive and false negative samples during the training of the process reward model.", "questions": "- What does the variable c represent in the Linear(c, 2c) at Line 206?\n\n- In Equation 6, what do the terms $y_i$ and $Score_i$ with the index $i$ represent?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- By sharing the backbone network, the proposed method reduces the inference cost of using the PRM to evaluate policy rollouts.\n\n- Experimental results show that the proposed SPRM achieves superior performance with the addition of fewer parameters.\n\n- The proposed method is simple and seems to be effective.", "weaknesses": "- Missing related work on process reward models. Several studies [1-4] also incorporate outcome labels to train a process reward model, which is highly relevant to this paper.\n\n- Other work [5] has introduced '\\n' as a step token. What is the rationale and benefit behind selecting '\\n\\n' instead? A concern is that if the policy model does not generate '\\n\\n', how would this method remain applicable?\n\n- Regarding line 219, I have a concern about the clarification: \"Since the representation in the last layer mainly captures the logits prediction for a single token, we use the hidden representations from the second-to-last layer of the policy model to provide richer contextual information.\" Why does the second-to-last layer provide richer contextual information than the last layer? More theoretical or empirical justification is needed for this assertion.\n\n[1] From r to Q: Your Language Model is Secretly a Q-Function, COLM 2024.\n\n[2] Discriminative Policy Optimization for Token-Level Reward Models, ICML 2025.\n\n[3] DPO Meets PPO: Reinforced Token Optimization for RLHF, ICML 2025.\n\n[4] Free Process Rewards without Process Labels, ICML 2025.\n\n[5] Let's Verify Step by Step, Arxiv 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3kfV3wvXyr", "reviewer": "Reviewer_RgpY", "summary": "This paper proposes RGM (Reflective Generative Model), a test-time scaling approach that shares the backbone network between the policy model and process reward model (PRM), adding only 50M parameters for trajectory scoring. The key innovation is a Self-supervised Process Reward Model (SPRM) that learns to evaluate reasoning steps using only outcome-level supervision. Experiments show strong empirical results across models of different sizes.", "questions": "- Is the \"Aha moment\" surprising? Isn't it just training dynamics?\n- Why did you not train a new ORM on the training data for SPRM?\n- Why are obvious baselines like majority voting/self-consistency missing from the paper?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Model overlap in the policy and reward model means that the inference overhead is minimal.\n- SPRM can be trained with just the outcome labels, and from the ablation experiments seems to correspond to process-level correctness. For example, score of the last step performs worse than the score of the entire sequence. \n- Good empirical results\n- Generalization beyond math to coding \n- The experimental setup is thorough and tested across multiple model sizes.", "weaknesses": "- Important baselines such as majority voting is missing. Moreover, there have been recent work such as \"GenSelect: A Generative Approach to Best-of-N\" (Toshniwal, 2025) and \"Learning to Reason Across Parallel Samples for LLM Reasoning\" (Qi, 2025) which demonstrate strong parallel reasoning performance. \n- The claims about beating 72B RM is somewhat misleading. The Qwen2.5-RM models were trained on short, non-reasoning CoT solutions, and are not suitable for scoring the long reasoning traces generated by models evaluated in this paper. The parameter count claim of 50M beating 72B RM in the abstract is also somewhat misleading because the RM is sharing the backbone.\n- MCTS results being slightly worse than Best-of-N suggests the RM is still not good enough to conduct search which begs the question if a process reward model is really buying any performance in this setup.  Ideally, an ORM trained on the same data as SPRM would be a more fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QYn4efgqW6", "reviewer": "Reviewer_65ZK", "summary": "The paper introduces Reflective Generative Models to enable test time scaling. Authors propose a unified interface where the policy model and PRM share the same backbone network, which saves parameter; they also introduce a Self-supervised PRM that learns to evaluate reasoning quality from only outcome-level supervision. Experiment results show that RGM with SPRM achieves performance comparable to OpenAI o3-mini on math benchmarks.", "questions": "* Can the authors provide some analysis on the model's robustness to initialization? \n* Is there a way to detect the aha moment online during training, or can you predict when it will occur based on model size or other factors?\n* Additional ablation studies or explanations of design choices regarding the model architecture would strengthen the paper. See weakness.\n* Given the dependency on training SPRM jointly with each policy model, have you explored whether SPRM trained on one model can transfer to similar architectures with fine-tuning? This would make the approach more practical for people who want to apply it to other base models without access to full retraining.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The unified interface design is elegant and addresses a real deployment challenge in test-time scaling. Achieving comparable performance to 72B parameter reward models with only 50M parameters is a solid contribution\n* The evaluation is thorough and well-designed with multiple base models and diverse architectures\n* Paper writing is clear and readable. The formalization of different inference paradigms provides a clean framework for understanding the contribution\n* The \"aha moment\" analysis is very interesting and provides valuable insights for future works", "weaknesses": "* The SPR loss introduces a bootstrapping process where the current model quality decides the psudolabel quality. I am curious if there is any formal analysis on the convergence properties. \n* Since the authors are proposing a novel architecture, I would expect more ablation studies on model design, for example the SPRM head MLP structure, and the geometric mean aggregation. \n* The method relies on '.\\n\\n' tokens as semantic boundaries, which assumes the policy model naturally produces well-segmented reasoning. This heuristic may not transfer to models with different output conventions, non-English languages, or domains where reasoning doesn't follow paragraph-like structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y8e2eftDwX", "reviewer": "Reviewer_uFte", "summary": "This paper focuses on parallel TTS techniques to enhance the reasoning capabilities of large models. The authors propose Reflective Generative Model (RGM), where the core idea is to have the policy model and the PRM share the same backbone network. This reduces the parameter and computational overhead associated with deploying a separate PRM. Experiments demonstrate that a 32B model equipped with just a 50M-parameter SPRM can outperform OpenAI's o3-mini on challenging reasoning benchmarks.", "questions": "Could you elaborate on how the \"Step-tokens\" are identified and inserted during the reasoning process?\n\nIs it possible to compare with the ImplicitPRM methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  clear performance improvement is observed as the number of candidates increases, proving the method's effectiveness. \n2. The shared-backbone design adds minimal parameter overhead, significantly reducing computational costs.\n3. As claimed by the authors, RGM enables efficient reasoning selection while eliminating the need for expensive process-level annotation.", "weaknesses": "1. Scalability Concerns: Like many PRM methods, this approach requires manually decomposing responses into \"Steps.\" This process relies heavily on human priors and empirical design, raising doubts about its scalability to new tasks.\n2. A core claimed contribution of the paper is removing reliance on process-level supervision. However, ImplicitPRM already claims they can fulfill this (learn PRM from ORM ). How do you position your work relative to this line of work, which suggests that a PRM can function without any extra parameters?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)