StepWiser: Stepwise Generative Judges for Wiser Reasoning

## Abstract

Abstract As models increasingly leverage multi-step reasoning strategies to solve complex problems, supervising the logical validity of these intermediate steps has become a critical research challenge. Process reward models address this by providing step-by-step feedback, but current approaches have two major drawbacks: they typically function as classifiers without providing explanations, and their reliance on supervised fine-tuning with static datasets limits generalization.
Inspired by recent advances, we reframe stepwise reward modeling from a classification task to a reasoning task itself.
We thus propose a generative judge that reasons about the policy model’s reasoning steps (i.e., meta-reasons), outputting thinking tokens before delivering a final verdict.
Our model, StepWiser ,
is trained by reinforcement learning using relative outcomes of rollouts.
We show it provides (i) better judgment accuracy on intermediate steps than existing methods; (ii) can be used to improve the policy model at training time; and (iii) improves inference-time search.

## 1 Introduction

As large language models (LLMs) increasingly tackle complex problems, they rely on multi-step reasoning strategies like Chain-of-Thought (CoT) (Wei et al., 2022) and ReAct (Yao et al., 2022) to decompose tasks and formulate better solutions. Consequently, ensuring these intermediate reasoning steps possess logical validity has become a critical research challenge. Process Reward Models (PRMs) have emerged as a potential tool to meet this need, providing step-by-step feedback for supervising learning, instead of relying on a single, often sparse, outcome-based reward (Lightman et al., 2023; Wang et al., 2023). However, this approach suffers from two major drawbacks. First, current PRMs typically function as “black-box” classifiers, providing a score or label without explaining why a step is correct or flawed. Second, their reliance on supervised fine-tuning (SFT) with static datasets can limit their ability to generalize to new reasoning patterns (Lightman et al., 2023; Luo et al., 2024; Wang et al., 2023; Xiong et al., 2024b; Zhang et al., 2024a).
In contrast, reasoning models themselves are trained to produce CoTs with reinforcement learning (RL) for best performance (DeepSeek-AI et al., 2025).

In this paper we propose to reward intermediate reasoning steps by first reasoning about those reasoning steps, before making a judgment – a meta-reasoning process which itself is trained by RL.
Our overall method (as shown in [Figure 1](#S1.F1)) to build such a stepwise generative judge involves 3 components: (1) a new self-segmentation technique to equip the base policy model with the ability to produce coherent and informative reasoning chunks (chunks-of-thought); (2) assignment of target rewards to chunks via relative outcomes of rollouts; and
(3) online training of judgment reasoning chains (i.e., reasoning about reasoning) and final reward judgments via RL. Our stepwise judge, termed StepWiser, can then be used to provide rewards either at training time or inference time in order to improve the reasoning ability of the policy model.

We conduct a comprehensive evaluation of our method across three key dimensions: (i) the judge’s classification accuracy on intermediate steps, e.g., via its score on ProcessBench (Zheng et al., 2024); (ii) its performance in a new inference-time search paradigm where the judge cleans up the reasoning history and re-samples – a method we propose for efficiently scaling sequential computation while maintaining the original generation length; and (iii) its utility in data selection for downstream model training. Our experiments demonstrate that our RL-trained generative stepwise judge significantly outperforms traditional SFT-based baselines and other existing methods across all axes of evaluation, where the ability to meta-reason – trained via RL – is the critical factor.

Figure: Figure 1: Overview of our StepWiser training method: we teach the model to segment its chain-of-thought (CoT) into coherent chunks. Then after each chunk, we generate Monte-Carlo rollouts to estimate the average success rate (i.e. Q-value) starting from that point. If the success rate goes up (or down) after a given chunk, we label it as good (or bad). Using these labels, we RL train a stepwise judge model that determines the quality of a given chunk after its own CoT reasoning.
Refer to caption: /html/2508.19229/assets/x1.png

## 2 Related Work

### 2.1 Process Reward models in LLM math reasoning

To improve the reliability of multi-step reasoning in LLMs, one can consider methods beyond
evaluating only the final answer, termed Outcome Reward Models (ORMs), by instead evaluating each intermediate step, a method pioneered by Process Reward Models (PRMs). Lightman et al. (2023) first demonstrated that a process-supervised model can significantly outperform an outcome-supervised one in guiding best-of-n sampling. However, their PRM800K dataset relied on intensive human annotation for each reasoning step, which is generally infeasible for larger, more diverse and challenging datasets.

Subsequent research has focused on automating this annotation process. Wang et al. (2023) proposed using Monte Carlo (MC) rollouts to estimate the Q-value of each step, while Luo et al. (2024) introduces a binary search method to efficiently identify faulty steps. Our work builds upon the MC-based annotation approach, exploring various methods for converting these Q-value estimates into effective learning signals.

In parallel, another line of work has established a theoretical connection between intermediate step values and the final outcome within the framework of KL-regularized Markov Decision Processes (Zhong et al., 2024; Rafailov et al., 2024). This result has been used to derive DPO-like objectives for learning an implicit PRM from outcome-only data (Xiong et al., 2024a; Cui et al., 2025; Zhou et al., 2025) or a KL-regularized version of the MC-based estimator (Zhang et al., 2024a). A recent work (Zha et al., 2025) prompts LLMs to evaluate each individual step before producing a final judgment, but supervises only the evaluation of the final answer. These methods share the goal of learning a stepwise judge from sparse signals. The major advantage of this line of work is that it is easy to train the judge in an online manner. A central question, which our work addresses, is whether the rich, explicit signals from extensive Monte Carlo sampling provide a more effective learning signals for RL-based reward training.

Concurrent work by He et al. (2025) uses a prompting approach to segment thought process into coherent chunks similar to ours. However, their stepwise judge is based only on prompting techniques that leverages hints in CoT like “Wait, I made a mistake”.
In contrast, our method focuses on training a judge using stepwise labels grounded in final verified answers.

### 2.2 Judge architectures

The process rewards described above can be used to train judges with different distinct architectures and training paradigms.

#### Discriminative PRMs

The most straightforward approach is to treat the task as a classification problem. This involves replacing the language model’s final layer with a linear head and fine-tuning it to predict a binary label for each step using a cross-entropy loss (Lightman et al., 2023). A more recent method formulates the task as next-token prediction, prompting the LLM to generate a pre-defined token (e.g., + or -) as its judgment (Wang et al., 2023; Xiong et al., 2024b). This approach further dates back to preference reward model training (Dong et al., 2024; Liu et al., 2023). Although this method uses a generative mechanism, its function remains purely discriminative, as it outputs a simple judgment without justification. We therefore group both under the discriminative category.

#### Generative judges with CoT reasoning

In sharp contrast, the second and most recent paradigm is the generative reasoning judge. Here, the evaluation itself is framed as a reasoning task. The judge first generates an explicit CoT to explain its rationale before outputting its final judgment. This approach was initially explored for preference learning and ORMs (Zhang et al., 2024b; Chen et al., 2025). There are also a few very recent works studying this paradigm shift in the context of stepwise judges, including Zhao et al. (2025); Zha et al. (2025); Khalifa et al. (2025). Though we share similar spirit of leveraging the inherent reasoning ability of the LLMs to train a stepwise judge, the algorithmic designs are distinctly different.

In contrast to Zhao et al. (2025) and Khalifa et al. (2025), who focus on offline rejection sampling fine-tuning, our investigation shows that such static training methods suffer from scalability issues, where the performance quickly stagnates after the initial few steps. In contrast, we cast the stepwise judgment as a reasoning task, and focus on online RL training. Zha et al. (2025) do use RL, but with sparse, trajectory-level supervision. Specifically, they prompt the LLMs to evaluate each individual step and final answer but only the final verification is supervised. Their approach assumes that to get an accurate evaluation of the final answer, models implicitly become a stepwise judge. In contrast, our framework is built on dense, stepwise supervision via rollouts. Our experiments will show that this explicit, online, stepwise signal is critical for training state-of-the-art generative judges.

## 3 Method: Training stepwise generative judges with RL

As depicted in [Figure 1](#S1.F1), our overall method StepWiser consists of three components:

- •
We equip the base policy model with the ability to self-segment Chain-of-Thoughts into coherent and informative reasoning chunks, called Chunks-of-Thought. This is done by creating SFT data with informative segments, so that the model can be trained to self-segment. We show that this causes no loss in performance for the base model.
- •
Given the chunks generated by the policy model, we annotate each chunk to create training data for our generative stepwise judge with binary target labels. This is done by comparing outcomes of rollouts starting before and after the given chunk using the outcome rewards.
- •
We perform online RL training using GRPO which trains our stepwise judge model to produce judgment reasoning chains (i.e., reasoning about reasoning) and reward final judgments that match the chunk labels from the previous step.

We describe the three components in detail in the following three subsections.

### 3.1 CoT Generation with Self-Segmentation (Chunks-of-Thought)

To train judges that can evaluate individual steps in a reasoning process, a key challenge is defining what a “step” is. While CoT reasoning enables models to reason step by step, properly segmenting this reasoning remains a difficult problem.

Current methods often segment reasoning trajectories using pre-defined tokens, like “Step 1, Step 2” or simply using double line breaks as delimiters. However, these heuristics frequently result in segments that are neither logically complete nor self-contained. Each segment contains only limited information, making it unsuitable as a standalone unit for a judge model to evaluate effectively. We present a representative example in [Table 2](#S3.T2) (left), where the model tends to insert double line breaks before and after a mathematical equation. This breaks an intuitively unified logical step into multiple different chunks, where one chunk contains a textual explanation, and the next with the corresponding equation.

Table: Table 1: Rules that we provide for an LLM to create segmented Chunks-of-Thought SFT data.

#### Achieving better step definition via self-segmentation

To mitigate this issue, we propose a method to teach the model to generate and simultaneously self-segment its own reasoning chains into more meaningful steps. First, we define the criteria for a high-quality reasoning step. The core idea is that each step should represent a complete logical leap or a self-contained part of the problem-solving process. Our definitions are given in [Table 1](#S3.T1). We then create our training data by:

- 1.
Generating a set of initial reasoning trajectories from the base model.
- 2.
Using an LLM prompted with our rules, to automatically segment these trajectories into logically coherent steps.

We fine-tune our base model on this data, thus teaching it to generate and simultaneously self-segment its own reasoning chains automatically. This self-segmentation ability is crucial for two main reasons. First, it produces more informative and logically complete steps, which provides better context for our judge model and improves its evaluation accuracy. Second, this method significantly reduces the total number of steps per trajectory. This reduction is also important because, as we will show, the process of annotating each step with a quality label is computationally expensive.

**Table 2: A representative example illustrating CoT segmentation methods based on a response from Qwen2.5-1.5B-it. Standard splitting by newlines (left) results in too many chunks, often separating textual explanations from their corresponding mathematical equations. In contrast, our self-segmentation approach (right) yields more coherent and meaningful chunks, with a clear, self-contained purpose—simplifying the second term in the previously established equation. Additional full examples are provided in the appendix.**
| Split by \n\n | LLM-based self-segmentation (Chunks-of-Thought) |
| --- | --- |
| …<br>chunk 9<br>Similarly, for the second term, we have:<br>chunk 10<br>$\left|\frac{z^{2}+4i-3}{z-i+2}\right|=\left|\frac{(x+yi)^{2}+4i-3}{x+yi-i+2}\right|.$<br>chunk 11<br>Since $z^{2}+4i-3=x^{2}-y^{2}+2xyi+4i-3=(x^{2}-y^{2}-3)+(2xy+4)i$, we get:<br>chunk 12<br>$z-i+2=x+(y-1)=x+yi-i+2=(x+2)+yi.$<br>chunk 13<br>Using the same logic as above, the magnitude is:<br>… | …<br>chunk 3<br>Similarly, for the second term, we have:<br>$\left|\frac{z^{2}+4i-3}{z-i+2}\right|=\left|\frac{(x+yi)^{2}+4i-3}{x+yi-i+2}\right|.$<br>Since $z^{2}+4i-3=x^{2}-y^{2}+2xyi+4i-3=(x^{2}-y^{2}-3)+(2xy+4)i$, we get:<br>$z-i+2=x+(y-1)=x+yi-i+2=(x+2)+yi.$<br>Using the same logic as above, the magnitude is:<br>$\displaystyle\sqrt{\left(x^{2}-y^{2}-3+x+y+2\right)^{2}+\left(2xy+4+x+y\right)^{2}}$<br>$\displaystyle=\sqrt{\left(x^{2}-y^{2}-1+x+y\right)^{2}+\left(2xy+x+y+4\right)^{2}}.$<br>chunk 4<br>… |

### 3.2 Stepwise Data Annotation

#### Stepwise data annotation via Q value estimation

Previous work has used human labelers to annotate correctness of each reasoning step (Lightman et al., 2023), although most such data is collected for proprietary models that we cannot access. Other works annotate steps automatically using methods like Monte Carlo estimation (Wang et al., 2023). We follow this second approach, using an estimated Q-value to measure the quality of each step.

For a given training prompt $x$ with verifiable outcome rewards, we generate a response from our policy model $\pi$ which segments its CoT into chunks $a=[a_{1},a_{2},\cdots,a_{H}]$, where $a_{i}$ is the i-th reasoning chunk. Then, the Q value of an individual step $a_{i}$ and its history is the expected final reward starting from that point:

$$ $Q^{\pi}\big([x,a_{1:i-1}],a_{i}\big):=Q^{\pi}(s_{i-1},a_{i})=\mathbb{E}_{a_{i+1:H}\sim\pi(\cdot|x,a_{1:i})}r^{\star}(x,a_{1:H}),$ (1) $$

where $s_{i}:=[x,a_{1:i-1}]$ is the history, and $r^{\star}$ is a final reward, which can be 1 for correct answers and 0 otherwise. We estimate this Q-value by generating $M$ full completions $a_{i+1:H}^{j}$ from that step $a_{i}$ and calculating the average final reward, i.e. the ratio of correct final answers:

$$ $\widehat{Q}^{\pi}\big(s_{i-1},a_{i}\big)=\frac{1}{M}\sum_{j=1}^{M}r^{\star}(x,a_{1:i},a_{i+1:H}^{j}).$ (2) $$

Following prior work (Wang et al., 2023; Xiong et al., 2024b), we can then assign a binary label to the step based on this Q-value:

$$ $\displaystyle y_{i}=$ $$

For convenience, we refer to this labeling approach as Absolute Q value thresholding (Abs-Q).

#### Rewarding the progress

One drawback of Abs-Q is its insensitivity to the dynamics of the reasoning process. For instance, it does not differentiate between a step that raises the success probability from 10% to 50% and one that drops it from 60% to 55%. To reward progress, we also explore methods that consider the change in Q-value.

Setlur et al. (2024) proposes to consider the change in value. Specifically, they define the notion of effective reward as a combination of Q value and advantage function of the best-of-n policy induced by $r^{\star}$:

$$ $Q^{\pi}(s_{i-1},a_{i})+\alpha\cdot A^{\mu}(s_{i-1},a_{i}),$ (3) $$

where $\alpha>0$ is a hyperparameter, and $A^{\mu}(s_{i-1},a_{i}):=Q^{\mu}(s_{i-1},a_{i})-Q^{\mu}(s_{i-2},a_{i-1})$. Here $\mu$ is taken as the best-of-n policy with $r^{\star}$. In other words, we generate $n$ responses from $\pi$ and use $r^{\star}$ to select the best one. In this case, $\mu$ satisfies that $Q^{\mu}(s_{i-1},a_{i})=1-(1-Q^{\pi}(s_{i-1},a_{i}))^{n}$(^1^11Assuming binary $\{0,1\}$ outcome rewards where $(1-Q^{\pi}(s_{i-1},a_{i}))^{n}$ is the probability of $n$ rollouts failing.). Therefore, the effective reward can also be estimated via Q value estimation. Accordingly, we consider an alternative approach of data annotation:

$$ $\displaystyle y_{i}=$ $$

where $\widehat{A}^{\mu}$ is the estimated advantage through the Monte-Carlo estimation of the Q value. We refer to this labeling approach as Relative Effective Reward Thresholding (Rel-Effective).

As a simpler alternative to capture relative improvement, we also consider a method based on the value ratio, where the label is determined as:

$$ $\displaystyle y_{i}=$ $$

Here $\gamma>0$ is a threshold and we refer this labeling approach as Rel-Ratio.

Using one of these methods, we can assign binary label $y_{i}$ to every step $a_{i}$ in a reasoning trajectory.
Since these labels come from unbiased estimates of the actual Q-values, they are likely to be more reliable compared to more ad-hoc methods.
For example, if a step $a_{i}$ is the first step with a mistake, rollouts starting after $a_{i}$ are more likely to fail compared to ones that start before the flawed step $a_{i}$.

### 3.3 Training the Judge via RL

At this stage we now have the recipe to create segmented (chunked) reasoning chains, each with a stepwise target label, across our training data. A straightforward approach would be to train a judge model as a classifier using standard SFT, as done in prior works (Wang et al., 2023; Xiong et al., 2024b). However, recent studies suggest that a more robust and effective (in their case, outcome-based) judge can be created by having it generate its own CoT analysis to evaluate the models’ responses (Zhang et al., 2024b; Chen et al., 2025; Whitehouse et al., 2025). In the meantime, this generative formulation naturally allows us to train a judge via reinforcement learning. We therefore frame the stepwise evaluation as a reasoning task where the judge model first generates an analytical rationale and then concludes with a final judgment. This approach is compelling also because it forces the judge to “show its work”, providing a more transparent and potentially more accurate evaluation process.

#### Task formulation and prompt dataset balancing

We decompose the full trajectories into step-level training prompts (judgment tasks). For each training prompt, the model is provided with the original problem $x$, the reasoning history $a_{1:i-1}$, and the new reasoning chunk $a_{i}$ to be evaluated. The model is prompted to generate its own CoT reasoning about the correctness of the step $a_{i}$, followed by a final judgment in a predefined format (e.g., enclosed in a box). Such CoT reasoning in the judge allows it to spend more compute and perform thorough analysis of a reasoning step $a_{i}$, which is likely necessary given $a_{i}$ itself is a part of CoT that performs non-trivial reasoning. See Table [3](#S3.T3) for the prompt template used.

A critical but often overlooked aspect of the stepwise judge training is that the stepwise labels can be highly imbalanced due to the data annotation process in Section [3.2](#S3.SS2). For example, with the Qwen2.5-1.5B-chunk model, $70.2\%$ of Abs-Q samples are labeled as correct. In our early experiments, we observe that these imbalanced training prompts can cause model degeneration, as the model can achieve a high score by simply predicting “correct” in most cases. To mitigate this, we make *prompt dataset balancing* an explicit part of our method: we down-sample the majority class so that the numbers of positive and negative prompts are equal. We find this balancing step to be essential for stable RL training, as it ensures the reward signal reflects the model’s discriminative ability rather than class frequency bias. We will study its impacts on the final model performance in Section [4.3](#S4.SS3).

#### Reward and RL training

The training signal for RL is direct and intuitive. For each step $a_{i}$, the judge model receives a reward of $1$ if its judgment aligns with the label $y_{i}$ (created by Monte-Carlo estimations), and 0 otherwise. We use GRPO (Shao et al., 2024) as our optimization algorithm due to its demonstrated effectiveness across multiple studies (Shao et al., 2024; DeepSeek-AI et al., 2025).

Table: Table 3: Prompt Template for our StepWiser judge.

## 4 Experiments

### 4.1 Experiment Setup

#### Model and data

We conduct experiments with the Qwen2.5-1.5B-it and Qwen2.5-7B-it instruction-tuned models (Yang et al., 2024) which have context window length of 8192. The ground-truth scores for solution verification are provided by Math-Verify(^2^22[https://github.com/huggingface/Math-Verify](https://github.com/huggingface/Math-Verify)). For our training data, we use mathematical problems from the NuminaMath-CoT dataset (Beeching et al., 2024). Before training, we preprocess the dataset by first removing duplicate prompts. We then use Math-Verify to extract the final answer from each reference solution and score it against the labeled ground truth. We discard any prompts where Math-Verify cannot successfully verify the answer. Unless otherwise specified, we focus on using the same base model to initialize both the policy and the judge. We also conduct experiments to study how the choice of base model affects the judge’s evaluation ability.

#### Self-segmentation fine-tuning

To create demonstration data for self-segmentation, we use a random subset of $20$k prompts from NuminaMath-CoT. First, we generate 16 responses for each prompt using the base policy model (i.e., Qwen2.5-1.5B-it). Next, we filter out incorrect responses, keeping up to $4$ correct solutions per prompt. We then prompt the strong Llama-3.1-70B-it (Meta, 2024) to segment these correct responses according to the rules in Table [1](#S3.T1). We generate 8 segmentations for each response and only keep those that perfectly reconstructed the original response and follow the required format. Finally, we fine-tune the base model on this collected data to create Qwen2.5-1.5B-chunk. For this fine-tuning, we use the open-source Axolotl package(^3^33[https://github.com/axolotl-ai-cloud](https://github.com/axolotl-ai-cloud)) with a learning rate of $1e-5$, a packing block size of $8192$ tokens, and a global batch size of $32$. The setup for Qwen2.5-7B-chunk is similar. We provide the prompt template we use for the response generation in Table [11](#A2.T11). Table [4](#S4.T4) reports both the average number of steps (evaluated on 10K randomly selected prompts from NuminaMath) and the average@32 performance on MATH500. After self-segmentation fine-tuning, the resulting models achieve comparable or slightly better test accuracy on MATH500, while maintaining similar response lengths. Notably, the number of steps decreases significantly compared to commonly used splitting by \n\n, from 9.6 to 6.0 with 1.5B model and from 9.9 to 6.8 with the 7B model, suggesting that the models generate responses in a more organized and structured manner. An illustrative example is shown in Table [2](#S3.T2), and additional case studies can be found in the Appendix. Meanwhile, we observe that for most current open-source thinking models that do long reasoning before answering, the number of steps exceeds 150 when trajectories are segmented using \n\n, with each step containing only about 30 tokens. Due to resource constraints, we do not experiment with these thinking models. However, we expect that our self-segmentation technique would be particularly beneficial in this setting, which we leave for future exploration.

#### Stepwise data annotation

We select a subset of $40$k prompts from NuminaMath for stepwise data annotation based on a pre-filtering process using the pass@k metric. Specifically, for each prompt, we generate 16 responses using our chunk-tuned models (e.g., Qwen2.5-1.5B-chunk). To ensure the selected prompts are of a suitable difficulty, we filter out prompts where the responses were either all correct or all incorrect. During generation, we use a temperature of 1.0 and set the maximum token limit to $8192$, or until the model produced a final answer. Then, for each intermediate step in a solution, we sample another $M=16$ completions staring from that step for estimating Q-values, as specified in [Equation 2](#S3.E2). The intermediate labels are then assigned according to the methods described in Section [3.2](#S3.SS2). Here we mainly follow the annotation framework from previous literature (Wang et al., 2023; Xiong et al., 2024b), which is well-suited to our specific research questions. While advanced engineering—such as employing model ensembles for generation or using powerful LLMs/human for label verification (Zhang et al., 2025)—may enhance results, these engineering strategies are orthogonal to our main contribution, and we believe they could be integrated for future improvements. The data annotation takes approximately $14$ days on 8 A100 GPUs using the Qwen2.5-7B-chunk model, making it computationally expensive. We notice that self-segmentation fine-tuning significantly reduces the number of chunks during trajectory splitting, thereby saving substantial compute and annotation time. We also present additional ablation results with and without chunking in Appendix [A.3](#A1.SS3).

**Table 4: Comparison of the base policy with and without self-segmentation fine-tuning. Overall performance is comparable, but self-segmentation results in less chunks than using split by \n \n. Here Avg@32 is the test accuracy averaged over 32 trajectories with random seeds.**
| Generator | Method | # Steps | # Tokens | Avg@32 on MATH500 |
| --- | --- | --- | --- | --- |
| Qwen2.5-1.5B-it | Split by \n\n | 9.6 | 686.7 | 44.2 |
| Qwen2.5-1.5B-chunk | Self-segmentation | 6.0 | 714.1 | 44.7 |
| Qwen2.5-7B-it | Split by \n\n | 9.9 | 733.0 | 73.3 |
| Qwen2.5-7B-chunk | Self-segmentation | 6.8 | 768.1 | 73.3 |

#### Judge RL training details

We implement the GRPO algorithm using the verl library (Sheng et al., 2024). During each training step, we use a per-prompt batch size of 1024 and a gradient update mini-batch size of 256. For the GRPO training process, the judge model generates $4$ responses for each prompt. The maximum prompt length is set to $3096$ tokens, and the model can generate up to $3096$ new tokens. The learning rate is set to $1e-6$.

In our initial experiments, we observe that the model’s entropy decreases rapidly, particularly for the 7B model. Since our verification task is a binary classification (correct/incorrect), low entropy causes the model to generate 4 responses with identical final judgments. This leads to zero gradients during the GRPO update, causing performance to get stuck after approximately 200 training steps. To mitigate this issue, we adopt the clip higher technique (Yu et al., 2025), using $\epsilon_{h}=0.28$ and $\epsilon_{l}=0.2$. We expect that more advanced methods, such as those proposed by Lanchantin et al. (2025), could further alleviate this issue.

We apply a heuristic filtering process to remove prompts that were overly short or excessively long. Additionally, we find that our threshold-based labeling method often results in an imbalance between positive and negative examples. Therefore, we downsample the majority class to create a balanced training set to stabilize training. We also include an ablation study on this process.

We run the training for 800 steps, which takes approximately $5$ days on 8 A100 GPUs (for Qwen2.5-7B-chunk models). We present a typical reward curve and an entropy loss curve in Appendix Figure [4](#A1.F4).

### 4.2 Evaluation on ProcessBench

**Table 5: ProcessBench results. The score of each subset is computed via [Equation 4](#S4.E4). Average accuracy (Avg) of our method StepWiser is better than all variants of our discriminative baselines, and existing baselines in the literature (first rows). Further comparisons are given in Appendix Table [10](#A2.T10).**
| Method | Learning signal | GSM8K | MATH | Olympiad | Omni-MATH | Avg $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Existing Reference Models |  |  |  |  |  |  |
| Math-Shepherd-PRM-7B | Abs-Q | 47.9 | 29.5 | 24.8 | 23.8 | 31.5 |
| RLHFlow-Llama3-8B-it | Abs-Q | 50.4 | 33.4 | 13.8 | 15.8 | 28.4 |
| Skywork-Qwen2.5-Math-7B-it | Abs-Q | 70.8 | 53.6 | 22.9 | 21.0 | 42.1 |
| Eurus-Qwen2.5-Math-7B-it (DPO) | Outcome | 56.6 | 43.0 | 27.3 | 26.8 | 35.1 |
| RL-TANGO-Qwen2.5-7B-it | Outcome | 53.1 | 48.2 | 37.8 | 36.3 | 43.9 |
| Qwen2.5-1.5B-chunk |  |  |  |  |  |  |
| Discriminative + SFT | Abs-Q | 39.3 | 32.1 | 19.3 | 18.9 | 27.2 |
| Discriminative + SFT | Rel-Effective | 40.8 | 37.2 | 18.7 | 20.1 | 29.2 |
| Discriminative + SFT | Rel-Ratio | 32.1 | 32.0 | 14.2 | 18.0 | 24.1 |
| Generative CoT + RL (StepWiser) | Abs-Q | 49.2 | 40.5 | 23.8 | 31.0 | 36.1 |
| Generative CoT + RL (StepWiser) | Rel-Effective | 48.2 | 43.6 | 22.1 | 25.3 | 34.8 |
| Generative CoT + RL (StepWiser) | Rel-Ratio | 46.9 | 43.4 | 26.3 | 28.4 | 36.2 |
| Qwen2.5-7B-chunk |  |  |  |  |  |  |
| Discriminative + SFT | Abs-Q | 54.8 | 45.9 | 28.0 | 26.9 | 38.9 |
| Discriminative + SFT | Rel-Effective | 55.6 | 48.7 | 26.4 | 28.3 | 39.7 |
| Discriminative + SFT | Rel-Ratio | 48.6 | 46.9 | 21.9 | 25.4 | 35.7 |
| Generative CoT + RL (StepWiser) | Abs-Q | 61.9 | 61.0 | 48.4 | 43.9 | 53.8 |
| + Maj@8 | Abs-Q | 65.5 | 62.1 | 49.7 | 45.7 | 55.8 (+2.0) |
| Generative CoT + RL (StepWiser) | Rel-Effective | 72.4 | 68.3 | 54.4 | 52.4 | 61.9 |
| + Maj@8 | Rel-Effective | 72.9 | 72.1 | 57.3 | 54.0 | 64.1 (+2.2) |
| Generative CoT + RL (StepWiser) | Rel-Ratio | 72.6 | 67.2 | 52.3 | 49.8 | 60.5 |
| + Maj@8 | Rel-Ratio | 74.3 | 69.0 | 53.8 | 50.2 | 61.8 (+1.3) |

We first evaluate our method StepWiser on ProcessBench (Zheng et al., 2024), a benchmark designed to test the ability to identify the first incorrect step in a reasoning trajectory, as labeled by human annotators. The benchmark contains 3500 problem-solution pairs from diverse math datasets (GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), Olympiad Bench (He et al., 2024a), and Omni-MATH (Gao et al., 2024)). Performance is measured by the harmonic mean(^4^44Note that while this score was referred to as the F1 score in the original paper, it is different from the standard F1 score.) of the accuracy on problems with correct final answers $\mathrm{acc}_{1}$ and those with incorrect final answers $\mathrm{acc}_{2}$, calculated as:

$$ $2\times\frac{\mathrm{acc}_{1}\times\mathrm{acc}_{2}}{\mathrm{acc}_{1}+\mathrm{acc}_{2}}.$ (4) $$

#### Our RL-trained StepWiser judge significantly outperforms SFT-trained discriminative judges

Our primary results on ProcessBench are presented in [Table 5](#S4.T5). The findings show that our RL-trained StepWiser judge significantly outperforms all variants of the SFT-trained discriminative judge. This holds true across all learning signals (Abs-Q, Rel-Ratio, Rel-Effective) and model scales. For instance, on the 7B model with the Rel-Effective signal, StepWiser achieves an average score of 61.9, far surpassing the discriminative baseline’s 39.7. This demonstrates that combining explicit reasoning generation with online RL training is a more effective strategy. As a reference, we also include several models from the open-source community that adopt similar discriminative training pipelines. These include model Math-Shepherd-PRM-7B (Wang et al., 2023), based on Mistral-7B, and RLHFlow-Llama3-8B-it (Xiong et al., 2024b), and Skywork-Qwen2.5-Math-7B-it (He et al., 2024b). Notably, the performance of these community-trained models is worse or comparable to that of our reproduced SFT-trained discriminative judge, and they similarly lag far behind our RL-trained StepWiser judge. Therefore, we conclude that the stepwise judge benefits from our proposed recipe of explicit reasoning traces and online reward optimization by RL.

#### Our RL-trained StepWiser judge significantly outperforms existing RL-trained judges

Furthermore, we benchmark StepWiser against other models trained with online methods like online DPO (Xiong et al., 2023; Xu et al., 2023) or GRPO (Shao et al., 2024) (e.g., Eurus-7B, RL-TANGO-7B). Unlike our method, these models are supervised at the
trajectory level, using only the final answer’s correctness as a reward signal, denoted by “Outcome” in [Table 5](#S4.T5). The StepWiser models, which are trained on explicit step-level signals from Monte-Carlo estimation, also surpass these baselines by a large margin. For example, our best 7B model scores 61.9 while RL-TANGO scores 43.9. This result strongly suggests that direct, step-level supervision provides a much richer and more effective learning signal than a sparse, outcome-only reward.

#### Test-time compute scaling via majority voting

Since StepWiser performs evaluation via CoT reasoning, a natural extension is to generate multiple judgments and use majority voting to decide the final judgment. We apply majority voting with the Qwen2.5-7B-chunk model, and the results are presented in [Table 5](#S4.T5). We can see that the majority voting shows consistent improvements in the ProcessBench score across various labeling methods. However, the overall gain from majority voting is modest compared to what is often observed in standard mathematical reasoning tasks. We hypothesize this is because our evaluation at each step is binary (correct vs. incorrect), resulting in a much narrower output space than the richer answer spaces typical in math reasoning. In such broader tasks, majority voting is more effective at reducing noise and improving robustness. In contrast, the binary nature of our judgments limits the potential benefit from aggregating multiple outputs.

**Table 6: Ablation study results on ProcessBench. The results show that both the generative CoT reasoning and RL components of our StepWiser method are important for overall results.**
| Method | GSM8K | MATH | Olympiad | Omni-MATH | Avg $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| Qwen2.5-1.5B-chunk |  |  |  |  |  |
| Discriminative + SFT (Baseline) | 32.1 | 32.0 | 14.2 | 18.0 | 24.1 |
| StepWiser (Generative Reasoning + RL) | 46.9 | 43.4 | 26.3 | 28.4 | 36.2 |
| – Ablate RL (use RS-FT) | 32.8 | 23.9 | 16.3 | 19.6 | 23.1 |
| – Ablate CoT (use Discriminative format + RL) | 42.0 | 43.2 | 23.6 | 28.7 | 34.3 |
| Qwen2.5-7B-chunk |  |  |  |  |  |
| Discriminative + SFT (Baseline) | 48.6 | 46.9 | 21.9 | 25.4 | 35.7 |
| StepWiser (Generative Reasoning + RL) | 72.6 | 67.2 | 52.3 | 49.8 | 60.5 |
| – Ablate CoT (use Discriminative format + RL) | 58.7 | 49.4 | 40.8 | 42.7 | 47.9 |
| – Ablate Prompt Balancing (Generative Reasoning + RL) | 58.8 | 54.8 | 41.0 | 36.9 | 47.9 |

### 4.3 Analysis of the Performance Gap

To better understand the source of the performance gap observed on ProcessBench, we isolate the contribution of each component by comparing StepWiser against three specialized baselines:

- •
Ablate RL. A StepWiser judge trained with rejection sampling fine-tuning (RS-FT, offline): We fine-tune the base model on a static dataset created via rejection sampling, a common offline approach. This isolates the effect of using the CoT format without online RL.
- •
Ablate CoT. A discriminative judge with RL (online): This baseline uses our full RL pipeline but trains the model to output a verdict token directly, without a CoT explanation first. This isolates the effect of online RL without the generative reasoning component.
- •
Ablate prompt dataset balancing. A StepWiser judge trained with our full RL pipeline but without prompt dataset balancing. For instance, for Rel-Ratio with a threshold of $0.8$, the proportion of positive samples now is 56.5% instead of 50%.

We evaluate performance using both the ProcessBench score and the in-distribution classification accuracy. For this study, we focus on the Rel-Ratio signal, where additional results with other labeling approaches are deferred to Appendix since the general trend is similar. The results, presented in Table [6](#S4.T6) and Figure [2](#S4.F2), clearly demonstrate that removing any of theses components leads to a significant drop in performance.

#### Online RL contributes to the performance improvement

The importance of online learning is evident when comparing our full StepWiser model to the RS-FT baseline. On ProcessBench using Qwen2.5-1.5B-chunk, the RS-FT model achieves an average score of only 23.1, which is substantially lower than StepWiser’s score of 36.2 and is even worse than the standard discriminative SFT baseline (24.1). To understand this phenomenon, we plot the loss curve of rejection sampling fine-tuning in [Figure 2](#S4.F2) (right). We notice that its training loss on a large, static dataset plateaus quickly. This trend is consistent across other learning signals and the larger 7B model, indicating that offline methods are insufficient to capture the complexity of CoT reasoning and reward modeling, making online RL a critical component.

#### StepWiser judge with CoT leverages intrinsic reasoning ability to obtain better evaluation

The benefit of the generative CoT format is illustrated by the “Ablate CoT” baseline. With the Qwen2.5-1.5B-chunk model, augmenting a discriminative-style judge with RL boosts the ProcessBench score from 24.1 (SFT) to 34.3 (RL), but it still falls short of the StepWiser model’s 36.2. Moreover, the in-distribution accuracy results in [Figure 2](#S4.F2) (left) show that the StepWiser model with CoT reasoning achieves higher accuracy on the held-out data. This suggests that generating explicit rationales provides a more expressive and informative structure for learning and modeling the stepwise reward signal. The gap between the generative CoT model and the discriminative model becomes much larger with the stronger Qwen2.5-7B-chunk. Specifically, the generative StepWiser model reaches an average score of 60.5, while the discriminative model only achieves 47.9. This is because we are leveraging the intrinsic reasoning ability of the base model through CoT in the judgment so the stronger model offers more advantages.

#### Prompt dataset balancing stabilizes training and mitigates overfitting

The practice of balancing the prompt dataset is also crucial for robust performance. Our ablation study on the Qwen2.5-7B-chunk model shows that removing this balancing step causes a substantial performance drop, with the average ProcessBench score dropping from 60.5 to 47.9. A deeper analysis reveals that while both the “Ablate CoT” ablation and the lack of dataset balancing hurt performance, their underlying failure modes are different. The “Ablate CoT” model suffers from a general decline in its ability to recognize correct and incorrect steps. In contrast, without balancing, the prompt dataset is heavily biased towards positive examples. This trains the model to be overly optimistic, developing a strong bias towards predicting any given step as correct. This bias is particularly enhanced during online training, which eventually leads to training instability and model collapse. A detailed analysis of this phenomenon is provided in the Appendix [A.4](#A1.SS4).

Figure: Figure 2: StepWiser ablation results. Left: Test stepwise accuracy of various stepwise judge setups. Both generative CoT and RL training are important for the best stepwise judge. Here we plot the results of Rel-Ratio using Qwen2.5-1.5B-chunk, other results are presented in the Appendix for completeness (see [Figure 5](#A1.F5)). Right: the training loss of rejection sampling fine-tuning, which saturates quickly.
Refer to caption: /html/2508.19229/assets/x2.png

### 4.4 Using the StepWiser judge to obtain better solutions

In this section, we evaluate the practical utility of our RL-trained StepWiser judge in two common applications: guiding an LLM’s reasoning process at inference time and selecting high-quality data for subsequent fine-tuning.

- 1.
Inference-Time Search via Chunk-Reset Reasoning.
To leverage the StepWiser judge for improved reasoning, we employ an inference-time search strategy.
The base policy model generates a solution “chunk-by-chunk”. After each chunk is produced, our StepWiser judge evaluates it. If the chunk is considered to be good, it is accepted, and the model proceeds to the next step. If it is rejected, the flawed chunk is discarded, and the policy model re-generates a new one from the same point (up to 5 attempts). This allows the model to self-correct and explore alternative reasoning paths without committing to an early mistake, enhancing the final solution’s quality. Meanwhile, this reasoning paradigm allows for scaling sequential compute (i.e., the compute used to extend a single inference trajectory with additional steps, rather than running many independent trajectories in parallel), while the overall number of accepted tokens remains similar.
- 2.
Training Data Selection via Stepwise Rejection Sampling Fine-tuning. Rejection Sampling Fine-tuning (Dong et al., 2023; Touvron et al., 2023) is a standard technique to improve a base policy by fine-tuning it on its own best outputs. The high-level intuition is that, when the models are allowed to generate $N$ responses per prompt, the pass@N is usually much higher than a randomly selected one (pass@1). Therefore, we can use a proxy reward model to select a training set from the self-generated responses, and train on this set to improve the base policy. However, for mathematical reasoning, verifying only the final answer provides a coarse, binary signal (correct/incorrect) that struggles to differentiate between multiple correct solutions. Here we use StepWiser to compute the scores for each individual reasoning chunk, and use the average score as a proxy to pick the best response from the correct ones.

**Table 7: Inference time search via Chunk-Reset Reasoning. We report results with both Qwen2.5-1.5B-chunk and Qwen2.5-7B-chunk, using them as both the response generators and the initialization checkpoints for the StepWiser judge. We see clear improvements using StepWiser across both model sizes, with similar accepted responses lengths (on MATH500). Rejected length is the number of tokens in removed chunks during inference time search.**
|  | Learning |  | NuminaMath |  | Accepted | Rejected |
| --- | --- | --- | --- | --- | --- | --- |
| Method | signal | MATH500 | Heldout-1K | Avg $\uparrow$ | length | length |
| Qwen2.5-1.5B-chunk | - | 44.7 | 17.6 | 31.2 | 616.0 | 0000.0 |
| Discriminative + SFT | Abs-Q | 47.7 | 19.1 | 33.4 | 625.2 | 0218.7 |
| Discriminative + SFT | Rel-Effective | 47.4 | 19.6 | 33.5 | 612.7 | 0302.4 |
| Discriminative + SFT | Rel-Ratio | 50.4 | 20.0 | 35.2 | 596.0 | 0475.8 |
| Generative CoT + RL (StepWiser) | Abs-Q | 51.4 | 19.8 | 35.6 | 599.1 | 1069.2 |
| Generative CoT + RL (StepWiser) | Rel-Effective | 52.1 | 21.2 | 36.7 | 602.0 | 0947.4 |
| Generative CoT + RL (StepWiser) | Rel-Ratio | 51.9 | 21.8 | 36.9 | 596.4 | 0884.7 |
| Qwen2.5-7B-chunk | - | 73.3 | 41.5 | 57.4 | 609.5 | 0000.0 |
| Discriminative + SFT | Abs-Q | 74.8 | 44.4 | 59.6 | 654.0 | 0168.2 |
| Discriminative + SFT | Rel-Effective | 76.9 | 46.1 | 61.5 | 654.6 | 0186.5 |
| Discriminative + SFT | Rel-Ratio | 76.7 | 45.8 | 61.3 | 641.4 | 0219.7 |
| Generative CoT + RL (StepWiser) | Abs-Q | 77.5 | 46.3 | 61.9 | 658.5 | 0345.7 |
| Generative CoT + RL (StepWiser) | Rel-Effective | 78.3 | 48.1 | 63.2 | 660.8 | 0425.8 |
| Generative CoT + RL (StepWiser) | Rel-Ratio | 79.0 | 47.5 | 63.3 | 653.0 | 0295.4 |

**Table 8: Data selection via Stepwise Rejection Sampling Fine-Tuning. Our StepWiser judge trained with RL provides better quality training data, as measured by final average test performance. The evaluation is with greedy decoding and a maximal generated length of 8192.**
| Method | Learning signal | MATH500 | NM-Heldout-1K | Average $\uparrow$ |
| --- | --- | --- | --- | --- |
| Qwen2.5-7B-chunk (greedy) | - | 75.6 | 44.6 | 60.1 |
| Outcome-based selection | - | 76.6 | 45.2 | 60.9 |
| Discriminative + SFT | Abs-Q | 78.4 | 45.3 | 61.8 |
| Discriminative + SFT | Rel-Effective | 78.2 | 45.2 | 61.7 |
| Discriminative + SFT | Rel-Ratio | 78.2 | 45.7 | 61.9 |
| Generative CoT + RL (StepWiser) | Abs-Q | 79.0 | 46.1 | 62.5 |
| Generative CoT + RL (StepWiser) | Rel-Effective | 79.4 | 46.7 | 63.0 |
| Generative CoT + RL (StepWiser) | Rel-Ratio | 79.0 | 46.8 | 62.9 |

The results for inference-time search and data selection are presented in Table [7](#S4.T7) and Table [8](#S4.T8), respectively. We summarize the key findings below.

#### Consistent performance improvements in both setups

In both applications, using our StepWiser judge leads to superior outcomes. For inference time search (Table [7](#S4.T7)), with the Rel-Ratio learning signal, our approach guides the 1.5B model to achieve an average accuracy of 36.9%, a significant improvement over the 31.2% of the base model. We also see clear trends of our StepWiser model being superior to the disciminative models across all learning signals. This trend holds for the 7B model, demonstrating the scalability of our method. Similarly, when used for data selection (Table [8](#S4.T8)), models fine-tuned on data selected by the StepWiser judge achieve the highest performance (63.0%), surpassing the original base model (60.1%), as well as data selected by a discriminative judge (61.9%) or outcome-based selection (60.9%).

#### Superior reasoning error detection using our approach

The “Accepted Length” column in Table [7](#S4.T7) shows that the final solutions from StepWiser are of a similar length to the baselines. However, the “Rejected Length” column, which corresponds to chunks that judged to be flawed, indicates that our model generates more total tokens to arrive at its final answer. We interpret this as direct evidence of StepWiser’s superior ability to identify incorrect or unproductive steps. This triggers the reset mechanism more effectively, forcing the model to discard flawed reasoning and find a better path. This is also consistent with the higher accuracy of identifying the error step on ProcessBench (see also Table [10](#A2.T10) in the appendix).

#### Relative signals are more effective for assigning chunk labels

We also observe that across both tables, training signals that reward relative progress (Rel-Effective, Rel-Ratio) consistently yield better judges than a signal that only measures a step’s absolute quality (Abs-Q). This appears to be consistent across all results in the paper, for all model sizes and evaluation setups.
For example, in inference time search, the Rel-Effective judge achieves a 64.3% average accuracy, outperforming the Abs-Q judge (61.9%). This pattern is also confirmed in data selection, where the Rel-Effective judge produces the best fine-tuning dataset, leading to a final model with 63.0% accuracy, again surpassing the Abs-Q judge (62.5%).

## 5 Conclusion

Reasoning models that output internal thought tokens before a final response have proven to outperform non-reasoning models.
In this paper we have shown that further improvements can be found by making models reason about the reasoning decisions made within those internal thoughts.
We provide a recipe to: (1) segment reasoning into chunks-of-thought;
(2) assign rewards to chunks via relative outcomes of rollouts; and (3) train a judge model to reason about the quality of CoT chunks via reinforcement learning (RL).

Our stepwise generative judge StepWiser is shown to be superior to existing methods on ProcessBench,
to provide improved inference time search, and better training time rewards for building better response models.
We show that both the use of reasoning during judgment, and training with RL in order to reason about reasoning, are important components to achieve this performance.

## References

- Beeching et al. (2024)
Edward Beeching, Shengyi Costa Huang, Albert Jiang, Jia Li, Benjamin Lipkin, Zihan Qina, Kashif Rasul, Ziju Shen, Roman Soletskyi, and Lewis Tunstall.
Numinamath 7b cot.
[https://huggingface.co/AI-MO/NuminaMath-7B-CoT](https://huggingface.co/AI-MO/NuminaMath-7B-CoT), 2024.
- Chen et al. (2025)
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al.
Rm-r1: Reward modeling as reasoning.
*arXiv preprint arXiv:2505.02387*, 2025.
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021.
- Cui et al. (2025)
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding.
Process reinforcement through implicit rewards, 2025.
URL [https://arxiv.org/abs/2502.01456](https://arxiv.org/abs/2502.01456).
- DeepSeek-AI et al. (2025)
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong
Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu,
Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- Dong et al. (2023)
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang.
RAFT: Reward ranked finetuning for generative foundation model alignment.
*Transactions on Machine Learning Research*, 2023.
ISSN 2835-8856.
URL [https://openreview.net/forum?id=m7p5O7zblY](https://openreview.net/forum?id=m7p5O7zblY).
- Dong et al. (2024)
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang.
Rlhf workflow: From reward modeling to online rlhf.
*arXiv preprint arXiv:2405.07863*, 2024.
- Gao et al. (2024)
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al.
Omni-math: A universal olympiad level mathematic benchmark for large language models.
*arXiv preprint arXiv:2410.07985*, 2024.
- He et al. (2024a)
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al.
Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.
*arXiv preprint arXiv:2402.14008*, 2024a.
- He et al. (2024b)
Jujie He, Tianwen Wei, Rui Yan, Jiacai Liu, Chaojie Wang, Yimeng Gan, Shiwen Tu, Chris Yuhao Liu, Liang Zeng, Xiaokun Wang, Boyang Wang, Yongcong Li, Fuxiang Zhang, Jiacheng Xu, Bo An, Yang Liu, and Yahui Zhou.
Skywork-o1 open series.
[https://huggingface.co/Skywork](https://huggingface.co/Skywork), November 2024b.
URL [https://huggingface.co/Skywork](https://huggingface.co/Skywork).
- He et al. (2025)
Tao He, Rongchuan Mu, Lizi Liao, Yixin Cao, Ming Liu, and Bing Qin.
Good learners think their thinking: Generative prm makes large reasoning model more efficient math learner.
*arXiv preprint arXiv:2507.23317*, 2025.
- Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the math dataset.
*arXiv preprint arXiv:2103.03874*, 2021.
- Khalifa et al. (2025)
Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, and Lu Wang.
Process reward models that think.
*arXiv preprint arXiv:2504.16828*, 2025.
- Lanchantin et al. (2025)
Jack Lanchantin, Angelica Chen, Shehzaad Dhuliawala, Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ilia Kulikov.
Diverse preference optimization.
*arXiv preprint arXiv:2501.18101*, 2025.
- Lightman et al. (2023)
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step.
*arXiv preprint arXiv:2305.20050*, 2023.
- Liu et al. (2023)
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu.
Statistical rejection sampling improves preference optimization.
*arXiv preprint arXiv:2309.06657*, 2023.
- Luo et al. (2024)
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al.
Improve mathematical reasoning in language models by automated process supervision.
*arXiv e-prints*, pp. arXiv–2406, 2024.
- Meta (2024)
Meta.
Introducing meta llama 3: The most capable openly available llm to date.
*Meta AI Blog*, 2024.
[https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/).
- Rafailov et al. (2024)
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn.
From r to q*: Your language model is secretly a q-function.
*arXiv preprint arXiv:2404.12358*, 2024.
- Setlur et al. (2024)
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar.
Rewarding progress: Scaling automated process verifiers for llm reasoning.
*arXiv preprint arXiv:2410.08146*, 2024.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Sheng et al. (2024)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
*arXiv preprint arXiv: 2409.19256*, 2024.
- Touvron et al. (2023)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
*arXiv preprint arXiv:2307.09288*, 2023.
- Wang et al. (2023)
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui.
Math-shepherd: Verify and reinforce llms step-by-step without human annotations.
*CoRR, abs/2312.08935*, 2023.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
*Advances in neural information processing systems*, 35:24824–24837, 2022.
- Whitehouse et al. (2025)
Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason Weston, Ilia Kulikov, and Swarnadeep Saha.
J1: Incentivizing thinking in llm-as-a-judge via reinforcement learning.
*arXiv preprint arXiv:2505.10320*, 2025.
- Xiong et al. (2023)
Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang.
Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf.
*arXiv preprint arXiv:2312.11456*, 2023.
- Xiong et al. (2024a)
Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, et al.
Building math agents with multi-turn iterative preference learning.
*arXiv preprint arXiv:2409.02392*, 2024a.
- Xiong et al. (2024b)
Wei Xiong, Hanning Zhang, Nan Jiang, and Tong Zhang.
An implementation of generative prm, 2024b.
- Xu et al. (2023)
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston.
Some things are more cringe than others: Preference optimization with the pairwise cringe loss.
*arXiv preprint arXiv:2312.16682*, 2023.
- Yang et al. (2024)
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al.
Qwen2. 5 technical report.
*arXiv preprint arXiv:2412.15115*, 2024.
- Yao et al. (2022)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
*arXiv preprint arXiv:2210.03629*, 2022.
- Yu et al. (2025)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al.
Dapo: An open-source llm reinforcement learning system at scale.
*arXiv preprint arXiv:2503.14476*, 2025.
- Zha et al. (2025)
Kaiwen Zha, Zhengqi Gao, Maohao Shen, Zhang-Wei Hong, Duane S Boning, and Dina Katabi.
Rl tango: Reinforcing generator and verifier together for language reasoning.
*arXiv preprint arXiv:2505.15034*, 2025.
- Zhang et al. (2024a)
Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, and Tong Zhang.
Entropy-regularized process reward model, 2024a.
URL [https://arxiv.org/abs/2412.11006](https://arxiv.org/abs/2412.11006).
- Zhang et al. (2024b)
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal.
Generative verifiers: Reward modeling as next-token prediction.
*arXiv preprint arXiv:2408.15240*, 2024b.
- Zhang et al. (2025)
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin.
The lessons of developing process reward models in mathematical reasoning.
*arXiv preprint arXiv:2501.07301*, 2025.
- Zhao et al. (2025)
Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, et al.
Genprm: Scaling test-time compute of process reward models via generative reasoning.
*arXiv preprint arXiv:2504.00891*, 2025.
- Zheng et al. (2024)
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin.
Processbench: Identifying process errors in mathematical reasoning.
*arXiv preprint arXiv:2412.06559*, 2024.
- Zhong et al. (2024)
Han Zhong, Guhao Feng, Wei Xiong, Li Zhao, Di He, Jiang Bian, and Liwei Wang.
Dpo meets ppo: Reinforced token optimization for rlhf.
*arXiv preprint arXiv:2404.18922*, 2024.
- Zhou et al. (2025)
Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li.
Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks.
*arXiv preprint arXiv:2503.15478*, 2025.

## Appendix A Additional Experiment Details and Results

### A.1 Discriminative Judge Training and RL Training

#### Discriminative stepwise judge training

We follow Xiong et al. (2024b) to formulate the discriminative stepwise judge as a multi-turn conversation task. Specifically, in every user turn, we provide a single step of reasoning, while in the next assistant turn, the model will decode either “+” or “-” token to indicate its judgment.
For training, we use standard SFT code. The data is packed into a block with length 8192 tokens. We use a learning rate of $1e-5$, a global batch size of $32$. We also mask out the user turn’s loss. We present the representative training loss curves in Figure [3](#A1.F3).

Figure: Figure 3: The training loss curves of discriminative stepwise judge under different learning signals. Left: 1.5B model, Right: 7B model.
Refer to caption: /html/2508.19229/assets/x4.png

#### Hyperparameter search for stepwise labeling

We conduct hyperparameter tuning for the learning signals labeling. We mainly search by training discriminative models and SFT training, as this is more computationally efficient than full RL training. For Rel-Ratio, we search over $\gamma\in\{0.6,0.7,0.8,1.0,1.2\}$, and for Rel-Effective, we search over $\alpha\in\{0.2,0.4,0.6,0.8,1.0\}$ with $\mu$ set as the best-of-4 policy induced by the base policy. For Qwen2.5-1.5B-chunk, we choose $\gamma=0.8$ and $\alpha=0.4$, while for Qwen2.5-7B-chunk, we use $\gamma=0.7$ and $\alpha=0.8$.

#### Learning curves of RL training

In [Figure 4](#A1.F4), we present a representative example of training reward, entropy loss, and response length with and without clip higher technique. The model is Qwen2.5-7B-chunk and the learning signal is Rel-Ratio with threshold 0.7. We can see that clip higher helps to encourage exploration and leads to a higher training curve.

Figure: Figure 4: A representative example of training reward, entropy loss, and response length with and without clip higher technique. The model is Qwen2.5-7B-chunk and the learning signal is Rel-Ratio with threshold 0.7.
Refer to caption: /html/2508.19229/assets/x6.png

### A.2 Additional Result on Classification Accuracy

We plot the stepwise classification accuracy under different learning signals in Figure [5](#A1.F5). For Rel-Ratio and Rel-Effective, we observe that the RL-trained generative StepWiser judge achieves significantly higher test accuracy on both intermediate steps and final answer correctness. This suggests that the additional reasoning steps enable a more expressive model class that better fits the learning signal. In contrast, for Abs-Q, the performance gap between the discriminative and generative judges is relatively small. We suspect this is because the Abs-Q dataset has a high proportion (70.2%) of positive samples. To stabilize RL training, we downsample the positive class, which may introduce distribution shift and affect test classification accuracy. Nevertheless, our StepWiser model still achieves substantially higher accuracy in evaluating final answer correctness.

Figure: Figure 5: The test stepwise accuracy of different stepwise judges. From left to right, we plot the results of Abs-Q, Rel-Ratio, Rel-Effective, respectively. The stars at step 10 represent the accuracy of recognizing the final answer.
Refer to caption: /html/2508.19229/assets/x9.png

### A.3 Additional Result on Self-segmentation Fine-tuning

**Table 9: The main ablation results on self-segmentation fine-tuning an chunking.**
| Method | Learning signal | # Steps | GSM8K | MATH | Olympiad | Omni-MATH | Ave |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Split by \n\n | Abs-Q + SFT | 5457820 | 33.7 | 37.1 | 20.2 | 18.9 | 27.5 |
| Split by \n\n | Abs-Q + RL | - | 46.3 | 38.4 | 19.0 | 25.8 | 32.4 |
| Split by \n\n | Rel-Ratio + SFT | - | 28.3 | 30.9 | 15.5 | 21.0 | 23.9 |
| Split by \n\n | Rel-Ratio + RL | - | 46.3 | 39.1 | 17.3 | 21.3 | 31.0 |
| Self-segmentation | Abs-Q + SFT | 3463520 | 39.3 | 32.1 | 19.3 | 18.9 | 27.2 |
| Self-segmentation | Abs-Q + RL | - | 49.2 | 40.5 | 23.8 | 31.0 | 36.1 |
| Self-segmentation | Rel-Ratio + SFT | - | 32.1 | 32.0 | 14.2 | 18.0 | 24.1 |
| Self-segmentation | Rel-Ratio + RL | - | 46.9 | 43.4 | 26.3 | 28.4 | 36.2 |

In terms of task performance, the benefits of self-segmentation are most apparent in the context of RL training. While the average scores for SFT are closer (27.5 vs. 27.2 for Abs-Q), models trained with RL show significant improvements. Specifically, the average score for Abs-Q + RL increased from 32.4 to 36.1, and the Rel-Ratio + RL score rose from 31.0 to 36.2. This disparity suggests that the self-segmentation process effectively filters out noisy and less informative steps from the reasoning trajectories. The resulting cleaner signal is highly beneficial for the RL training process, which is more sensitive to data quality. This observation is also consistent with our findings from the prompt filtering and balancing experiments. Conversely, SFT appears more robust to this type of noise, and thus its performance is less impacted.

### A.4 Additional Ablation Result on CoT and Prompt Dataset Balancing

#### Prompt dataset balance technique

In our early experiments, we observed that the prompt sets could be highly imbalanced. For example, using the Qwen2.5-1.5B-chunk model, the proportion of positive samples for Rel-Ratio with a threshold of 0.8 is 56.5%, while for Abs-Q, it is 70.2%. Such imbalance may cause model degeneration, as the model can achieve a high score by simply predicting “correct” in most cases. To address this, we consider a prompt balance technique that down-samples the majority class.

#### Generative judge without CoT

Another key component of our approach is the use of CoT reasoning before producing the final judgment. To assess its impact on performance, we also experiment with prompting the model to generate the final judgment directly, while assigning a format penalty of $-1.0$ whenever the model fails to follow the requirement. Under this setting, the model quickly learns to produce judgments without CoT reasoning in fewer than 20 training steps.

#### Results

We present the detailed results of ablation on CoT and prompt dataset balancing in [Table 10](#A2.T10). The results reveal that while the absence of either component degrades the final F1 score, the underlying reasons for the performance drop are fundamentally different.

The removal of CoT appears to weaken the model’s overall ability to discriminate between correct and incorrect reasoning steps. This is evidenced by a general decline in accuracy for both the “Correct” and “Error” classifications across the datasets. In contrast, removing dataset balancing introduces a strong class bias. Without balancing, the model is trained on a dataset where correct steps are overrepresented, causing it to overfit and develop a tendency to classify most steps as correct. This is clearly demonstrated by a sharp increase in accuracy for the “Correct” class, coupled with a significant decrease in accuracy for the “Error” class. While the model becomes proficient at identifying correct steps, it largely loses its ability to detect errors. This trade-off is ultimately detrimental, as shown by the identical drop in the average F1 score from 60.5 to 47.9 for the Rel-Ratio model.

## Appendix B Template, Example, and Additional Tables

**Table 10: Judge performance on ProcessBench, broken down by four subsets. Each subset reports Error (%), Correct (%), and F1 score (%). The final column is the average F1 across all subsets. We remark that the F1 score here is indeed the harmonic mean of the accuracies on two classes.**
| Method | Learning | GSM8K | MATH | Olympiad | Omni-MATH | Avg. F1 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Signal | Error | Correct | F1 | Error | Correct | F1 | Error | Correct | F1 | Error | Correct | F1 |  |
| Qwen2.5-1.5B-chunk |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Discriminative + SFT | Abs-Q | 26.0 | 80.0 | 39.3 | 22.2 | 57.6 | 32.1 | 14.2 | 30.2 | 19.3 | 13.2 | 28.2 | 18.0 | 27.2 |
| Discriminative + SFT | Rel-Effect | 28.5 | 72.0 | 40.8 | 28.6 | 53.0 | 37.2 | 16.4 | 21.8 | 18.7 | 15.8 | 27.6 | 20.1 | 29.2 |
| Discriminative + SFT | Rel-Ratio | 22.5 | 56.0 | 32.1 | 26.2 | 41.0 | 32.0 | 14.0 | 14.4 | 14.2 | 15.2 | 22.0 | 18.0 | 24.1 |
| Generative + CoT + RL | Abs-Q | 42.5 | 58.5 | 49.2 | 36.4 | 45.6 | 40.5 | 31.4 | 19.2 | 23.8 | 32.8 | 29.4 | 31.0 | 36.1 |
| Generative + CoT + RL | Rel-Effect | 38.5 | 64.5 | 48.2 | 37.8 | 51.6 | 43.6 | 23.2 | 21.0 | 22.1 | 24.0 | 26.8 | 25.3 | 34.8 |
| Generative + CoT + RL | Rel-Ratio | 35.0 | 71.0 | 46.9 | 37.8 | 50.8 | 43.4 | 27.0 | 25.6 | 26.3 | 28.0 | 28.8 | 28.4 | 36.2 |
| Gen + RL (no CoT) | Rel-Ratio | 28.5 | 79.5 | 42.0 | 37.0 | 51.8 | 43.2 | 24.4 | 22.8 | 23.6 | 28.6 | 28.8 | 28.7 | 34.3 |
| Gen + CoT + RL (no Chunk) | Rel-Ratio | 36.0 | 65.5 | 46.5 | 37.0 | 39.8 | 38.4 | 25.4 | 15.2 | 19.0 | 29.4 | 23.0 | 25.8 | 32.4 |
| Qwen2.5-7B-chunk |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Discriminative + SFT | Abs-Q | 41.0 | 80.5 | 54.3 | 36.0 | 66.4 | 46.7 | 28.8 | 43.4 | 34.6 | 21.8 | 39.6 | 28.1 | 40.9 |
| Discriminative + SFT | Rel-Effect | 40.5 | 80.0 | 53.8 | 36.8 | 69.6 | 48.1 | 27.0 | 36.2 | 30.93 | 24.6 | 41.0 | 30.8 | 38.7 |
| Discriminative + SFT | Rel-Ratio | 37.5 | 78.5 | 50.8 | 36.6 | 63.8 | 46.5 | 24.0 | 35.8 | 28.7 | 24.2 | 35.6 | 28.8 | 38.7 |
| Generative + CoT + RL | Abs-Q | 59.5 | 64.5 | 61.9 | 63.2 | 59.0 | 61.0 | 53.0 | 44.6 | 48.4 | 44.4 | 43.4 | 43.9 | 53.8 |
| Generative + CoT + RL | Rel-Effect | 70.5 | 74.5 | 72.4 | 69.2 | 67.4 | 68.3 | 61.4 | 48.8 | 54.4 | 54.4 | 50.6 | 52.4 | 61.9 |
| Generative + CoT + RL | Rel-Ratio | 66.5 | 80.0 | 72.6 | 62.6 | 72.6 | 67.2 | 57.2 | 48.2 | 52.3 | 49.4 | 50.2 | 49.8 | 60.5 |
| Qwen2.5-7B-chunk Ablation |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Gen + CoT + RL (no Balancing) | Abs-Q | 31.5 | 94.0 | 47.2 | 34.0 | 79.6 | 47.7 | 25.0 | 58.0 | 34.9 | 23.2 | 43.2 | 30.2 | 40.0 |
| Gen + CoT + RL (no Balancing) | Rel-Effect | 45.0 | 94.0 | 60.9 | 44.8 | 79.0 | 57.2 | 35.8 | 59.2 | 44.6 | 27.0 | 48.8 | 34.8 | 49.4 |
| Gen + CoT + RL (no Balancing) | Rel-Ratio | 42.5 | 95.5 | 58.8 | 41.6 | 80.2 | 54.8 | 29.8 | 65.8 | 41.0 | 29.4 | 49.6 | 36.9 | 47.9 |
| Gen + RL (no CoT) | Rel-Ratio | 45.5 | 82.5 | 58.7 | 37.6 | 72.0 | 49.4 | 36.0 | 47.0 | 40.8 | 40.2 | 45.6 | 42.7 | 47.9 |

Table: Table 11: Prompt template for solver model to produce self-segmented CoTs.

**Table 12: A representative example illustrating CoT segmentation methods based on a response from Qwen2.5-1.5B-it. Standard splitting by newlines (left) produces too many chunks without a clear logical step. For example, chunk 3 and chunk 4 are logically a whole, but they are separated, resulting in poor independence of each step and the inability to form a complete thinking unit.**
| Split by \n\n | LLM-based self-segmentation (Chunks-of-Thought) |
| --- | --- |
| chunk 1<br>To find the value of $f(5)$ for the given function<br>$f(x)=\begin{cases}x-3,&x\geq 6\\<br>f(f(x+5)),&x<6\end{cases},$<br>we need to follow the recursive definition of the function for $x<6$.<br>chunk 2<br>Since $5<6$, we use the second case of the piecewise function:<br>$f(5)=f(f(5+5))=f(f(10)).$<br>chunk 3<br>Now, we need to evaluate $f(10)$. Since $10\geq 6$, we use the first case of the piecewise function:<br>$f(10)=10-3=7.$<br>chunk 4<br>So, we have:<br>$f(5)=f(7).$<br>chunk 5<br>Next, we need to evaluate $f(7)$. Again, since $7\geq 6$, we use the first case of the piecewise function:<br>$f(7)=7-3=4.$<br>chunk 6<br>Thus, we have:<br>$f(5)=4.$<br>chunk 7<br>The final answer is:<br>$\boxed{4}$<br>. | chunk 1 (Initial Expansion)<br>To find the value of $f(5)$ for the given function<br>$f(x)=\begin{cases}x-3,&x\geq 6\\<br>f(f(x+5)),&x<6\end{cases},$<br>we need to follow the recursive definition of the function for $x<6$.<br>Since $5<6$, we use the second case of the piecewise function:<br>$f(5)=f(f(5+5))=f(f(10)).$<br>chunk 2 (Calculate Inner Function and Substitute)<br>Now, we need to evaluate $f(10)$. Since $10\geq 6$, we use the first case of the piecewise function:<br>$f(10)=10-3=7.$<br>So, we have:<br>$f(5)=f(7).$<br>chunk 3 (Final Calculation)<br>Next, we need to evaluate $f(7)$. Again, since $7\geq 6$, we use the first case of the piecewise function:<br>$f(7)=7-3=4.$<br>Thus, we have:<br>$f(5)=4.$<br>The final answer is:<br>$\boxed{4}$ |

**Table 13: A representative example illustrating CoT segmentation methods based on a response from Qwen2.5-1.5B-it. Standard splitting by newlines (left) produces too many chunks without a clear logical step.**
| Split by \n\n | LLM-based self-segmentation (Chunks-of-Thought) |
| --- | --- |
| chunk 1<br>To solve the given problem, we start by expressing $z$ in its standard form $z=x+yi$, where $x$ and $y$ are real numbers. The given equation is:<br>chunk 2<br>$\left|\frac{z^{2}+1}{z+i}\right|+\left|\frac{z^{2}+4i-3}{z-i+2}\right|=4.$<br>chunk 3<br>First, let’s express the magnitudes of the complex numbers. For the first term, we have:<br>chunk 4<br>$\left|\frac{z^{2}+1}{z+i}\right|=\left|\frac{(x+yi)^{2}+1}{x+yi+i}\right|.$<br>chunk 5<br>Since $(x+yi)^{2}=x^{2}-y^{2}+2xyi$, we get $z^{2}+1=x^{2}-y^{2}+2xyi+1=(x^{2}-y^{2}+1)+2xyi$. Therefore, the magnitude is:<br>chunk 6<br>$\displaystyle\left|(x^{2}-y^{2}+1)+2xyi+i(x+yi)\right|$<br>$\displaystyle=\left|(x^{2}-y^{2}+1+x+y)+(2xy+x+y)i\right|.$<br>chunk 7<br>This magnitude can be simplified to the square root of the sum of the squares of the real and imaginary parts:<br>chunk 8<br>$\sqrt{(x^{2}-y^{2}+1+x+y)^{2}+(2xy+x+y)^{2}}.$<br>chunk 9<br>Similarly, for the second term, we have:<br>chunk 10<br>$\left|\frac{z^{2}+4i-3}{z-i+2}\right|=\left|\frac{(x+yi)^{2}+4i-3}{x+yi-i+2}\right|.$<br>chunk 11<br>Since $z^{2}+4i-3=x^{2}-y^{2}+2xyi+4i-3=(x^{2}-y^{2}-3)+(2xy+4)i$, we get:<br>chunk 12<br>$z-i+2=x+(y-1)=x+yi-i+2=(x+2)+yi.$<br>chunk 13<br>Using the same logic as above, the magnitude is:<br>…<br>chunk 21<br>$\boxed{0}.$ | chunk 1 (Express the Equation in Terms of Real and Imaginary Parts)<br>To solve the given problem, we start by expressing $z$ in its standard form $z=x+yi$, where $x$ and $y$ are real numbers. The given equation is:<br>$\left|\frac{z^{2}+1}{z+i}\right|+\left|\frac{z^{2}+4i-3}{z-i+2}\right|=4.$<br>chunk 2 (Simplify the first Term)<br>First, let’s express the magnitudes of the complex numbers. For the first term, we have:<br>$\left|\frac{z^{2}+1}{z+i}\right|=\left|\frac{(x+yi)^{2}+1}{x+yi+i}\right|.$<br>Since $(x+yi)^{2}=x^{2}-y^{2}+2xyi$, we get $z^{2}+1=x^{2}-y^{2}+2xyi+1=(x^{2}-y^{2}+1)+2xyi$. Therefore, the magnitude is:<br>$\displaystyle\left|(x^{2}-y^{2}+1)+2xyi+i(x+yi)\right|$<br>$\displaystyle=\left|(x^{2}-y^{2}+1+x+y)+(2xy+x+y)i\right|.$<br>This magnitude can be simplified to the square root of the sum of the squares of the real and imaginary parts:<br>$\sqrt{(x^{2}-y^{2}+1+x+y)^{2}+(2xy+x+y)^{2}}.$<br>chunk 3 (Simplify the Second Term and Formulate Equations)<br>Similarly, for the second term, we have:<br>$\left|\frac{z^{2}+4i-3}{z-i+2}\right|=\left|\frac{(x+yi)^{2}+4i-3}{x+yi-i+2}\right|.$<br>Since $z^{2}+4i-3=x^{2}-y^{2}+2xyi+4i-3=(x^{2}-y^{2}-3)+(2xy+4)i$, we get:<br>$z-i+2=x+(y-1)=x+yi-i+2=(x+2)+yi.$<br>Using the same logic as above, the magnitude is:<br>$\displaystyle\sqrt{\left(x^{2}-y^{2}-3+x+y+2\right)^{2}+\left(2xy+4+x+y\right)^{2}}$<br>$\displaystyle=\sqrt{\left(x^{2}-y^{2}-1+x+y\right)^{2}+\left(2xy+x+y+4\right)^{2}}.$<br>chunk 4 (Solve the System of Equations to Find z)<br>Given that these two magnitudes sum to 4, and recognizing that the equality $a^{2}+b^{2}=4$ indicates that each term must separately be 2 (because the maximum magnitude of any complex number is 2), we derive:<br>$(x^{2}-y^{2}+1+x+y)^{2}+(2xy+x+y)^{2}=4,$<br>$\left(x^{2}-y^{2}-3+x+y+2\right)^{2}+\left(2xy+4+x+y\right)^{2}=4.$<br>This leads to $x^{2}-y^{2}+x=1$ and $x^{2}-y^{2}-x=-1$. Solving these equations simultaneously, we get $x=1$ and $y=0$, so $z=1$.<br>chunk 5 (Calculate the Final Minimum Value)<br>Finally, we need to find the minimum value of $|z-1|$. This simplifies to:<br>$|z-1|=|1-1|=0.$<br>Thus, the minimum value of $|z-1|$ is:<br>$\boxed{0}.$ |

**Table 14: A representative example illustrating CoT segmentation methods based on a response from Qwen2.5-1.5B-it. Standard splitting by newlines (left) produces too many chunks without a clear logical step.**
| Split by \n\n | LLM-based self-segmentation (Chunks-of-Thought) |
| --- | --- |
| chunk 1<br>Let’s start by analyzing the given conditions:<br>1. When 5655 is divided by $N$, the remainder is 11. This can be written as:<br>$5655=qN+11$<br>where $q$ is some integer.<br>chunk 2<br>2. When 5879 is divided by $N$, the remainder is 14. This can be written as:<br>$5879=mN+14$<br>where $m$ is some integer.<br>chunk 3<br>We can express these equations as:<br>$5655-11=5644=Nq$<br>$5879-14=5865=Nm$<br>chunk 4<br>We know that $N$ is a two-digit positive integer. Therefore, we need to find a value of $N$ that satisfies the above conditions for integer $q$ and $m$.<br>chunk 5<br>First, let’s determine the possible values of $N$ by factoring 5644 and 5865:<br>$5644=2^{2}\cdot 7\cdot 11^{2}$<br>$5865=3\cdot 5\cdot 13^{2}$<br>chunk 6<br>We need to find a common divisor for both $5644$ and $5865$. Let’s check the common divisors:<br>$\text{Common divisors are: }1,7,11,143$<br>chunk 7<br>Since $N$ is a two-digit positive integer, we consider the possible values:<br>$N=11\quad\text{(if 11 is a divisor of both 5644 and 5865)}$<br>chunk 8<br>Let’s test $N=11$:<br>1. Check if 11 divides 5644 exactly:<br>$5644=11\times 513\quad\text{Yes}$<br>2. Check if 11 divides 5865 exactly:<br>$5865=11\times 533\quad\text{Yes}$<br>chunk 9<br>Since $N=11$ divides 5644 and 5865 exactly, $N=11$ satisfies the conditions. Now we have:<br>$Nq=5644\quad\text{and}\quad Nm=5865$<br>chunk 10<br>Using the Law of Cosines (not needed for this case, but for completeness):<br>$\cos(\theta)=\frac{a^{2}+b^{2}-c^{2}}{2ab}$<br>where $a=5644$, $b=5865$, and $c$ is the hypotenuse (since 11 * 11 * 1 is our $N$).<br>chunk 11<br>Sum of the digits of $N$:<br>$1+1=2$<br>chunk 12<br>Thus, the sum of the digits of $N$ is:<br>$\boxed{2}$ | chunk 1 (Formulating the Equations)<br>Let’s start by analyzing the given conditions:<br>1. When 5655 is divided by $N$, the remainder is 11. This can be written as:<br>$5655=qN+11$<br>where $q$ is some integer.<br>2. When 5879 is divided by $N$, the remainder is 14. This can be written as:<br>$5879=mN+14$<br>where $m$ is some integer.<br>We can express these equations as:<br>$5655-11=5644=Nq$<br>$5879-14=5865=Nm$<br>chunk 2 (Finding Common Divisors via Factorization)<br>We know that $N$ is a two-digit positive integer. Therefore, we need to find a value of $N$ that satisfies the above conditions for integer $q$ and $m$.<br>First, let’s determine the possible values of $N$ by factoring 5644 and 5865:<br>$5644=2^{2}\cdot 7\cdot 11^{2}$<br>$5865=3\cdot 5\cdot 13^{2}$<br>We need to find a common divisor for both $5644$ and $5865$. Let’s check the common divisors:<br>$\text{Common divisors are: }1,7,11,143$<br>chunk 3 (Verifying the Candidate N)<br>Since $N$ is a two-digit positive integer, we consider the possible values:<br>$N=11\quad\text{(if 11 is a divisor of both 5644 and 5865)}$<br>Let’s test $N=11$:<br>1. Check if 11 divides 5644 exactly:<br>$5644=11\times 513\quad\text{Yes}$<br>2. Check if 11 divides 5865 exactly:<br>$5865=11\times 533\quad\text{Yes}$<br>Since $N=11$ divides 5644 and 5865 exactly, $N=11$ satisfies the conditions.<br>chunk 4 (Final Calculation)<br>Now we have:<br>$Nq=5644\quad\text{and}\quad Nm=5865$<br>Using the Law of Cosines (not needed for this case, but for completeness):<br>$\cos(\theta)=\frac{a^{2}+b^{2}-c^{2}}{2ab}$<br>where $a=5644$, $b=5865$, and $c$ is the hypotenuse (since 11 * 11 * 1 is our $N$).<br>Sum of the digits of $N$:<br>$1+1=2$<br>Thus, the sum of the digits of $N$ is:<br>$\boxed{2}$ |

Table: Table 15: An illustrative example from the Process-Bench dataset, where the ground-truth label is “correct.” The judge model is StepWiser with Rel-Ratio and Qwen2.5-7B-chunk.

Table: Table 16: An illustrative example from the Process-Bench dataset, where the ground-truth label is “correct.” The judge model is StepWiser with Rel-Ratio and Qwen2.5-7B-chunk.

Table: Table 17: An illustrative example from the Process-Bench dataset, where the ground-truth label is “incorrect.” The judge model is StepWiser with Rel-Ratio and Qwen2.5-7B-chunk. We highlight the key parts of the reasoning trajectory in red.

Table: Table 18: An illustrative example from the Process-Bench dataset, where the ground-truth label is “incorrect.” The judge model is StepWiser with Rel-Ratio and Qwen2.5-7B-chunk. We highlight the key parts of the reasoning trajectory in red.

Table: Table 19: An illustrative example from the Process-Bench dataset, where the ground-truth label is “incorrect.” The judge model is StepWiser with Rel-Ratio and Qwen2.5-7B-chunk. We highlight the key parts of the reasoning trajectory in red.

Table: Table 20: An illustrative example from the Process-Bench dataset, where the ground-truth label is “incorrect.” The judge model is StepWiser with Rel-Ratio and Qwen2.5-7B-chunk. We highlight the key parts of the reasoning trajectory in red.

Table: Table 21: An illustrative example of chunk-reset reasoning. The judge model is StepWiser with Rel-Effective and Qwen2.5-7B-chunk. In the first attempt of chunk 3, it contains a significant algebraic mistake in the formula relating $\lambda$ and $r$.

Table: Table 22: An illustrative example of chunk-reset reasoning. The judge model is StepWiser with Rel-Effective and Qwen2.5-7B-chunk. In the first attempt of chunk 3, the model fails to verify the condition.

Table: Table 23: An illustrative example of chunk-reset reasoning. The judge model is StepWiser with Rel-Effective and Qwen2.5-7B-chunk. The idea is to use the commutative law of multiplication and division to perform large-scale reduction, while the rejected chunks focus on the approximation for simplifying the expression.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - How does segmentation quality and downstream judge performance change when using the same-scale model (e.g., Qwen2.5-1.5B or 7B)?
soundness: 3
strengths: - The idea of reframing stepwise evaluation from classification to reasoning is interesting. 
- The paper is well written. Each component of their methods is well explained.  Strong ablations clearly isolate the contribution of each component and demonstrate improvements in both training-time reward modelling and inference-time search.
- The results show that their method provides improved inference-time search and better training-time rewards.
summary: The paper proposes STEPWISER, a generative stepwise judge that evaluates intermediate reasoning steps of LLMs by reasoning about them. Unlike traditional process reward models that treat step evaluation as classification, STEPWISER reframes it as a reasoning problem trained via reinforcement learning (RL). The proposed method comprises three components: (1) a Self-segmentation method, (2) step-wise reward annotation, and (3) online training of judgment reasoning chains. The authors show that STEPWISER can be used to provide rewards at either training or inference time. The authors conducted experiments on ProcessBench and demonstrated improved performance over a classification baseline.
weaknesses: - Despite its framing, the approach does not fundamentally innovate beyond concurrent work on generative process reward models. The “reasoning about reasoning” claim is largely conceptual—since the judge’s reasoning trace is not used in the loss or supervision signal. The RL reward depends solely on the final verdict label.

- All results focus on final-answer correctness. There is no manual evaluation of the judge’s faithfulness or calibration. Annotating a small set of reasoning steps and comparing judge labels to human ratings would have made the claims more convincing.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 4
questions: I have listed several major questions in the Summary and Weakness sections. Some additional minor questions are as follows:

1. Is the StepWiser model susceptible to prompt attacking, and how robust is it?
2. If process-level judgment is removed, would training a Reasoning RM with the same computational budget and method yield comparable performance?

For other questions, see weakness.
soundness: 3
strengths: 1. The paper is well written, and the method is easy to understand.
2. The experiments show significant performance improvements.
3. The ablation shows that the proposed components are valid.
summary: This paper proposes StepWiser, a step-wise GenRM trained by online technique, following SFT. The trained RM can segment steps more reasonably after distillation and score at the step level. The author also introduces a relative improvement approach on step-level estimation. Experiments and analysis have demonstrated the effectiveness of the method. The ablation study also demonstrates that RL contributes significantly to the performance improvement. 

However, due to the risk of data pollution in the backbone model used and the lack of experiments on other models, I believe that this paper is not strongly acceptable; we cannot determine whether the improvement in the effectiveness of generative RM is due to reasoning ability by data pollution or the strong judging ability of the model. The ability to stimulate models on already trained data is what RL excels at.

I know it's too much to ask for more model experiments, but the excessive improvement makes me have these concerns: a comprehensive validation for multiple model experiments should have been available when submitting the paper, or it will raise my concern that the score is improved by the model's reasoning ability (and data pollution).

A score of 4 indicates that there are many key issues that need clarification, not the method itself.
Therefore, if this concern cannot be solved, I will tend not to accept this paper and retain my score. On the contrary, if the method is universal and if the author can argue that such a large cost is reasonable, I am willing to recommend it as a poster.
weaknesses: 1. The main experiments are only conducted based on Qwen2.5, which involves data contamination on several datasets.
2. The method is costly (around 5 days on 8*A100 GPUs), and the paper does not provide fair or cost-comparable baselines, like an RL-trained reasoning GenRMs (cost-competitive reasoning GenRMs without process-level reward).
3. The authors do not prove whether the improvement of the model's judging ability is related to data pollution, and whether the model's judging ability strongly depends on its reasoning ability.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. Data contamination:
Since all experiments use Qwen2.5 models, which are known to contain math-heavy data, how did the authors ensure there is no benchmark leakage (e.g., GSM8K, MATH, ProcessBench)? Would the method still perform well on cleaner model families such as LLaMA or DeepSeek?

2. Generalization:
The paper focuses only on mathematical reasoning. Can the authors provide evidence or discussion on whether STEPWISER generalizes to other reasoning domains, such as logic, commonsense, or code?

3. Robustness:
Are the generative judge’s CoT analyses consistent under paraphrased or slightly perturbed inputs? Have the authors evaluated stability across multiple runs or sampling temperatures?

4. Theoretical grounding:
The paper argues that dense step-level rewards are more effective than sparse ones. Can the authors formalize or provide intuition on why this improves sample efficiency under GRPO?

5. Integration with policy training:
Since STEPWISER can serve both as a training-time reward and inference-time controller, have the authors considered joint optimization with the policy model? What challenges would this introduce?
soundness: 2
strengths: This paper presents a well-motivated and technically sound contribution to the emerging area of process-level reward modeling for reasoning-intensive LLMs. Its central innovation lies in reframing stepwise evaluation as a generative reasoning task, where the judge model is trained to “reason about reasoning.” The proposed STEPWISER framework is conceptually elegant and empirically convincing: it combines (1) self-segmentation of chains-of-thought into coherent “chunks of thought,” (2) Monte-Carlo-based relative outcome labeling for stepwise supervision, and (3) reinforcement learning of a generative judge through GRPO.
The experimental evaluation is extensive and well controlled. Results on ProcessBench and inference-time search clearly demonstrate that the RL-trained generative judge significantly outperforms strong discriminative and SFT baselines. The ablation studies convincingly isolate the impact of each design component, showing that online RL and CoT-based generation both contribute materially to performance.
summary: The paper addresses the supervision of intermediate steps in multi-step reasoning. Traditional Process Reward Models (PRMs) are typically designed as discriminative classifiers, which assign scores to each reasoning step but fail to explain the rationale behind their judgments. Moreover, their dependence on static supervised fine-tuning (SFT) datasets limits both generalization and interpretability.

To overcome these limitations, the authors propose reframing the judgment of reasoning itself as a reasoning task—allowing the model to “reason about reasoning”, i.e., to generate its own chain of thought before delivering a final verdict.

Core Contribution: The STEPWISER Framework
STEPWISER is a generative judge designed for evaluating intermediate reasoning steps. It is trained via reinforcement learning (RL) to produce well-grounded, explanatory judgments after analyzing reasoning chunks. The framework consists of three main components:

1. Self-Segmentation of Reasoning Chains – enabling the model to divide its reasoning process into coherent “chunks of thought.”

2. Monte Carlo–based Stepwise Labeling – using relative outcome rollouts to assign dynamic rewards to reasoning chunks.

3. RL Training of a Generative Judge – teaching the model to reason about the quality of each chunk through its own chain-of-thought before producing a verdict.
weaknesses: While the proposed framework is conceptually strong, several limitations reduce its scientific rigor and generality.
First, all experiments are conducted exclusively on Qwen2.5-based models (1.5B and 7B). These models are known to have been exposed to extensive mathematical corpora during pretraining, leading to potential data contamination and evaluation leakage on benchmarks such as GSM8K, MATH, and ProcessBench. This raises concerns about the true generalization of the proposed method beyond math-specific reasoning or Qwen’s internal distributional biases.

Second, although the authors claim that dense, step-level supervision provides a richer learning signal than outcome-level RLHF, the paper lacks formal theoretical analysis—for example, no proof or derivation is given to quantify the sample-efficiency advantage or convergence behavior of GRPO under stepwise credit assignment. The argument remains mostly empirical.

Third, the evaluation diversity is limited. The framework is only tested on mathematical reasoning tasks; extending it to symbolic reasoning, logic puzzles, or real-world procedural reasoning (e.g., code synthesis or multi-hop QA) would provide stronger evidence of its generality.

Finally, although the generative CoT-based judge improves interpretability, the paper does not analyze the robustness or consistency of the generated rationales—e.g., whether the judge’s reasoning chains are stable under paraphrased inputs, adversarial perturbations, or different sampling temperatures. This limits our understanding of how much of the gain stems from genuine meta-reasoning versus overfitting to specific reasoning patterns.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness.
soundness: 3
strengths: 1. The paper creatively shifts the paradigm from a simple classification task (discriminative PRMs) to a generative reasoning task, forcing the judge to "meta-reason" by producing a CoT.
2. The authors provide comprehensive ablations that clearly isolate the individual performance gains from using RL over SFT, generative CoT over discriminative formats, and dataset balancing. The practical utility is convincingly shown through superior results in both inference-time search ("Chunk-Reset Reasoning") and RFT data selection, proving the judge's effectiveness.
3. This paper is well-written and structured. Figure 1 provides a clear, high-level overview of the entire pipeline, and concepts like "Chunks-of-Thought" are well-defined and motivated.
summary: This paper introduces STEPWISER, a novel generative judge designed to supervise the multi-step reasoning of LLMs. Unlike traditional Process Reward Models (PRMs) that act as simple classifiers, STEPWISER reframes evaluation as a reasoning task itself. It first generates its own "meta-reasoning" (a Chain-of-Thought explaining its analysis) before delivering a final verdict on a reasoning step.Key contributions include:
1. A "Chunks-of-Thought" self-segmentation technique to create logically coherent reasoning steps.
2. An RL training framework (using GRPO) where the judge is rewarded based on the relative outcome of Monte-Carlo rollouts (i.e., whether a step improves the chance of success, $\Delta Q^{\pi}$).
3. Demonstrating that this generative, RL-trained judge significantly outperforms SFT baselines on ProcessBench, enables effective inference-time self-correction ("Chunk-Reset Reasoning"), and improves training data selection.
weaknesses: 1. The paper's primary weakness is its incremental novelty. The core components (generative verifiers, MC rollouts, RL training) have been individually explored in recent works. However, the authors provide a comprehensive and systematic integration of this entire pipeline. I will give a positive view of its contribution.
2. Significant Computational and Latency Overheads. The multi-stage pipeline—requiring extensive MC rollouts for data annotation (14 days) followed by a full online RL run (5 days)—makes replication and practical application extremely expensive.
3. Inference Latency: Chunk-Reset Reasoning requires running the judge model after every generated chunk and potentially re-sampling up to 5 times. This introduces substantial latency, making it unsuitable for real-time applications. The high "Rejected length" in Table 4  confirms that this re-sampling occurs frequently. The authors should discuss these practical limitations and potential trade-offs (e.g., performance vs. compute).