Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models

## Abstract

Abstract Reinforcement Learning (RL) has shown remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). Process-Supervised RL (PSRL) has emerged as a more effective paradigm compared to outcome-based RL. However, existing PSRL approaches suffer from limited exploration efficiency, both in terms of branching positions and sampling. In this paper, we introduce a novel PSRL framework (AttnRL), which enables efficient exploration for reasoning models. Motivated by preliminary observations that steps exhibiting high attention scores correlate with reasoning behaviors, we propose to branch from positions with high values. Furthermore, we develop an adaptive sampling strategy that accounts for problem difficulty and historical batch size, ensuring that the whole training batch maintains non-zero advantage values. To further improve sampling efficiency, we design a one-step off-policy training pipeline for PSRL. Extensive experiments on multiple challenging mathematical reasoning benchmarks demonstrate that our method consistently outperforms prior approaches in terms of performance and sampling and training efficiency.

## 1 Introduction

Large Language Models (LLMs) have achieved remarkable progress in recent years (OpenAI, 2023; Hurst et al., 2024; Anthropic, 2023), particularly in their reasoning capabilities (OpenAI, 2024; DeepSeek-AI et al., 2025). With the success of DeepSeek-R1 (DeepSeek-AI et al., 2025), Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective post-training paradigm for further strengthening the reasoning abilities of LLMs (Shao et al., 2024; Zeng et al., 2025; Luo et al., 2025; Yu et al., 2025; Liu et al., 2025d; Hu et al., 2025; He et al., 2025a; An et al., 2025; Zhang et al., 2025a; Wang et al., 2025b; Zheng et al., 2025a; Wang et al., 2025a).

Common RLVR approaches, such as Group Relative Policy Optimization (GRPO) (Shao et al., 2024) and its variants (Yu et al., 2025; Liu et al., 2025d; Yue et al., 2025), assign uniform training signals to all tokens within the same response, thereby overlooking fine-grained reasoning quality. In contrast, Process-Supervised RL (PSRL) methods refine credit assignment with Monte Carlo (MC) sampling to estimate step-level advantages (Hou et al., 2025; Guo et al., 2025; Yang et al., 2025b; Zheng et al., 2025b; Li et al., 2025). However, existing PSRL methods suffer from several limitations: (1) they segment responses by fixed token length or entropy, ignoring the semantic meaning of model outputs; (2) they adopt uniform sampling across prompts and responses, leading to inefficient exploration; (3) they typically rely on two-step sampling per update, which significantly increases computational cost.

To overcome these limitations, we introduce AttnRL, a novel PSRL framework that improves both exploration and training efficiency. Our approach is motivated by the observation that attention scores serve as a meaningful metrics for identifying important reasoning behaviors in the model output. We therefore introduce an attention-based branching strategy for Monte Carlo sampling. To further enhance efficiency, we design an adaptive sampling mechanism that prioritizes difficult problems while filtering easier ones, and an adaptive batch sampling strategy that guarantees non-zero advantage values across batches. The experimental results on mathematical reasoning tasks demonstrate that AttnRL outperforms strong outcome-based and process-based baselines with great efficiency.

The contributions of this work can be summarized as follows:

- •
We analyze the relationship between attention scores and reasoning behaviors, and propose attention-based branching method for PSRL.
- •
We develop an adaptive sampling mechanism that balances exploration across problems of varying difficulty and ensure valid training batches without zero advantage values(^1^11In the following sections, we use “valid token/batch” to denote the tokens/batches with non-zero advantage values for training.).
- •
Empirical results on six mathematical benchmarks demonstrate the superiority of our method beyond the baselines in both performance and efficiency.

## 2 Preliminaries

### 2.1 LLM Reasoning as a Step-Level Markov Decision Process

Following Sutton and Barto (2018); Zhang et al. (2025b), we formulate LLM reasoning as a Markov Decision Process (MDP) defined by the tuple $(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathcal{P}:\mathcal{S}\times\mathcal{A}\mapsto\mathcal{S}$ is the transition dynamics, $\mathcal{R}:\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}$ is the reward function, and $\gamma\in[0,1]$ is the discount factor.
In the LLM setting with a prompt dataset $\mathcal{D}$, the initial state is $s_{1}=q\sim\mathcal{D}$. The state transition is deterministic, since the next state is formed by concatenating the current state with the generated action: $s_{k+1}=[s_{k},a_{k}]$, where $[\cdot,\cdot]$ denotes string concatenation.
For process-level supervision of LLMs (Zhang et al., 2025b; Liu et al., 2025b), actions are defined at the step level, where each action $a_{t}$ corresponds to a semantically coherent segment such as a sentence or a paragraph, rather than a single token. In this paper, we adopt this step-level MDP formulation.

### 2.2 Outcome-Supervised and Process-Supervised RL

##### Outcome-Supervised RL.

Group Relative Policy Optimization (GRPO) (Shao et al., 2024) is an Outcome-Supervised RL (OSRL) method that eliminates the need for an explicit critic model by estimating the advantage using the rewards $\{R_{1},\cdots,R_{G}\}$ of $G$ sampled rollouts $\{o_{1},\cdots,o_{G}\}$. The normalized advantage is computed as $\hat{A}_{i,t}=\frac{R_{i}-\operatorname{mean}(\{R_{i}\}_{i=1}^{G})}{\operatorname{std}(\{R_{i}\}_{i=1}^{G})}$.
The GRPO objective is then given by:

$$ $\displaystyle\mathcal{J}_{\text{GRPO}}(\theta)=$ $\displaystyle\ \mathbb{E}_{q\sim\mathcal{D},\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)}$ (1) $\displaystyle\hskip-40.00006pt\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\bigg(\min\left(r_{i,t}(\theta)\hat{A}_{i,t},\operatorname{clip}\big(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon\big)\hat{A}_{i,t}\right)-\beta\mathbb{D}_{\text{KL}}(\pi_{\theta}\|\pi_{\text{ref}})\bigg)\right],$ $$

where $r_{i,t}=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})}$ is the importance sampling ratio, and $\beta$ controls the strength of the KL divergence penalty that regularizes the policy towards the reference policy $\pi_{\text{ref}}$.

##### Process-Supervised RL.

For PSRL, the sampling process usually includes two stages: (1) Initial Sampling: Sample multiple responses to the problem; (2) Monte Carlo Sampling: Select several tokens as division points and rollout twice starting from these branching positions (Hou et al., 2025; Guo et al., 2025; Yang et al., 2025b).
In this paper, we follow the setting of TreeRL (Hou et al., 2025), which proposes a tree-based advantage estimation method. For each node, the value is computed as the average accuracy of its all children:

$$ $V(s_{k})=\frac{1}{|L(s_{k})|}\sum_{l\in L(s_{k})}\bm{1}(l\text{ is correct}),$ (2) $$

where $L(s_{k})$ denotes the children of node $s_{k}$.
The final advantage is the summation of global advantage ($V(s_{k})-V(s_{1})$) and local advantage ($V(s_{k})-V(p(s_{k})$):

$$ $\hat{A}_{i,k}=\frac{1}{\sqrt{|L(s_{k})|}}\left(V(s_{k})-V(s_{1})+V(s_{k})-V(p(s_{k}))\right),$ (3) $$

where $\sqrt{|L(s_{k})|}$ is used to reduce the optimization strength of the non-leaf steps to prevent overfitting (Hou et al., 2025) and $p(s_{k})$ is the parent node of $s_{k}$.
Then the policy is optimized using the loss function in ([1](https://arxiv.org/html/2509.26628v1#S2.E1)), which is the same as that of OSRL but differs at the advantage granularity.

### 2.3 Attention Mechanism

Modern LLMs are typically decoder-only Transformer-based architectures (Vaswani et al., 2017; Yang et al., 2024, 2025a), and the core operation inside each Transformer block is the (masked) self-attention mechanism.
For a given layer $l$ and head $h$, the model first computes query $Q^{l,h}$, key $K^{l,h}$ and value matrices. Then the attention score $\alpha$ is computed as:

$$ $\alpha^{l,h}=\operatorname{softmax}\bigg(\frac{Q^{l,h}{K^{l,h}}^{\top}}{\sqrt{d_{k}}}+M\bigg),$ (4) $$

where $d_{k}$ is the per-head dimensionality and $M$ is the causal mask. In vanilla causal attention, $M$ blocks access to all future tokens by assigning them $-\infty$, while past and current tokens remain unmasked with 0.

## 3 Method

In this section, we present AttnRL, an exploration-efficient method for process-supervised RL.
We begin by examining the role of massive attention values and leverage them for attention-based tree branching (ATB) (Section [3.1](https://arxiv.org/html/2509.26628v1#S3.SS1)). Next, we propose an adaptive sampling (ADS) strategy that enables more efficient exploration (Section [3.2](https://arxiv.org/html/2509.26628v1#S3.SS2)).
Finally, we introduce our efficient training pipeline based on one-step off-policy learning (Section [3.3](https://arxiv.org/html/2509.26628v1#S3.SS3)).

### 3.1 Branching at Massive Attention Values

Prior work has demonstrated that massive attention values in self-attention mechanisms play a critical role in contextual knowledge understanding (Jin et al., 2025), as they highlight tokens most relevant for answering questions. Additionally, Bogdan et al. (2025) finds that some heads in reasoning models narrow attention toward specific sentences and these sentences are related to reasoning roles. Motivated by these two works, we investigate two key questions: (1) What effects do the steps with massive attention values have? and (2) how can they be effectively utilized in PSRL?

#### 3.1.1 Massive Attention Values in LLMs

##### Step 1: Segmenting and computing step-level attention scores.

Following prior work on process supervision (Wang et al., 2024a; Liu et al., 2025b), we first segment the entire response into multiple steps using two consecutive line breaks (“\n\n”), yielding $T_{k}$ steps: $o=(o_{1},o_{2},\dots,o_{T_{k}})$.
Next, we obtain token-to-token attention scores via a single forward process. By aggregating these scores at the step level, we get step-to-step attention matrices $\alpha^{l,h}\in\mathbb{R}^{T_{k}\times T_{k}}$, where $\alpha_{j,k}^{l,h}$ denotes the attention weight of step $j$ attending to step $k$ at layer $l$ and head $h$.

##### Step 2: Computing the Forward Context Influence (FCI) score.

To quantify the influence of a given step on subsequent tokens, we sum the attention scores over the subsequent steps at layer $l$ and head $h$:

$$ $y_{k}^{l,h}=\sum_{j=k+\Delta}^{T_{k}}\alpha_{j,k}^{l,h},$ (5) $$

where $\Delta$ is a hyperparameter that restricts the scope to sufficiently distant parts of the response, set to $4$ following Bogdan et al. (2025).
We then aggregate across layers and heads by taking the maximum value, obtaining Forward Context Influence (FCI) as follows:

$$ $y_{k}=\max_{l,h}\{y_{k}^{l,h}\}.$ (6) $$

The resulting FCI score $y_{k}$ captures the degree to which step $k$ influences the downstream context at the attention level.
An illustrative visualization of steps with high FCI values is provided in Figure [2](https://arxiv.org/html/2509.26628v1#S3.F2). From this figure, we can see that most steps with high FCI scores or peak FCI values are related to reasoning behaviors, such as planning and self-verification (Bogdan et al., 2025). The full response are listed in Table [5](https://arxiv.org/html/2509.26628v1#A3.T5) in Appendix [C](https://arxiv.org/html/2509.26628v1#A3).

Figure: Figure 2: The visualization of steps with high FCI scores.
Refer to caption: x3.png

#### 3.1.2 The Effects of Steps with High FCI Scores

After identifying and qualitatively analyzing steps with high FCI scores, we conduct quantitative experiments to examine the impact of disrupting attention values on performance. Specifically, we select a step: (1) randomly from the top 20% of steps ranked by FCI scores (denoted as “FCI Top 20%”), (2) randomly from the remaining steps (denoted as “FCI 20%–100%”), or (3) randomly from the top 20% of steps ranked by step-level entropy (denoted as “Entropy Top 20%”). For the chosen step, we set its corresponding attention values to zero. We hypothesize that disrupting attention at key steps (with high FCI scores) will cause greater performance degradation compared to disrupting other steps.
We test this hypothesis on AIME24 (MAA, 2024) and AIME25 (MAA, 2025) using DS-R1-Distill-Qwen-1.5B, with each problem sampled eight times. The results shown in Figure [3](https://arxiv.org/html/2509.26628v1#S3.F3)(a) show that all disruption types lead to a drop in accuracy. Among all disruption types, disrupting steps with top 20% FCI scores leads to the largest drop in accuracy, while disrupting steps with top 20% step entropy leads to a drop between that of 20%–100% FCI and top 20% FCI scores, demonstrating the steps with high FCI scores are more important than the steps with high entropy.
Furthermore, we investigate the effect of disruption position. We divide the disruption positions relative to the original response length into five uniform bins. As shown in Figure [3](https://arxiv.org/html/2509.26628v1#S3.F3)(b), accuracy exhibits an increasing trend as the disruption position moves later in the sequence, indicating that disruptions at earlier positions have a larger negative impact on final performance.

Figure: Figure 3: Disruption results on AIME24 and AIME25. (a) Normalized average accuracy of different disruption types. (b) Average accuracy of different disruption positions.
Refer to caption: x4.png

#### 3.1.3 Attention-based Tree Branching

Based on the analysis in Section [3.1.1](https://arxiv.org/html/2509.26628v1#S3.SS1.SSS1) and [3.1.2](https://arxiv.org/html/2509.26628v1#S3.SS1.SSS2), we have identified that steps with high FCI scores
are related to reasoning behaviors and have strong influences on the the reasoning performance. Now we propose Attention-based Tree Branching (ATB), which builds the branches of the tree at steps with high FCI scores.

Specifically, we compute the FCI score for each step using ([6](https://arxiv.org/html/2509.26628v1#S3.E6)) after initial sampling to enable effective exploration. We then select the top 20% of the steps with the highest FCI scores for branching:

$$ $C=\{k\mid k\geq\operatorname{Quantile}(y_{1},\dots,y_{T_{k}},\rho)\},$ (7) $$

where $\rho=0.2$ is the quantile level. However, randomly selecting steps with high FCI scores as branching points can be suboptimal, as misleading initial steps may lead the reasoning process in incorrect directions and we have found that earlier steps have more influence on the final result. Similar phenomenons have also been found in Wen et al. (2025), which identifies these as “Tunnel Vision”.
To mitigate this, we select the top $N$ ($N=2$ following Hou et al. (2025)) earliest steps from $C$ as branching points, ensuring that diverse reasoning paths are explored through attention-based branching.

### 3.2 Adaptive Sampling

#### 3.2.1 Difficulty-aware Exploration

##### Attention-based Filtering.

Previous PSRL approaches explore all problems uniformly (Hou et al., 2025), which is highly inefficient. In particular, problems that are easy (i.e., achieving an accuracy of 100% at initial sampling) have a high probability (about 70% - 80%, shown in Figure [7](https://arxiv.org/html/2509.26628v1#S5.F7)(a)) of being correct at both sampling stages, leading to limited learning opportunities.

Figure: Figure 4: Average FCI scores of all problems during the training process of TreeRL on DeepScaleR dataset.
Refer to caption: x5.png

To address this, we propose an attention-based filtering method to identify problems that are too easy to sample an incorrect response. We compute the average FCI scores for all problems in the DeepScaleR (Luo et al., 2025) dataset using DS-R1-Distill-Qwen-1.5B. As shown in Figure [4](https://arxiv.org/html/2509.26628v1#S3.F4), we empirically find that problems with lower FCI scores tend to have zero advantage values, indicating that all samples are correct.
Therefore, we filter out problems with low FCI scores and only retain those with FCI scores above the average value. The filtered problem set for MC sampling is:

$$ $\mathcal{D}_{\text{MC}}=\{q\mid\frac{1}{G}\sum_{i=1}^{G}\frac{1}{T_{i,k}}\sum_{k=1}^{T_{i,k}}y_{i,k}\geq\text{mean value}\},$ (8) $$

where $y_{i,k}$ is the FCI score for the $k$-th step in response $i$.

##### Difficulty-aware Expansion.

After attention-based filtering, we expand different number of trees according to problem difficulty since it is more difficult to rollout correct responses for hard problems.
Let the difficulty score be $z_{n}=\frac{1}{G}\sum_{i}\bm{1}(o_{i}\text{ is correct})$.
Then the number of trees expanded for each problem is determined by the difficulty score:

$$ $\text{actual tree numbers}=\operatorname{Round}(\exp(-z_{n})\times\text{original tree numbers}),$ (9) $$

where $\operatorname{Round}(\cdot)$ denotes rounding to the nearest integer, and original tree numbers is set to 6 following Hou et al. (2025).

#### 3.2.2 Adaptive Batch Sampling

After initial sampling and MC sampling, a large proportion of responses contribute nothing to training because their advantages are zero (detailed in Figure [7](https://arxiv.org/html/2509.26628v1#S5.F7)(b)). To ensure that each training batch remains effective, we introduce an adaptive batch size mechanism.

Let the target training batch size be $B^{\prime}$, current valid training batch size be $B^{\prime\prime}$, and the sampled prompt batch size at step $m$ be $B_{m}$. The sampling batch size at step $m$ is updated as:

$$ $B_{m}=\operatorname{Round}(\lambda B_{m-1}+(1-\lambda)\frac{B^{\prime}}{B^{\prime\prime}}B_{m-1}),$ (10) $$

where $\lambda$ is the weight balancing historical and current batch sizes.
After MC sampling, responses with zero advantages are discarded, ensuring that all samples in the final batch have non-zero advantages, which improves training efficiency.

Our adaptive batch sampling differs from the dynamic sampling used in DAPO (Yu et al., 2025) in two key ways:
(1) It requires only a single round of prompt sampling and generation per training step.
(2) It avoids inefficiency from discarding valid responses when their number exceeds $B$.
As a result, the actual batch size naturally fluctuates around the target $B$ while maintaining high training efficiency.

### 3.3 Efficient Training with One-Step Off-Policy

Prior process-supervised RL methods typically require two sampling procedures per training iteration (Hou et al., 2025; Yang et al., 2025b; Guo et al., 2025; Zheng et al., 2025b). This is highly inefficient, as sampling often dominates the overall training time. To address this, we propose a one-step off-policy learning framework for PSRL, inspired by recent advances in efficient RL training (Noukhovitch et al., 2025; Fu et al., 2025; meituan search, 2025).

Figure: Figure 5: Training pipeline of AttnRL. Our method (left) only needs one-time generation per training iteration, while previous methods (right) require to sample twice and are inefficient.
Refer to caption: x6.png

In our approach, only a single sampling operation is performed at each training step. Concretely, at training step $m$, we conduct initial sampling for the $(m{+}1)$-th problem batch while simultaneously performing MC sampling for the $m$-th problem batch. This design ensures that the initial sampling for a batch occurs at step $m{-}1$, followed by its MC sampling at step $m$, thereby eliminating redundant sampling. As a result, the overall sampling cost is substantially reduced, leading to improved training efficiency. The full training pipeline of AttnRL is illustrated in Figure [5](https://arxiv.org/html/2509.26628v1#S3.F5).

## 4 Experiments

### 4.1 Setup

##### Models and Baselines.

Following Hou et al. (2025), we adopt two supervised fine-tuned models, which are also reasoning models, as base models: DS-R1-Distill-Qwen-1.5B and DS-R1-Distill-Qwen-7B (DeepSeek-AI et al., 2025). We compare against the following baselines:
(1) GRPO (Shao et al., 2024): A representative OSRL method.
(2) TreeRL (Hou et al., 2025): A PSRL approach with tree-based branching and advantage estimation.
(3) DeepScaleR-Preview-1.5B (Luo et al., 2025): A strong RL-trained model at the 1.5B scale.

##### Evaluation and Metrics.

We evaluate all methods on six widely used mathematical reasoning benchmarks: AIME24 (MAA, 2024), AIME25 (MAA, 2025), AMC23 (MAA, 2023), MATH-500 (Lightman et al., 2024), Minerva Math (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024).
We report both Pass@1 and Pass@K, where $K=32$ for AIME24, AIME25, and AMC23, and $K=4$ for the remaining benchmarks.
Evaluation is performed with a maximum response length of 32,768 tokens.
For verification, we use a hybrid of DeepScaleR’s verifier and Math-Verify(^2^22https://github.com/huggingface/Math-Verify) to ensure correctness (He et al., 2025a).

##### Implementation Details.

We train all methods using DeepScaleR-Preview-Dataset (Luo et al., 2025), following Luo et al. (2025); Liu et al. (2025c), which contains 40.3k mathematical reasoning problems.
We set the training batch size to 64, the PPO minibatch size to 32, and the learning rate to $1\times 10^{-6}$.
For all methods, we adopt token-level policy loss and apply Clip-Higher with $\varepsilon_{\text{high}}=0.28$, following Yu et al. (2025). We use KL loss with weight 0.001 following Liu et al. (2025a); Wang et al. (2025b).
For AttnRL, we set $\lambda=0.9$ (a common EMA value (Kingma, 2014)) and $\rho=0.2$.

The training is conducted using verl (Sheng et al., 2025), and rollouts are generated using vLLM (Kwon et al., 2023) with a maximum response length of 8,192 tokens, top-$p$ of 1.0, and temperature of 1.0 for both DS-R1-Distill-Qwen-1.5B and DS-R1-Distill-Qwen-7B.
Experiments for DS-R1-Distill-Qwen-1.5B are conducted on a single node with 8$\times$ NVIDIA H100 GPUs, and experiments for DS-R1-Distill-Qwen-7B are run on three nodes, each with 8$\times$ NVIDIA H800 GPUs.

### 4.2 Main Results

**Table 1: Evaluation results on mathematical benchmarks. The results of AttnRL are shaded and the highest values are bolded.**
| Method | AIME24 | AIME25 | AMC23 | MATH-500 | Minerva | Olympiad | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DS-R1-Distill-Qwen-1.5B | 28.3 | 23.0 | 71.8 | 84.8 | 35.6 | 54.9 | 49.7 |
| $\mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\thinspace\lower-5.32501pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{}}{}<br>{{}{}}{}<br>{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{}{{<br>{\pgfsys@beginscope\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@setdash{\pgf@temp}{\the\pgf@x}\pgfsys@roundcap\pgfsys@roundjoin{}<br>{}{}{}<br>{}{}{}<br>\pgfsys@moveto{-1.42497pt}{1.89996pt}\pgfsys@curveto{-1.30621pt}{1.18747pt}{0.0pt}{0.11874pt}{0.35623pt}{0.0pt}\pgfsys@curveto{0.0pt}{-0.11874pt}{-1.30621pt}{-1.18747pt}{-1.42497pt}{-1.89996pt}\pgfsys@stroke\pgfsys@endscope}}<br>}{}{}{{}}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{-3.40001pt}\pgfsys@curveto{0.0pt}{-4.28368pt}{0.71634pt}{-5.00002pt}{1.6pt}{-5.00002pt}\pgfsys@lineto{5.38377pt}{-5.00002pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope}}{{}}}}<br>\pgfsys@invoke{ }\pgfsys@endscope}<br>\pgfsys@invoke{ }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{ }\pgfsys@endscope\hss}}\endpgfpicture}}}$ GRPO | 36.9 | 27.2 | 77.7 | 88.4 | 39.5 | 60.4 | 55.0 |
| $\mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\thinspace\lower-5.32501pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{}}{}<br>{{}{}}{}<br>{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{-3.40001pt}\pgfsys@curveto{0.0pt}{-4.28368pt}{0.71634pt}{-5.00002pt}{1.6pt}{-5.00002pt}\pgfsys@lineto{5.38377pt}{-5.00002pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope}}{{}}}}<br>\pgfsys@invoke{ }\pgfsys@endscope}<br>\pgfsys@invoke{ }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{ }\pgfsys@endscope\hss}}\endpgfpicture}}}$ DeepScaleR-Preview-1.5B | 40.5 | 28.3 | 81.0 | 89.5 | 38.1 | 61.8 | 56.5 |
| $\mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\thinspace\lower-5.32501pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{}}{}<br>{{}{}}{}<br>{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{-3.40001pt}\pgfsys@curveto{0.0pt}{-4.28368pt}{0.71634pt}{-5.00002pt}{1.6pt}{-5.00002pt}\pgfsys@lineto{5.38377pt}{-5.00002pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope}}{{}}}}<br>\pgfsys@invoke{ }\pgfsys@endscope}<br>\pgfsys@invoke{ }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{ }\pgfsys@endscope\hss}}\endpgfpicture}}}$ TreeRL | 36.7 | 27.1 | 78.9 | 88.5 | 38.7 | 60.9 | 55.1 |
| $\mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\thinspace\lower-5.32501pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{}}{}<br>{{}{}}{}<br>{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{-3.40001pt}\pgfsys@curveto{0.0pt}{-4.28368pt}{0.71634pt}{-5.00002pt}{1.6pt}{-5.00002pt}\pgfsys@lineto{5.38377pt}{-5.00002pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope}}{{}}}}<br>\pgfsys@invoke{ }\pgfsys@endscope}<br>\pgfsys@invoke{ }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{ }\pgfsys@endscope\hss}}\endpgfpicture}}}$ AttnRL | 39.7 | 28.5 | 83.2 | 90.0 | 40.3 | 61.4 | 57.2 |
| DS-R1-Distill-Qwen-7B | 54.0 | 40.0 | 89.8 | 94.1 | 48.1 | 70.0 | 66.0 |
| $\mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\thinspace\lower-5.32501pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{}}{}<br>{{}{}}{}<br>{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{-3.40001pt}\pgfsys@curveto{0.0pt}{-4.28368pt}{0.71634pt}{-5.00002pt}{1.6pt}{-5.00002pt}\pgfsys@lineto{5.38377pt}{-5.00002pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope}}{{}}}}<br>\pgfsys@invoke{ }\pgfsys@endscope}<br>\pgfsys@invoke{ }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{ }\pgfsys@endscope\hss}}\endpgfpicture}}}$ GRPO | 54.9 | 39.6 | 90.8 | 94.3 | 48.6 | 69.7 | 66.3 |
| $\mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\thinspace\lower-5.32501pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{}}{}<br>{{}{}}{}<br>{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{-3.40001pt}\pgfsys@curveto{0.0pt}{-4.28368pt}{0.71634pt}{-5.00002pt}{1.6pt}{-5.00002pt}\pgfsys@lineto{5.38377pt}{-5.00002pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope}}{{}}}}<br>\pgfsys@invoke{ }\pgfsys@endscope}<br>\pgfsys@invoke{ }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{ }\pgfsys@endscope\hss}}\endpgfpicture}}}$ TreeRL | 55.4 | 40.0 | 92.2 | 94.3 | 49.0 | 70.7 | 66.9 |
| $\mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\thinspace\lower-5.32501pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{}}{}<br>{{}{}}{}<br>{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{-3.40001pt}\pgfsys@curveto{0.0pt}{-4.28368pt}{0.71634pt}{-5.00002pt}{1.6pt}{-5.00002pt}\pgfsys@lineto{5.38377pt}{-5.00002pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope}}{{}}}}<br>\pgfsys@invoke{ }\pgfsys@endscope}<br>\pgfsys@invoke{ }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{ }\pgfsys@endscope\hss}}\endpgfpicture}}}$ AttnRL | 59.3 | 42.5 | 92.5 | 95.4 | 49.3 | 73.3 | 68.7 |

##### AttnRL outperforms the base model.

As shown in Table [1](https://arxiv.org/html/2509.26628v1#S4.T1), AttnRL outperforms the base model across all six benchmarks, achieving an average improvement of 7.5% for DS-R1-Distill-Qwen-1.5B. AttnRL surpasses the base model significantly on AIME24 benchmark, achieving an improvement of 11.4% and 5.3% for 1.5B and 7B models, respectively.

##### AttnRL outperforms PSRL and strong RLVR baselines.

As reported in Table [1](https://arxiv.org/html/2509.26628v1#S4.T1), AttnRL surpasses GRPO and TreeRL by an average of 1.9% and 1.8% across all benchmarks at 1.5B scale, confirming its effectiveness. Moreover, AttnRL outperforms DeepScaleR-Preview-1.5B, which is trained with a three-stage context extension (8K $\to$ 16K $\to$ 24K) over 1750 steps. In contrast, AttnRL achieves superior results with only 500 steps at an 8K response length, highlighting both its effectiveness and efficiency.

### 4.3 Ablation Study

To evaluate the contribution of each component, we conduct an ablation study on the six mathematical benchmarks using DS-R1-Distill-Qwen-1.5B.
As shown in Table [2](https://arxiv.org/html/2509.26628v1#S4.T2), incorporating ATB alone improves performance over TreeRL by an average of 1.2%, while combining ATB with adaptive sampling allows AttnRL to achieve the highest performance.
Importantly, filtering out problems whose responses are all correct after initial sampling results in a slight performance drop, as even “easy” problems can produce incorrect responses under Monte Carlo sampling, providing valuable training signals that enhance overall model performance.

**Table 2: Results of ablation study on mathematical benchmarks. The results of AttnRL are shaded and the highest values are bolded.**
| Method | AIME24 | AIME25 | AMC23 | MATH-500 | Minerva | Olympiad | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| TreeRL | 36.7 | 27.1 | 78.9 | 88.5 | 38.7 | 60.9 | 55.1 |
| $\mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\thinspace\lower-5.32501pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{}}{}<br>{{}{}}{}<br>{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{-3.40001pt}\pgfsys@curveto{0.0pt}{-4.28368pt}{0.71634pt}{-5.00002pt}{1.6pt}{-5.00002pt}\pgfsys@lineto{5.38377pt}{-5.00002pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope}}{{}}}}<br>\pgfsys@invoke{ }\pgfsys@endscope}<br>\pgfsys@invoke{ }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{ }\pgfsys@endscope\hss}}\endpgfpicture}}}$ w/ATB | 39.1 | 27.2 | 81.4 | 89.2 | 40.1 | 61.0 | 56.3 |
| $\mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\thinspace\lower-5.32501pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{}}{}<br>{{}{}}{}<br>{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{-3.40001pt}\pgfsys@curveto{0.0pt}{-4.28368pt}{0.71634pt}{-5.00002pt}{1.6pt}{-5.00002pt}\pgfsys@lineto{5.38377pt}{-5.00002pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope}}{{}}}}<br>\pgfsys@invoke{ }\pgfsys@endscope}<br>\pgfsys@invoke{ }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{ }\pgfsys@endscope\hss}}\endpgfpicture}}}$ w/ATB + ADS (w/o attention-based filtering) | 38.4 | 29.1 | 81.0 | 89.8 | 38.7 | 61.2 | 56.4 |
| $\mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\thinspace\lower-5.32501pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{}}{}<br>{{}{}}{}<br>{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{-3.40001pt}\pgfsys@curveto{0.0pt}{-4.28368pt}{0.71634pt}{-5.00002pt}{1.6pt}{-5.00002pt}\pgfsys@lineto{5.38377pt}{-5.00002pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope}}{{}}}}<br>\pgfsys@invoke{ }\pgfsys@endscope}<br>\pgfsys@invoke{ }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{ }\pgfsys@endscope\hss}}\endpgfpicture}}}$ w/ATB + ADS (w/o difficulty-aware expansion) | 39.6 | 28.2 | 82.0 | 90.3 | 39.6 | 61.0 | 56.8 |
| $\mathrel{\hskip 3.00003pt\hbox to6.65pt{\vbox to5.65pt{\pgfpicture\makeatletter\hbox{\thinspace\lower-5.32501pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{}}{}<br>{{}{}}{}<br>{}{}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@setlinewidth{\the\pgflinewidth}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{}{}{}{}{{}}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{0.0pt}{-3.40001pt}\pgfsys@curveto{0.0pt}{-4.28368pt}{0.71634pt}{-5.00002pt}{1.6pt}{-5.00002pt}\pgfsys@lineto{5.38377pt}{-5.00002pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{5.38377pt}{-5.00002pt}\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@invoke{ }\pgfsys@endscope}}{{}}}}<br>\pgfsys@invoke{ }\pgfsys@endscope}<br>\pgfsys@invoke{ }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{ }\pgfsys@endscope\hss}}\endpgfpicture}}}$ AttnRL | 39.7 | 28.5 | 83.2 | 90.0 | 40.3 | 61.4 | 57.2 |

## 5 Analysis

### 5.1 Sampling

##### How does ATB outperform entropy-based tree branching?

The results in Table [2](https://arxiv.org/html/2509.26628v1#S4.T2) show that TreeRL w/ATB outperforms TreeRL, which branches at tokens with highest entropy values. To further understand the effects of ATB, we plot four sampling curves during training process in Figure [6](https://arxiv.org/html/2509.26628v1#S5.F6). For Figure [6](https://arxiv.org/html/2509.26628v1#S5.F6)(a) and (b), we visualize the solve all ratio (i.e., the ratio of problems whose outputs are all correct) and solve none ratio (i.e., the ratio of problems whose outputs are all wrong) of MC sampling, respectively. These two subfigures demonstrate that ATB enables more effective sampling at both easy and hard problems. Figure [6](https://arxiv.org/html/2509.26628v1#S5.F6)(c) and (d) show the valid ratio (i.e., the ratio of problems whose outputs are either not all correct nor all wrong) of MC sampling and both sampling, respectively. The results also demonstrate the effectiveness of ATB.

Figure: Figure 6: The sampling statistics of ATB and entropy-based branching. The curves are smoothed using EMA for better visualization.
Refer to caption: x7.png

##### Adaptive Sampling.

To better understand the effects of our proposed adaptive sampling method, we visualize the training curves related to the sampling process. The results in Figure [7](https://arxiv.org/html/2509.26628v1#S5.F7)(a) show that our method significantly reduces the ratio of both samples of two sampling steps are correct given the initial sampling results are correct, by filtering out prompts with low FCI scores (shown in Figure [7](https://arxiv.org/html/2509.26628v1#S5.F7)(c)). Additionally, AttnRL benefits from maintaining a valid training batch by dynamically adjust the prompt batch size (shown in Figure [7](https://arxiv.org/html/2509.26628v1#S5.F7)(d)), resulting in a training batch with all tokens having non-zero advantage values (shown in Figure [7](https://arxiv.org/html/2509.26628v1#S5.F7)(b)).

Figure: Figure 7: Curves related to sampling information statistics of all methods. The curves are smoothed using EMA for better visualization.
Refer to caption: x8.png

### 5.2 Training Dynamics and Efficiency

##### Training Dynamics.

The training dynamics of GRPO, TreeRL, and AttnRL are visualized in Figure [8](https://arxiv.org/html/2509.26628v1#S5.F8). Figure [8](https://arxiv.org/html/2509.26628v1#S5.F8)(a) shows that the entropy curve of GRPO decreases along the training process, while PSRL methods first decreases then increases. Compared with TreeRL, AttnRL shows higher entropy, enabling more diverse exploration during training.
Figure [8](https://arxiv.org/html/2509.26628v1#S5.F8)(b)-(c) show AttnRL learns faster with less training steps and Figure [8](https://arxiv.org/html/2509.26628v1#S5.F8)(d) shows the response length of AttnRL is shorter than that of TreeRL, demonstrating that AttnRL outperforms TreeRL at both final performance and reasoning conciseness.

Figure: Figure 8: The training dynamics curves of all methods. The curves are smoothed using EMA for better visualization.
Refer to caption: x9.png

##### Training Efficiency.

As shown in Table [3](https://arxiv.org/html/2509.26628v1#S5.T3), the training efficiency of the introduced one-step off-policy reduces the training time by 8% compared with original TreeRL implementation.
AttnRL outperforms TreeRL with less wall-clock training time, more valid tokens for training (i.e., token with non-zero advantage values), and better overall performance significantly under the same computational resources. These strong efficiency improvements are achieved through especially at our adaptive sampling mechanism, which samples a dynamic batch of problems, filters out some low-value easy problems, and keeping a relatively stable size of batch with all samples useful for training.

**Table 3: Comparison of performance and training efficiency among AttnRL and baselines. The results of AttnRL are shaded and the best values are bolded.**
| Method | # Training Steps | Wall-clock Time | # Valid Tokens | Performance |
| --- | --- | --- | --- | --- |
| GRPO | 800 | 54.0 | 5.2B | 55.0 |
| TreeRL | 800 | 67.7 | 5.0B | 55.1 |
| TreeRL w/one-step off-policy | 800 | 62.2 | 5.0B | 55.3 |
| AttnRL | 500 | 62.6 | 5.6B | 57.2 |

## 6 Related Work

### 6.1 Reinforcement Learning for LLM

Reinforcement Learning has shown great success for enhancing the reasoning abilities of LLMs (OpenAI, 2024; DeepSeek-AI et al., 2025). With the success of OpenAI o1 (OpenAI, 2024) and DeepSeek-R1 (DeepSeek-AI et al., 2025), RLVR has become an efficient method for improving reasoning abilities of LLMs (Yu et al., 2025; Liu et al., 2025d; Chu et al., 2025; Yue et al., 2025; He et al., 2025a; Luo et al., 2025; Chen et al., 2025b; Liu et al., 2025a; Chen et al., 2025a; An et al., 2025; Wang et al., 2025b; Zheng et al., 2025a).
These works focus on outcome-based rewards that are inefficient for RL training, while our method focus on RL with process rewards.

### 6.2 Process Supervision for LLM

Process supervision has demonstrated superiority than outcome-based feedback (Uesato et al., 2022; Lightman et al., 2024; Wang et al., 2024b).
A line of works focus on token-level process rewards (Yuan et al., 2025; Cui et al., 2025; Fei et al., 2025), using DPO-like rewards (Rafailov et al., 2023, 2024) for policy learning.
For PRM-based methods, a line of works (Wang et al., 2024b; Setlur et al., 2025; Cheng et al., 2025; Zha et al., 2025; Ye et al., 2025) use discriminative PRMs for RL training, while another line of works use generative PRMs (Zhao et al., 2025) to provide process rewards for RL training (Zou et al., 2025; He et al., 2025b; Xie et al., 2025).
To mitigate reward hacking and avoid training an online PRM, some works use online Monte Carlo sampling to estimate process rewards (Kazemnejad et al., 2025; Hou et al., 2025; Guo et al., 2025; Yang et al., 2025b; Zheng et al., 2025b; Li et al., 2025; Dong et al., 2025).
Our method belong to the category which leveraging MC sampling to estimate process rewards. However, previous methods mainly focus on non-reasoning models and is inefficient from the perspective of both branching points, sampling mechanism, and two-step generation, while our work proposes effective and efficient methods of process supervision for reasoning models.

## 7 Conclusion

In this paper, we propose AttnRL for PSRL in reasoning models, which leverages attention information to find reasoning-related steps and branches at these positions for efficient exploration. Additionally, we introduce adaptive sampling based on problem difficulty and maintaining valid training batch size.
Experimental results on mathematical reasoning benchmarks demonstrate the effectiveness and efficiency of our method.

## References

- An et al. (2025)
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong.
Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025.
URL [https://hkunlp.github.io/blog/2025/Polaris](https://hkunlp.github.io/blog/2025/Polaris).
- Anthropic (2023)
Anthropic.
Introducing claude, 2023.
URL [https://www.anthropic.com/index/introducing-claude/](https://www.anthropic.com/index/introducing-claude/).
- Bogdan et al. (2025)
Paul C Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy.
Thought anchors: Which llm reasoning steps matter?
*arXiv preprint arXiv:2506.19143*, 2025.
- Chen et al. (2025a)
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al.
Minimax-m1: Scaling test-time compute efficiently with lightning attention.
*arXiv preprint arXiv:2506.13585*, 2025a.
- Chen et al. (2025b)
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping.
Acereason-nemotron: Advancing math and code reasoning through reinforcement learning.
*arXiv preprint arXiv:2505.16400*, 2025b.
- Cheng et al. (2025)
Jie Cheng, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Gang Xiong, Yisheng Lv, and Fei-Yue Wang.
Stop summation: Min-form credit assignment is all process reward model needs for reasoning.
*arXiv preprint arXiv:2504.15275*, 2025.
- Chu et al. (2025)
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang.
Gpg: A simple and strong reinforcement learning baseline for model reasoning.
*arXiv preprint arXiv:2504.02546*, 2025.
- Cui et al. (2025)
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al.
Process reinforcement through implicit rewards.
*arXiv preprint arXiv:2502.01456*, 2025.
- DeepSeek-AI et al. (2025)
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Dong et al. (2025)
Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al.
Agentic reinforced policy optimization.
*arXiv preprint arXiv:2507.19849*, 2025.
- Fei et al. (2025)
Wu Fei, Hao Kong, Shuxian Liang, Yang Lin, Yibo Yang, Jing Tang, Lei Chen, and Xiansheng Hua.
Self-guided process reward optimization with redefined step-wise advantage for process reinforcement learning.
*arXiv preprint arXiv:2507.01551*, 2025.
- Fu et al. (2025)
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al.
Areal: A large-scale asynchronous reinforcement learning system for language reasoning.
*arXiv preprint arXiv:2505.24298*, 2025.
- Guo et al. (2025)
Yiran Guo, Lijie Xu, Jie Liu, Dan Ye, and Shuang Qiu.
Segment policy optimization: Effective segment-level credit assignment in rl for large language models.
*arXiv preprint arXiv:2505.23564*, 2025.
- He et al. (2024)
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun.
OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 3828–3850, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[10.18653/v1/2024.acl-long.211](https:/doi.org/10.18653/v1/2024.acl-long.211).
URL [https://aclanthology.org/2024.acl-long.211/](https://aclanthology.org/2024.acl-long.211/).
- He et al. (2025a)
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al.
Skywork open reasoner 1 technical report.
*arXiv preprint arXiv:2505.22312*, 2025a.
- He et al. (2025b)
Tao He, Rongchuan Mu, Lizi Liao, Yixin Cao, Ming Liu, and Bing Qin.
Good learners think their thinking: Generative prm makes large reasoning model more efficient math learner.
*arXiv preprint arXiv:2507.23317*, 2025b.
- Hou et al. (2025)
Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong.
TreeRL: LLM reinforcement learning with on-policy tree search.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 12355–12369, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
URL [https://aclanthology.org/2025.acl-long.604/](https://aclanthology.org/2025.acl-long.604/).
- Hu et al. (2025)
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum.
Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.
*arXiv preprint arXiv:2503.24290*, 2025.
- Hurst et al. (2024)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Jin et al. (2025)
Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du, Zirui Liu, and Yongfeng Zhang.
Massive values in self-attention modules are the key to contextual knowledge understanding.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=1SMcxxQiSL](https://openreview.net/forum?id=1SMcxxQiSL).
- Kazemnejad et al. (2025)
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux.
VinePPO: Refining credit assignment in RL training of LLMs.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=Myx2kJFzAn](https://openreview.net/forum?id=Myx2kJFzAn).
- Kingma (2014)
Diederik P Kingma.
Adam: A method for stochastic optimization.
*arXiv preprint arXiv:1412.6980*, 2014.
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the 29th Symposium on Operating Systems Principles*, SOSP ’23, page 611–626, New York, NY, USA, 2023. Association for Computing Machinery.
ISBN 9798400702297.
[10.1145/3600006.3613165](https:/doi.org/10.1145/3600006.3613165).
URL [https://doi.org/10.1145/3600006.3613165](https://doi.org/10.1145/3600006.3613165).
- Lewkowycz et al. (2022)
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra.
Solving Quantitative Reasoning Problems with Language Models.
In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, *Advances in Neural Information Processing Systems (NeurIPS)*, volume 35, pages 3843–3857. Curran Associates, Inc., 2022.
URL [https://proceedings.neurips.cc/paper_files/paper/2022/file/18abbeef8cfe9203fdf9053c9c4fe191-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2022/file/18abbeef8cfe9203fdf9053c9c4fe191-Paper-Conference.pdf).
- Li et al. (2025)
Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, Tianshun Xing, Shuyue Guo, Tianyu Zheng, Xin Zhou, Xingwei Qu, Wangchunshu Zhou, Zheng Zhang, Wei Shen, Qian Liu, Chenghua Lin, Jian Yang, Ge Zhang, and Wenhao Huang.
Treepo: Bridging the gap of policy optimization and efficacy and inference efficiency with heuristic tree-based modeling.
*arXiv preprint arXiv:2508.17445*, 2025.
- Lightman et al. (2024)
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=v8L0pN6EOi](https://openreview.net/forum?id=v8L0pN6EOi).
- Liu et al. (2025a)
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong.
Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.
*arXiv preprint arXiv:2505.24864*, 2025a.
- Liu et al. (2025b)
Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou.
Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling.
*arXiv preprint arXiv:2502.06703*, 2025b.
- Liu et al. (2025c)
Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang, and Junxian He.
Learn to reason efficiently with adaptive length-based reward shaping.
*arXiv preprint arXiv:2505.15612*, 2025c.
- Liu et al. (2025d)
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin.
Understanding r1-zero-like training: A critical perspective.
*arXiv preprint arXiv:2503.20783*, 2025d.
- Luo et al. (2025)
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica.
Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl.
[https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2](https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2), 2025.
Notion Blog.
- MAA (2023)
MAA.
American mathematics contest 12 (amc 12), November 2023.
URL [https://artofproblemsolving.com/wiki/index.php/AMC_12_Problems_and_Solutions](https://artofproblemsolving.com/wiki/index.php/AMC_12_Problems_and_Solutions).
- MAA (2024)
MAA.
American invitational mathematics examination (aime), February 2024.
URL [https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions](https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions).
- MAA (2025)
MAA.
American invitational mathematics examination (aime), February 2025.
URL [https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions](https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions).
- meituan search (2025)
meituan search.
Recipe: One step off policy async trainer.
[https://github.com/volcengine/verl/tree/main/recipe/one_step_off_policy](https://github.com/volcengine/verl/tree/main/recipe/one_step_off_policy), 2025.
GitHub repository.
- Noukhovitch et al. (2025)
Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville.
Asynchronous RLHF: Faster and more efficient off-policy rl for language models.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=FhTAG591Ve](https://openreview.net/forum?id=FhTAG591Ve).
- OpenAI (2023)
OpenAI.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- OpenAI (2024)
OpenAI.
Learning to reason with llms, 2024.
URL [https://openai.com/index/learning-to-reason-with-llms](https://openai.com/index/learning-to-reason-with-llms).
- Rafailov et al. (2023)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
*Advances in neural information processing systems*, 36:53728–53741, 2023.
- Rafailov et al. (2024)
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn.
From r to q star: Your language model is secretly a q-function.
In *First Conference on Language Modeling*, 2024.
URL [https://openreview.net/forum?id=kEVcNxtqXk](https://openreview.net/forum?id=kEVcNxtqXk).
- Setlur et al. (2025)
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar.
Rewarding progress: Scaling automated process verifiers for LLM reasoning.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=A6Y7AqlzLW](https://openreview.net/forum?id=A6Y7AqlzLW).
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Sheng et al. (2025)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
In *Proceedings of the Twentieth European Conference on Computer Systems*, EuroSys ’25, page 1279–1297, New York, NY, USA, 2025. Association for Computing Machinery.
ISBN 9798400711961.
[10.1145/3689031.3696075](https:/doi.org/10.1145/3689031.3696075).
URL [https://doi.org/10.1145/3689031.3696075](https://doi.org/10.1145/3689031.3696075).
- Sutton and Barto (2018)
Richard S Sutton and Andrew G Barto.
*Reinforcement learning: An introduction*.
MIT press, 2018.
- Uesato et al. (2022)
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins.
Solving math word problems with process-and outcome-based feedback.
*arXiv preprint arXiv:2211.14275*, 2022.
- Vaswani et al. (2017)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc., 2017.
URL [https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf).
- Wang et al. (2025a)
Jiakang Wang, Runze Liu, Lei Lin, Wenping Hu, Xiu Li, Fuzheng Zhang, and Guorui Zhou.
Aspo: Asymmetric importance sampling policy optimization.
[https://rogue-canopy-54a.notion.site/Asymmetric-Dual-Clipping-Unleashing-the-Full-Potential-of-RL-in-LLM-Training-2650e4c8c16a8034a5d3dfec358c9021](https://rogue-canopy-54a.notion.site/Asymmetric-Dual-Clipping-Unleashing-the-Full-Potential-of-RL-in-LLM-Training-2650e4c8c16a8034a5d3dfec358c9021), 2025a.
Notion Blog.
- Wang et al. (2025b)
Jiakang Wang, Runze Liu, Fuzheng Zhang, Xiu Li, and Guorui Zhou.
Stabilizing knowledge, promoting reasoning: Dual-token constraints for rlvr.
*arXiv preprint arXiv:2507.15778*, 2025b.
- Wang et al. (2024a)
Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al.
Openr: An open source framework for advanced reasoning with large language models.
*arXiv preprint arXiv:2410.09671*, 2024a.
- Wang et al. (2024b)
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui.
Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 9426–9439, Bangkok, Thailand, August 2024b. Association for Computational Linguistics.
[10.18653/v1/2024.acl-long.510](https:/doi.org/10.18653/v1/2024.acl-long.510).
URL [https://aclanthology.org/2024.acl-long.510/](https://aclanthology.org/2024.acl-long.510/).
- Wen et al. (2025)
Hao Wen, Yifan Su, Feifei Zhang, Yunxin Liu, Yunhao Liu, Ya-Qin Zhang, and Yuanchun Li.
Parathinker: Native parallel thinking as a new paradigm to scale llm test-time compute.
*arXiv preprint arXiv:2509.04475*, 2025.
- Xie et al. (2025)
Guofu Xie, Yunsheng Shi, Hongtao Tian, Ting Yao, and Xiao Zhang.
Capo: Towards enhancing llm reasoning through verifiable generative credit assignment.
*arXiv preprint arXiv:2508.02298*, 2025.
- Yang et al. (2024)
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report.
*arXiv preprint arXiv:2412.15115*, 2024.
- Yang et al. (2025a)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025a.
- Yang et al. (2025b)
Zhicheng Yang, Zhijiang Guo, Yinya Huang, Xiaodan Liang, Yiwei Wang, and Jing Tang.
Treerpo: Tree relative policy optimization.
*arXiv preprint arXiv:2506.05183*, 2025b.
- Ye et al. (2025)
Chenlu Ye, Zhou Yu, Ziji Zhang, Hao Chen, Narayanan Sadagopan, Jing Huang, Tong Zhang, and Anurag Beniwal.
Beyond correctness: Harmonizing process and outcome rewards through rl training.
*arXiv preprint arXiv:2509.03403*, 2025.
- Yu et al. (2025)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al.
Dapo: An open-source llm reinforcement learning system at scale.
*arXiv preprint arXiv:2503.14476*, 2025.
- Yuan et al. (2025)
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng.
Free process rewards without process labels.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=8ThnPFhGm8](https://openreview.net/forum?id=8ThnPFhGm8).
- Yue et al. (2025)
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al.
Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.
*arXiv preprint arXiv:2504.05118*, 2025.
- Zeng et al. (2025)
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He.
Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.
*arXiv preprint arXiv:2503.18892*, 2025.
- Zha et al. (2025)
Kaiwen Zha, Zhengqi Gao, Maohao Shen, Zhang-Wei Hong, Duane S Boning, and Dina Katabi.
Rl tango: Reinforcing generator and verifier together for language reasoning.
*arXiv preprint arXiv:2505.15034*, 2025.
- Zhang et al. (2025a)
Kaiyan Zhang, Runze Liu, Xuekai Zhu, Kai Tian, Sihang Zeng, Guoli Jia, Yuchen Fan, Xingtai Lv, Yuxin Zuo, Che Jiang, Ziyang Liu, Jianyu Wang, Yuru Wang, Ruotong Zhao, Ermo Hua, Yibo Wang, Shijie Wang, Junqi Gao, Xinwei Long, Youbang Sun, Zhiyuan Ma, Ganqu Cui, Lei Bai, Ning Ding, Biqing Qi, and Bowen Zhou.
Marti: A framework for multi-agent llm systems reinforced training and inference, 2025a.
URL [https://github.com/TsinghuaC3I/MARTI](https://github.com/TsinghuaC3I/MARTI).
- Zhang et al. (2025b)
Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, Yu Fu, Xingtai Lv, Yuchen Zhang, Sihang Zeng, Shang Qu, Haozhan Li, Shijie Wang, Yuru Wang, Xinwei Long, Fangfu Liu, Xiang Xu, Jiaze Ma, Xuekai Zhu, Ermo Hua, Yihao Liu, Zonglin Li, Huayu Chen, Xiaoye Qu, Yafu Li, Weize Chen, Zhenzhao Yuan, Junqi Gao, Dong Li, Zhiyuan Ma, Ganqu Cui, Zhiyuan Liu, Biqing Qi, Ning Ding, and Bowen Zhou.
A survey of reinforcement learning for large reasoning models.
*arXiv preprint arXiv:2509.08827*, 2025b.
- Zhao et al. (2025)
Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, et al.
Genprm: Scaling test-time compute of process reward models via generative reasoning.
*arXiv preprint arXiv:2504.00891*, 2025.
- Zheng et al. (2025a)
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al.
Group sequence policy optimization.
*arXiv preprint arXiv:2507.18071*, 2025a.
- Zheng et al. (2025b)
Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, et al.
First return, entropy-eliciting explore.
*arXiv preprint arXiv:2507.07017*, 2025b.
- Zou et al. (2025)
Jiaru Zou, Ling Yang, Jingwen Gu, Jiahao Qiu, Ke Shen, Jingrui He, and Mengdi Wang.
Reasonflux-prm: Trajectory-aware prms for long chain-of-thought reasoning in llms.
*arXiv preprint arXiv:2506.18896*, 2025.

## Appendix A Experimental Details

##### Evaluation.

For evaluation, we use the prompt listed in Table [4](https://arxiv.org/html/2509.26628v1#A1.T4), following Luo et al. (2025).

Table: Table 4: Prompt Template.

## Appendix B Additional Experimental Results

### B.1 Full Test Curves

The test curves of six mathematical benchmarks are shown in Figure [9](https://arxiv.org/html/2509.26628v1#A2.F9).

Figure: Figure 9: Test curves of GRPO, TreeRL, and our method on six benchmarks.
Refer to caption: x10.png

## Appendix C Cases

The full response of Figure [2](https://arxiv.org/html/2509.26628v1#S3.F2) is listed in Table [5](https://arxiv.org/html/2509.26628v1#A3.T5) and steps with high FCI scores are shown in red.

Table: Table 5: Full response.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: see weakness above
soundness: 3
strengths: ## Originality

The paper demonstrates strong originality by being the first to systematically leverage attention mechanisms for identifying branching points in process-supervised RL. While attention analysis and PSRL exist independently, the Forward Context Influence (FCI) metric elegantly connects attention patterns to reasoning behaviors, moving beyond naive entropy-based heuristics. The disruption experiments (Figure 1) provide compelling validation that high-FCI steps correspond to critical reasoning behaviors like planning and verification, not just predictive uncertainty. The adaptive sampling strategy that dynamically allocates compute based on problem difficulty addresses a practical inefficiency overlooked by prior uniform-sampling approaches, showing creative problem-solving beyond incremental improvements.

## Quality

The technical quality is high with comprehensive evaluation across six mathematical benchmarks and two model scales (1.5B, 7B), demonstrating consistent improvements. The experimental design is sound, using appropriate baselines (GRPO, TreeRL, DeepScaleR-Preview) and reporting both performance (Pass@K) and efficiency metrics (wall-clock time, valid training tokens). Ablation studies systematically isolate component contributions (ATB: +1.2%, full method: +1.8%) and reveal nuanced insights, such as filtering all easy problems slightly hurting performance. 

## Clarity

The paper is well-structured with clear motivation and logical progression from problem to solution. Figure 1 effectively visualizes FCI scores with annotated reasoning steps. Mathematical formulations are precise and implementation details comprehensive, facilitating reproducibility.
summary: This paper proposes AttnRL, a novel Process-Supervised Reinforcement Learning (PSRL) framework that improves exploration efficiency for reasoning models. The authors identify critical limitations in existing PSRL methods like TreeRL: entropy-based branching ignores semantic meaning of reasoning steps, uniform sampling wastes computation on uninformative problems, and two-step sampling per iteration creates computational bottlenecks. AttnRL addresses these issues through attention-guided branching, adaptive difficulty-aware sampling, and efficient training pipelines, achieving superior performance with better sample efficiency.
weaknesses: ## Limited Theoretical Foundation

The paper lacks rigorous theoretical justification for why attention-based branching should outperform entropy-based selection. While the FCI metric is intuitively motivated and empirically validated through disruption experiments, there is no formal analysis establishing under what conditions high attention scores correspond to high-value branching points for credit assignment. The connection between "massive attention values" and reasoning behaviors relies primarily on empirical correlation (Figure 1) without explaining the underlying mechanism—why do planning and verification steps exhibit high attention? What attention patterns distinguish productive reasoning from unhelpful verbosity?

## Questionable Distribution Shift

The **selective branching based on FCI scores creates significant state distribution shift** that the paper completely ignores: by branching only from top 20% high-FCI states, the method oversamples certain states by 5-10× compared to the natural policy distribution $d^\pi(s)$. This creates $d^{branching}(s) \neq d^\pi(s)$, which technically requires importance weight corrections $w(s) = d^\pi(s)/d^{branching}(s)$ for unbiased policy gradients. 

## Inconsistent Results and Missing Statistical Analysis

The results show concerning inconsistencies without proper statistical analysis. Table 1 shows AttnRL's improvements vary wildly across benchmarks: +4.0 points on AIME24 but only +0.2 on MATH-500 for the 1.5B model, and sometimes underperforms TreeRL (e.g., AMC23: 91.9 vs 92.2 for 7B). No error bars, confidence intervals, or significance tests are provided—are these differences within noise? The paper doesn't specify number of training runs, random seeds, or variance across runs, making it impossible to assess reliability. The paper should report results with multiple seeds, provide statistical significance tests (e.g., bootstrap confidence intervals), and better control for confounds when comparing to published baselines.

## Computational Cost and Scalability Concerns

While the paper claims "lightweight" and improved efficiency, the actual computational overhead of AttnRL is underspecified. Computing FCI scores requires extracting and aggregating attention matrices across all layers and heads for every sequence ($\max_{l,h}\{\sum_j \alpha^{l,h}_{j,k}\}$), which could be expensive for long sequences or large models. How much wall-clock time does FCI computation add per sample?

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: * How is a tree constructed from just identify important steps? I believe the tree construction is never actually discussed.
* How does Figure 3 show that the advantage is zero if it shows FCI scores?
  * If the FCI score is defined for a given step, how is it applied to a whole problem?
* Isn't the total token generation the limiting factor instead of the multiple sampling?
* How different can the DeepScaleR-Preview-Dataset be seen compared to the evaluation benchmarks/datasets?
* Are TreeRL and AttnRL trained using PPO as a "PPO minibatch size" is used? Previously I was under the impression that AttnRL mechanism is applied to GRPO.
soundness: 3
strengths: * improvement of (exploration in) reasoning chains in RLMs is a timely topic
* selection of important reasoning steps based on attention scores instead of for example on entropy seems like a novel idea
* additionally a number of improvements are suggested to improve the sampling efficiency
* writing is mostly clear and seems to be well motivated, where certain decisions are based on or validated with micro-benchmarks
* evaluation:
  * clear description of the used evaluation setting and the hyperparameters for the most part
  * training, while being done in fewer steps, shows improvements on a selection of evaluation benchmarks
summary: The manuscript proposes AttnRL to make exploration in Reasoning Language Models (RLMs) more efficient. The authors start from an analysis that shows that the decomposition of reasoning steps, which they define on a paragraph level, correlates with high attention scores. By using a new metric called FCI score, which reflects how much a given reasoning step influences the subsequent parts of the trajectory, they select the highest relevant reasoning steps as well as the first two reasoning steps, where additional exploration, i.e. branching should take place. Additionally the authors propose a number of techniques to improve the sampling efficiency, for example by filtering out easy problems that provide zero advantage in the Markov Decision Process to improve efficiency as well as determining the magnitude of exploration by the complexity of the problem. Another optimization is the  dynamic selection of the batch size to improve training efficiency and that the two sampling phases are combined together, so that presumably optimizations can applied to the sampling phase as a whole.
weaknesses: My main concern is the limited evaluation:
* only two different and smaller models (1.5B and 7B) are trained, which are from the same model family
* unclear whether the approach generalizes to larger models, since the improvements to the 7B model are already small
* I found it a bit unclear which policy optimization algorithm is used for training AttnRL
* ablation study seems a bit limited, albeit I have no concrete suggestion for additional experiments to run
* description of the baselines could be a bit clearer

4.3: Table 4.2 seems to suggest that AttnRL is implemented on top of TreeRL or is based on it, but alters certain key aspects? -> I think this should be made more clearer at the beginning of chapter 4.
  * best results on the individual benchmarks are not bolded, despite claiming in the caption to do so, probably because AttnRL is only best in three of them 
  * performance drop for filtering easiest is not completely true -> is best in two of the benchmarks

related work: if there is that much work with outcome-based reward modeling, I assume there is a reason for it - I found the efficiency claim not convincing in its brevity.

Additionally while the manuscript is clearly written, it also omits a lot of details/assumes prior knowledge, so it might be at times hard to understand for the reader. But maybe that is just my personal lack of knowledge.

minor issues:
* 4.1 
  * line 310: footnote is not on the same page
  * line 314/315: Luo et al. (2025) is cited twice in the same sentence
* related work, 6.2: abbreviation PRM is never introduced
* references:
  * no "accessed date for weblinks:
    * [An et al. 2025]
    * [Anthropic 2023]
    * [Luo et al. 2025]
    * [MAA 2023/2024/2025]
    * [meituan search 2025]
    * [OpenAI 2024]
  * consider capitalizing the titles to be consistent, especially abbreviations and proper names (which is sometimes done)
  * [Luo et al. 2025] - URL cuts into the margin (consider using the xurl package)
  * [MAA 2024] and [MAA 2025] have the same URL

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: - In the abstract, AttnRL is claimed to resolve the “limited exploration efficiency”. How does AttnRL improve the exploration efficiency?
- In figure 1, from my understanding, those sentences (e.g., “Let me compute that”) are steps. Why are only some words colored in red? What does the color indicate? Why are those words related to “reasoning”?
- In line 252, What do you mean “more difficult to rollout”?
- In line 272, what is $B$ and how is it involved in the filtering? 
- In experiment section, why do different training have different steps? If so, is the comparison fair?
soundness: 2
strengths: 1, I appreciate their in-depth empirical analysis into the effects of attention score/step position (section 3.1.1 and 3.1.2).

2, In section 4, the proposed method (AttnRL) can significantly outperform three baselines: the base model, GRPO, and TreeRL.
summary: This work first conducts empirical analysis on how high attention score reasoning steps are connected to the reasoning process. Then, they propose the following methods: 1, branching at massive attention values; 2, filtering out responses whose advantage values are 0; 3, a multi-process method to sample trajectories on adjacent steps.
weaknesses: 1, Some presentations in this paper could be improved, making some of its point unclear to me. I will list my questions in the Questions section.

2, Although the work does some empirical analysis on token attention scores, it is still unclear to me why the branching points should be selected according to the attention scores.

3, Some of the designs are claimed without explanation:
- equation 3
- equation 7 (btw, what is the definition of $C$; how is it used?)
- equation 9
- The proposed method does not significantly outperform DeepScaleR-Preview-1.5B

4, in Line 240 to 241, the claim that zero advantage indicates all samples are correct is wrong. If all samples are wrong, their advantages are also 0.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Q1. What is the relationship between FCI scores and step position? For example, do they exhibit a certain correlation?

Q2. Do you expand starting at the high-FCI step itself (i.e., keep that step fixed and sample continuations from it), or do you expand from its parent step (i.e., resample the action at that step as well)? Please clarify which you use and why. If possible, please also explain your intuition: you mention that steps with high FCI scores are related to reasoning behaviors such as planning and self-verification—why does branching in this way help?
soundness: 3
strengths: 1. Clear motivation and insightful analysis: Figure 2’s analysis (e.g., how perturbing high-FCI steps impacts performance) is interesting and provides useful intuition for why attention-guided branching should help.

2. Method completeness: attention-guided step selection + adaptive sampling + off-policy pipeline form a coherent system, the one-step pipeline reduces the training-efficiency issues brought about by TreeRL.

3. Empirical evidence: the authors report consistent gains over GRPO/TreeRL across multiple math benchmarks (1.5B/7B).
summary: The paper proposes AttnRL for process-supervised reasoning RL. It uses attention-guided branching to pick a small set of step-level decision points to expand; a one-step off-policy pipeline is used to improve training efficiency; plus difficulty-aware adaptive sampling so harder problems receive more rollouts. On several math reasoning benchmarks (1.5B/7B models), AttnRL shows better accuracy and faster learning curves.
weaknesses: 1. The main claim is using attention scores to guide process RL, but current experiments may not sufficiently isolate this contribution. The final selection combines the top 20% FCI steps plus the earliest steps—are there ablations separating these two factors? Would a simple baseline that branches only on the first 2 steps close the gap?

2.  The paper reports a w/ ATB ablation with 1.2 point gains in average (in Table 2), does TreeRL and w/ATB in Table 2 use the same rollout budget? Is this gain large enough to support the claim? Similar ablations are missing on DS-R1-Distill-Qwen-7B—are the conclusions consistent there?

2. Complexity vs. gain: Although simpler than TreeRL, training complexity still increases compared to plain GRPO. The gains at 7B are smaller than at 1.5B; considering cost vs. benefit, the overall value is questionable.


Decision: Accept (Poster)