Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Models
Authors: Runze Liu
Jiakang Wang
Yuling Shi
Zhihui Xie
Chenxin An
Kaiyan Zhang
Jian Zhao
Xiaodong Gu
Lei Lin
Wenping Hu
Xiu Li
Fuzheng Zhang
Guorui Zhou
Kun Gai
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Reinforcement Learning
Process Supervision
Primary area: reinforcement learning

Abstract: Reinforcement Learning (RL) has shown remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). Process-Supervised RL (PSRL) has emerged as a more effective paradigm compared to outcome-based RL. However, existing PSRL approaches suffer from limited exploration efficiency, both in terms of branching positions and sampling. In this paper, we introduce a novel PSRL framework (AttnRL), which enables efficient exploration for reasoning models. Motivated by preliminary observations that steps exhibiting high attention scores correlate with reasoning behaviors, we propose to branch from positions with high values. Furthermore, we develop an adaptive sampling strategy that accounts for problem difficulty and historical batch size, ensuring that the whole training batch maintains non-zero advantage values. To further improve sampling efficiency, we design a one-step off-policy training pipeline for PSRL. Extensive experiments on multiple challenging mathematical reasoning benchmarks demonstrate that our method consistently outperforms prior approaches in terms of performance and sampling and training efficiency. Our code is available at https://github.com/RyanLiu112/AttnRL.

Published as a conference paper at ICLR 2026
ATTENTION AS A COMPASS: EFFICIENT EXPLORATION FOR PROCESS- SUSSSSISSEER RL IN REASONING MOD- ELS
Runze Liu Jiakang Wang", Yuling Shi Zhihui Xie Chenxin An', Kaiyan Zhang Jian Zhao', Xiaodong Gu", Lei Lin?, Wenping Hu?, Xiu Li Fuzheng Zhang", Guorui Zhou?t, Kun Gai
TTinnhua University, Kuaiihhuu Technology. Shaanhai Jiao Tong University, "The University of Hong Kong, Frontis.Al
ABSTRACT
Reinforcement Learning (RL) has shown remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). Process-Supervised effective paradigm compared outcome-
RL (PSRL) has emerged as based RL. However, existing PSRL approaches suffer from limited exploration efficiency, both in terms of branching positions and sampling. In this paper, we introduce a novel PSRL framework (AttRLL), which enables efficient exploration for reasoning models. Motivated by preliminary observations that steps exhibiting high attention scores correlate with reasoning behaviors, we propose to branch from positions with high values. Furthermore, we develop an adaptive sampling strategy that accounts for problem difficulty and historical batch size, ensuring that the whole training batch maintains non-zero advantage values. To further improve sampling efficiency, design one-step off-police training pipeline for PSRL. Extensive experiments on multiple challenging mathematical reasoning benchmarks demonstrate that method consistently outperforms prior approaches in terms of performance and sampling and training efficiency. Our code is available at https::/ /gi hub. com Ryanniannlllliiii ul12 AttnRL.
57.54
Most efficient
Branching Point
Branching Point
57.0-
92
0.5
as
Accuracy (%)
DeepScalel-Pneviee. 1.58
56.5
56.0
GAPO
452
464
496
528
560
=2000
GPU Hours
(b)
Figure 1: An Illustration of AunRL. (a) AttRI branches at steps with high attention scores. (b)
AttnRI outperforms the baselines with great efficiency.
INTRODUCTION
Large Language Models (LLMs) have achieved remarkable progress in recent years (OpenAI, 2023; Hurst et al., 2024; Anthropic, 2023), particularly in their reasoning capabilities (OpenAI, 2024;
Work done during an internship at Kuaishou Technology Corresponding authors: Xiu Li (li.xiu@sz.. wwi.husu.siggg...cu.....mmmm Guorui Zhou (zhouguorui @ kuaishou.com)

---

Published as a conference paper at ICLR 2026
Guo et al., 2025a). With the success of DeepSeek-RI R1 (Guo et al., 2025a), Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective post-training paradigm for further strengthening the reasoning abilities of LLMs (Shao et al., 2024; Zeng et al., 2025; Luo et al., 2025; Yu et al., 2025; Liu et al., 2025c; Hu et al., 2025; He et al., 2025a; An et al., 2025; Zhang et al., 2026; Wang al., 2025; Zheng al., 2025a).
Common RLVR approaches, such Group Relative Policy Optimization (GRPO) (Shao et al., 2024) and its variants (Yu et al., 2025; Liu et al., 2025c; Yue et al., 2025), assign uniform training signals to all tokens within the same response, thereby overlooking fine-grained reasoning quality. In contrast, Process-Super SupeessSupe RL (PSRL) methods refine credit assignment with Monte Carlo (MC) sampling to estimate step-level advantages (Hou et al., 2025; Guo et al., 2025b; Yang et al., 2025b; Zheng et al., 2025b; Li et al., 2025). However, existing PSRL methods suffer from several limitations: (1) they segment responses by fixed token length entropy, ignoring the semantic meaning of model outputs; (2) they adopt uniform sampling across prompts and responses, leading to inefficient exploration; (3) they typically rely on two-step sampling per update, which significantly increases computational cost.
To overcome these limitations, introduce AttnRL, novel PSRL framework that improves both exploration and training efficiency. Our approach is motivated by the observation that attention scores serve meaningful metrics for identifying important reasoning behaviors in the model output. We therefore introduce an ittention-basee branching strategy for Monte Carlo sampling. To further enhance efficiency, design an adaptive sampling mechanism that prioritizes difficult problems while filtering easier ones, and an adaptive batch sampling strategy that guarantees non-zero advantage values across batches. The experimental results on mathematical reasoning tasks demonstrate that AttnRL outperforms strong outcome- outcome-based and process-based based basclines with great efficiency. An overview of AttnRI. is shown Figure 1.
The contributions of this work be summarized follows:
We analyze the relationship between attention scores and reasoning behaviors, and propose attention-based branching method for PSRL.
We develop an adaptive sampling mechanism that balances exploration across problems of varying difficulty and ensure valid training batches without zero advantage values.
Empirical results on six mathematical benchmarks demonstrate the superiority of our method beyond the baselines in both performance and efficiency.
PRELIMINARIES
2.1 LLM REASONING STEE-LEVEL MARKOV DECISION PROCESS
Following Sutton & Barto (2018); Zhang et al. (2025), we formulate LLM reasoning as a Markov Decision Process (MDP) defined by the tuple (S,A, P. R, R),,, where S is the state space, A is the action space, P S x A S is the transition dynamics, R : S x A + IR is the reward function, and 7 [o, 1] is the discount factor. In the LLM setting with prompt dataset D, the initial state is 81 D. The state transition is deterministic, since the next state is formed by concatenating the current state with the generated action: S = [sk, axl, where [. denotes string concatenation. For process-level supervision of LLMs (Zhang et al., 2025; Liu et al., 2025b), actions are defined at the step level, where each action a corresponds to a semantically coherent segment such as a sentence or a paragraph, rather than single token. In this paper, we adopt this step-level MDP formulation.
2.2 WUTTCOME-SUEERVEE AND PROCESS- PROCESS-SUPERVISEE SUPERVISED RL
Outcome-Supervised RL. Group Relative Policy Optimization (GRPO) (Shao et al., 2024) is an Outcome-S Supervised RL (OSRL) method that eliminates the need for an explicit critic model by estimating the advantage using the rewards ... Ro) ) of G sampled rollouts {0i,... .a.). The
normalized advantage is computed as A = Re-mean( ( (
The GRPO objective is then given
std((R,)
2

---

Published as a conference paper at ICLR 2026
by:
(o) = -E
G
loil
min (rre(0)Aau,, clip (rr,(0), -,,1+), - KL (Tell (itelaaet)
G
0i
(1)
20l0n111 where ri, Te (9,,119%),c))) the importance sampling ratio, and controls the strength of the KL divergence penalty that regularizes the policy towards the reference policy Tee.-
Process-Supervise RL. For PSRL, the sampling process usually includes two stages: (1) Initial
Sampling:
Sample multiple responses the problem: (2) Monte Carlo Sampling: Select several tokens as division points and rollout twice starting from these branching positions (Hou et al., 2025;
Guo al., 2025b; Yang et al., 2025b). In this paper, we follow the setting of TreeRL (Hou al., 2025), which proposes a tree-based advantage estimation method. For each node, the value is computed as the average accuracy its all children:
((s))
Σ (EL(Su)
1(1 is correct),
[[(sk)]
where L(sa) denotes the children of node 8g. The final advantage is the summation of global advantage (V(s)) V(s1)) and local advantage (V(s)) - (()(ss)):
A
V((x) V(s)+ - (p((s)))
(3)
V(L(s
global advantage
local advantage
where v [(s)) is used reduce the optimization strength of the non-leaf steps to prevent overfit- ting (Hou et al., 2025) and p(sk) is the parent node of sμ. Then the policy is optimized using the loss function in equation which is the same that of OSRL but differs at the advantage granularity.
2.3 ATTENTION MECHANISM
Modern LLMs are typically decoder-only Transformer-baat architectures (Vaswani et al., 2017; Yang al., 2024; 2025a), and the operation inside each Transformer block is the (masked) self-attention mechanism. For a given layer and head h, the model first computes query Q key kth and value matrices. Then the attention score a is computed as:
(Q. KLA ddl
L.h softmax
mask
where de is the per-head dimensionality. vanilla causal attention, attention mask blocks access to all future tokens by assigning them -00, while past and current tokens remain unmasked with O.
METHOD
In this section, we present AttnRL, an exploration-efficcen method for process-supervised RL. We begin by examining the role of massive attention values and how they be leveraged to identify branching points via attention scores to explore at important branches (Section 3.1). To enable more efficient exploration, easy problems and ensures each training batch contains only samples with non-zero advantage (Section 3.2). Finally, introduce our efficient training pipeline based one-step off-policy learning (Section 3.3).
propose adaptive sampling strategy that avoids oversampling
3.1 BRANCHING AT MASSIVE ATTENTION VALUES
Prior work has demonstrated that massive attention values in self-attention mechanisms play critical role contextual knowledge understanding (Jin et al., 2025), as they highlight tokens most relevant for answering questions. Motivated by this insight, investigate two key questions: (1) Do massive attention values consistently appear in complex reasoning tasks? (2) What impact do these massive attention values have, and how can they be effectively utilized in RI. training?
3

---

Published as a conference paper at ICLR 2026
3.1.1 MASSIVE ATTENTION VALUES IN LLMS
Step Segmenting and computing step-level attention scores. Following prior work on process supervision (Wang et al., 2024a; Liu et al., 2025b), we first segment the entire response into multiple steps using two consecutive line breaks \n"",, yielding Tg steps: o (01, 02,...... or,). Next, we extract token-to-token attention scores via a single forward pass. By aggregating these scores at the step level, obtain step-to-step attention matrices a E RTxx,, where jjk denotes the attention weight of step j attending to step k at layer and head h.
I.
Step 2: Computing the Forward Context Inftuence (FCI) score. To quantify the influence of a given step on subsequent tokens, we define the Forward Context Influence (FCI) score at layer and head h by summing the attention scores over the subsequent steps:
Lh
J.k
R++AA
where A is a hyperparameter that restricts the scope to sufficiently distant parts of the response, set to 4 following Bogdan al. (2025). We then aggregate across layers and heads by taking the maximum value:
Vk max 1.A)
(6)
The resulting FCI score yk captures the degree to which step influences the downstream context at the attention level. An illustrative visualization of steps with large FCI values is provided in Figure From this figure, we can see that most steps with high FCI scores peak FCI values are related to reasoning behaviors, such as planning and self-verification (Bogdan et al., 2025). The full response are listed in Table 5 in Appendix C.
net compute that.
0.10
0.00
Number
Wait, let verify that. Maybe list...
0.06
Airight, the next step ...
Alternatively, maybe using...
0.04
Then,
0.02
0.00
5
10
15
20
25
30
Step Indices
Figure The visualization of steps with high FCI scores. The words in red denote reasoning
behaviors.
3.1.2 THE EFFECTS OF STEPS WITH HIGH FCI SCORES
After identifying and qualitatively analyzing steps with high FCI scores, we conduct quantitative experiments examine the impact of disrupting attention values on performance. Specifically. we select step either (1) randomly from the top 20% of steps ranked by FCI scores (denoted as "Top 20%") or (2) randomly from the remaining steps (denoted as "20%-100%"), For the chosen step, we set its corresponding attention values to zero. We hypothesize that disrupting attention at key steps (with high FCI scores) will cause greater performance degradation compared to disrupting other steps. We test this hypothesis on AIME24 (MAA, 2024) using DS--R-Disttll--- problem sampled four times. The results shown in Figure 3(a) confirm that disrupting top 20% steps leads to significant drop in accuracy. Furthermore, We divide the disruption positions relative to the original response length into five uniform bins. As shown in Figure 3(b), accuracy exhibits increasing trend the disruption position moves later in
with each
investigate the effect of disruption position.
4

---

Published as a conference paper at ICLR 2026
the sequence, indicating that disruptions at earlier positions have larger negative impact on final performance.
Disruption Acc. (Original Acc.
0.35
0.30
0.25
Accuracy
0.20
0.15
0.10
6
0.05
0.00
FCI 20%
100%
FCI Top 20%
Entropy Top 20%
0.0
0.2
0.4
0.6
0.8
Disruption Type
Disruption Position
(a)
(b)
Figure 3: Disruption results on AIME24 and AIME25. (a) Normalized average accuracy of different disruption types. (b) Average accuracy of different disruption positions.
3.1.3 ATTENTION-BASE BBSEDD TREE BRANCHING
Based on the analysis in Section 3.1.1 and 3.1.2, have identified that steps with high FCI scores are related to reasoning behaviors and have strong influences on the the reasoning performance. Now propose Attention-based Tree Branching (ATB), which builds the branches of the tree steps with high FCI scores.
Specifically, we compute the FCI score for each step using equation after initial sampling to enable effective exploration. We then select the top 20% of the steps with the highest FCI scores for branching:
C= {A: k Quanile(pt.....w... p)),
where p 0.2 is the quantile level. However, randomly selecting steps with high FCI scores as branching points can be suboptimal, as misleading initial steps may lead the reasoning process in incorrect directions and we have found that earlier steps have more influence on the final result. Similar phenomenons have also been found in Wen et al. (2025), which identifies these as "Tunnel Vision". To mitigate this, we select the top N (N following Hou al. (2025)) earliest steps from C' as branching points, ensuring that diverse reasoning paths are explored through attention-based branching.
3.2 ADAPTIVE SAMPLING
3.2.1 DUFFICULTY-AWARR EXPLORATION
Attention-based Filtering. Previous PSRL approaches explore all problems uniformly (Hou et al., 2025), which is highly inefficient. In particular, problems that easy (i.e., achieving accuracy of 100% at initial sampling) have a high probability (about 70% 80%, shown Figure 7(a)) of being correct both sampling stages, leading limited learning opportunities.
To address this, propose attention-based filtering method to identify problems that are too easy to sample an incorrect response. We compute the average massive attention values for all problems in the DeepScaleR (Luo al., 2025) dataset. As shown in Figure 4, empirically find that, for problems whose initial samples are all correct, they have lower attention values, they will tend to have zero advantage values, indicating that all samples correct. Therefore, filter out problems with low attention values and only retain those with attention values above the average attention values:
DMC = {g
Wi.k 2 mean value),
(8)
T
i=1
where Vi is the attention score for the k-tth step in problem i.

---

Published as a conference paper at ICLR 2026
Invalid (Iritial)
14
Invalid (Iritial + MC)
Invalid (Iritial) Valid (MG)
Value
10
Densiyy
0.00
0.05
0.10
0.15
0.20
FCI
Figure 4: Average FCI scores all problems during the training process of TreeRL on DeepScaleR dataset. "Invalid" means that the advantage is zero for all responses of that problem.
Difficulty-aware Expansion. After irrention-based filtering, we expand different number of trees according to problem difficulty since it is more difficult to rollout correct responses for hard problems. Let the difficulty score be žn à Eg ((; is correct). Then the number of trees expanded for each problem M is determined by the difficulty score:
M = Round(exp(-2z) x M'),
(9)
where (-und(.) denotes rounding to the nearest integer and M' denotes original tree numbers and is set to 6 following Hou al. (2025).
3.2.2 ADAPTIVE BATCH SAMPLING
After initial sampling and MC sampling, large proportion of responses contribute nothing training because their advantages are zero (detailed in Figure 7(b)). To ensure that each training batch remains effective, we introduce an adaptive batch size mechanism.
Let the target training batch size be B', valid training batch size step be B" and the sampled prompt batch size at step m be Bm- The sampling batch size at step m is updated as:
B'
Bm+1 Round(AB, + (1 - A)
Bm);
(10)
B"
where À is the weight balancing historical and current batch sizes. After MC sampling, responses with zero advantages are discarded, ensuring that all samples in the final batch have non-zer zerz advantages, which improves training efficiency.
Our adaptive batch sampling differs from the dynamic sampling used in DAPO (Yu et al., 2025) in two key ways: (1) It requires only a single round of prompt sampling and generation per training step. (2) It avoids inefficiency from discarding valid responses when their number exceeds B', As result, the actual batch size naturally fluctuates around the target D' while maintaining high training efficiency.
3.3 EFFICIENT TRAINING WITH ONE-STEP OFF-POLICY
Prior srrcess-superviser RL methods typically require two sampling procedures per training iter- ation (Hou et al., 2025; Yang et al., 2025b; Guo et al., 2025b; Zheng et al., 2025b). This is highly inefficient, as sampling often dominates the overall training time. To address this, we propose one-step off- off-policy learning framework for PSRL, inspired by recent advances in efficient RL training (Noukhovitch et al., 2025; Fu et al., 2025; Meituan-search, 2025).
In our approach, only a single sampling operation performed at each training step. Concretely, at training step m, conduct initial sampling for the (m++1)tth problem batch while simultaneously performing MC sampling for the m-th problem batch. This design ensures that the initial sampling for batch occurs at step m-1, followed by its MC sampling at step m, thereby eliminating redundant sampling. As a result, the overall sampling cost is substantially reduced, leading to improved training efficiency. The full training pipeline of AtmRL is illustrated in Figure 5.

---

Published as a conference paper at ICLR 2026
One-Step Off-Policy Training (Ours)
Previous Method
Initial Sampling
2nd
m+1-th
1st
2nd
m-hh batch
MC Sampling
1st batch
m-th batchh
1st batch
2nd batch
m-th batch
Training
beitch
..4
beich
batch
battch
beichh
Training Process
Training Process
Prompt
Generation Process
Training Batch (Initial samples MC Samples of m-th Step)
Figure 5: Training pipeline of AttnRL. Our method (left) only needs one-time generation per training iteration, while previous methods (right) require sample twice and are inefficient.
4 EXPERIMENTS
4.1 SETUP
Models and Baselines. Following Hou et al. (2025), we adopt two supervised fine-tuned models, which also reasoning models, as base models: DS-RI DS-RI-Diise --istill-Qwen-I1 1.5B and DS-RI DS-RR-Disstin Qwen-7B (Guo et al., 2025a). We compare against the following baselines: (1) GRPO (Shao et al., 2024): A representative OSRL method without critic model training. (2) TreeRL (Hou et al., 2025): The method is based on GRPO but differs that TreeRI. samples with tree- rree-based branching and estimates advantage values at segment-level. (3) --reScileRR-reviw-----e (Luo et al., 2025): A strong RL-trained model with iterative context expansion at 1.5B scale.
Evaluation and Metrics. We evaluate all methods on six widely used mathematical reasoning benchmarks: AIME24 (MAA, 2024), AIME25 (MAA, 2025), AMC23 (MAA, 2023), MATH- 500 (Lightman et al., 2024), Minerva Math (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024). We report both Pass and Pass (K, where and K = 4 for the remaining benchmarks. Evaluation is performed with maximum response length of 32,768 tokens. For verification, we use hybrid of DeepScaleR's verifier and Math- Math-Verify" to ensure correctness (He et al., 2025a).
32 for AIME24, AIME25, and AMC23,
Implementation Details. We train all methods using DeepScaleR-Prees Pweeee----eee---eeeeee Preview-Dataser following Luo al. (2025); Liu al. (2026), which contains 40.3k mathematical reasoning problems. We set the training batch size to 64, the PPO minibatch size to 32, and the learning rate to 1 x 10-6, For all methods, we adopt teven-level policy loss and apply Clip-Higher with Ehigh 0.28, following Yu et al. (2025). We use KL loss with weight 0.001 following Liu et al. (2025a); Wang et al. (2025). AttnRI is implemented based on TreeRI. (Hou et al., 2025) and GRPO is used for policy optimization. We set X = 0.9 (a standard EMA value (Kingma, 2014)) and 0.2.
The training is conducted using verl (Sheng et al., 2025), and rollouts are generated using vLLM (Kwon al., 2023) with a maximum response length of 8.192 tokens, top-p of 1.0, and temperature of 1.0 for both DS-Ri--Dssill-Qwen--133 Q.5 and DS-RI-Distill Diittill-Qeen-B... Experiments for DS-RI-Distill- Qwen- 1.5B are conducted on single node with 8x NVIDIA H100 GPUs, and experiments for DS-RI R-R---itilll-uPe Qwee--eeeennneeeee 7B are run on three nodes, each with 8x NVIDIA H800 GPUs.
4.2 MAIN RESULTS
AttnRL outperforms the base model. As shown in Table 1, AttRR outperforms the base model across all six benchmarks, achieving an average improvement of 7.5% for -e----Dissill--e 1.5B....B.
7

---

Published as a conference paper at ICLR 2026
Table I: Evaluation results mathematical benchmarks, The results of AttnRL are shaded and the highest values are bolded.
Method
AIME24
AIME25
AMC23
MATH-500
Minerva
Olympiad
Avg.
DS-RR-Distill-Qwen--na
23.0
84.8
35.6
49.7
GRPO
27.2
88.4
39.5
60.4
55.0
CaaScaleR-Preview..
28.3
81.0
89.5
56.5
TreeRI.
36.7
27.1
78.9
88.5
38.7
60.9
55.1
AttnRI.
39.7
28.5
83.2
90.0
40.3
61.4
57.2
DS-R ---R--Disill--weeeeee
54.0
40.0
89.8
94.1
48.1
70.0
66.0
4 GRPO
54.9
39.6
90.8
94.3
48.6
69.7
66.3
4 TreeRL
55.4
40.0
92.2
94.3
49.0
70.7
66.9
4 AttnRI.
59.3
42.5
92.5
95.4
49.3
73.3
68.7
AttnRL surpasses the base model significantly AIME24 benchmark, achieving improvement of 11.4% and 5.3% for 1.5B and 7B models, respectively".
AttnRL outperforms PSRL and strong RLVR baselines. As reported Table I, AttnRL sur- passes GRPO and TreeRL by average of 1.9% and 1.8% across all benchmarks at 1.5B scale, confirming its effectiveness. Moreover, AttnRL outperforms DeepScaleR- Prreeeew---- which is trained with three-stage context extension (8K - 16K - 24K) over 1750 steps. In contrast, AttnRL achieves superior results with only 500 steps at an 8K response length, highlighting both its effectiveness and efficiency.
4.3 ABLATION STUDY
To evaluate the contribution of each component, conduct an ablation study on the six mathematical benchmarks using DS-R ----isiilll-weer 1.5B. As shown in Table 2, incorporating ATB alone improves performance over TreeRL by an average of 1.2%. while combining ATB with adaptive sampling allows AttnRL to achieve the highest performance. Importantly, filtering out problems whose responses are correct after initial sampling results in a slight performance drop, as even "easy" problems can produce incorrect responses under Monte Carlo sampling, providing valuable training signals that enhance overall model performance.
Table Results of ablation study on mathematical benchmarks. The results of AttnRL are shaded and the highest values are bolded.
Method
AIME24
AIME25
AMC23
MATH-500
Minerva
Olympiad
Avg.
TreeRL
36.7
27.1
78.9
88.5
38.7
60.9
55.1
wifiist step branching
35.6
28.9
79.5
89.4
38.7
60.5
55.4
4 w/TT
39.
27.2
81.4
89.2
40.1
61.0
56.3
w/ATB ADS (wlo intention-hssed filtering)
38.4
29.1
81.0
89.8
38.7
61.2
56.4
4 wATB ADS (woo difficult-awware expansion)
39.6
28.
82.0
90.3
39.6
61.0
56.8
5
61.4
57.2
5 ANALYSIS
5.1 SAMPLING
How does ATB outperform entropy-based tree branching? The results in Table 2 show that TreeRL w/ATB outperforms TreeRL, which branches tokens with highest entropy values. To further understand the effects of ATB, we plot four sampling curves during training process in Figure 6. For Figure 6(a) and (b), we visualize the solve all ratio (i.e., the ratio of problems whose outputs are all correct) and solve none ratio (i.e., the ratio of problems whose outputs are all wrong) of MC sampling, respectively. These two subfigures demonstrate that ATB enables more effective sampling both easy and hard problems. Figure 6(C) and (d) show the valid ratio (i.e., the ratio of problems whose outputs are either not correct nor all wrong) of MC sampling and both sampling, respectively. The results also demonstrate the effectiveness of ATB.
"The performance gain of 7B model is smaller than that of 1.5B model, which may because DeepScaleR- Preview-Datanl Dataset is originally used fine-tune -e--istill--wen- and is relatively easy for DS-R1- Distill-Qwen-7B, which is stronger and has less for improvement.
8

---

Published as a conference paper at ICLR 2026
w/ATB
0.250
0.75
0.225
0.70
0.70
Ratio
0.25
0.200
Ratio
0.65
0.63
0.20
0.175
0.60
0.40
0.150
0.35
0.55
Steps
Steps
Stops
Ratio (MC)
None Ratio
Prompt Ratio
Valid Ratio (initial
Figure 6: The sampling statistics of ATB and entropy-based branching. The curves smoothed using EMA for better visualization.
Easy filttred
Method
0.6
175
150
0.4
0.8
325
100
0.0
200
4.00
400
Steps
Steps
Stops
Stops
(8) Invalid Once Twice Valid Ratio
(b) Valid Token Ratio Per Batch
(c) Prompt Filtered Ratio
(d) Prompt Batch Size
Figure 7: Curves related to sampling information statistics of all methods. The curves are smoothed using EMA for better visualization.
GRPO
TrecRL
Our Methad
0.90
0.54
5400
0.55
0.52
5000
0.48
0.80
4800
0.75
0.40
0.44
4600
0
2.00
400
600
200
400
800
200
40.0
600
2000
400
600
(a) Entropy
Reward
(c) Average Test Accuracy
Average Test Response Length
Figure 8: The training dynamics curves of all methods. The curves are smoothed using EMA for better visualization.
Adaptive Sampling. To better understand the effects of our proposed adaptive sampling method, visualize the training curves related the sampling process. The results in Figure 7(a) show that our method significantly reduces the ratio of both samples of two sampling steps are correct given the initial sampling results are correct, by filtering out prompts with low FCI scores (shown in Figure 7(c)). Additionally, AttnRL benefits from maintaining a valid training batch by dynamically adjust the prompt batch size (shown in Figure 7(d)), resulting training batch with all tokens having zor-zero advantage values (shown in Figure 7(b)).
5.2 TRAINING DYNAMICS AND EFFICIENCY
Training Dynamics. The training dynamics of GRPO, TreeRI, and AttnRL are visualized in Figure 8. Figure 8(a) shows that the entropy curve of GRPO decreases along the training process, while PSRL methods first decreases then increases. Compared with TreeRL, AttnRL shows higher entropy, enabling more diverse exploration during training. Figure 8(b)-(c) show AtmRL learns faster with less training steps and Figure 8(d) shows the response length of AtmRL is shorter than that of TreeRL, demonstrating that AttnRL outperforms TreeRL at both final performance and reasoning conciseness.

---

Published as a conference paper at ICLR 2026
Training Efficiency. As shown in Table 3, the training efficiency of the introduced one-step off- policy reduces the training time by 8% compared with original TreeRL implementation. AttmLL outperforms TreeRL with less wall-clock training time, more valid tokens for training (i.e., to- ken with non-zero advantage values), and better overall performance significantly under the same computational resources. These strong efficiency improvements our adaptive sampling mechanism, which samples dynamic batch of problems, filters out some low-value easy problems, and keeping relatively stable size of batch with all samples useful for training.
achieved through especially at
Table Comparison of training efficiency among AttnRL and baselines. The results of AttnRL are shaded and the best values are bolded.
Method
Wall-clock Time
Valid Training Tokens
Performance
GRPO
54.0
656.0M
55,0
TreeRL
67.7
274.6M
55.1
TreeRI. w/one-step off-policy
62.2
269.1M
55.3
AttaRL
62.6
930.4M
57.2
6 RELATED WORK
6.1 REINFORCEMENT LEARNING FOR LLM
Reinforcement Learning has shown great success for enhancing the reasoning abilities of LLMSs (Ope- nAI, 2024; Guo et al., 2025a). With the success of OpenAl ol (OpenAI, 2024) and DeepSeek-RI (Guo et al., 2025a), RLVR has become an efficient method for improving reasoning abilities of LLMs (Yu et al., 2025; Liu et al., 2025c; Chu et al., 2026; Yuc et al., 2025; He al., 2025a; Luo et al., 2025; Chen et al., 2025b; Liu al., 2025a; Chen et al., 2025a; An et al., 2025; Wang et al., 2025; Zheng al., 2025a). These works focus outcome-based rewards that inefficient for RL training, while our method focus on RL with process rewards.
6.2 PROCESS SUPERVISION FOR LLM
Process supervision has demonstrated superiority than outcome- based feedback in mathematical reasoning, especially Process Reward Models (PRMs) (Uesato al., 2022; Lightman al., 2024; Wang et al., 2024b). A line of works focus on token-level process rewards (Yuan et al., 2025; Cui al., 2025; Fei et al.. 2025), using DPO-like rewards (Rafailov et al., 2023; 2024) for policy learning. For PRM-based methods, line of works (Wang et al.. 2024b; Setlur et al., 2025; Cheng et al., 2025; Zha et al., 2025; Ye al., 2025) use discriminative PRMs for RL training, while another line of works use generative PRMs (Zhao al., 2026) to provide process rewards for RL training (Zou et al., 2025; He et al., 2025b; Xie et al., 2025). To mitigate reward hacking and avoid training an online PRM, some works use online Monte Carlo sampling to estimate process rewards (Kazemnejad et al., 2025; Hou et al., 2025; Guo et al., 2025b; Yang et al., 2025b; Zheng et al., 2025b; et al., 2025; Dong al., 2026). Our method belong the category which leveraging MC sampling estimate process rewards. However, previous methods mainly focus on non non-reassning models and incfficient from the perspective of both branching points, sampling mechanism, and two-step generation, while work proposes effective and efficient methods of process supervision for reasoning models.
7 CONCLUSION
In this paper, propose AttRR for PSRL in reasoning models, which leverages attention in- formation to find reasoning-related steps and branches at these positions for efficient exploration. Additionally, introduce adaptive sampling based on problem difficulty and maintaining valid training batch size. Experimental results on mathematical reasoning benchmarks demonstrate the effectiveness and efficiency of our method.
REPRODUCIBILITY STATEMENT
The implementation details of our method are discussed Section 4 and Appendix A. The link of code implementation is provided in Abstract.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the STI 2030-Major Projects under Grant 2021ZD0201404 and Shenzhen Key Laboratory of New Generation Interactive Media Technology Innovation (ZDSYS2001U02330200000))
REFERENCES
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: A post-training recipe for scaling reinforcement learning advanced reasoning models, 2025. URL https:// paattpa:/hhkklp.... githu
b.io/blog/2025 /
Accessed: 2025-09-01.
Anthropic. Introducing claude, 2023. URL oducing-claadee Accessed: 2025-09-01.
innr
Paul C Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. Thought anchors: Which I1m reasoning steps matter? arXiv preprint arXiv:2506.19) 19/43, 2025.
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-ml: Scaling test-time compute efficiently with lightning attention. arXiv preprint urXiv:2506.1330 2025a.
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shocybi, Bryan Catanzaro, and Wei Ping. decreason-nemotree learning. In Advances in Neural Information Processing Systems (NeurIPS), 2025b. URL https: h //openreeieww... net / forum? i beerum?dd-Egaaee d--gg--bn0n.......
Advancing math and code reasoning through reinforcement
Jie Cheng, Gang Xiong, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Yisheng Lv, and Fei- Yue Wang. Stop summation: Min-form credit assignment all process reward model needs for reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2025. URL net. forum? Lo
Xiangxiang Chu, Hailang Huang. Xiao Zhang, Fei Wei, and Yong Wang. GPG: A simple and strong reinforcement learning baseline for model reasoning. In International Conference on Learning Representations (ICLR), 2026. URL ht /ttttp:::/opeee hpwp//veppeeeeeeeeeeeeeeeeeeeeee nwt. www.rooooooinncooooooooooo 8x.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint vrXiv:2502.01456 2025.
Guanting Dong. Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. In International Conference on Learning Representations (ICLR), 2026. URL https: : /openre view.
Wu Fei, Hao Kong, Shuxian Liang, Yang Lin, Yibo Yang, Jing Tang. Lei Chen, and Xiansheng Hua. Self-guided process reward optimization with redefined step-wise advantage for process reinforcement learning. arXiv preprint arXiv:2507.00 arX/v:2507.0155/, 0/55/, 2025.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei. Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.24228, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in llms through reinforce- ment learning. Nature, 645(808) )):63-668, 2025a.
Yiran Guo, Lijie Xu, Jie Liu, Ye Dan, and Shuang Qiu. Segment policy optimization: Effective segment-level credit assignment in RL for large language models. In Advances in Neural Informa tion Processing Systems (NeurIPS), 2025b. URL https:///////cc...t..tt /ttp:///ppe/eieieee//m/e net/ / forum?? id= 90svTOYbT4.
11

---

Published as a conference paper at ICLR 2026
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agı with olympiad-leve bilingual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 3828-3850, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024 acl-lcng.211111 URL https: /ttpg://aacaaatitaaeae 211/.
/////alaanttt htwp//oooaattoooo.oooooooooooooa org/2024 acl-
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.22311 2025a.
Tao He, Rongchuan Mu, Lizi Liao, Yixin Cao, Ming Liu, and Bing Qin. Good learners think their thinking: Generative prm makes large reasoning model more efficient math learner. arXiv preprint arXiv:2507.23317, 2025b.
Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong. Treerl: Lim reinforcement learning with on-policy tree search. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 12355-12369, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 979-8-89176-255 // //aclanthologyy.o org/ 2025 acl- con..nnnn...... 604/-
URL https:
Jingcheng Hu, Yinmin Zhang. Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. In Advances in Neural Information Processing Systems (NeurIPS), 2025. URL https: net //tpperevieww..////////////////t net /otrrforuumi forum? www.ummiiinffai NEMMMMM5CV0SS
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du Zirui Liu, and Yongfeng Zhang. Massive values in self-attention modules are the key to contextual knowledge understanding. International Conference on Machine Learning (ICML), 2025. URL https: //openreview wtwp:eewwwww.eeeoon.ne..e..... 1ccddddds
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Refining credit assignment in RL training of LLMs. In International Conference on Machine Learning (ICML), 2025. URL https: // /ppenreview net/ www.:/ee//ounnne
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141.v:141..6980. 2014.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP '23, pp. 611-626, New York. NY, USA, 2023. Association for Computing Machinery. ISBN 0798400702297. doi: 10.1145/3600006.36133666 URL https: /ow.. 5/10.1144 1110.1145/36000000 3613116
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), in Neural Information Processing Systems (NeurIPS), volume 35, pp. 3843-3857 Curran Associates, Inc., 2022. URL https: http::/pppppppp.oceine...s////// neurips cc paper files /ttees/pppe///0/// 2/ e/file// 2ttaeeoeooeeeoerr00000000033e4e0 Porererr .df..
12

---

Published as a conference paper at ICLR 2026
Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, Tianshun Xing, Shuyue Guo, Tianyu Zheng, Xin Zhou, Xingwei Qu, Wangchunshu Zhou, Zheng Zhang, Wei Shen, Qian Liu, Chenghua Lin, Jian Yang, Ge Zhang, and Wenhao Huang. Treepo: Bridging the gap of policy optimization and efficacy and inference efficiency with heuristic tree-based modeling. arXiv preprint arXiv:2508. 1745,, 2025.
Hunter Lightman, Vineet Kosaraju. Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, 1lya Sutskever, and Karl Cobbe. Let's verify step by step. In International Conference on Learning Representations (ICLR), 2024. URL https: httpchtsss///peeeeeeeeeeeeeeeeee /ttp///peeeeeett/////
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. ProRL: Prolonged reinforcement learning expands reasoning boundaries large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2025a. URL https: //open reviee ew net. fo rum? d==YP.Jaa5x22
Runze Liu, Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. Can lb I1m surpass 405b Ilm? rethinking compute-optimal test-time scaling. arXiv preprint arXiv:2502.06703 2025b.
Wei Liu, Ruochen Zhou, Yiyun Deng, Yuzhen Huang, Junteng Liu, Yuntian Deng, Yizhe Zhang, and Junxian He. Learn to reason efficiently with adaptive length-basee reward shaping. In International Conference Learning Representations (ICLR), 2026. URL https: openreview / j 9expaxQ1.
Zichen Liu, Changyu Chen, Wenjun Li. Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: critical perspective. Conference on Language Modeling (COLM), 2025c. URL https: http:s:///ppeeeeeeeeeee net wtte:/oorumidddpPP///
Michael Luo. Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o -pevviww with a 1.5b model by scaling rl. https: http::///etttttyyrteeee
not http::/snnoiiiiinnnnineeeeeoeeee
81902c1 0000022446600000ccaa330001101100
2025. Accessed: 2025-09-01.
MAA. American mathematics contest 12 (ame 12), November 2023. URL httts: 12:00_____1000
12_ Prob_Probllssss and Solutions. com/ www.wwoio//////oooo.oo...o.weewe wwwwwiiidedd hatpeeeeppaaaMa____eeeeeeeeeeeee
Accessed: 2025-09-01.
MAA. American invitational mathematics examination (aime), February 2024. URL https: cettoprroblescolvieee 09-01.
AIME I. Accessed: 2025-
MAA. American invitational mathematics examination (aime), February 2025. URL https:
// pertoffrobbemsoll wiigg wwtp:wwwwwwwww.e.. http:////w/////eeeeep/p///22//// 09-01.
Accessed: 2025-
Meituan-search. Recipe: One step off policy async trainer. htwineerhtttpss//g//tt http://hhhuu.coooo/oooenngeeeeee
e/ s/ver1/tree//miin weriiin/eeriiii http:/pepooeeepeeeeeeeeeee fone_step_off__oliie 2025. Accessed: 2025-09-01.
Michael Noukhovitch, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous RLHF: Faster and more efficient off-policy rl for language
models. International Conference on Learning Representations (ICLR), 2025. URL https:
//openreview.oo net/ forum?id-Fh httpr//reeeee..ee.oooooooooe//an
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
OpenAl, Learning to reason with mims, 2024. URL pttestts:///peeee
g- toorresoo-wt th-1lms 1mms. Accessed: 2025-09-01.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems (NeurIPS), 36:53728-53745 2023.
13

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to q star: Your language model is secretly a q q-function. In Conference on Language Modeling (COLM), 2024. URL
https: wpw.review.eeeeeeee wwt. mett/forumldd::Eelltttttt.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Hisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for LLM reasoning. In International Conference on Learning Representations (ICLR), 2025. URL https: /ttp///ppeeeeeeeeeeeeeeeeeeeeeee net http://immmmmmmmmmmmmm 2LM.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys '25, pp. 1279-1297, New York, NY, USA, 2025, Association for Computing Machinery. ISBN 1798400711961 doi: 1.144///668933336666690 URL https://doi. org/ 100 10. 1145/ 368 9031 3696075.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-basee feedback. arXiv preprint arXiv:2211.14275. 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 30. Curran Associates, Inc., 2017. URL https://proceee ings neurips cc happppppperrffi wtt.fi/ssppppee/1/////..... 243547dee www.eeed00eeeaaeeccce Pap.aa..a.pppp............ pdf.
Jiakang Wang, Runze Liu, Fuzheng Zhang, Xiu Li, and Guorui Zhou. Stabilizing knowledge, promoting reasoning: Dual-token constraints for rlvr. arXiv preprint arXiv:2507.15778. 2025.
Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. Openr: An open source framework for advanced reasoning with large language models. arXiv preprint arXiv:24 2024a.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 9426-9439, Bangkok, Thailand, August 2024b. Association for Computational Linguisties. doi: 10. 1/2533/1/2022.cc11 510.510. URL https: /aaaacaaaantoooooooo
org 2024 acl-long long.ong..... 510/.
Hao Wen, Yifan Su, Feifei Zhang, Yunxin Liu, Yunhao Liu, Ya-Qin Zhang, and Yuanchun Li. Parathinker: Native parallel thinking new paradigm to scale Ilm test-time compute. arXiv preprint (t..ix:250990447777 2025.
Guofu Xie, Yunsheng Shi, Hongtao Tian, Ting Yao, and Xiao Zhang. Capo: Towards enhancing I1m reasoning through verifiable generative credit assignment. arXiv preprint (tti::20080029994 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men. Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
14

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.09 503888....8., 2025a.
Zhicheng Yang, Zhijiang Guo, Yinya Huang, Xiaodan Liang, Yiwei Wang, and Jing Tang. Treerpo: Tree relative policy optimization. arXiv preprint ar arXiv:2506.05 00..001883, 183, 2025b.
Chenlu Ye, Zhou Yu, Ziji Zhang, Hao Chen, Narayanan Sadagopan, Jing Huang, Tong Zhang, and Anurag Beniwal. Beyond correctness: Harmonizing process and outcome rewards through rl training. arXiv preprint urXiv:2509.03400 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Juncai Liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Ru Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Wei-Ying Ma, Ya-Qh Zhang, Lin Yan, Yonghui Wu, and Mingxuan Wang. DAPO: An open- source LLM reinforcement learning system at scale. In Advances in Neural Information Processing Systems (NeurIPS), 2025. URL https: /tpp//pppeeeeeeeeeeeee ew, net/ forum? i d=2a3 6EMSSTp.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. In International Conference on Machine Learning (ICML), 2025. URL https: http:::///ppeeeviewwe wwtp://te/ooommmmtttttttt. hGm8.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofci Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05/18, 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun MA, and Junxian He. SimpleRL.- 700: Investigating and taming zero reinforcement learning for open base models in the wild. In Conference on Language Modeling (COLM), 2025. URL : operrevieeeeeeee review. net/ for um um:2dd
Kaiwen Zha, Zhengqi Gao, Maohao Shen, Zhang- Zhei Hong, Duane S Boning, and Dina Katabi. RL tango: Reinforeing generator and verifier together for language reasoning. In Advances Neural Information Processing Systems (NeurIPS), 2025. URL / net for um7id=JRkFZI DTJ2.
Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia Pengfei Li, Yu Fu, Xingtai Lv, Yuchen Zhang, Sihang Zeng. Shang Qu, Haozhan Li, Shijie Wang, Yuru Wang, Xinwei Long, Fangfu Liu, Xiang Xu, Jiaze Ma, Xuekai Zhu, Ermo Hua, Yihao Liu, Zonglin Li, Huayu Chen, Xiaoye Qu, Yafu Li, Weize Chen, Zhenzhao Yuan, Junqi Gao, Dong Li, Zhiyuan Ma. Ganqu Cui, Zhiyuan Liu, Biqing Qi. Ning Ding, and Bowen Zhou. A survey of reinforcement learning for large reasoning models. arXiv preprint 2025.
Kaiyan Zhang, Kai Tian, Runze Liu, Sihang Zeng, Xuekai Zhu, Guoli Jia, Yuchen Fan, Xingtai Lv, Yuxin Zuo, Che Jiang, Yuru Wang, Jianyu Wang, Ermo Hua, Xinwei Long, Junqi Gao, Youbang Sun, Zhiyuan Ma, Ganqu Cui, Ning Ding, Biqing Qi, and Bowen Zhou. MARTI: A framework for multi-agent I.LM systems reinforced training and inference. In International Conference on Learning Representations (ICLR), 2026. URL https:/ /al:://oppeee //tp//pppeeeieeeeeoe wttrriiwwwwweeeeeeeeeeeeeeeeeeee wtt.e//oroiiia///eeeeeeeeee jZgo0A50.
Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, and Bowen Zhou. GenPRM: Scaling test-time compute of process reward models via generative reasoning. In AAAI Conference on Artificial Intelligence (AAAI), 2026. URL https:/ hppp/eeeeeevieeeeeeeeeee/eeeeeee ew net forum? d= 9VGbRupgiP.
net/forumm
15

---

Published as a conference paper at ICLR 2026
Chujie Zheng, Shixuan Liu, Mingze Li, Kiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.8807, 2025a.
Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, et al. First return, entropy-eliciting explore. arXiv preprint arXiv:250..07017, 2025b.
Jiaru Zou, Ling Yang. Jingwen Gu, Jiahao Qiu, Ke Shen, Jingrui He, and Mengdi Wang. Reasonflux- PRM: Trajectory-aware PRMs for long chain-of-thought reasoning in LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2025. URL https: htttt:///ppeeeeeee// /wwpn/oveeeeww..ooe/ee ew. net FFormmiddffee k0bv2..
16

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL DETAILS
Evaluation. For evaluation, we use the prompt listed in Table 4, following Luo et al. (2025).
Table 4: Prompt Template.
(problem) Let's think step by step and output the final answer within \boxed
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 FULL TEST CURVES
The test curves of six mathematical benchmarks are shown in Figure 9.
GRPO
TreaRL
Our Method
0.375
0.325
0.26
6.250
0.22
0.650
0.825
200
400
600
200
400
500
200
4.00
500
Steps
Steps
Steps
(a) AIMEZ4 (Avg@)2)
AIMEZS
AMC23 (Aug@322)
0.575
0.86
0.550
0.84
0.500
0.80
400
630
200
400
600
Steps
(Avg@4)
(d) (Avgg4)
(c) Dlympiad (Avg@4)
Figure 9: Test curves of GRPO, TreeRL, and our method on six benchmarks.
C CASES
The full response of Figure 2 is listed in Table and steps with high FCI scores shown in red.
Table 5: Full response.
Okay, so need to figure out how many positive three-digit integers are divisible by both 11 and 5. Hmm, let's see. remember that if number is divisible by both 11 and 5, it must be divisible by their least common multiple, right? But wait, should double-check that because sometimes numbers share common factors, maybe I need the least common multiple correctly.
First, let recall that 11 is prime number, and 5 is also prime number. Since they both prime and different, the least common multiple (LCM) them should just be their product. So, LCM 11 * = 55. That means we're looking for three-digit numbers divisible by 55. Alright, so the next step is to find out how many three-digit numbers are divisible by 55. Let's figure out the smallest and largest three-digit numbers that are divisible by 55. The smallest three-digit number is 100, and the largest is 999.
need to find the smallest three-digit number that's 's divisible by 55. So, me divide 100 by 55 to see if it's a whole number. 55 55, 55 2 110. So, 55 2 1 110 is the first number
17

---

Published as a conference paper at ICLR 2026
above 100 that is divisible by 55. Is that right? Let me divide 100 by 55 to find how many
times 55 goes into 100. 100 / 55 approximately 1.818. So, yes, the first multiple is 55 2 110. But 110 is the first three-digit number?
Wait, hold on. What's 55 * That would 55, which is not three-digit number. So, we have to start at 55 2. So, the first multiple is 110, which is 55 2. But do we need to check if 55 * is two-digit, which it is, so yes, we have to skip that. Now, for the upper limit, the largest three-digit number is 999. So, let's divide 999 by 55 to get how many times 55 goes into 999. So, 999 55 is equal to approximately 18. 16. So, 55 990, and 55 19 1045. Wait, 1045 is four-digit number, which is too big. So, the largest multiple within three digits is 990, which is 55 * 18.
Wait, no no-actually, 55 times 18 is 990, which is still less than 999. So, 18 okay. Wait, but 55 19 is 1045, which is more than 999, so 990 is indeed the largest three-digit multiple.
So, our sequence of three-digit numbers divisible by 55 goes from 110 up to 990, with common difference of 55 each time. So, to find the number of terms in this sequence, we can use the formula for the number of terms in an arithmetic sequence: Number of terms = (Last term First term) Common difference So plugging in the numbers:
First term, al 110
Last term, an = 990
Common difference, d=
So, (990- 110) / 55 +
Let compute that. First, calculate the difference: 990 110= 880. Then, divide by 55: 880 /50/55= 16. Then, add 1: 16 + 17.
Therefore, there are 17 three-digit numbers divisible by both and
Wait, let me verify that. Maybe list few and count.
Starting from 110,
110 (Ist term) 165 (2nd term) 220 (3rd term) 275 (4th term) 330 (5th term) 385 (6th term) 440 (7th term) 495 (8th term) 550 (9th term) 605 (10th term) 660 (1 1t term) 715 (12th term) 770 (13th term) 825 (14th term) 880 (15th term) 935 (16th term) 990 (1 (17th term)
Hmm, so that's terms. Looks correct. So, the number is 17. Alternatively, maybe using the formula with the general term:
The n-th term of the sequence given by a_n al + (n l)d. So, since the last term 990, let's set 990 110 + (n 1)*55. Subtract 10: 880 = (n 15..
Divide by 55: 880 /55 16.
So, n-1=16 =16. Therefore, n=17. Yes, same result. Seems solid.
Therefore, the number of positive three-digit integers divisible by both 11 and 5 is 17. **Final Answer*
The number of positive three-digit integers divisible by both 11 and 5 is 17 </think>
To determine how many positive three-digit integers are divisible by both I and 5, we first find the least common multiple (LCM) of 1l and 5. Since 11 and 5 are both prime, their LCM 11 x 5 = 55. Therefore, we need to find the three-digit integers divisible by 55.
The smallest faree-dight number divisible 55 is found by dividing 100 by 55, which gives approximately 1.818. The next whole number is 2, so the smallest term is 55 x 2 = 110.
The largest three-digit number 999. Dividing this by 55 gives approximately 18.16. The largest whole number is 18, so the largest term is 55 x 18 = 990.
To find the number terms in this sequence, we use the formula for the number of terms an arithmetic sequence:
Last term First term
Number of terms
+1
Common difference
18

---

Published as a conference paper at ICLR 2026
Substituting the values, we get:
990
110
880
Number of terms
+ +1
+ 16 1 17
55
55
Thus, the number of positive three-digit integers divisible by both 11 and 5 is 17
D THE USE OF LARGE LANGUAGE MODELS
In this paper, LLMs are only used to polish the draft of some paragraphs, improving the clarity of the sentences. The key ideas, figures, and experiments all from the human authors' contributions without the help of LLMs.
E ADDITIONAL DETAILS ON PROCESS- SERIEESSSSPERRISSEI RL
In this section, we provide additional preliminaries about process-supervised RL, especially TreeRL (Hou al., 2025).
Tree Constuction in TreeRL. TreeRL contains two-step sampling processes, including initial sampling and Monte Carlo sampling. For initial sampling, TreeRL. samples 6 responses for each prompt. Starting from the prompt, we now have branches at the tokens with the highest entropy for each response and sample 2 times at each branching point. After the branching, the tree has depth of 3 and 30 leaf nodes (6 responses + responses x 2 branching points 2 samples). The responses at these leaf nodes are used for process reward estimation and policy training.
with depth I and 6 leaf nodes. Then, TreeRL
19


Reviews:
[{"review_id": "FWhKbX5mTQ", "reviewer": "Reviewer_jGGo", "summary": "This paper proposes AttnRL, a novel Process-Supervised Reinforcement Learning (PSRL) framework that improves exploration efficiency for reasoning models. The authors identify critical limitations in existing PSRL methods like TreeRL: entropy-based branching ignores semantic meaning of reasoning steps, uniform sampling wastes computation on uninformative problems, and two-step sampling per iteration creates computational bottlenecks. AttnRL addresses these issues through attention-guided branching, adaptive difficulty-aware sampling, and efficient training pipelines, achieving superior performance with better sample efficiency.", "questions": "see weakness above", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Originality\n\nThe paper demonstrates strong originality by being the first to systematically leverage attention mechanisms for identifying branching points in process-supervised RL. While attention analysis and PSRL exist independently, the Forward Context Influence (FCI) metric elegantly connects attention patterns to reasoning behaviors, moving beyond naive entropy-based heuristics. The disruption experiments (Figure 1) provide compelling validation that high-FCI steps correspond to critical reasoning behaviors like planning and verification, not just predictive uncertainty. The adaptive sampling strategy that dynamically allocates compute based on problem difficulty addresses a practical inefficiency overlooked by prior uniform-sampling approaches, showing creative problem-solving beyond incremental improvements.\n\n## Quality\n\nThe technical quality is high with comprehensive evaluation across six mathematical benchmarks and two model scales (1.5B, 7B), demonstrating consistent improvements. The experimental design is sound, using appropriate baselines (GRPO, TreeRL, DeepScaleR-Preview) and reporting both performance (Pass@K) and efficiency metrics (wall-clock time, valid training tokens). Ablation studies systematically isolate component contributions (ATB: +1.2%, full method: +1.8%) and reveal nuanced insights, such as filtering all easy problems slightly hurting performance. \n\n## Clarity\n\nThe paper is well-structured with clear motivation and logical progression from problem to solution. Figure 1 effectively visualizes FCI scores with annotated reasoning steps. Mathematical formulations are precise and implementation details comprehensive, facilitating reproducibility.", "weaknesses": "## Limited Theoretical Foundation\n\nThe paper lacks rigorous theoretical justification for why attention-based branching should outperform entropy-based selection. While the FCI metric is intuitively motivated and empirically validated through disruption experiments, there is no formal analysis establishing under what conditions high attention scores correspond to high-value branching points for credit assignment. The connection between \"massive attention values\" and reasoning behaviors relies primarily on empirical correlation (Figure 1) without explaining the underlying mechanism—why do planning and verification steps exhibit high attention? What attention patterns distinguish productive reasoning from unhelpful verbosity?\n\n## Questionable Distribution Shift\n\nThe **selective branching based on FCI scores creates significant state distribution shift** that the paper completely ignores: by branching only from top 20% high-FCI states, the method oversamples certain states by 5-10× compared to the natural policy distribution $d^\\pi(s)$. This creates $d^{branching}(s) \\neq d^\\pi(s)$, which technically requires importance weight corrections $w(s) = d^\\pi(s)/d^{branching}(s)$ for unbiased policy gradients. \n\n## Inconsistent Results and Missing Statistical Analysis\n\nThe results show concerning inconsistencies without proper statistical analysis. Table 1 shows AttnRL's improvements vary wildly across benchmarks: +4.0 points on AIME24 but only +0.2 on MATH-500 for the 1.5B model, and sometimes underperforms TreeRL (e.g., AMC23: 91.9 vs 92.2 for 7B). No error bars, confidence intervals, or significance tests are provided—are these differences within noise? The paper doesn't specify number of training runs, random seeds, or variance across runs, making it impossible to assess reliability. The paper should report results with multiple seeds, provide statistical significance tests (e.g., bootstrap confidence intervals), and better control for confounds when comparing to published baselines.\n\n## Computational Cost and Scalability Concerns\n\nWhile the paper claims \"lightweight\" and improved efficiency, the actual computational overhead of AttnRL is underspecified. Computing FCI scores requires extracting and aggregating attention matrices across all layers and heads for every sequence ($\\max_{l,h}\\{\\sum_j \\alpha^{l,h}_{j,k}\\}$), which could be expensive for long sequences or large models. How much wall-clock time does FCI computation add per sample?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vAhN51m5GI", "reviewer": "Reviewer_n5ER", "summary": "The manuscript proposes AttnRL to make exploration in Reasoning Language Models (RLMs) more efficient. The authors start from an analysis that shows that the decomposition of reasoning steps, which they define on a paragraph level, correlates with high attention scores. By using a new metric called FCI score, which reflects how much a given reasoning step influences the subsequent parts of the trajectory, they select the highest relevant reasoning steps as well as the first two reasoning steps, where additional exploration, i.e. branching should take place. Additionally the authors propose a number of techniques to improve the sampling efficiency, for example by filtering out easy problems that provide zero advantage in the Markov Decision Process to improve efficiency as well as determining the magnitude of exploration by the complexity of the problem. Another optimization is the  dynamic selection of the batch size to improve training efficiency and that the two sampling phases are combined together, so that presumably optimizations can applied to the sampling phase as a whole.", "questions": "* How is a tree constructed from just identify important steps? I believe the tree construction is never actually discussed.\n* How does Figure 3 show that the advantage is zero if it shows FCI scores?\n  * If the FCI score is defined for a given step, how is it applied to a whole problem?\n* Isn't the total token generation the limiting factor instead of the multiple sampling?\n* How different can the DeepScaleR-Preview-Dataset be seen compared to the evaluation benchmarks/datasets?\n* Are TreeRL and AttnRL trained using PPO as a \"PPO minibatch size\" is used? Previously I was under the impression that AttnRL mechanism is applied to GRPO.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* improvement of (exploration in) reasoning chains in RLMs is a timely topic\n* selection of important reasoning steps based on attention scores instead of for example on entropy seems like a novel idea\n* additionally a number of improvements are suggested to improve the sampling efficiency\n* writing is mostly clear and seems to be well motivated, where certain decisions are based on or validated with micro-benchmarks\n* evaluation:\n  * clear description of the used evaluation setting and the hyperparameters for the most part\n  * training, while being done in fewer steps, shows improvements on a selection of evaluation benchmarks", "weaknesses": "My main concern is the limited evaluation:\n* only two different and smaller models (1.5B and 7B) are trained, which are from the same model family\n* unclear whether the approach generalizes to larger models, since the improvements to the 7B model are already small\n* I found it a bit unclear which policy optimization algorithm is used for training AttnRL\n* ablation study seems a bit limited, albeit I have no concrete suggestion for additional experiments to run\n* description of the baselines could be a bit clearer\n\n4.3: Table 4.2 seems to suggest that AttnRL is implemented on top of TreeRL or is based on it, but alters certain key aspects? -> I think this should be made more clearer at the beginning of chapter 4.\n  * best results on the individual benchmarks are not bolded, despite claiming in the caption to do so, probably because AttnRL is only best in three of them \n  * performance drop for filtering easiest is not completely true -> is best in two of the benchmarks\n\nrelated work: if there is that much work with outcome-based reward modeling, I assume there is a reason for it - I found the efficiency claim not convincing in its brevity.\n\nAdditionally while the manuscript is clearly written, it also omits a lot of details/assumes prior knowledge, so it might be at times hard to understand for the reader. But maybe that is just my personal lack of knowledge.\n\nminor issues:\n* 4.1 \n  * line 310: footnote is not on the same page\n  * line 314/315: Luo et al. (2025) is cited twice in the same sentence\n* related work, 6.2: abbreviation PRM is never introduced\n* references:\n  * no \"accessed date for weblinks:\n    * [An et al. 2025]\n    * [Anthropic 2023]\n    * [Luo et al. 2025]\n    * [MAA 2023/2024/2025]\n    * [meituan search 2025]\n    * [OpenAI 2024]\n  * consider capitalizing the titles to be consistent, especially abbreviations and proper names (which is sometimes done)\n  * [Luo et al. 2025] - URL cuts into the margin (consider using the xurl package)\n  * [MAA 2024] and [MAA 2025] have the same URL", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KF5Wu6GXKg", "reviewer": "Reviewer_RRBB", "summary": "This work first conducts empirical analysis on how high attention score reasoning steps are connected to the reasoning process. Then, they propose the following methods: 1, branching at massive attention values; 2, filtering out responses whose advantage values are 0; 3, a multi-process method to sample trajectories on adjacent steps.", "questions": "- In the abstract, AttnRL is claimed to resolve the “limited exploration efficiency”. How does AttnRL improve the exploration efficiency?\n- In figure 1, from my understanding, those sentences (e.g., “Let me compute that”) are steps. Why are only some words colored in red? What does the color indicate? Why are those words related to “reasoning”?\n- In line 252, What do you mean “more difficult to rollout”?\n- In line 272, what is $B$ and how is it involved in the filtering? \n- In experiment section, why do different training have different steps? If so, is the comparison fair?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1, I appreciate their in-depth empirical analysis into the effects of attention score/step position (section 3.1.1 and 3.1.2).\n\n2, In section 4, the proposed method (AttnRL) can significantly outperform three baselines: the base model, GRPO, and TreeRL.", "weaknesses": "1, Some presentations in this paper could be improved, making some of its point unclear to me. I will list my questions in the Questions section.\n\n2, Although the work does some empirical analysis on token attention scores, it is still unclear to me why the branching points should be selected according to the attention scores.\n\n3, Some of the designs are claimed without explanation:\n- equation 3\n- equation 7 (btw, what is the definition of $C$; how is it used?)\n- equation 9\n- The proposed method does not significantly outperform DeepScaleR-Preview-1.5B\n\n4, in Line 240 to 241, the claim that zero advantage indicates all samples are correct is wrong. If all samples are wrong, their advantages are also 0.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7dvV2IqUSG", "reviewer": "Reviewer_iknd", "summary": "The paper proposes AttnRL for process-supervised reasoning RL. It uses attention-guided branching to pick a small set of step-level decision points to expand; a one-step off-policy pipeline is used to improve training efficiency; plus difficulty-aware adaptive sampling so harder problems receive more rollouts. On several math reasoning benchmarks (1.5B/7B models), AttnRL shows better accuracy and faster learning curves.", "questions": "Q1. What is the relationship between FCI scores and step position? For example, do they exhibit a certain correlation?\n\nQ2. Do you expand starting at the high-FCI step itself (i.e., keep that step fixed and sample continuations from it), or do you expand from its parent step (i.e., resample the action at that step as well)? Please clarify which you use and why. If possible, please also explain your intuition: you mention that steps with high FCI scores are related to reasoning behaviors such as planning and self-verification—why does branching in this way help?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation and insightful analysis: Figure 2’s analysis (e.g., how perturbing high-FCI steps impacts performance) is interesting and provides useful intuition for why attention-guided branching should help.\n\n2. Method completeness: attention-guided step selection + adaptive sampling + off-policy pipeline form a coherent system, the one-step pipeline reduces the training-efficiency issues brought about by TreeRL.\n\n3. Empirical evidence: the authors report consistent gains over GRPO/TreeRL across multiple math benchmarks (1.5B/7B).", "weaknesses": "1. The main claim is using attention scores to guide process RL, but current experiments may not sufficiently isolate this contribution. The final selection combines the top 20% FCI steps plus the earliest steps—are there ablations separating these two factors? Would a simple baseline that branches only on the first 2 steps close the gap?\n\n2.  The paper reports a w/ ATB ablation with 1.2 point gains in average (in Table 2), does TreeRL and w/ATB in Table 2 use the same rollout budget? Is this gain large enough to support the claim? Similar ablations are missing on DS-R1-Distill-Qwen-7B—are the conclusions consistent there?\n\n2. Complexity vs. gain: Although simpler than TreeRL, training complexity still increases compared to plain GRPO. The gains at 7B are smaller than at 1.5B; considering cost vs. benefit, the overall value is questionable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)