Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards
Authors: Shangyu Xing
Siyuan Wang
Chenyuan Yang
Xinyu Dai
Xiang Ren
Venue: ICLR 2026 Poster (2026)
Keywords: RLVR
GRPO
rollout
LLM
reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR), particularly with algorithms like Group Relative Policy Optimization (GRPO), has proven highly effective in enhancing the reasoning capabilities of large language models. However, a critical bottleneck in current pipelines lies in the limited diversity of sampled trajectories during group rollouts. Homogeneous trajectories and their associated rewards would diminish the return signals for policy updates, thereby hindering effective policy learning. This lack of diversity stems primarily from token-level stochastic sampling, where local variations are likely to collapse into near-identical reasoning paths. To address this limitation, we propose Lookahead Tree-Based Rollouts (LATR), a novel rollout strategy designed to explicitly promotes trajectory-level diversity by enforcing branching into different candidate tokens likely to yield distinct continuations. Specifically, LATR iteratively operates in three stages: (1) branching at high-uncertainty generation steps, (2) performing lookahead simulation for each new branch, and (3) pruning branches that exhibits prolonged similarity during simulation. Compared with Stochastic Sampling, LATR accelerates policy learning by 131% on average and improves final pass@1 performance by 4.2% on both GRPO and Dynamic sAmpling Policy Optimization (DAPO) algorithms across different reasoning tasks. Our code and data are available at https://github.com/starreeze/latr.

Published as a conference paper at ICLR 2026
LOOKAHEAD TREE-BASED ROLLOUTS FOR ENHANCED TRAJECTORY-LEVEI EXPLORATION IN REINFORCE- MENT LEARNING WITH VERIFIABLE REWARDS
Shangyu Xing
Siyuan Wang"*
Chenyuan Yang*
Xinyu Dai
Xiang Ren
Nanjing University
UUniversity of Southern California
Fudan University
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR), particularly with al- gorithms like Group Relative Policy Optimization (GRPO), has proven highly effective in enhancing the reasoning capabilities of large language models. How- ever, critical bottleneck in current pipelines lies in the limited diversity of sampled trajectories during group rollouts. Homogeneous trajectories and their associated rewards would diminish the return signals for policy updates, thereby hindering effective policy learning. This lack of diversity stems primarily from token-level stochastic sampling, where local variations are likely to collapse into near-identical reasoning paths. To address this limitation, we propose Lookahead Tree- Based Rollouts (LATR), novel rollout strategy designed to explicitly promotes trajectory- level diversity by enforcing branching into different candidate tokens likely yield distinct continuations. Specifically, LATR iteratively operates in three stages: (1) branching at high-uncertainty generation steps, (2) performing lookahead sim- ulation for each new branch, and (3) pruming branches that exhibits prolonged similarity during simulation. Compared with Stochastic Sampling. LATR ac- celerates policy learning by 131% and improves final pass performance by 4.2% on both GRPO and Dynamic sAmpling Policy Optimization (DAPO) gorithms across different reasoning tasks. Our code and data available
https: htt:::/ggtttu.... com wtw.m//tttreeeeeeememmeeee
First, moybe addition and subtraction 60 35 is 31 carswer 51 </onswar>
+5.7 Final Score
Siillarty
wemesy Vollcation Score
Let's consider additiion and subtraction first. 60 51 Then 35 is 31 <onewer> 60 35 St 4 +/onswerr
20
Simulation
x2 Training Speed
First, maybe addition
Thes
15
15
Prusing
10
DAPO Stochastic DAPO w LATA
Enhanced Diversity
First, moybe addition subtraction, 60 Maybe multiplying.. ... corower- (60 35) 4 51 </onswer>
100
400
Steg
Samsting branching, simulztionn
Gencrated
higher
dynomis:
Figure 1: Comparison of conventional token-level stochastic sampling and our proposed method LATR on sampling process, rollout sequence diversity, and performance on DAPO Math dataset.
INTRODUCTION
Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models recently (DeepSeek-Al al. 2025 Yang et al. 2025: OpenAl 2025). By leveraging sequence rollouts and updating policies according
"Correspondence sw_641 @usc. edu and aiggrenuuucc.ee

---

Published as a conference paper at ICLR 2026
to appropriate rewards, RLVR can significantly improve performance across diverse reasoning tasks, including mathematical problem solving, code generation, and multi-step logical deduction (Pan et al. 2025). Algorithms such as Group Relative Policy Optimization (GRPO) (Shao et al. 2024) have become central to this approach, enabling stable model training through in-group trajectory comparisons to learn from high-quality responses while penalizing low-rewarded ones.
A key challenge in these methods lies in the limited diversity of trajectories sampled during the rollout phase (Wang et al. 2025) Zhu et al. 2025). When trajectories within a group exhibit high similarity, the estimated relative advantage and learning signals tend to diminish. As a result, the policy updates become less informative, ultimately hindering the effective scaling. Recent efforts have sought to mitigate this issue through various approaches, including increasing sampling temperature (Liu et al.) 2025) and dynamically filtering out groups with highly similar samples (Yu al. 2025). However, the former focuses on token-level variation without ensuring trajectory-level divergence, while the latter relies on post hoc filtering that provides only limited over-generation. Both methods therefore yield only modest improvements in rollout diversity under constrained generation budget.
diversity at the cost of excessive
We argue that such diversity limitation stems from the predominate reliance token-level stochastic sampling strategies, where each sequence in group is generated independently by sampling tokens from the model's output distribution at each decoding step. While simple and widely adopted, this approach ignores the contrast among sequences within the group and fails to enforce distinction or complementarity at the trajectory level, thus exhibiting inherently myopic limitation. Specifically, token-level variations typically occur without lookahead ability, making local deviations (e.g., substi- tuting "compute" with "calculate") easily collapse back into nearly identical reasoning paths, leading to redundant exploration and diminishing returns.
To address these limitations, we propose Lookahead Tree-Based Rollout (LATR), a strategy designed to explicitly promote trajectory-level diversity within a group by maintaining rollouts in tree structure. At token positions with high model uncertainty, LATR enforces branching into different candidate tokens that are highly likely to yield distinct continuations. To guarantee that each selected candidate token can lead different reasoning path, LATR performs lookahead simulation by continuing generation for fixed length, and removes those candidates failing to diverge from others. This branching, simulation and pruning procedure is iteratively repeated until the target number of rollouts is reached, after which all surviving partial sequences continue to be extended in parallel under standard stochastic sampling. This ensures that the generated trajectories are reasonably distinct from each other, thereby enriching the in-group rollout diversity.
We apply LATR strategy to both GRPO and DAPO algorithms and evaluate across 5 datasets involving mathematical and logical reasoning. Our experiments demonstrate that LATR consistently accelerates policy learning by an average of 131% while simultaneously improving final task performance of pass @ I by averagely 4.2% across different tasks. Our contributions are summarized as follows:
1) We introduce a novel tree-based rollout algorithm LATR that explicitly optimizes for trajectory- level diversity, which can be integrated seamlessly into any policy update algorithms.
2) We provide extensive empirical validation across tasks and training configurations, demonstrating consistent improvements over existing sampling strategies in RLVR pipelines.
2 PRELIMINARY
We adopt Group Relative Policy Optimization (GRPO) (Shao et al., 2024) as the foundational RL. algorithm for policy refinement. Unlike Proximal Policy Optimization (PPO) (Schulman al., 2017). which relies on a learned value function to estimate advantages, GRPO eschews explicit value modeling and instead computes advantages directly from group-relative rewards. This design simplifies training dynamics and enhances stability in reward-sparse or high-variance environments.
Each training iteration in GRPO consists of two phases: (1) rollout, where multiple candidate responses are sampled per prompt, and (2) policy update, where the policy is optimized using group-normalized advantages and clipped surrogate objective.
Rollout, Given a prompt drawn from the dataset D. a group of k candidate sequences {st]/_ are generated via autoregressive sampling from policy ny. Each sequence is constructed token-by-token
2

---

Published as a conference paper at ICLR 2026
Prompt: Using the numbers (51, 4, 60, 35). create on expression that equals 49,
Answer: 60+35-51-4
You can basic arithmetic operations (+. ") and each number only be used once.
Reward: 0.0
35 is 31, 9 31 40. Not enough
Answer: (60-35)44-51
Reward: 1,0
<thinlo First, maybe con try addition and subtraction, 60
Let's try multiplying. 4 100. Then subtract 51
35 25, Then 25 26. Then 26 4 is 30. enough.
Answer: [INWALID]
I high similarity
Reward: 0,0
oken probel iy inbiiin
24
Moybe multiplying.
Answer: Veft(660
25 * 4. 100. Then subtract 51
351right) Atimes 4 -51 Reward:
"Mayee" "Haee"
"Wat"
Branching high probability tokens
Simulating & Pruning branches with high similarities
Diverse answers & CoT
Figure 2:
An overview of LATR. A dynamic search tree built by branching on model uncertainty,
simulating and pruning similar branches, resulting in diverse answers and reasoning paths.
through stochastic sampling from the model's predicted next-token distribution. This process is identical to inference-time generation.
Formally, let St denote the multiset of partial sequences of length 1 generated for prompt p. The rollout process is recursively defined as:
So (c,6,...,6) S
U {s ② 1 1~ no(: p p))))
(1)
sES:
where the empty sequence, @ denotes token concatenation, and sampling terminates when all sequences reach an end-of-sequence token or maximum length n. The final output is the group
= (.,,..., This group-based sampling enables direct comparison of responses under the same prompt, forming the basis for relative advantage estimation.
Policy Update. Policy update aims to refine policy и by maximizing expected cumulative rewards. Similar to PPO, GRPO adopts clipped objective, together with directly imposed KL penalty term:
JGRPO (0) BpDDD, ~e
leel
min (r.,(0)Arr, (,, BDDL ( (Toll (mrollaae))
(2)
where the advantage is calculated by normalizing the group-level rewards {Ri and the ratio compares the likelihood of token 8; under the current and old policies:
Rg - mean - (R.
To(3i. P. St,c))
Å. std (
Te (8i.1
Py 81,<1)
Variations of GRPO. Building upon this, DAPO et. 2025) improves GRPO in several aspects. In rollout stage, DAPO oversamples data batches and filters out groups with identical rewards. If the retained groups are insufficient to fill batch, additional rollouts are iteratively sampled. This mechanism trades computational efficiency for higher response diversity and more informative gradients. In policy update stage, DAPO addresses GRPO's limitations in long-form generation tasks by implementing token-level loss calculation to mitigate length bias, and employs decoupled clipping without RL penalty to encourage exploration. Formally, the objective
(P) Bp (pp)
(s.)
lea
min (0),())A,,, clip (r), 1 Elows 1 Enigh) A.t)
(4)
Σ
|ss|
1m1
These enhancements make DAPO robust and effective algorithm for complex reasoning tasks.
3

---

Published as a conference paper at ICLR 2026
3 LOOKAHEAD TREE-BASE ROLLOUT
To address the limited diversity of conven- tional token-level sampling during the roll- out phase, we introduce Lookahead Tree- Based Rollout (LATR), structured explo- ration strategy inspired by Monte Carlo Tree Search (Silver et al. 2016). LATR achieves diverse trajectory generation by enforcing branching at candidate tokens that are highly likely yield distinet continuations.
Algorithm Lookahead Tree-Based Rollouts
Require:
Policy model rollout number k prompt absolute branching threshold Tabs relative threshold Tiel, pruning threshold Ted. lookahead step r, max length n.
Ensure: Set rollouts S {s1...., sk).
1: Initialize <- {e} > Single root branch
for 1 to n
3: Smext t- 0
Specifically, LATR operates through three it-
erative stages: (1) Branching, which creates
4:
5:
(------ Branching logic--.----
new trajectories at token positions with high
6:
for branch 8; E $ do
model uncertainty; (2) Lookahead Simula-
7:
P: + TT- @ s;)
Prob distribution
tion, where the new branch is extended for tokens; and (3) Pruming, where simulated sequences that fail diverge from others are removed. This process repeats until the target number of roll- outs is reached, ensuring their diversity. We provide the complete algorithm in Algorithm T and an illustration in Figure 2
8:
t argmax, P.[cl
> Top candidate
fixed lookahead window
9:
gestend + 8; $ Ci
Extend main
10:
Snext f Snext U gattend }
11:
C. + fo c I P
Tabs and Trl
P.let] - Pi
12:
for c E C, do
13:
if Snext < k then
14:
Snew f 8 +
New branch
Snew new.parent + St
3.1 BRANCHING
16:
Snew nnw.birth +
17:
S SmExt Saext U {Spew)
LATR begins with a root node corresponding
18:
end if
the input prompt. At each generation stepl. I every active branch is extended by its highest-
19:
end for
20:
end for
probability token to ensure progress along the re-
21:
most likely trajectory. These branches
22:
-..----Prunnng logic -...-- for sj E Snext with sj. birth = 1 do
graded as parent branches. Simultaneously. other candidate tokens satisfy both the abso-
23:
24:
if EditDist &j; 8j-parent) < Ted then
lute probability threshold Tabs and the relative probability threshold Teel, child branches
25:
Remove with its descendants end
are instantiated. This dual-threshold mecha-
27:
end for
nism targets reasoning crossroads where the model is genuinely uncertain between seman-
28:
29:
S 5 Smext
tically distinct continuations, while prevent- ing the branches diverging too far from the policy distribution. Branching allows LATR to maintain multiple distinct reasoning paths in parallel, significantly increasing the probability of discovering high- diverse solutions.
end for
31: return Pad S to exactly k sequences
Formally, let St denote the set of active branches at step I, and for each branch St, let P. denote its next-token distribution, the most likely token c = arg max, P.[c]. and C, is the set of all remaining candidates excluding et. A new child branch e created if:
P.íc > Tabs and
Palct] - P&[c] <
Trel-
(5)
The set of branches after expansion is the union of the parent branches with their new children:
s = U U (secl C., conditions of (5) hold, |S|| < k.))
(6)
SE
the rollout budget is reached, candidate branches are prioritized by descending probability P.[c]. This ensures that more plausible alternatives more likely to be explored.
4

---

Published as a conference paper at ICLR 2026
3.2 SIMULATION & PRUNING
While the above branching strategy effectively enables structured parallel exploration, it faces two challenges: (1) unconstrained branching leads exponential growth, quickly exhausting the rollout budget and limiting exploration sequentially; (2) branches started from token-level variations easily collapse back into nearly identical reasoning paths, struggling to ensure trajectory-Ievel diversity.
To address these issues, LATR incorporates a lookahead simulation and pruning phase. After branching, each new trajectory continues generation for fixed lookahead window of tokens. These continuations are then evaluated for divergence using normalized edit distance, and branches exhibiting insufficient divergence from their parents are pruned.
Specifically, at each step I, LATR identifies all branches created at step / - and computes the normalized edit distance over their most recent r tokens relative their parents' corresponding segment. If the distance falls below threshold Ted, the branch and all its descendants are removed: r, EditDist(s{
S Spmine
{s E S/, s.birth
parept :]) Teah:
S {s St, s..ume).
(8)
where EditDist() indicates normalized edit distance, i.c., the Levenshtein distance between token ID sequences, divided by sequence length. This ensures that only branches exhibiting meaningful divergence within the lookahead window preserved. We also explore similarity measures other than edit distance in Appendix C.2] and find that their performance are very close. Notably, LATR is backtracking-free, so the number of forward passes required by a group rollout is bounded by ((n)), where & is the rollout number (tree width) and n is the maximum completion length (tree depth).
Through lookahead simulation and pruning, LATR preserves only diverse branches that more likely to yield distinct reasoning paths. The final output consists of & surviving branches, padded if necessary meet the rollout number requirements. The entire procedure compatible with any autoregressive language model and can be integrated seamlessly into existing policy update algorithms and RLVR frameworks without modifications.
3.3 FURTHER OPTIMIZATIONS
Early Stopping. When the tree width reaches the rollout number k. LATR has already produced k sequences that are likely to lead to diverse reasoning paths. At this point, the generation process is switched to standard stochastic sampling for all remaining steps. This allows surviving branches to continue exploring the solution space stochastically while maintaining the diversity benefits from LATR. Analyses on the stopping length is in Appendix C.5
Hybrid Rollout for RL Training. While LATR excels at promoting diverse exploration during RL training, its explicit divergence objective world inference, models typically generate a single trajectory using greedy or stochastic decoding. prioritizing correctness and coherence over diversity. However, policy updates with LATR tries to maximize the reward from the LATR-generated diverse rollout group. Training exclusively with LATR throughout the entire process may thus bias the policy toward over-exploration patterns that do not generalize. To bridge this gap, we adopt hybrid sampling strategy during RL training. At each training step, we allocate fraction 7) of rollouts LATR and the remainder standard Stochastic Sampling:
create mismatch with test-time behavior. At real-
KLAIR Lnnk], hys = k KLAIR
where 1.1 denotes rounding to the nearest integer. We anneal 17 exponentially over training step
(10) with decay rate 7 < This ensures stale exploration benefits from LATR's diversity, while later stages increasingly mimic test-time behavior reduce train-test discrepancy.
" 170 -7'
4 EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
To rigorously assess LATR's 's performance in reasoning-intent intensive environments, we evaluate it on two canonical domains suited for RLVR: logical reasoning and mathematical problem solving.
5

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison of test correctness and average completion length on the Count- down dataset. 1 indicates higher is better, while & indicates lower is better. Relative improvement of LATR to Stochastic Sampling with the same policy update algorithm is marked in the parentheses, where green indicates positive improvements and red otherwise. Best results are in bold.
Correctness (%)
Average Length 1 Pass@8
Method
Pass@
P
Pass@!
Qwen2.5-3B
1.1
5.5
543
975
+ GRPO w Stochastic
65.9
73,9
473
610
DAPO w Stochastic
70.7
78.0
483
630
GRPO w LATR
70.9
77.4 (43.5)
378 (-20%)
469 (-23%)
DAPO w LATR
74.7 (+4.0)
81.5 (+3.5)
367 (-24%)
453 (-28%)
Table 2: Performance comparison on DAPO Math and AMC 2023 dataset.
DAPO-Math (val)
AMC-2025
Method
Correctness (56)
Average Length
Correctness (%)
Average Length
Pass 1
P
Pass G
Pas.
Pass
Pass@8
Pas@1
Pas
Qwen2.5-3B GRPO w Stoch.
5.6
20.1
938
2203
20.7
963
2255
24.1
51.3
880
1732
32.8
59.7
833
1622
+ DAPO w Stoch.
26.8
53,1
1024
2022
37.8
62.7
1075
2116
GRPO w LATR
28.4 (+4.3)
51.9 (+0.6)
853 (:3%)
1556 (-10%)
35.6 (+2.8)
60.3 (+0.6)
838 (+1%)
1537 (:5R)
+ DAPO w LATR
32.5 (+5.7)
54.1 (+2.8)
896 (-13%)
1880 (-7%)
45.3 (47.5)
65.0 (+2.3)
883 (-18%)
1920 (-9%)
Logical Reasoning. We adopt the Countdown dataset for both training and evaluation. Following
prior work (Pan et al., 2025). we use reward R = 0.1 Riormat + 0.9 Reorrectess- where Rsormat encourages outputs with proper form and Reorrectness assigns full reward for logically correct solutions.
Mathematical Problem Solving. Models are trained on the DAPO-Math dataset and evaluated on three additional benehmarks: MATH-500 (Hendrycks al.) 202T). AMC-2023 (MAA 2023), and Olympiad-Bere Bendhh-Behhhhee (He et al.) 2024). Consistent with Yu et al. (2025), the reward is binary: R = 1.0 for correct final answers, and otherwise.
For each test instance, we sample 8 independent completions. We report Pass@] I and @ass@8 correctness scores along with the average completion length to assess solution conciseness. All implementation details, including dataset descriptions, hyperparameters, and environment configurations, are provided in Appendix B
Evaluation Protocol.
4.2 TERMINATING PERFORMANCE
We provide a comprehensive comparison between LATR and Stochastic Sampling in Table I 2 and for Countdown and Math tasks, reporting their performance and completion length test datasets after the complete 500 steps of training. Observations are summarized as follows:
LATR delivers consistent gains in correctness across various benchmarks on both GRPO and DAPO. Across all task-policy combinations, LATR outperforms Stochastic Sampling in final Pass@1 1 scores. On the Countdown dataset, L.TRR improves accuracy by an average of 4.5% under both GRPO and DAPO. On the Math dataset, gains are averagely 3.8%. Notably, GRPO + LATR achieves comparable or even higher performance than DAPO + Stochastic Sampling despite DAPO's computationally intensive mechanisms such group filtering. Moreover, DAPO + LATR achieves state-of-the aofftheeartt art performance on both benchmarks, reinforcing that trajectory diversity during rollout is the primary driver of performance gains. This finding aligns with ablation studies in DAPO (Yu et al.) 2025). which identified rollout group filtering as the most effective component of their framework.
LATR consistently reduces inference cost while enhancing performance. Beyond accuracy, LATR significantly reduces the average length of generated reasoning trajectories at test time. On Countdown, completion length decreases by 22% under both GRPO and DAPO; on math datasets, we observe 8.3% reduction. We attribute this dual benefit to LATR's core mechanism: by encouraging exploration of diverse reasoning paths during training, it exposes the policy to a broader distribution of solutions, guiding the model to internalize efficient reasoning strategies. In contrast, Stochastic Sampling tends to traverse the reasoning space sequentially within independent trajectories due to its insufficient parallel exploration, often resulting in verbose, redundant, over-elaborated chains.

---

Published as a conference paper at ICLR 2026
Table Performance comparison on MATH-500 and Slympiad-Benche dataset.
MATH-500
Olympiad-Bench
Method
Correctness (%6
Average Length
Correciness (%) f
Average Length J
Pass #1
P
Pass 6f
Pass
Pass 6f
Pass GP
Pass
Pass
Qwen2.5-3B + GRPO w Stoch.
24.7
748
25.8
1088
2413
58.4
76.7
657
1207
27.2
47.4
1058
2014
DAPO w Stoch.
60.4
700
1283
28.1
47.0
1162
2193
GRPO w LATR
(+3.5)
77.5 (40.8)
594 (-10%)
952 (-25%)
29.5 (+2.3)
48.2 (+0.8)
954 (-10%)
1728 (-14%)
+ DAPO w LATR
62.6 (+2.2)
79.0 (-0.2)
653 (-7%)
1217 (-5%)
30.4 (+2.3)
47.8 (+0.8)
1105 (-5%)
2354 (+7%)
0.8
Average Valdation Reward
Average Validation Reward
0.30
0.7
0.25
GRPO w Stochastic
GRPD w Stschastic
0.5
DAPO Stochastic
DAPO Stochastic
GRPO w LATR
GRPO w LATR
DAPO
0.05
100
300
100
200
Step
Step
Figure 3: Learning curve comparison Countdown (left) and DAPO-Math (right) datasets.
4.3 TRAINING DYNAMICS
To further investigate the RL training process with LATR and Stochastic Sampling, we analyze training dynamics by plotting validation accuracy against training step in Figure The results reveal that LATR not only converges to a better solution, but does so considerably faster.
Under DAPO, Stochastic Sampling requires 450 steps reach peak performance the Countdown task, whereas LATR achieves the same level of accuracy by step 150, resulting in a 3x acceleration in training efficiency. On the math task, compared to step 500 for Stochastic Sampling, DAPO LATR reaches same performance at step 240, yielding 2x speedup. Crucially, the acceleration provided by LATR exceeds that gained by upgrading from GRPO DAPO, despite DAPO's heavier data requirements and computational overhead per step. This suggests that LATR's enhanced exploration of diverse trajectories is able translate into more informative policy updates per training iteration. In effect, LATR increases the sample efficiency of the RL process, enabling faster learning without architectural changes or additional data.
5 DISCUSSIONS
To evaluate the behavior and advantages of LATR under varying conditions, we conduct compre- hensive set of controlled experiments. Unless otherwise specified, all analyses in this section are performed using the DAPO algorithm on the Countdown dataset, with all other hyperparameters and architectural settings held consistent with the main experiments. Comparison with other rollout strate- gies, impact of different similarity metrics for pruning, impact of branching and pruning thresholds, analyses on efficiency, and key statistics of LATR are provided in Appendix C
5.1 DIVERSITY COMPARISON
To empirically validate that LATR promotes greater diversity among reasoning trajectories within each rollout group, we conduct a comparative analysis between LATR and the baseline method Stochastic Sampling. We evaluate three variants of the Qwen2.5-3B architecture: Qwen2.5-3B, Qwen2.5-3B-Instnul and Qwen2.5-3B trained with GRPO LATR, which we name it Qwen2.5- LATR. This progression, from raw pretrained model to an instruction-tunne variant and finally to policy-optimized model incorporating LATR, enables a nuanced assessment of how LATR influences diversity across different stages of model development. In addition to standard performance metrics (Pass and Pass Pass@8), 8), we also evaluate the average number of distinct final answers per rollout group. Two answer expressions of Countdown are considered distinct if their evaluated numerical
7

---

Published as a conference paper at ICLR 2026
outcomes differ. This ensures that diversity is measured in terms of semantic rather than syntactic variation.
Table 4: Diversity comparison between Stochastic Sam- pling and LATR.
As shown in Table 4 LATR consis- tently yields higher Pass scores and greater number of distinct answers per rollout group across all three model vari- ants compared Stochastic Sampling. These results support our claim that LATR enhances groua trajectory diversity, thereby facilitating more ef- fective policy learning through broader exploration of the solution space.
Method
@ass@
Pass@8
ff Ans.
Qwen2.5-3B Stoch.
5.8
28.9
6.3
Qwen2.5-3B LATR
6.1
30.7
6.9
Qwen2.5.3UD Benn2.5-3B-nnstruee + Stoch.
9.4
35.2
6.4
Qwen2.5-3B-Insmeee + LATR
10.9
40.6
6.9
(wen2.5-3B-LAIT Stoch.
70.9
77.4
2.6
Qwen2.5.3B B-LTT LATR
68.9
79.9
3.0
5.2 EFFECT OF DIFFERENT COMPONENTS
We dissect the contributions of LATR's core components through an ablation study. Specifically, evaluate four variants of
Table 5: Performance comparison Stochastic Sampling and variants of LATR.
LATR: (1) random branching, (2) random
Method
Pass @
Pass
pruning, (3) pruning, and (4) token-level looka-
lookahead in place of trajectory-les head. In the random variants, the branching or pruning ratio is matched to the average ratio observed in the full LATR throughout training. In the token-level lookahead variant, pruning decisions are made solely based on the next token: a branch is pruned if the next tokens across trajectories are identical. This design
Stochastic
70.7
78.0
LATR rand branch
69.6
75.8
LATR rand prune
72.5
79.2
LATR w/o prune
71.0
78.7
LATR w token-level lookahead
72.1
80.4
LATR
74.7
81.5
enables us to isolate the effects of structured branching and similarity-bar -based pruning on overall performance. Our findings summarized below.
Random branching leads to unstable training and degrades final performance. As shown in 5 LATR with random branching performs even worse than Stochastic Sampling, We observe that the KL divergence between the policy model and the base (reference) policy rises as high as 1.0 within just 50 training steps, signaling severe off-policy behavior. This instability stems from uncontrolled branching. Without the probability thresholds imposed by our method, the model may generate extremely low-probability sequences that diverge significantly from the base policy, thereby disrupting the learning process.
Both random and no pruning yield suboptimal results. The variants of LATR without pruning and with random pruning achieve only modest improvements over Stochastic Sampling, confirming that branching alone enhances exploration by diversifying rollout trajectories. However, the full LATR outperforms both. This performance gap is primarily attributable to trajectory-level redundancy, as rollout groups generated without pruning follow similar reasoning paths, reducing effective diversity and leading to inefficient policy updates. Moreover, the comparison between the no no-pruning and random -arhoonng-nnnggggg variants highlights budget exhaustion as another critical factor. Without pruning, the fixed rollout budget is quickly depleted in early generation steps, leaving insufficient capacity for exploration in later stages.
with random pruning frequently contain sequences that
Token-level lookahead underperforms trajectory-level lookahead. Although token-level looka- head outperforms both Stochastic Sampling and the no-pruning variant, it falls significantly short of the full LATR model. This deficit stems from its limited ability to capture trajectory divergence. Pruning decisions based solely the next token often inaccurate, leading the premature removal of potentially valuable branches and degrading rollout quality.
In summary, while branching provides robust mechanism for exploration, dynamic, similarity pruning serves as a crucial regulator: it ensures that the exploration budget is allocated meaningfully across the generation process and effectively mitigates redundant trajectories.
8

---

Published as a conference paper at ICLR 2026
75
70
70
60
65
Stochastic Pass@8
Stochastic Pass@g
LATR Pass@l
LATR Pass@1
Passge
LATR
k-4 k-8
k-12
k-16
t-0.8
t-1.0
t-12
t-1.5
Figure 4: Comparison of test correctness with different rollout number k and temperature t (%).
5.3
SCALABILITY WITH ROLLOUT NUMBER
The rollout budget k fundamentally constrains the scope of exploration in RLVR training. We evaluate how LATR and Stochastic Sampling scale with increasing k E {4, 12, 16). Results in Figure reveal two critical trends:
1) LATR consistently outperforms Stochastic Sampling at every value of k, demonstrating robustness to budget constraints.
2) While Stochastic Sampling performance plateaus at h: = LATR continues improve up to
k 12, indicating a higher effective capacity for leveraging additional rollouts.
This suggests that LATR not only uses its budget more efficiently but also raises the performance ceiling of the system, enabling gains from larger k values that Stochastic Sampling cannot exploit.
5.4 IMPACT OF DIFFERENT SAMPLING TEMPERATURES
In standard RL frameworks with Stochastic Sampling, the sampling temperature t governs the exploration-exploitttoo trade-off: higher increases stochasticity and thus exploration, but risks degrading rollout quality. In contrast, LATR delegates exploration primarily to branching-ae pruning mechanism, using / only modulate stochastic fallback and hybrid rollouts, which is described in Section
We evaluate performance across 1 (0.8, 1.0, 1.2, 1.5). As shown in Figure 4 both methods peak near t = 1.2, suggesting this is optimal for the base policy. Notably, LATR achieves superior performance every and exhibits lower variance across temperatures.
This robustness stems from LATR's architectural decoupling: exploration is driven by structural diversity (branching pruning), not sampling noise. Consequently, LATR less sensitive to suboptimal temperature tuning.
5.5 GENERALIZABILITY ACROSS DIFFERENT BASE MODELS
To evaluate the generalizability LATR across diverse base models, we conduct additional ex- periments using additional models from different series and scales, specifically Qwen2.5-7B and Qwen3-1.7B-B Base. As shown in T LATR consistently outperforms Stochastic Sampling across all evaluated models, demonstrating the broad applicability and robustness of our proposed method.
Table 6: Performance comparison of LATR and Stochastic Sampling different base models.
Correctness (%)
Average Length J.
Method
Pass@ 1
Pass@8 @
Pass@] I
Pass@ 8
Qwen2.5-7B Qwen2.5-77 + Stochastic
1.1
556
997
70.9
79.8
522
722
Qwen2.5-7B + LATR
76.0
82.1
396
508
Qwen3- 1.7B-Base
2.0
9.5
542
Qwen3- Bovn3-1.77B-Ban 1.77--BBse + Stochastic
66.0
75.5
521
Qwen3-1. 1an33-1.B--Baar + LATR
67.8
77.6
494
662

---

Published as a conference paper at ICLR 2026
6 RELATED WORK
6.1 REINFORCEMENT LEARNING WITH VERIFIABLE REWARDS
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as powerful alternative for tasks with verifiable results (Lambert et al. (002)). In RLVR, the reward signal is derived from an external verifier, providing an objective measure of a trajectory's s success. Within RLVR, GRPO (Shao et al. 2024) have become the state-of-the-art solution. Rather than relying on a learned value model, it compares trajectories within a sampling group and updates policy based on relative success.
Following this line of research, many seek to improve the performance of GRPO. DAPO introduces clip higher technique and removes RL constraints to enable aggressive policy updates towards correct reasoning. GSPO (Zhng et al., 2025) proposes sequence-level rewards to smooth and stabilize learning. These innovations policy update are orthogonal the rollout strategy, LATR is fully compatible with these methods. Replacing vanilla rollouts with LATR- gennrrttdd trajectories yields additive performance improvements, as we demonstrate empirically.
few works have also touched upon the rollout strategy, though typically as a secondary component. DAPO (Yu et al., 2025) proposes group filtering strategy to oversample and discard groups with identical rewards. ProRL (Liu et al., 2025) increases the sampling temperature to obtain more diverse rollout sequences. Despite these advancements, these methods only address trajectory-level in groupu diversity indirectly. Their reliance on token-level stochastic sampling is prone to generating semantically redundant reasoning paths, a limitation our work directly confronts.
More recently, two contemporaneous works integrate tree search into RLVR. TreeRL (Hou et al. 2025) and TreePO (Li et al. 2025) propagate sparse binary outcome rewards backward through the reasoning tree to derive dense process rewards that guide policy updates. TreePO additionally enhances generation efficiency by reusing shared prefixes and pruning unpromising branches early in the rollout process. While both methods leverage tree-based structures, their objectives differ fundamentally from ours, as they primarily aim to refine reward estimation or improve computational efficiency. In contrast, we adopt tree-search to explicitly foster and compare diverse reasoning trajectories within a rollout group. This trajectory-level diversity enriches the reward signal by capturing broader spectrum of potential outcomes, thereby enhancing policy learning.
6.2 LOOKAHEAD REASONING FOR LLMS
Recent work has increasingly explored lookahead-baad reasoning strategies in LLMs, with their majorly focus on inference-time approaches or offline data construction. Tree-of- Thoughts (ToT) (Yao et al. (203)) pioneered this direction by generating multiple reasoning branches at each step and selecting the most promising path using an external reward model. Subsequent methods such as MCTS-DPO (Xie et al. 2024) and ReST-MCTS (Zhang et al.] (024) extend this idea by integrating Monte Carlo tree search with lookahead estimation to decompose sparse, instance-level rewards into dense, step-level supervision signals. Quiet-STaR (Zelikman al.) 2024) also leverages a lookahead mechanism, generating token-wise rationales that anticipate future text and optimizing them based on their contribution to correct continuations.
While these works share the common ingredient of lookahcad search, their objectives differ funda- mentally from ours. Our primary goal in employing lookahead tree search is not reward propagation or step-level supervision, but rather to explicitly compare and promote trajectory-level diversity among rollouts for the same problem. This explicit focus trajectory-level diversity distinguishes our method from prior lookahead approaches in LLMs.
7 CONCLUSION
In this work, we present Lookahead Tree-Based Rollout, a novel rollout strategy that explicitly promotes trajectory-level diversity in RLVR by dynamically branching at high-uncertainty tokens and pruning non-divergent paths via lookahead simulation. By moving beyond token-level sampling heuristics, LATR enriches policy learning signals, accelerating training convergence while improving final performance by a large margin across different benchmarks. Our work demonstrates that trajectory-level rollout diversity is key scaling RLVR effectively and efficiently.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To better support reproducibility, we explain all the details to reproduce our results in Section B.3 including the parameters for our methods, training details, environment and framework versions.
ACKNOWLEDGMENTS
This research is supported in part by the Office of the Director National Intelligence (ODNI, Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022- 22072200006, the Defense Advanced Research Projects Agency with award HR001 1 12220046, and NSF IIS 2048211, We would like thank all the collaborators in USC INK research lab for their constructive feedback on the work.
REFERENCES
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li. H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni. Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. CoRR, abs/2501 12948, 2025. doi: 0044550/ARXIV.250... 12248. URL htRLh:tp:://ddt .rg. 10. (18550/aaXii. 2501 .1..0....29994 12948
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. ciation for Computational Linguistics, 2024. doi: 1L11663/N12024AAC--LOOO https://doi org. 1ttp///...11531112200000
Asso-
URL
211
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sii-Kit Yeung (eds.). Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https:
weweeett--eehnncee httpo://daatsstss-eennmmaaatat pww.eeeng.....nnniiiisi..os..ss. nwwrins.unn..... haa.:/e88aaccc007777222o00111223
html
Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong. Treerl:
LLM reinforcement learning with on-policy tree search. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 August 1, 2025, pp. 12355-12366 Association for Computational Linguistics, 2025. URL https: / antho wgw.oog/......o.o acl. 6044
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model
11

---

Published as a conference paper at ICLR 2026
serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace (eds.), Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26 26, 2023, pp. 61 626. ACM, 2023. doi: 10.1145/ 3600006.3613165 Utt.tttp::///et
o org 16.
3613165
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jianqiang Wang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang. Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu Pushing Frontiers in Open Language Model Post-Training, 2024.
Yizhi Li, Qingshui Gu, Zhoufutu Wen, Ziniu Li, Tianshun Xing. Shuyue Guo, Tianyu Zheng, Xin Zhou, Xingwei Qu, Wangchunshu Zhou, Zheng Zhang, Wei Shen, Qian Liu, Chenghua Lin, Jian Yang, Ge Zhang, and Wenhao Huang. Treepo: Bridging the gap of policy optimization and efficacy and inference efficiency with heuristic tree-based modeling, 2025, URL https /arxiv.org/abs//..... 250. 2508.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. CoRR, abs/2505 24864, 2025. doi: 1044550/RRII.2555 224664. URL /Rhhttps://dii.rgg
10. 45.46850/arkx arxivv 2505. 24864
MAA. American mathematics competitions. http:///wwggngggccc..wwwsssssswt
OpenAI. Introducing gpt-5.
2025.
URL.
Introducing-gpt-
Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://ittabb.co//liiiiaaaaai ZantinyZeeo,,, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347 2017. URL htttp://axxv..rrg/ abs/
1707.06347
Ruichen Shao, Bei Li, Gangao Liu, Yang Chen, Xiang Zhou, Jingang Wang, Xunliang Cai, and Peng Li. Earlier tokens contribute more: Learning direct preference optimization from temporal decay perspective, 2025. URL https:: / /ww.xvv.vv./..///////...///.vtva www.iiioorrraeeee wtrg/abs... 2502. 14340
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukeuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nat., 529(7587):484 2016. doi: 10.1038/NATRRE16661 URL TaLLhttps::/ htttpps:/dbi.oog/1110 htwp://aeee...///eaaeeeeeeeeeeee 16961
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Bap -annittpistt Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawezyk, Cosmo Du, Ed Chi, Heng-Tze Tze Cheng, Eric Ni. Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Ittycheriah, Mahdis Mahdieh, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan,
12

---

Published as a conference paper at ICLR 2026
Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, Hyunleong Choe, Ágoston Weisz, Tao Zhu, Yifeng Lu, and et.al. Gemini: A family of highly capable multimodal models, 2025. URL https: /ttp:/..xxii/o//////..
abs 20s/22112 2312. 11805
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for LLM reasoning. CoRR, abs/2506.01939, 2025. doi: 10.48550/ 01939
ARXIV.2506.0193. URL 9BP.:ttpp..//ooo.rrr..000..////0
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy P. Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. CoRR, abs/2405 00451, 2024. doi: 10.4850/ARXV..405..000
URL URLhhtts:://oi.oo
10. 40558855//a2 20t000/rXiv.. w4w.:xxwv..4..0000 00451
Yixuan Even Xu, Yash Savani, Fei Fang, and J. Zico Kolter. Not all rollouts are useful: Down- sampling rollouts in Ilm reinforcement learning, 2025. URL httttt://aax.....rtt.t wwg.bbi.oorg/b.......
2504. 13818
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang. Jingren Zhou, Jumyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang. Zhipeng Zhou, and Zihan Qiu. 2025. doi: 40448500/RRXIV.55555 009388. URL Qwen3 technical report. CoRR, www.s//arxvvv... 2505. 09388
https://dot /ow... www.o////....//////...//./...//
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 n niss........ cc wwtpcc/cpppppeeeeeee les /ttpe///pppp///2///aaaaeee
16, 2023, 2023. URL htt.:http////ppee
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhijin Lin, Bole Ma, Guangming Sheng, Yuyuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Zheng Zhang, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yonguan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: An Open-Source LLM Reinforcement Learning System at Scale, 2025.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet- star: Language models can teach themselves to think before speaking. CORR, abs/2403.00629, 2024. doi: 10.44550/ARXIV.240..000 2403. 09629
URL https: org 10. 48550/arXiv
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*; LLM self-training via process reward guided tree search. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL /tttt::://pappr... nips cc/paper files/ httpee//pppe/222//eeeee Abstract Conference hor.aaa-asaaaatt-ooereee....... .tml
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy
13

---

Published as a conference paper at ICLR 2026
optimization. CoRR, abs 2025. doi: 10.4850/AARXII2250.18007 URL https
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising
effectiveness negative reinforcement LLM reasoning. CoRR, abs/2506.01347, 2025. doi: 10. 48550/ARXIV.2506.01..2 URL https: http:////...ooo....ooo.///////// 2551..88888502222225555...0000 01347
A
LLM USAGE
In the course of preparing this manuscript and supporting materials, we leveraged large language models (LI.Ms) auxiliary tools enhance the efficiency and quality of non-core research tasks. Specifically, LLMs were employed in two primary capacities:
1) Language polishing: We used LLMs to assist in proofreading, grammatical correction, and stylistic refinement of the manuscript's 's prose.
2) Boilerplate and utility code generation: For ancillary implementation tasks, such file I/O wrappers, format converters, or logging utilities, we used LLMs to accelerate prototyping.
B DETAILS ON EXPERIMENT SETUP
In this section, we detail the datasets, evaluation protocols, and implementation configurations.
B. DATASETS AND TASK FORMULATIONS
Logical Reasoning. We adopt the Countdown dataset (Pan et al. 2025). which challenges models to construct arithmetic expressions from given set of integers that evaluate exactly target number. Following Pan et al. (2025), we define a two-component reward function:
R 0.1 Trrmat + 0.9 Iconects
(11)
where Tiormat indicates syntactic validity and Tearree indicates semantic correctness. Models are trained on the training split and evaluated the official test set.
Mathematical Problem Solving. For mathematical reasoning, we train on the DAPO Math dataset Yu et 2025). a curated collection of problems drawn from diverse sources. Consistent with Yu et al. (2025). ) the reward is binary:
R leoorectt
(12)
awarding 1. only for exact numerical matches.
To ensure broad generalization, we evaluate not only on DAPO Math's held-out validation set, which manually partitioned with 1,024 samples, but also on three established external benchmarks, including MATH500 (Hendrycks et al. 2021). AMC2023 (MAA 2023), and OlympiadBench (He
ct al. 2024).
To maintain consistency across datasets with heterogeneous answer formats, following Yu et al. (2025). we apply a standardized answer normalization pipeline that maps all results to integers. We construct comprehensive few-shot prompt that instructs Gemini-2.5-pro (Team et al. 2025) apply a set of deterministic heuristics according the original answer's 's format. These heuristics include: (1) for structured non-inter con-integer answers like fractions (p/q) q) radicals (k m/n), rephrasing the question to ask for the sum of their components (e.g., P q or + m + n); (2) for symbolic expressions, either summing the coefficients of simple polynomials or evaluating complex functions when assigning the variables (e.g., x = 2); and (3) for multi-part or multiple-choice answers, asking for the sum of solutions or the 0-indexed position of the correct choice. The few shot prompt applied is provided in Figure 6
B.2 EVALUATION PROTOCOL
We perform stochastic sampling on the trained model for fixed times for each sample in the evaluation datasets, and report Pass@) 1 (the average accuracy over single sampled completion per
14

---

Published as a conference paper at ICLR 2026
question) and Pass @8 (the accuracy of the best solution among independently sampled completions per question). In addition to correctness, we also include average completion length for both Pass @1 and Pass@8 to quantify test-time computational cost and efficiency.
B.3 IMPLEMENTATION DETAILS
Sampling and Rollout Parameters. We set our sampling parameters following Yu et al. 2025) )
Pan et al. (2025). During training. we sample rollouts with temperature 1.0, top-k -1, and top-p 1.0 to encourage exploration. During evaluation, we use temperature 0.6, top-k 20, and top-p 0.95 for calibrated diversity. Each training step involves ht rollouts per prompt. Maximum completion lengths are set to 1,024 tokens for Countdown and 8,192 tokens for math problems.
Algorithmic Parameters for LATR. Hybrid rollout coefficient To = 1.0, decaying per-step via N ?/u -7', with Y 0.985 (Countdown) and y 0.995 (Math). For branching thresholds, absolute probability threshold Tabe = 0.25, relative probability threshold Tiel 0.15, and edit-distance threshold Ted 0.4. Lookback step r is {20, 30, 50), which means we enforce conditions all of the 3 lookback windows, and all should be satisfied for a branch to be kept.
Training Hyperparameters. For training parameters, global data batch size 256, global mini batch size is 256, local micro batch size is 8 for Countdown and for DAPO Math, clip ratio is 0.2, KL penalty 3 0.01. For DAPO, clip ratio high 0.28, low is 0.2, and oversampled data generation batch size is 384. We train the Qwen2.5-3B base model on both datasets for fixed 500 steps with AdamW optimizer and a constant learning rate of le-6. All our experiments are performed with VeRL-005.0 framework (Sheng et al. 2024 8x NVIDIA H200 GPUs with mixed precision.
C ADDITIONAL ANALYSES
C.I COMPARISON WITH ALTERNATIVE ROLLOUT STRATEGIES
To further demonstrate the effectiveness of LATR, compare it against two other baseline rollout strategies:
1) Rollout Down-sampling (RDS) Xu al. 2025): Similar the group filtering in DAPO, RDS also seeks to enhance trajectory diversity in a post-hoc manner. Specifically, it first generates 16 trajectories via standard rollout and then selects the most diverse trajectories for policy updates. The selection is implemented greedy, using the average of sentence-level BLEU and ROUGE scores as trajectory similarity measures.
2) Entropy Guided Tree Search (EPTreee (Hou et al. 2025): Proposed in TreeRL, EPTree constructs rollout tree to support fine-graing reward estimation and optimization during policy updates. After generating M complete sequences, it identifies the top-N tokens with the highest entropy and re-generates continuations T times from each of these tokens, yielding a total of M x (N T + 1) sequences. Following the setup in TreeRL, we use (M,N, T) 24, 1). resulting in 10 sequences per rollout. To ensure a fair comparison, we randomly sample trajectories from these 10 for policy updates. We directly the official code from TreeRI. and integrates EPTree rollout into the VeRL training framework.
As shown in Table 7 LATR consis- tently outperforms both RDS and EP- Tree across both GRPO and DAPO policy update algorithms. Notably, while GRPO combined with RDS yields improvements over Stochastic Sampling due to enhanced trajectory diversity, the same combination under DAPO fails to surpass Stochastic Sam- pling's performance. Further analysis reveals that DAPO RDS leads to un- stable training dynamics, marked by performance degradation and sharp in- creases in KI. divergence during later
Table Performance comparison of different rollout strategies on the Countdown dataset.
Correctness (%) +
Average Length
Method
Pass@1
Pass@8 8
Pass @
Pass@8
GRPO Stoch.
65.9
73.9
473
610
GRPO RDS
68.7
75.7
365
GRPO w EPTree
65.3
73.5
471
GRPO w LATR
70.9
77.4
378
469
DAPO Stoch.
70.7
78.0
483
630
DAPO RDS
68.5
74.4
348
462
DAPO EPTree
66.3
74.6
450
595
DAPO W LATRR
74.7
81.5
367
453
15

---

Published as a conference paper at ICLR 2026
training stages. This instability likely stems from the diversity-oriente se-
lection mechanism, which biases towards selecting low-probabilityy trajectories, thereby increasing off- off-policy risk. When combined with DAPO, which already promotes diversity through group-based filtering, this effect is amplified, ultimately contributing to model collapse.
In contrast, the underwhelning performance of EPTree suggests that the gains reported in TreeRL primarily arise from its novel policy update mechanism rather than its rollout strategy. Specifically, TreeRL employs Monte Carlo Tree Search (MCTS) to estimate fine-grained rewards for individual tree branches by propagating sparse binary outcome rewards backward through the tree, enabling targeted optimization of intermediate reasoning steps. By contrast, LATR improves RL performance by enhancing trajectory-leve diversity without requiring modifications the underlying policy update algorithm.
C.2 IMPACT OF DIFFERENT SIMILARITY METRICS
As described in Equation the main experiments employ edit distance as the similarity measure to identify and prune redundant trajectory branches. In principle, however, numerous alternative metrics could be used assess the divergence between partial sequences. To investigate this, we evaluate three additional similarity measures in this section:
1) ROUGE-L: defined as the ratio of the length of the longest common subsequence to the sequence length.
2) Suffix matching: defined as the ratio of the length of the longest suffix one sequence that appears anywhere in the other sequence to the sequence length.
3) Embedding--bsset the cosine similarity of the sequence embeddings calculated by the model Qwen3-Enee -aaete--emeddn.......
For each metric, we conduct experiments while tuning the pruning threshold to identify its optimal value. The best results, summarized in Table 9 show that embedding- based similarity yields the weakest performance, while all other token-level metrics achieve comparable and better final accuracy. The failure of embedding- -aase similarity is likely stem from the inability for embedding models to capture fine-grained logical distinctions, since these models are usually trained to discern topic-level differences. Therefore, Given its simplicity and competitive efficacy, retain edit distance as our pruning criterion.
C.3 IMPACT OF BRANCHING AND PRUNING THRESHOLDS
To investigate the impact of the pruning threshold Ted, we conduct a series of experiments with Ted E f0.3, 0.4, 0.6, 0.63. The results are summarized Table g We observe that Ted = 0.4 yields the best performance, while both lower and higher values lead to a decline in results. This behavior can be attributed to the trade-off imposed by the pruning threshold: excessively high values result in overly similar trajectories, which consume the rollout budget without promoting exploration, whereas excessively low values impose overly stringent constraints, leading to an insufficient number of viable trajectories.
Table 8: Performance comparison of different thresholds.
Threshold
Pass @
Pass @as
Tahs 0.2
71.4
77.2
Tabs 0.25
74.7
81.5
Taha = 0.3
72.3
78.5
Tral 0.1
72.0
77.7
Trel 0.15
74.7
81.5
Trat 0.2
72.1
77.6
For branching thresholds Tobss Treis experiment by fix- ing one threshold while adjusting the other one. As shown in Table (Tabe: Trai) (0.25, 0.15) yields the best per- formance, corroborating the suitability of our selected hyperparameters.
Ted 0.3
73.8
80.5
Ted = 0.4
74.7
81.5
Ted 0.5
74.1
81.4
Ted = 0.6
73.4
79.8
C.4 EFFICIENCY ANALYSIS
The search tree in LATR is bounded by maximum width corresponding to the rollout number k. Unlike Stochastic Sampling, which perform forward passes each sequence independently at
16

---

Published as a conference paper at ICLR 2026
Table 9: Performance comparison of different similarity metrics.
Correctness (%) t
Average Length Pass
Method
Pass
Pass@8
Pass @ 8
Edit Distance
74.7
81.5
367
453
ROUGE-L
73.9
80.5
390
486
Suffix Matching
74.9
81.7
388
493
Embedding-
72.9
79.8
369
445
each step, LATR dynamically branches and prunes sequences, resulting in sparser tree structure, particularly during early generation stages. Consequently, the actual number of FLOPs consumed by LATR is strictly less than that of Stochastic Sampling for the same settings.
Empirically, LATR exhibits a modest slowdown in generation speed during RL training compared to Stochastic Sampling, shown in Figure Specifically, LATR runs approximately 10% slower per step than Stochastic Sampling with the same configuration. However, compared with DAPO with Stochastic Sampling, GRPO with LATR is able to achieve comparable performance in shorter training time, Additionally, considering the averagely 2.3x training speedup (as introduced in Section 4.3). LATR is able achieve higher performance in less total training time. This suggests that the algorithmic gains of LATR outweigh its runtime penalties in end-to-end training scenarios.
Further profiling reveals that the runtime overhead in LATR primarily stems from the sequential computation patterns during the branching and pruning phase. Unlike Stochas- tic Sampling, which processes contiguous batched inputs, LATR dynamically inserts and removes sequences during tree expansion and pruning, so indexing and comparisons are performed per sequence rather than fully batched manner. Targeted optimizations, analogous to PagedAtten- tion (Kwon et al. 2023) for Stochastic Sampling, are likely to mitigate the overhead. While promising, such engineering improvements lic outside the scope this work and are left to future efforts.
100
50
0
GRPO
DAPO
Stochastic
LATR
Figure 5: Comparison of average consumed time per training step un- der different settings (second).
C.5 ADDITIONAL. STATISTICS FOR LATR
To further elucidate the behavior of our proposed method, we re- port two key statistics: the aver- age branching ratio, which the proportion of tokens at which new reasoning branches are initiated rel- ative to the total number of gener- ated tokens, and the average satu- ration length, defined the number of tokens generated before early stopping is triggered. Following the setup in Section we present these metrics for the same three model variants: Qwen2.5-3B, Swwe2.5--B-Inntteee and Qwen2.5-LATR, enabling consistent and comprehensive analysis across
Table 10: Key statistics for LATR.
Model
Branching Ratio
Saturation Length
Qwen2.5-3B
0.101
1ree2.55-3-IInsteeee
0.039
Qwcn2.5-3B-LAY
0.044
132
model stages.
As shown in Table 10 the branching ratios consistently low across all models, indicating conservative branching behavior. Moreover, the average saturation length is notably shorter than the maximum completion length of 1,024 tokens. This observation aligns with prior findings (Shao et al. 2025), which suggest that the initial segments of reasoning chain are often most critical in determining the final outcome.
17

---

Published as a conference paper at ICLR 2026
You expert Al assistant specializing in mathematics, You must strictly return the result in valid JSON format.
IMPORTANT: In all JSON strings, every literal backslash 'V' for LaTeX commands must be escaped as 'W. For example, '\frac' must be written as "lifrac".
Task
Given a math problem with its non-integer answer, rephrase it into a new problem whose solution is single integer.
Guidelines
1. "FFor non-integer numerical answers"*, convert them a single integer base using the following rules:
"*Fractions:** For simplified fraction SWisaacaa{b}S, ask for the value of + b. For mixed number $alVfrac(bHc]S, ask for form $k mlisqrt(n]s, ask value k m n. If
""Radicals:" For an answer
k=0, ask for m n.
**Complex Numbers:" For complex number bi, ask for the value of a b. *CCordinate Points:** For ordered pair Sileft( a, bliright]S, ask for the value of
*Muitiples Pi."* an answer k pi, ask value
**ther Number Bases:** For an answer in base (e.g., 52_ 8), ask for its value base
10.
2. **For Symbolic Expressions and Functions** choose the appropriate strategy based the expression's type:
"((Strateyy A) Component Analysis for Simple Polynomials and Fixed Structures:"" For answers that are simple linear combinations. polynomials, or have a fixed symbolic structure, prompt for the sum of their key integer components. "This should be prioritized for simple expressions." **
**Applies to:** $a lodot p b lodot q$, $n^2 + an + b$
**And fixed forms like:"* $a^bS. $al$. 'Sa! bĄc$
**Example Action:" For $2n + 3S, ask for the sum of coefficients and constants ('2+3").
(Sirrategy B) Evaluation for Complex or Non-PPlynomial Functions:**
For answers involving complex functions (e.g., binomials, floors) or defined as pairs functions, prompt for the expression's numerical value at small, specified integer inputs. "Applies to:** $1ninom((n)(n]S, $mn Wifloor m/2 Wrrfoorr,, function pairs like Sf(n),
g(n)S.
"Example Action:"* For $ibbinom22K(n)5.. ask for value at $n=2$. 3. ""For multiple-chen ceatiplle-chiceeeeee or named a Convert the list of possible choices into 0-indexed list and ask for the index of the correct
Requirements
1. Please modify the instruction of the output format if necessary. Except that, do not change the content of the original problem.
2. Adhere the JSON format: ("question": "The complete rewritten question text", "answer": The integer answer)
Examples
(Examplees)
Now, please process the following problem
Original Problem:
Figure 6: Prompt for data transformation.
18


Reviews:
[{"review_id": "dHmhA2ztxK", "reviewer": "Reviewer_gVkW", "summary": "This paper introduces a tree-based algorithm to increase the sampling diversity in the setting of RLVR.  The method focuses in generating trajectory-level diversity during the sampling process by rolling out and pruning semantically similar trajectories. The show that including this sampling strategy with GRPO and DAPO, the performance increases given the new variety of sampled reasoning trajectories. They evaluate their method on math and logical reasoning tasks.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally well written and the method is explained clearly.\n- The empirical evaluation shows the benefits of the enhanced strategy with respect to token-level stochastic sampling", "weaknesses": "- The paper claims to care about semantic similarity but they prune based on Edit distance which doesn’t seem to me to be a good measure of semantic similarity. Maybe could the authors explain why this is working in their evaluation tasks?\n- It seems that the new sampling process introduces could potentially introduce off-policy issues? Is this being taken into account in $\\pi_{old}$, or am I misunderstanding  something?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ddWz7RUtF", "reviewer": "Reviewer_cvuy", "summary": "This paper introduces an enhancement to the reinforcement learning with verifiable rewards algorithm.\n\nMotivation\n\nPrevious existing methods for RLVR, such as GRPO and DAPO, suffer from limited diversity in the sampled trajectories. Follow-up works mitigate this by (i) including increasing sampling temperature or (ii) filtering out and rejecting sequences with very similar samples. However the authors argue that  (i) i focuses on token-level diversity, rather than trajectory-leval diversity, and (ii) needs multiple rollouts leading to high cost, and  lacks withing-group diversity.\n\nMethod\n\nTo mitigate this, the authors propose  Lookahead Tree-Based Rollout (LATR) algorithm, inspired by MCTS that works as follows: \nstarting from a root branch that corresponds to the prompt:\n- first there is a branching stage, where each branch is extended with its highest-probability token (this is the parent branch) , as well as other tokens that have a \"high-enough probability\"  (these are the children branches), This is done until we have a set of K sequences.\n- then, there is a pruning stage: given a lookahead step r , they look at each children branch that already had r rollouts from their parent branch, and they look at how different this children branch is from the parent branch (using edit distance). If differrent enough, the children branch and its descendants are kept. If not, it is removed along with its descendants.\n\nThey introduce further optimizations eg.. slowly decreasing the ratio of sequences generated with LATR during training in favor of standard stochastic sampling (more exploration in the start, more exploitation in the end of training) \n\nExperiments \n\nThey run experiments on the CountDown,  DAPO-math, ACM23, MATH-500, Olympiad-Bench datasets. They run GRPO w LATR / DAPO w LATR, and they compare them to the baselines od GRPO w stochastic sampling / DAPO w stochastic sampling. Performance is measured in terms of (i) average correctness of the provided answer and (ii) average length of the answer provided (as well as format for CountDown). In almost all cases, LATR shows better performance in both metrics compared to the baselines. Taining dynamic results reveal faster learning (in terms of avg val reward) with LATR. \n\nThey also run ablation studies on the diversity comparison across answers with / wo LATR, the effect of the pruning, and of sampling at different temperatures.", "questions": "- Given point 1 above, Could you show ablation studies on the lookahead step r, in particular with r = 1 (i.e. only one-step lookahead but with the pruning that removed redundancy ) ? I would be curious to know if trajectory-level lookahead is important, or if token-level lookahead with diversity pruning is enough. \n- Given point 2 above, do you think edit distance a good measure of diversity between sequences? You could have 2 sequences that have high edit distance but mean the same thing because there are synonym words. Have you thought of using another distance e.g.  distance between the embeddings of the sequence, to make sure that they are semantically different? \n- In the diversity experiments, \"two answer expressions ...  if their evaluated numerical outcomes differ\". I am confused: for each math question there is only one correct answer, and you can have multiple diverse ways to reach this correct answer. Why aren't you looking at this diversity instead (rather than diversity over final answers, for which high diversity means that some answers are incorrect! )\n- For the temperature-ablation experiments, do you keep the same theresholds tau_abs and tau_rel across temperatures? If yes, could you elaborate why is LATR less sensitive to temperature tuning? The branching quality in LATR will still be affected by the increase in temperature (because of the increased stochasticity in the new branches ), no?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Good presentation of the paper, clarity and ease of reading\n- Good contextualization of the paper in previous work, helpful for those who are not experts in this field\n- Simplicity of the introduced changes and strong evidence that it yields improvements \n- Clear experiments section with ablation studies", "weaknesses": "- I think an important ablation study is missing to show the claim of the paper that trajectory level lookahead is important and yields improvements over token-level lookahead.  \n- Also, the authors mention that \"token-level variations typically occur without lookahead ability, making local deviations (e.g., substituting “compute” with “calculate”)  \" however this can also happen in trajectory level variations (with multiple such subsitutions). The authors use the edit distance to quantity diversity, however this distance assigns high value to word substitutions that mean the same thing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NvZHtheKok", "reviewer": "Reviewer_mqHB", "summary": "The paper presents LATR: Lookahead Tree-based rollouts for improving the diversity of the rollouts in algorithms like GRPO.\nLATR operates in 3 stages: Branching which creates new trajectories when the model is highly uncertain, (2) Lookahead simulation to extend a bnrach and Pruning when branches that are similar are pruned.\n\nBranching occurs at each point with the highest logit-based token being extended to capture the most likely trajectory. To encoureage diversity, if other logits satisfy a threshold, child branches are created. The authors use dual-thresholds to prevent diverging too far from the models distribution.\n\nSince it is computationally expensive to branch at each logit/token, the authors uses lookahead simulation. After a branch, each trajectory along the branch is generated sequentially for a window or r tokens. Finally, pruning is performed for these sequential generations so that branches that are not sufficiently different are pruned.\n\nThe authors explore more optimizations like early stopping and hybrid rollouts akin to epsilon in RL settings.\n\nThe authors then conduct an empirical evaluation and compare policy optimization methods with LATR and compare to stochastic sampling. The authors also provide ablations to contrast the effect of different components.", "questions": "Overall, this paper is interesting and the results seem okay.  There are a few weaknesses that the authors can clarify before I can increase my score. Happy to discuss further.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clear and well-written\n\n2. The proposed technique is intuitive and explained well. Using tree-based methods to focus on diverse trajectories based on model certainty is intuitive.\n\n3. The experiments showcase model improvements in performance but significant improvements in the inference costs.", "weaknesses": "1. I think one major weakness in the paper is the empirical evaluation which is my primary reason for a lower score.\n\nThere is only 1 model considered. I think to better understand the performance of LATR you would need to evaluate more models.\n\n2. It is not clear how statistically significant the results are since standard deviations are not provided. How many times were the models trained for these experiments.\n\n3. Other relevant baselines like TreeRL are not considered. I believe TreeRL also seeks to improve diversity by employing entropy.\n\n4. What is the threshold for EditDistance used in the experiments?\n\n5. Im not sure how useful Ablation 5.3 is. You are comparing two different methods. While LATR is not sensitive to temperature and this is not super surprising, the interesting analysis IMO is to showcase the sensitivity to all these new hypermeters/thresholds you have introduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d1XQjHbG8d", "reviewer": "Reviewer_Vq12", "summary": "This paper introduces Lookahead Tree-Based Rollout (LATR) to enhance trajectory-level diversity among rollouts in a group to improve the performance of GRPO and DAPO in policy refinement.", "questions": "1- The paper states that the tree-search approach is inspired by MCTS. Could you clarify how your tree search is similar to the MCTS used in AlphaZero [1]?\n\n\n\n1- Mastering the game of Go with deep neural networks and tree search.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1- The paper is well written and easy to follow.\n2- The diversity problem in RL is well known, and this paper clearly identifies it within the RLVR paradigm.", "weaknesses": "1- The literature review is not thorough. Many approaches that combine look-ahead reasoning with LLMs are not mentioned.\n\n2- The baselines are neither strong nor state-of-the-art. Although the paper cites works addressing diversity at the token level, the comparisons use only stochastic sampling; it is unsurprising that look-ahead search would outperform simple stochastic sampling.\n\n3- The performance gains are not substantial, especially in terms of correctness.\n\n4- Although inference cost is reduced, there is no comparison of training costs between the stochastic and look-ahead approaches. I would expect the look-ahead method to require significantly more training time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)