DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Tree-based Search
Authors: Fang Wu
Weihao Xuan
Heli Qi
Aaron Tu
Ximing Lu
Li Erran Li
Yejin Choi
Venue: ICLR 2026 Poster (2026)
Keywords: MCTS
RLVR
Primary area: foundation or frontier models, including LLMs

Abstract: Although Reinforcement Learning with Verifiable Rewards (RLVR) has become an essential component for developing advanced reasoning skills in language models, contemporary studies have documented training plateaus after thousands of optimization steps, i.e., notable decreases in performance gains despite increased computational investment. This limitation stems from the sparse exploration patterns inherent in current RLVR practices, where models rely on limited rollouts that often miss critical reasoning paths and fail to provide systematic coverage of the solution space. We present DeepSearch, a framework that integrates Monte Carlo Tree Search (MCTS) directly into RLVR training. In contrast to existing methods that rely on tree search only at inference, DeepSearch embeds structured search into the training loop, enabling systematic exploration and fine-grained credit assignment across reasoning steps. Through training-time exploration, DeepSearch addresses the fundamental bottleneck of insufficient exploration, which leads to diminishing performance gains over prolonged training. Our contributions include: (1) a global frontier selection strategy that prioritizes promising nodes across the search tree, (2) selection with entropy-based guidance that identifies confident paths for supervision, and (3) adaptive replay buffer training with solution caching for efficiency. Experiments on mathematical reasoning benchmarks show that DeepSearch achieves an average accuracy of 62.95% and establishes a new state-of-the-art reasoning model, while using 5.7x fewer GPU hours than extended training approaches. These results highlight the importance of strategic exploration over brute-force scaling and demonstrate the promise of algorithmic innovation for advancing RLVR methodologies. DeepSearch establishes a new direction for scaling reasoning capabilities through systematic search rather than prolonged computation.

Published as a conference paper at ICLR 2026
DEEPSEARCH: OVERCOME THE BOTTLENECK OF REINFORCEMENT LEARNING WITH VERIFIABLE RE- WARDS VIA TREE-BASED SEARCH
Fang Wu ON
Weihao Xuan v.4*
Heli Qj
Aaron Tu
Ximing Lu
Li Erran Li
Yejin Choi
Stanford University University of Tokyo
ARIKEN AIP UC Berkeley
University of Washington + Amazon AWS
ABSTRACT
Although Reinforcement Learning with Verifiable Rewards (RLVR) has become an essential component for developing advanced reasoning skills language models, contemporary studies have documented training plateaus after thousands of optimization steps, i.e., notable decreases in performance gains despite in- creased computational investment. This limitation stems from the sparse explo- ration patterns inherent in current RLVR practices, where models rely on lim- ited rollouts that often miss critical reasoning paths and fail provide system- atic coverage of the solution space. We present DeepSearch, a framework that integrates Monte Carlo Tree Search (MCTS) directly into RLVR training. In con- trast to existing methods that rely on tree search only at inference, DeepSearch embeds structured search into the training loop, enabling systematic exploration and fine-grained credit assignment across reasoning steps. Through training-time exploration, DeepSearch addresses the fundamental bottleneck of insufficient ex- ploration, which leads to diminishing performance gains prolonged training. Our contributions include: (1) a global frontier selection strategy that prioritizes promising nodes across the search tree, (2) selection with entropy-based guidance that identifies confident paths for supervision, and (3) adaptive replay buffer train- ing with solution caching for efficiency. Experiments on mathematical reasoning benchmarks show that DeepSearch achieves an average accuracy of 62.95% and establishes a new state-of of-tth-art reasoning model, while using 5.7x fewer GPU hours than extended training approaches. These results highlight the importance of strategic exploration over brute-force scaling and demonstrate the promise of algorithmic innovation for advancing RLVR methodologies. DeepSearch estab- lishes a new direction for scaling reasoning capabilities through systematic search rather than prolonged computation.
http:://ghhha.cccmmmmmmmee/////p
INTRODUCTION
Large language models (LLMs) have recently achieved notable progress complex reasoning tasks (DeepSeek-A] 2025 Yang et al. 2024 Wu et al.] 2025a Wang et al. 2026 Xia et al. 2025
driven in part by test- time computation scaling strategies (Li al. 2023 Yao al. 2023 Bi al. 2024 Zhang et al.] 2024a Guan et al. 2025) such as tree search with process-level evaluation. While effective, these methods typically treat structured search an inference-only mechanism, leaving untapped potential to integrate systematic exploration into the training process itself.
This separation between training and inference imposes fundamental limitations of reinforcement learning with verifiable rewards (RLVR) for reasoning. Current RLVR approaches
the scalability
Equua contributions.
Corresponding author. Email: rejinssattooordeee

---

Published as a conference paper at ICLR 2026
DeepSSeacc-MMCTS
Adaptive Training
Repeat Karations
Intermeditte
Solection
Expansion
Score backup
Fo KD
Reerttve poliy Enihiigwhhh Truss GRPO
Intomndistcrod: frobairranter)
Optinal global frontior
P.
Endinods coonec incomplett)
Node rewand
Endindde (conec)
Update Policy
Filter trainingset
Glosal from relection with frontio prlarity score A Ppprree + A
Viea
Reid
Expans Ra( lo..
Select etthrr cornect must wnnorrtoommtttenggoooo for backgr spagiton
valu
MCTTiaa othanwise
Replay buffer updete withenolectryy from AXCTS ACCTS
volution
onci PasttkklK myo) it
If FC
a Sha /ta xpp))) d:0).
Reploy Buffer
Figure I: DeepSearch framework overview with three key components.
remain constrained by sparse exploration patterns during training [Wu et al. 2025b Liu et 2025c Tu et al., 2025). while models are expected to demonstrate sophisticated search behaviors only at inference time. Even recent advances in prolonged RL training (Liu et al. 2025a) have shown that performance plateaus after thousands of steps, with diminishing returns to allocating more compute to deeper training. This suggests that simply scaling the number of training steps, the primary axis explored in prior work, may not be sufficient to fully realize RLVR's Rss potential.
We address this gap by introducing DeepSearch, a framework that embeds Monte Carlo Tree Search (MCTS) (Metropolis & Ulam 1949) directly into RLVR training, representing fundamental shift from scaling training depth to scaling training breadth. By coupling structured search with verifiable rewards during training, DeepSearch enables models learn not only from correct solutions but also from the systematic exploration process itself, providing richer supervision than outcome-based or direct rollout methods (Lyu al. 2025 He et al. 2025b).
The core insight driving us is to focus on training-time exploration as the driver of improved rea- soning. While traditional RLVR relies on limited rollouts that may miss critical reasoning paths, DeepSearch systematically expands the reasoning frontier during training through principled tree search. This design advances three key objectives: (i) expanding reasoning coverage beyond what direct policy rollouts can achieve, (ii) providing fine- Ene-grained credit assignment to intermediate rea- soning steps through tree-structured backpropagation, and (iii) maintaining computational efficiency through intelligent node selection and solution caching strategies.
Towards these goals, DeepSearch introduces several key innovations. First, global frontier selection strategy prioritizes the most promising nodes across the entire search tree, moving beyond tradi- tional root-to-leaf Upper Confidence Bounds for Trees (UCT) traversals that can be computationally wasteful and myopic. Second, selection with entropy-based guidance systematically identifies fident incorrect reasoning paths for supervision. Finally, an adaptive training strategy with replay buffers progressively filters challenging problems and caches verified solutions, thereby avoiding redundant computation across training iterations.
We evaluate DeepSearch on math reasoning benchmarks, where it significantly outperforms strong RLVR baselines (Liu et al.) 2025a Luo et al. 20256). Our results show that DeepSearch achieves 62.95% average accuracy on challenging mathematical tasks, representing new state-of- the-rr for 1.5B reasoning models. Importantly, these gains are achieved while maintaining computational efficiency through progressive filtering and intelligent reuse of solutions, demonstrating that search- augmented training can be both more effective and more practical than conventional approaches.
The implications extend beyond math reasoning: by bridging the gap between inference- time search capabilities and training-time learning, DeepSearch establishes new approach for scaling RLVR that emphasizes systematic exploration over prolonged training. This work suggests that the future of reasoning model development lies not just in scaling model parameters or training steps, but in fundamentally rethinking how we structure the learning process to mirror the sophisticated reasoning patterns expect at inference time.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORKS
Search-based reasoning. Structured search has become a standard strategy for scaling test-time compute in LLMs (Snell et al. including tree-based (Yao et al.
2024 Wu et al. 2024 Zhang et al. 2024c), with diverse methods 2023 Zhang et al. 2024b Qi et al. 2024 and random sampling approaches (Wang et al. (022). Recently, search-based reasoning has evolved into sophisticated frameworks that integrate three core components: policy models, reward models, and search algo- rithms. Drawing inspiration from game-playing systems like AlphaGo (Silver et al. 2016). works have explored Monte Carlo Tree Search (MCTS) and beam search to guide L.Mss through structured reasoning processes (Chen ct al., 2024, Zhang et al. 2024a|c), particularly following OpenAl's ol release (Jacch et al., 2024). These frameworks enable exploration of multiple solution paths during inference, trading compute resources for improved accuracy on challenging tasks such as math rea- soning. Key design considerations include outcome- supperiiide versus process-supervised reward models, discriminative versus generative reward architectures, and search strategies ranging from local selection to global exploration (Lightman et al., 2023 Wang et al. (023). However, most methods restrict search to inference and do not integrate exploration signals into training, leaving the potential for jointly optimizing search and learning largely unexplored.
Reinforcement learning from verifiable rewards. RLVR has emerged as a transformative ap- proach for aligning and enhancing LLMs by addressing critical challenges across instruction fol- lowing (Su et al.) 2025; Gunjal et al. 2025), ethical alignment (Wang et al. 2025a), and reason- ing capabilities (Wang et al.J 2025b). Recent extensions (Guo et al. 2025 Yu et al. 2025; Wan et al. 2025) have improved training stability and efficiency by incorporating critic-free optimiza- tion, dynamic sampling, and adaptive weighting mechanisms. While these approaches demonstrate the promise of RLVR, they predominantly rely direct rollouts, which can constrain systematic exploration of the solution space (Wu et al., 2025b Yue et al. 2025).
Monte Carlo Tree Search. MCTS is powerful search paradigm for complex decision-mate maakin problems, extensively explored across diverse fields like games (Silver et al.] 2016: Ye et al. 2021)) I)
robotics (Best et al.] [2019] Dam et al. 2022). theorem proving (Lample et al. 2022), and matrix multiplication (Fawzi et al. 2022). Early work, such as AlphaGio integrated MCTS with deep learn- ing (Kemmerling et al.] (223)), achieving superhuman performance in board and video games (Ye) eet a 20011) Recently, MCTS has been applied to path finding and train timetabling problems (Pi- tanov al. 2023 Yang 2023), while Valleia et al. (2024) integrated MCTS into physies-informed planning networks for robot control. Despite the demonstrated potential of MCTS for heuristic exploration, remains unclear how to effectively employ it during RLVR training.
3 DEEPSEARCH WITH MCTS
Given a problem r and a policy model na. we adopt a modified MCTS framework to build search tree for incremental step-by-step solution exploration. We replace traditional root-to-leaf selection with global frontier-based node selection. The root node represents the question z, and child nodes correspond to intermediate steps generated by To. A root-to-leaf path ending at a terminal node Send forms a trajectory = @ @ $2 o.... ① Send, where each step is assigned a q-value ((;;). Then we extract solution trajectories T = {t', .t.,t?...,tt (n 2 1) from the search tree T. where t' can be correct, incorrect or incomplete. The depth of any node is denoted as d(s) E 2 (s) and {(s) denote the number of visits to and the number of children nodes of s, respectively. Starting from the root node our MCTS iterations are conducted through three subsequent components.
3.1 EXPANSION WITH ENTROPY-BASHE GUIDANCE
In step i, we collect the latest reasoning trajectory o; = x + 81 + as the current state, ie., observation. Based on this state, we prompt the policy model (s:o))) generate n candidates for the next-step reasoning trail {8t, ) We repeat this expansion behavior until we reach the terminal nodes Send Send, either by arriving at the final answers or by hitting the maximum depth Send.
of the tree dr, which yields an ordered sequence
During each expansion, let s(A) denote the set of newly generated terminal nodes at iteration k. We evaluate the correctness of each terminal node using a verification function V Send > {0, 1). where (ss) 1 indicates correct solution and (ss) 0 indicates an incorrect incomplete
3

---

Published as a conference paper at ICLR 2026
solution. Then we partition the terminal nodes into correct and incorrect/incooplete subsets:
S correct
{s
end V(s) = 1},
((k) incverect
{s E Send
V(s) - 0}.
correct = 0, we employ an entropybaased selection to identify the most confident wrong rollout.
The terminal node with the lowest average entropy along its root-to-leaf trajectory is selected:
neg = arg min
Htt(s)),
(2)
8E aES ircemntt
where t(s) (2) 81, :2, 8) represents the unique trajectory from root x terminal node s, and the average trajectory entropy defined as H(t(s)) = [t()]] [(() (TT((s; o.))., where H (TT((:: o:)) Ea,, ma(a,, 0i, (i, log no (ai,a os, di,ck) is the Monte Carlo estimation of the Shannon entropy of the token distribution at step i. dj,A is the k-th token of step 8;, and dj.ck denotes the tokens preceding 0g,k- This strategy prioritizes incorrect reasoning sequences with low decision uncertainty, targeting areas where the model is most confident in its decisions and would benefit from additional training and supervision. We find that this most-confident incorrect selection consistently outperforms random and least-confident selection across all benchmarks, see Table
3.2 HEURISTIC SCORE BACKUP
Let to denote the selected trajectory for backpropagation, which is either a correct solution trajectory or the most confident negative trajectory t($iie) identified through entropy- selection. Let (x;) denote the q-value for node 8; E t" after the m-th rollout backpropagation. We define the iterative q-value update rule for nodes along the selected trajectory:
(m))
(s.) + (i, (m("))((())))))) (Sena),
where (i. 2 x - [0, 1] the depth decay function that assigns higher weights to nodes defined as (i, max ( Tmin). where is the current node
closer the terminal node. index in the trajectory, I is the terminal node index, and Tmin = 0.1 is the minimum decay threshold.
The q-value initialization is g (s:) = 0 for all 8t E T. Terminal node rewards are assigned according to the verification function's result:
+1 if (Sena) (correct),
((eadd)
(4)
if ((sua)
(incorrect) d(send) dr (incomplete).
To ensure positive q-values (e.g., Jcorrect = 0.1) for nodes on correct reasoming paths while penalizing nodes leading to incorrect or incomplete solutions, we enforee the constrained update rule:
((m-)((s))) + (i, -q (m))
if q(tm-") (st) ((m) 2
(m) (s:)
11, 9 (Sena)
elif o (m)) (Send) > 0,
(5)
9 (m--1)(s)
clif (m (s;;) > O.
This constraint preserves the invariant that ((m) 2 O for all intermediate nodes 8; E T Send leading to correct solutions, while allowing negative values only for nodes observed on any correct trajectory under the current search process. More justification is elucidated in Appendix B.2
3.3 HYBRID SELECTION STRATEGY
UCT (Kocsis Szepesvári 2006) is the standard selection rule used classical MCTS to balance exploitation of high-value nodes and exploration of under- ones. Our MCTS employs hybrid selection strategy that combines traditional UCT-based local selection with novel global frontier selection, each serving distinct purposes in the search process.
Local Selection for Sibling Comparison During the expansion of selected node, we generate multiple candidate children and must determine which to add to the tree. For this local sibling comparison, we follow the traditional MCTS protocol and employ the UCT algorithm asUCT(s)
Q(s) + X In Nowen(s) where Q(s) = q(s)
N(a) represents the average reward per visit, Nparem (s) is
N(s)
the number of visits from the parent node, and À balances exploitation and exploration. This local selection ensures that we make optimal decisions when choosing among sibling nodes that share the same parent and context.
4

---

Published as a conference paper at ICLR 2026
Global Frontier Selection for Next Expansion After completing the first score backup phase, need to identify the most promising node across the entire search tree for the next expansion round. This is where our novel global frontier selection mechanism operates.
Unlike traditional MCTS, which performs root-to-lcaf traversals using UCT at each level, our global approach directly compares all frontier nodes simultaneously. We maintain global view of all leaf nodes across the entire search tree T and prioritize promising expansion points globally:
F = {s E T (() = 0, & Send, d(s) < dr}.
(6)
For each frontier node E F. and its associated observation (prefix) Os, we compute a frontier priority score:
F(s) A1 tanh tanh(( pareet (s)) + A2 x (TQ(( oa)) + D(d(s))
Quality Potential
Uncertainty Bonus
Depth Bonus
Here, the quality potential term pareor)))))) encourages the selection of nodes whose parents have demonstrated high value. using the tanh transformation to smoothly handle negative Q-values and map them to the range 1-1,1] 1]. The uncertainty bonus term H (TTos os)) provides exploration guidance by adjusting priority according to the policy's entropy; the sign of its coefficient can be utilized to steer selection toward regions with high confidence or uncertainty. The depth bonus term (d((s) encourages deeper exploration by providing additional priority to nodes at greater depths, where we empirically find D(d(s))= /d(s)/dr to be most effective among other variants including d(s) and log (d(s) ) + 1). The node with the highest frontier score is selected for the next expansion: s" arg m Fs)).
Rationale for Hybrid Approach This hybrid design leverages complementary strengths: local UCT selection ensures principled sibling comparisons within subtrees, while global frontier selec- tion mitigates UCT's myopia by allocating resources across subtrees. The approach achieves three key advantages: (1) Computational efficiency by eliminating redundant root-to-leaf traversals, (2) Enhanced exploration coverage by preventing the algorithm from getting trapped in locally promis- ing but globally suboptimal subtrees, and (3) Uincetainty-guidee
search that leverages the policy's 's entropy to target regions expected benefit from additional training supervision, with the bonus coefficient controlling the direction of this preference.
4 ADAPTIVE TRAINING STRATEGY WITH REPLAY BUFFER
While MCTS offers fine-grained credit assignment, applying it to every training example is com- putationally infeasible. To address this, adopt iterative filtering strategy with a replay buffer mechanism that focuses MCTS computation on challenging examples while preventing catastrophic forgetting of solved problems. The complete pipeline is depicted in Algorithm
4.1 ITERATIVE TRAINING WITH PROGRESSIVE FILTERING
Our training process follows an iterative approach that progressively refines the training subset based on model performance. We begin by using the base RL model to perform an initial screening on the entire dataset Dhard creating the first training subset (()) bard for MCTS-based RL training.
Specifically, the iterative training process proceeds as follows:
Initial Subset Construction: Given the base policy Tgro), we evaluate its performance on the full training set Drain using direct rollouts and construct the initial hard subset:
D (00
{ar Drrain Pass (K(,,, Totos) (0)),
(8)
where Pass10K(1. x) represents the success rate when sampling K 4 solutions for problem using policy म. and ((0) E (0, 1) is the initial filtering threshold.
Iterative Refinement: After each training phase i, we re-evaluate the updated policy T on the current hard subset and apply threshold-based filtering create the next iteration's 's training set:
D ((++1) {a D hand Pass1 @K(x, Toca) 880).
5

---

Published as a conference paper at ICLR 2026
The filtering threshold 8() is typically set 25%, ensuring that only problems with insufficient success rates remain in the active training set. This progressive filtering concentrates computational resources on increasingly challenging problems the model improves.
4.2 REPLAY BUFFER WITH CACHED SOLUTIONS
To prevent catastrophic forgetting and efficiently leverage previously discovered solutions, we main- tain a replay buffer R that stores correct reasoning trajectories from earlier training phases.
Buffer Population. During each training iteration i, we identify problems that obtained correct solutions through MCTS rollouts but still fail to meet the filtering threshold after training:
(
candidates =
{z, teomoo) x E D Atoorca T(x), Pass (K(2 TTO)) < =
(10)
These candidate trajectories are added to the replay buffer, attaining R(++1) = (() (())
candidates
Cached Solution Usage. Instead of randomly sampling from the replay buffer, employ de- terministic strategy that directly utilizes cached solutions when available. For each problem æ in the current training iteration, first check whether correct solution has been previously cached. This approach eliminates redundant MCTS computation for problems with known solutions while directing computational resources toward truly challenging, unsolved problems.
Hybrid Rollout Strategy. When processing problems the current hard subset ((1) different rollout strategies based on cache availability:
hard we apply
Rollout (:) teached U (ireecRollluus((..3) if (I, teached) E R((,, otherwise.
(11)
(CTS
For problems with cached solutions, we directly incorporate the stored correct trajectory trached and supplement it with DirectRollouts(r,, 3). which samples 33. additional solution attempts from the current policy (-(rr)) where 3 E [0, 1] is implicitly determined by the number of cached solutions per problem (we allocate fewer DirectRollouts as more correct trajectories are cached) and is the standard sampling budget. For problems without cached solutions, we apply the complete MCTS search process (u))). Moreover, among the incorrect samples, we remove data containing garbled text or infinite repetitions. Based on empirical evidence, optimizing policies on such prob- lematic data frequently leads to training collapse (Bai et al. (025). The training dataset for each iteration is then constructed as:
(teached U (irecttollouss(,,3))) U)
U
MCTS
(: (eottdd
(oxcee (
(12)
Cached problems
Unsolved problems
This eliminates the need for artificial sampling ratios complex batch composition strategies, as training data naturally incorporates both preserved knowledge and fresh exploration, tailored to problem--pecific requirements. This has three key benefits: (1) Computational efficiency by avoid- ing redundant MCTS computation, (2) Solution preservation by guaranteeing the inclusion cached correct trajectories, and (3) Continued exploration at minimal computational cost.
4.3 TREE-GRPO TRAINING OBJECTIVE
After constructing search tree T for sample question in the dataset D we develop our Tree- GRPO training objective. This objective combines valuu regularization with policy optimization to learn effectively from tree-structured reasoning traces.
Q-Value Soft Clipping. To address the q-value explosion problem for intermediate nodes while preserving meaningful gradients, first apply soft clipping using the hyperbolic tangent function:
((3)) tanh ( (1 (s))/en)
yynax for all 8j E T Send
(13)
where kmax is the maximum rollout iterations, c, = 1.0 is the temperature parameter, and 9max defines the maximum allowable value magnitude.

---

Published as a conference paper at ICLR 2026
This soft clipping approach prevents q-value explosion by maintaining all intermediate node qr values within -9max Imax]] while offering several key advantages: (i) naturally bounds q-values without hard discontinuities, (ti) it preserves gradients everywhere, preventing the cero-gradient problem that occurs with hard clipping when all values hit the same bound, and (äi) maintains the relative ordering of q-values while compressing extreme outliers. Terminal node q-values remain unchanged as defined in Eq. 4 In practice, fewer than about 5% of intermediate node q-values fall into the saturation region and less than 0.5% lie near the boundaries, so clipping mainly compresses pathological tails rather than altering the bulk of the distribution.
Training Objective. jective as:
With regularized g-values, we formulate and maximize the Tree-GRPO ob-
layl min (p).((0))).,, clip (p)x(0), Elow, 1 + Ehigh) Aμ,z)
((0) = (o
(14)
where Pj. Tean (ag,k loy ck)ck)
the importance ratio. The parameters <high and flow follow the
Clip-Higher strategy of DAPO (Yu et al. 2025). while we also remove the KL regularization term DEL to naturally diverge (Luo et al. 2025al He et al., 2025a). An overlong buffer penalty is applied
to responses that exceed predefined maximum of 4096. The advantage function for node sj in
trajectory t; is computed using sequence-level normalization (Chu et al. 2025):
AJ,x = {(3)) Pt:
(15)
where It is the average reward of the terminal nodes Sent throughout the tree This normalization is crucial in practice, particularly for mitigating uncontrolled growth in response length. Notably, Tree-GRPO can be degraded to the vanilla DAPO if we consistently leverage the outcome reward (Senn) {(5;) for all intermediate nodes.
EXPERIMENTS
5.1
BENCHMARK PERFORMANCE EVALUATION
Datasets and Base Models. We train DeepSearch based on Nemotron-Researdee hectron---ssaarrc---annnnnnnnn Qwen- 1.5B v2 (Liu al. 2025al and employ DeePMath-103K (He et al. (025c) the dataset.
DeepMath-108 103K is a large-scale mathematical dataset designed to be highly challenging and rig- orously decontaminated across numerous benchmarks. We evaluate DeepSearch against state-of- the-art 1.5B reasoning models on six mathematical benchmarks: AIME 2024/2025, AMC2023, MATH500 (Hendrycks al. 202T). Minerva (Lekkowyzz al. 2022). and Olympiad (He al.,
2024). More experimental details are described in Appendix
Baselines. We compare against recent 1.5B models spanning different mechanisms: base models
(Qwen2.5-Math variants). RL-trained models DDeepSeckRR1-Dsstee STILL-3 (Tem 2025 Open-
RS series (Dang Ngo 2025), advanced RL methods (DeepScaleR (Luo al. 2025b), Nemotron variants), and search- based approaches (Qwen2.5 Mathh 1---Mah-OOtt-eee Oat- (Liu et al. 2025b)). Our evalua-
tion methods and results are consistent with Hochlehnert al. (0025).
Results. Table [ shows DeepSearch- 1.5B achieves 62.95% average accuracy, outperforming all baselines, including the previous best -eeetrch Reasoning -wen--.-eeeeBBB (61.70%). DeepSearch- 1.5B demonstrates consistent improvements across all benchmarks, with notable gains on AIME 2024 (53.65% vs 51.77%) and AMC (90.39% vs 88.83%). The 1.25 percentage-poing improvement over the previous state of the art validates the effectiveness of integrating structured search into RLVR training rather than restricting inference-only use.
5.2 TRAINING EFFICIENCY ANALYSIS
To evaluate the practical viability of DeepSearch, compare computational costs with extended training approaches that scale solely with the number of additional training steps. As shown in Ta- ble 2 extended training exhibits diminishing returns: 325 additional steps achieve 61. 61.78% accuracy

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison of 1.5B-scale language models on standard mathematical rea-
soning benchmarks. We report asssl1 accuracy estimated with = 32 samples. The best-
performing results highlighted in bold. All evaluations were conducted on 28X0H10 96G
cluster.
Model
AIME24
AIME25
AMC23
MATH
Minerva
Olympiad
Avg
Owen2.5-Math-L55B
8.33
6.35
44.06
66.67
18.42
30.74
29.10
Swen2.5-Math-LLSB--ntteee
10,10
8.85
55.08
74.83
29.32
40.00
36.37
DeepStek-RI-Distill Q.we-ll.---e
31.15
24.06
72.81
32.18
51.55
49,46
STILL-3-11.5B
31.46
25.00
75.08
86.24
32.77
53.84
50.73
Qwea225-Maab--55B-oot-t
20.00
10.00
52.50
74,20
26.84
37.78
36.89
Open-RS1-1.58
30.94
22.60
73.05
29,92
52.82
49.04
Open-R52-1.58
28.96
24.37
73.52
85,06
29.74
52.63
49.05
Open-RS3-I.5B
30.94
24.79
72.50
84.47
29.11
52.25
49.01
DeepScaleR-1.5B
38.54
30.52
80.86
88.79
36.19
58.95
55,64
Nemotron- Rew.aeeeeeeeeeeeeeeeeeee Reasoning- Baaasoning-Qwee--.ee
Qwenn- 1.5
45.62
33.85
85.70
92.01
39.27
64.56
60.17
Nemotron- Rescarch-Reasoning Swee---eeennna v2
51.77
32,92
88.83
92.24
39.75
64.69
61.70
DeepSearch-1.55
53.65
35,42
90.39
92.53
40,00
65.72
62.95
Table 2: Comparison of methods on efficiency and performance, which are trained from DeepSeek- LI--Ditiil--Qee Bestil-Qween-L.BB
Method
RLVR
Steps
Samples
Time
GPU Hours
Math Score
DeepSeck RI Ei-Distill- Qwen-1.5B
49.46
Cemotron-Rescardee descarch-Reasooingg Rcasoning-Qwert Qwen-1.55 v1
DAPO
2000
16000
60.10
Nemotron Research- Reaell-nnnnngggg Qwen-
DAPO
3000
24000
61,70
Extended Training
DAPO
+325
665.6
20.4
326.4
61.78
Extended Truining
DAPO KL
+785
1607.7
49.3
788.8
62.08
Extended Training
DAPO
+1875
3840.0
117.7
1883.2
62.02
DeepSearch 1.5B
Tree GRPO
+50
102.4
20.6
330
62.95
using 326.4 GPU hours, whereas 1,875 steps plateau 62.02% accuracy despite consuming 1,883.2 GPU hours. This reveals the fundamental limitation of depth- forth-first scaling, where performance gains become marginal as computational investment grows exponentially.
DeepSearch achieves superior results through algorithmic innovation rather than brute-force com- putation. With only 50 additional training steps, DeepSearch reaches 62.95% accuracy using 330 GPU cuurs-oupeeffrming the most extensive baseline (1,883.2 hours) while using 5.7 fewer resources. This efficiency stems from structured search that extracts maximum value from each training step by systematically exploring diverse solution paths.
Figure illustrates the training dynamics over 20 hours following 3K RLVR training. DAPO exhibits gradual linear improvement with shallow slope, while DeepSearch demonstrates more efficient learning through structured ex- ploration. The superior convergence properties suggest that RLVR bottlenecks stem from the quality of exploration rather than from insuffi- cient training time.
60.5
DAPO
60.0
DeepSearch
59.5
Average (96)
59.0
These results challenge the common assump- tion that improving RLVR performance re- quires proportional increases in computa- tional budget. Compared with the training of Nemotron-Ressaade -ertoo---eeeaacc--eessnnnn Breasoninn--wenn--e v2, DeepSearch-1.5D achieves 72x effi- ciency improvement, highlighting a qualitative shift toward algorithmically driven optimiza- tion rather than resource-drives scaling. This suggests that systematic exploration can offer a more practical and scalable path for advancing RLVR capabilities than extended high-compute training.
57.0
Time (Hours) (after training)
Figure 2: Average performance (AIME 2024, AIME 2025, and AMC 2023) of DAPO and DeepSearch after 3K RLVR training. Markers denote evaluations, while dotted lines indicate lin- ear trends.
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study of different search strategies in DeepSearch. We compare vanilla UCT with our proposed global frontier selection under varying depth bonus functions (()(s).. Reported met- rics include search statistics such as average search depth, trajectory entropy, and trajectory reward, as well as computational cost, measured by the number of iterations, average per-iteration time (in seconds), and per-tree time (in seconds). Results are presented as mean + standard deviation.
Methed
D(d(e)
Search Metries
Computationa Cost
Depth
Entrpy
Reward
Num.
Iter.
Time Per Tree
UCT Global Frontier Selection
20.11
1.23 0.20
0.82= 0.37
20086
5,83 0.21
1170.600004
As
0.4
20.28 4.80
1,23 +: 0.25
-0.63 =
187.7 16.2
5.76 + 0.19
1087.7 = 105.0
0.4,2 0.01
log(dia)
20.33 4.77
1,23 0.30
-0.65
185.5 4 15.9
5,85 4: 0.19
1080.3 102.2
0.4.A
0.85
85.7 $5.71.7777
0.4,3s
0.4,
0.01
20.83
4.71
1.31 0.30
-0.79 0.82
92.5 +: 22.5
5.48 + 0.13
506.2 114.8
0.4,A,
0.01
20.29
4.83
1.24
+ 0.76
189.3 14.7
1070.7
87.3
5.3
SEARCH STRATEGY ABLATION
Table compaaes our global frontier selection against vanilla UCT under different configurations on 1.2K samples from extremely hard ShepMathh1033k problems.
Global vs. Local Selection. Our global frontier selection (A1 = 0.4) reduces iterations by 10.4% (209.6 - 187.7) and improves trajectory rewards (-0.82 -0.65) compared to vanilla UCT, while maintaining similar search depth and entropy. This demonstrates that comparing frontier nodes across the entire tree is more efficient than traditional root-to-leaf UCT traversals.
Depth Bonus Impact. We evaluate three depth bonus functions D(d(s)): (i) Logarithmic log(d(s) + 1) provides minimal improvements, (ii) Linear d(s)) achieves the most aggressive ef- ficiency gains with 59% reduction in per-tree time (1179.6s - 480.9s) and deepest exploration (21.55 depth), but at cost of solution quality ( (-0.76 reward), (iii) Square root dd(s)/dr offers the best balance, maintaining search quality (-0.65 reward) with significant computational savings.
Uncertainty Bonus. Adding uncertainty weighting (A2 0.4) increases exploration diversity (en- tropy 1.23 + 1.31) by prioritizing high-uncertainty policy regions, but introduces computational variability (92.5 + 22.5 iterations).
Configuration Selection. We adopt /d(s)/dr with \ = 0.4, As = 0.01 as our default, balancing computational efficiency (189.3 iterations), search quality (-0.65 reward), and stable performance. This configuration eliminates UCT's redundant traversals while maintaining principled exploration through quality potential and depth guidance.
5.4 ALGORITHM EVOLUTION AND COMPONENT CONTRIBUTIONS
To understand the individual contributions of each component, we present systematic ablation study of our DeepSearch algorithm in Table 4 Starting from the Nemotron-Qwen- v2 base, incrementally add components and analyze their impact:
(i) Vanilla DeepSearch Foundation. We begin with a basic MCTS integration using simple q- value update rule:
(x)) = (mm)))(s)))) - (i. g((m)) (Send) mmx (1 (s))) + ((, I) qm(m) ((m)( (Scnd) 0) otherwise.
if gf ((m=)))(s)) (Seems) 2 0,
(8:)
This assigns constant values to nodes along correct reasoning paths but yields only limited im- provement over the baseline. (ii) Enhanced Q-Value Updates with Outcome Rewards. We re- place the simple update with our constrained backup rule (Eq.5) 5) and use outcome- advantages كز = ((*ema) for all nodes. This provides more stable credit assignment and yields meaningful im- provements. (iii) Fine-Grained Node-Level Advantages. Moving beyond outcome-only rewards, assign node-specific advantages Å q(s;) based on each node's individual q-value. This enables more precise credit assignment across different reasoning steps. (iv) Standard Advantage Normalization. We implement standard normalization as A₁ -(a,)-He where de is the standard a5+5 deviation of the rewards of the terminal nodes Send throughout the tree T. The constant prevents numerical instability when the variance is small. This stabilizes training but introduces variance- based scaling. (v) Mean-Only Normalization. We adopt mean-only normalization (Eq. 15). This

---

Published as a conference paper at ICLR 2026
Table Ablation study illustrating the step-by-step evolution of DeepSearch. Starting from Vanilla DeepSearch with a simple q-update, we progressively add wewtomeerewwrd-bseee and fine-grained advantages, standard or mean-only normalization, and frontier node selection.
Model Change
AIME24
AIME25
AMC23
MATH
Minerva
Olympiad
Avg
Nemotron Reshoonn-eeeach---e-ee Rescarch-Reasooing-- Deeasoning-wwert Qwen-I-eeBBBB
51.77
32.92
88.83
92.24
39.75
64.69
61.70
+
Vanilla DeepSearch
51.98
34.06
86.64
87.00
37.96
64.00
60.27
+ New y Update Coarse-grained Token Scores
51.04
35.73
86.48
90.66
39.14
65.23
61.38
y Updare Fine- ginengraneed Token Scores
50.52
35.52
91.70
39.71
61.85
Standard Advantages Normalization
52.60
35.00
89.30
92.44
39.29
64.99
62.27
+ Mean-only Advantages Normalization
51.98
35.73
89.06
91.88
39.58
65.71
62.32
Fmnntier Selection
90.39
92.53
40.00
65.72
Table 5: Ablation on different trajectory selection strategies. We compare using the most confi-
dent incorrect trajectory, a randomly selected incorrect trajectory, and the least confident incorrect trajectory. Confidence is measured via average trajectory entropy.
Method
AIME24
AIME25
AMC23
MATH
Minerva
Olympiad
Avg
Nemotron- Rechroon-eecechh--- Research-Reasonnng Seeeiingggww---neeeee Qwen-1.5B v2
51.77
32.92
88.83
92.24
39.75
64.69
61.70
Random incorrect trajectory
52.41
34.23
89.31
92.21
39.97
64.41
62.09
Least confident incorrett trajectory
52.16
33,77
88 68
91.96
40.11
64.73
61.90
Most confident incorrect trajectory
53.65
35.42
90.39
92.53
40.00
65.72
62.95
addresses miscalibration issues in GRPO while maintaining stable advantage scaling. (Bereket & Leskovec 2025). (vi) Global Frontier Selection. Finally, we integrate our novel frontier selec-
tion strategy (Eq. 6 which prioritizes promising expansion candidates across the entire search tree
rather than following traditional root-to-leaf UCT-like traversals.
Moreover, to justify our choice of prioritizing the most confident incorrect trajectory when no cor- rect solution is found, we compare three selection strategies in Tab. []] Selecting the trajectory with the lowest average entropy, representing the model's 's most confident mistake, achieves the best av- erage performance (+0.86 over random and +1.05 over least confident), and is the only method that consistently improves results across all benchmarks. This confirms that high-confidence incorrect trajectories expose systematic reasoning errors that benefit most from targeted supervision, whereas random or low-confidence errors tend to be noisy or uninformative.
The results demonstrate that each component contributes meaningfully to the final performance, with frontier selection providing the largest single improvement. The cumulative effect shows that systematic exploration and fine-grained credit assignment are essential for maximizing the benefits of search-augmented RLVR.
5.5 DISCUSSION: WIYY BASSED EXPLORATION HELPS
Direct rollouts from me behave like blind sampling: they quickly collapse into high-probability but low-diversity regions, rarely reaching deeper reasoning paths. As result, they fail to uncover the informative trajectories required to correct systematic reasoning errors. In contrast, MCTS provides structured exploration. Instead of distributing compute uniformly at random, allocates search effort according to frontier quality, uncertainty, and depth. This yields three intuitive benefits:
(1) Promising branches are expanded first. Frontier scoring prioritizes nodes with high value potential, preventing premature collapse into shallow modes. (2) Overconfident mistakes become learnable. When no correct solution exists, we select the lowest-entropy incorrect trajectory, expos- ing systematic reasoning failures that direct rollouts rarely sample. (3) Deep reasoning chains are reachable. Depth bonuses encourage exploration of multi-step trajectories that direct rollouts fail reach due to early-toker drift.
CONCLUSION
We introduced DeepSearch to integrate tree-based search into RLVR training, thereby addressing exploration bottlenecks that lead to performance plateaus. It achieves average accuracy of 62.95% on mathematical reasoning benchmarks, establishing a new state-of state-of-the-art for 1.5B models while using 5.7x fewer GPU hours. This suggests that systematic exploration during training can, in many cases, provide greater gains than simply extending training time, highlighting the value of scaling training breadth through algorithmic innovation rather than relying solely deeper compute.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was supported in part by RS-2024-00457882, National Al Research Lab Project.
REFERENCES
Lei Bai, Zhongrui Cai, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, Yongkang Chen, et al. Intern-sl: A scientific multimodal foun- dation model. arXiv preprint arXiv:2508. 2025.
Michael Bereket and Jure Leskovec. Uncalibrated reasoning: Grpo induces overconfidence for stochastic outcomes. arXiv preprint arXiv:2508. / 1800, 2025.
Graeme Best, Oliver M Cliff, Timothy Patten, Ramgopal R Mettu, and Robert Fitch. Dec-mets: Decentralized planning for multi-robot active perception. The International Journal of Roboties Research, 38(2-3)3316-33 2019.
Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. torest-of-thought: Scaling test-time compute for enhancing I1m reasoning. arXiv preprint arXiv:2412.09077. 2024.
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process. Advances in Neural Information Processing Systems, 37:27689-27724, 2024.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint 2504v:2500.02546, 2025.
Tuan Dam, Georgia Chalvatzaki, Jan Peters, and Joni Pajarinen. Monte-carlo robot path planning. IEEE Robotics and Automation Letters, 7(4):1 1211-111-11111 11200, 2022.
Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning small 1lms: and what doesn't. 't. urXiv preprint arXiv:2503. 16211 2025.
What works
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in iims via reinforcement learning,
2025. /tLbttpss//aaxivvoog////////////
Alhussein Fawzi, Matej Balog, Aja Huang. Thomas Hubert, Bernardino Romera-Paredes, Moham- madamin Barekatain, Alexander Novikov, Francisco R. Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 67057930):47-53, 2022.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rsthr-math: Small 1lms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519. 2025.
Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17746, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in llms through reinforce- ment learning. Nature, 645(808 1000811):633383 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint 40X00:2402.14008, 2024.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.22311 2025a.
Shenghua He, Tian Xia, Xuan Zhou, and Hui Wei. Response-level rewards are all you need for online reinforcement learning in 1lms: A mathematical perspective. arXiv preprint 020.22500002533... 2025b.

---

Published as a conference paper at ICLR 2026
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath- 103k: A scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504. / 1456, 2025c.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.088433 2021.
Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look progress in language model reasoning: Pitfalls and paths to reproducibility. arXiv preprint arXiv:2504.0700 07086, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint urXiv:2412.167200 2024.
Marco Kemmerling, Daniel Lütticke, and Robert H Schmitt. Beyond games: a systematic review of neural monte carlo tree search applications. arXiv preprint arXiv:2303.08060 2023.
Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pp. 282-293. Springer, 2006.
Guillaume Lample. Timothee Lacroix, Marie-Anne Lachaux, Aurelien Rodriguez, Amaury Hayat, Thibaut Lavril, Gabriel Ebner, and Xavier Martinet. Hypertree proof search for neural theorem proving, Advances in neural information processing systems, 35:26337-26349, 2022.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 5315- 5333, 2023.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong- Prorl: Prolonged reinforcement learning expands reasoning boundaries large language models. arXiv preprint arXiv:2505.24880 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2072 207883. 2025b.
Zihe Liu, Jiashun Liu, Yancheng He, Weixun Wang, Jiaheng Liu, Ling Pan, Xinyu Hu, Shaopan Xiong, Ju Huang, Jian Hu, et al. Part i: Tricks or traps? deep dive into rl for I1m reasoning. arXiv preprint ar arXiv:2508.082
2025c.
Michael Luo, Sijun Tan, Roy Huang. Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, et al. Deepcoder: A fully open-source 14b coder at 03-mini level. Notion Blog, 2025a.
Michael Luo. Sijun Tan, Justin Wong. Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol ol-preview with 1.5b model by scaling rl, 2025b. Notion Blog.
Chenggi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao. Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2502.0678.1 2025.
12

---

Published as a conference paper at ICLR 2026
Nicholas Metropolis and Stanislaw Ulam. The monte carlo method. Journal of the American statis- tical association, (A(247):335-341) 1949.
Yelisey Pitanov, Alexey Skrynnik, Anton Andreychuk, Konstantin Yakovlev, and Aleksandr Panov. Monte-carlo tree search for multi-agent pathfinding: Preliminary results. In International Con- ference on Hybrid Artificial Intelligence Systems, pp. 649-660. Springer, 2023.
Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reason- ing makes smaller Iims stronger problem-solvent arXiv preprint arXiv:2408.06199, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint arXiv: 2409.19256 2024.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittieseer, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587)4844488 2016.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint
2024.
Yi Su, Dian Yu, Linfeng Song. Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23822 2025.
RUCAIBox STILL Team. S:il-3-1..b-previie Enhancing slow thinking abilities of small models through reinforcement learning. 2025. URL https: thu. com/
Thinking_win wi with_ii
Aaron Tu, Weihao Xuan, Heli Qi, Xu Huang. Qingcheng Zeng, Shayan Talaei, Yijia Xiao, Peng Xia, Xiangru Tang, Yuchen Zhuang, et al. Position: The hidden costs and measurement gaps of reinforcement learning with verifiable rewards. arXiv preprint arXiv:2509.2188) 2025.
Harshil Vagadia, Mudit Chopra, Abhinav Barnawal, Tamajit Banerjee, Shreshth Tuli, Souvik Chakraborty, and Rohan Paul. Phyplan: Compositional and adaptive physical task reasoning with physics-informed skill networks for robot manipulators. arXiv preprint 15777 2024.
Zhongwei Wan, Zhihao Dou, Che Liu, Yu Zhang, Dongfei Cui, Qinjian Zhao, Hui Shen, Jing Xiong, Yi Xin, Yifan Jiang, et al. Srpo: Enhancing multimodal Ilm reasoning via reflection-aware rein- forcement learning. arXiv preprint arXiv:2506.01713, 2025.
Peisong Wang, Ruotian Ma, Bang Zhang, Xingyu Chen, Zhiwei He, Kang Luo, Qingsong Lv, Qingxuan Jiang, Zheng Xie, Shanyi Wang, et Rlver: Reinforcement learning with verifiable emotion rewards for empathetic agents. arXiv preprint arXiv:2507.03//2 2025a.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Iims step-by-step without human annotations. arXiv preprint arXiv:2312.08933 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint urXiv::203.11 /11 2022.
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with training example. arXiv preprint arXiv:2504.2057/ 2025h..
13

---

Published as a conference paper at ICLR 2026
Zehong Wang, Fang Wu, Hongru Wang, Xiangru Tang, Bolian Li, Zhenfei Yin, Yijun Ma, Yiyang Li, Weixiang Sun, Xiusi Chen, et al. Why reasoning fails to plan: A planning- analysis of long- monghooizon decision making in Ilm agents. arXiv preprint arXhv:2001.22311, 2026.
Fang Wu, Vijay Prakash Dwivedi, and Jure Leskovec. Large language models are good relational learners. In Proceedings of the 63rd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 7835-7854, 2025a.
Fang Wu, Weihao Xuan, Ximing Lu, Zaid Harchaoui, and Yejin Choi. The invisible leash: Why rlvr may not escape its origin. arXiv preprint arXiv:2507.14844 2025b.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of conpute-optimal inference for problem-solving with language models. arXiv preprint ar arXiv:2408.000 00724, 2024.
Peng Xia, Kaide Zeng, Jiaqi Liu, Can Qin, Fang Wu, Yiyang Zhou, Caiming Xiong, and Huaxiu Yao. AgentO:: Unleashing self-evolving agents from zero data via tool-integrated reasoning. arXiv preprint MXiv:25/1.1604: 2025.
Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical ex- pert model via self- iafroopprveettt.. arXiv preprint X4i::2409.121222 2024.
Feiyu Yang. An integrated framework integrating monte carlo tree search and supervised learning for train timetabling problem. arXiv preprint arXiv:2311.0mm 2023.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36:1 1809-11822, 2023.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in neural information processing systems, $4:25476-25488, 2021.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, al. Dapo: An open-source I1m reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity llms beyond the base model? arXiv preprint arXiv:2504.13888 2025.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets * Lim self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735-64772 2024a.
Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with Ilama-3 8b. arXiv preprint arXiv:2406.07334 07394, 2024b.
Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. Llama-berry Pairwise optimization for ol-like olympiad-level mathematical reasoning. arXiv preprint 2xXiv:2410.0222 028844, 2024c.
14

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL DETAILS
This section provides comprehensive details of our experimental setup, including system implemen- tation, training configurations, MCTS parameters, optimization strategies, and evaluation protocols used in our DeepSearch framework.
A.1 TRAINING DATA AND CONFIGURATION
We implement our DeepSearch system using the veRI. framework (Sheng et al., 2024). Train- ing is conducted on 16xH100 GPUs, each with 96GB of memory; evaluation is run a larger 128xH100 cluster to reduce wall-clock time. The policy model is initialized with Nemotron- Rescarch-Reasoning--e -wen--..-eee v2 (Liu et al. (2025a) (updated July 23rd). To ensure fair compar- ison with a well-aligned policy, we additionally perform DAPO-based extended training using the Ammotron-Researee Research-Rensoning Reasoning- Qwen-1.5BB initialization, with the same training configuration as DeepSearch.
Our training methodology the DeepMath-1033 (He et al. 2025c) dataset Dwain and employs a Deep prompt template that instructs the model to "Let's think step by step and out- put the final answer within (boxed() To manage computational constraints, we truncate the left side, keeping the most recent tokens with a maximum prompt length of 2,048 tokens and limiting response generation 16,384 tokens. The training process utilizes global batch size of 256 sam- ples, implemented through the DAPO-style Dynamic Batching strategy Yu et al. 2025 to optimize memory utilization and training efficiency.
A.2 MONTE CARLO TREE SEARCH IMPLEMENTATOON
Our MCTS implementation incorporates several strategic design choices balance search efficiency and solution quality. The exploration coefficient (A) for UCT Local Selection is set to 2.0, providing an optimal exploration-exploiaaee ture operates with a maximum depth of 64 levels, with each node allocated 256 tokens and expanding 8 children during the expansion phase. For entropy-based selection, we estimate average trajectory entropy using only tokens that appear in the response, rather than the entire per-position vocabu- lary, for computational efficiency. This average token negative log-probability along the sampled trajectory serves as proxy for decision confidence.
trade-off for mathematical reasoning tasks. The search architec-
To enhance search effectiveness, the system employs Global Frontier Selection for backtracking and applies a square-root function to depth-based bonuses, encouraging deeper exploration when beneficial. The global Xa parameter is configured to 0.01 for our frontier priority scoring, while an overlong buffer of 4,096 tokens with penalty factor of 1.0 accommodates lengthy reasoning chains typical in complex mathematical problems.
A.3 ADVANTAGE ESTIMATION AND OPTIMIZATION
For advantage estimation, we implement the Grouped Relative Policy Optimization (GRPO) (Shao et al. (2024) estimator with sibling mean normalization to ensure stable learning dynamics. The Q- value soft clipping mechanism operates temperature of 1.0 with the maximum q-value magnitude set to 1.0, while incomplete trajectories receive a penalty score of - -1.0 to discourage premature termination. Standard deviation normalization is disabled to prevent numerical instability during training.
The actor model is optimized using AdamW with a conservative learning rate of 1 x 10-6 and 10 warmup steps, along with weight decay of 0.1 and gradient clipping at 1.0 to stabilize convergence. We follow the Clip-Higher strategy in DAPO (Yu al., 2025) and set the lower and upper clip- ping thresholds to 0.2 and 0.28, respectively, with a clipping ratio of 10.0. Training proceeds with mini-batches of 32 samples per policy update using token-mean loss aggregation, while dynamic batch sizing accommodates up to 18,432 tokens per GPU. The entropy coefficient is set 0 for pure exploitation, and the KL divergence loss is disabled to maximize performance on the target mathematical reasoning tasks.
15

---

Published as a conference paper at ICLR 2026
A.4 SAMPLING AND REWARD CONFIGURATION
During rollout generation, we set the sampling temperature to 1.0 and top.p p to 1.0, and disable top k filtering to maintain diverse response generation. The system generates rollouts per prompt, aligning with the expansion width parameter, within a context length of 18,432 tokens. This config- uration ensures comprchensive exploration of the solution space while maintaining computational feasibility. During evaluation, uniformly use low temperature of 0.6 and top.p of 0.95.
custom mathematical scoring function based on from the math.dapo. pp module, designed to evaluate mathematical reasoning accuracy. We extract the final boxed answer by locating the last occurrence of (boxed{] in the trajectory and apply the same text-normalization logic as veRL's DAPO recipe to both prediction and ground-truth. The reward mechanism handles responses up to 16,384 tokens, following ProRL. (Liu et al. 2025a) and ensuring consistent evaluation across varying response lengths.
Our reward system implements
A.S TRAINING PROTOCOL
The complete training protocol spans 100 steps with model checkpointing performed every steps. This frequent checkpointing strategy ensures robust model preservation and enables detailed analysis of learning progress throughout training.
B JUSTIFICATION OF DEEPSEARCH DESIGN CHOICE
B.1 MAXIMUM DEPTH AND EXPANSION LENGTH
Data-driven determination of maximum search depth. We set the maximum search depth through two jointly defined factors: rollout budget 16,384 tokens and an expansion length of 256 tokens per node, yielding an approximate upper bound of 16,384/250 a 64 expansion steps. In practice, many trajectories terminate earlier, so this acts as an upper bound rather than a hard requirement. Most importantly, this limit is derived from empirical statistics of the reasoning- length distribution in Deep 103K, which confirms that 16,384 tokens comfortably cover full reasoning chains for nearly all problems. The cap, therefore, reflects principled trade-off between modeling requirements and training efficiency-not an arbitrary constant.
Empirical validation of 256-token expansion length. We evaluate several expansion lengths (128, 256, 512, 1024 tokens) and observe clear trade-off:
Search-space exploration: Longer expansions reduce tree breadth and make MCTS less likely to discover correct solution paths.
Computational cost: Shorter expansions significantly increase prefix re-encoding ("pre- fill") costs, which dominate GPU time in MCTS exploration.
The 256-token step size consistently gives the best balance between coverage, training stability, and computational efficiency across all experiments.
Summary. Both design choices of the depth cap and the expansion length arise from data-driven analysis and controlled comparisons. Together, they ensure that DeepSearch maintains wide search coverage while avoiding unnecessary computational overhead.
B.2 RATIONALE BEHIND THE ASYMMETRIC Q- VALUE BACKUP RULE
Our constrained backup rule in Eq. maintains for every intermediate node scalar q-value repre- senting its estimated tendency to lead toward a correct solution under the current policy and search process. When terminal trajectory returns reward signal (+1 for correct, -1 for incorrect or incomplete), the update is asymmetric, depending on the relationship between the node's 's currett q-value and the reward.
16

---

Published as a conference paper at ICLR 2026
Same-sign reinforcement. When node's existing q q-value and the terminal reward share the same sign, the update simply adds the depth- septh-decayed reward. Thus, nodes that frequently appear on correct trajectories naturally accumulate stronger positive evidence, while nodes that are consistently involved incorrect trajectories accumulate stronger negative evidence.
Negative-to-posie -o-piiiive transitions. If node previously held negative q-value-iidicatiigg it only appeared in incorrect trajectories-but it now lies on correct trajectory, we discard the accumulated negative value and overwrite it with the decayed positive reward. Intuitively, once node is ever shown to lead to a valid solution, we reclassify it as potentially valuable rather than penalizing it for past failures.
Desitive-tonnegativ suppression. If a node already has a positive q-value, meaning it has ap- peared least once in successful reasoning path, do not propagate new negative signals to it. A node proven capable of contributing to a correct solution should not be downgraded due to occasional failed attempts, which could arise from stochastic expansions or unrelated branches.
Guaranteed invariants.
This asymmetric update mechanism enforces two key invariants:
1. Any node that has ever appeared correct trajectory retains a non-negative q-value.
2. Only nodes that have never observed on a correct trajectory are allowed to accumulate stable negative values.
Together, these invariants ensure that the search process robustly separates "potentially useful" nodes from those that consistently lead to dead ends, thereby stabilizing MCTS exploration and preventing destructive interference from noisy, incorrect rollouts.
B.3 EVOLUTION OF CACHED Vs. UNSOLVED PROBLEMS DURING ADAPTIVE TRAINING
To quantify how adaptive training reallocates computation over time, we track the ratio of cached problems (those with at least one verified correct trajectory discovered by MCTS) to unsolved prob- lems (those with no verified correct trajectory). Because Round begins with an empty replay buffer, the cached proportion is initially 0:1. As training progresses, MCTS increasingly cracks problems in the hard subset, causing cached solutions to accumulate round by round.
Table 6: Number of cached unsolved problems during the 5-round adaptive training.
Round
# Cached
Unsolved
Cached Ratio
0
13,658
0%
765
12,893
5.6%
3
1,452
9,423
13.4%
2,243
7,423
23.2%
5
2,894
5,829
33.2%
The monotonic increase in cached solutions demonstrates that DeepSearch progressively transfers problems from the unsolved pool into the replay buffer as MCTS uncovers verifiable solution paths. By Round 5, one-third of the hard subset has at least one cached solution, while the unsolved set contracts to concentrated collection of extremely difficult tail cases. This behavior validates our description in Sec. 4.2: adaptive training naturally shifts computational effort toward the most chal- lenging remaining problems, enabling more efficient use of MCTS and contributing to the strong sample efficiency shown in Table 2 and Figure
B.4 ON THE CHOICE OF FIXED FILTERING THRESHOLD 8
The filtering threshold 8 25% used in adaptive training pipeline is simple yet empirically effective heuristic. At each round, we compute each problem's 's success rate using Passl0k with K 4 direct rollouts and retain only those samples whose success rate falls below 5 for inclusion in the next hard set. This mechanism filters out problems the model has effectively mastered while preserving sufficiently challenging cases that benefit most from training-time MCTS exploration.
17

---

Published as a conference paper at ICLR 2026
Although one could design adaptive 8 schedule, such as based on training progress the evolving difficulty distribution, we intentionally avoid such complexity for two reasons. First, adap- tive thresholding would introduce additional schedules and hyperparameters (e.g., functional forms across rounds and smoothing coefficients), complicating the design space and obscuring the specific contribution of MCTS- S-based exploration. Second, our primary aim is to demonstrate that even der fixed, straightforward hard-sample selection rule, integrating structured tree search into RLVR training already yields stable and significant gains.
Thus, while adaptive thresholding remains a promising direction aligned with curriculum-style learning, we prioritize methodological simplicity and clear attribution improvements in this work.
C COMPUTATION PROFILING
To better understand the computational characteristics of DeepSearch, we profile the per-tree run- time of our final configuration (λ₃ 0.4, A3 0.01). with components ranked from smallest to largest cost in Tab. 7 The results show that all CPU-side operations-iicluding prompt construc- tion, terminal-state checking, rank synchronization, input preparation, node expansion, and output cost-processing- collectivvlyy account for only few milliseconds and are negligible relative to the total cost. contrast, GPU-based policy inference overwhelmingly dominates the runtime, with Egenerate 1060.76 76 ms comprising more than 99% of the per-tree computation. This profiling firms that the primary bottleneck lies in sequence generation rather than in MCTS logic, implying that further speedups will mainly come from optimizing model inference (e.g., batching, KV-cache reuse, speculative decoding) rather than from modifying the search algorithm's 's structure.
Table 7: Computation time breakdown per MCTS tree under the final search configuration (Aı 0.4, A3 0.01). Components ranked from smallest to largest cost. Times are reported in milliseconds as mean = standard deviation.
Component
Time
Output post-processing ((proc.outuui
0.01 + 0.00
Rank synchronization (fsynerranks)
0.07 0.01
Terminal-state checking (teheck.terminal)
0.09 + 0.02
Prompt construction (fget.promppss)
prompts)
0.38 0.08
Node expansion (Fexpand_nnde)
0.96 0.18
Input preparation (fprep inputs)
1.89 + 0.36
Policy inference / generation (1 (Egenerate)
1060.76 + 106.59
D PSEUDOCODE OF DEEPSEARCH
Algorithm [ presents the complete DeepSearch framework, integrating MCTS-based exploration with adaptive training and replay buffer management. The algorithm operates via iterative finement, progressively allocating computational resources to challenging problems while preserv- ing previously solved solutions through intelligent caching. This integrated approach focuses training-tine time exploration, enabling models to learn from both correct solutions and systematic ex- ploration processes rather than relying solely outcome-based supervision.
18

---

Published as a conference paper at ICLR 2026
Algorithm DeepSearch with Global Frontier Selection and Iterative Filtering
Require: Initial policy Tg training set Prain verifier filtering threshold
1: Initialize (()) - { E Drain Pass 1 @K(2,, TTpr)) < ), J. R()) = 0
2: for training iteration
0, 0, ..... do 3: Initialize training trajectories T Ø
4:
for each batch g( D hard do for each problem (3 do
5:
6:
(z, trached)
( then
Use cached solution
7:
Tx f- ((acched] } DirectRollouts(r,
T T( T U.x
8:
9:
else
Apply full MCTS search
MCTS Search:
11:
Initialize search tree T with root node
12:
for rollout iteration & = 1, ....... do
13:
if k then
>> Initial expansion from root
14:
Select root node s = x for expansion
15:
else
16:
Global Frontier Selection:
17:
Compute frontier set F = {s T [(s) 0, & Send, d(s) dr}
18:
Compute frontier priority scores (Eq. Select node s" arg max (s)) for expansion
20:
end if
Local Expansion with UCT Selection:
22:
Generate candidates {sy Jン=ı Tt (. o,-) from s*
Continue expansion until terminal nodes ((k) are reached
23:
24:
Evaluation with Entropy-based Guidance
Partition: (k) = {s E s(k) V(s) = 1,. ((k) S {s E &(())
25:
end V(s)
0}
26:
if | correel 2 1 then
Extract trajectories T(x) from search tree
28:
T - T T(x)
else
30:
Select most confident negative: Sieg = arg min ((k) At(ss))
end if
32:
Heuristic Score Backup:
33:
Select trajectory t* (correct solution or t(siig))
34:
Assign terminal rewards (Eq.
35:
for each node s, t" do
36:
Update Q-values using constrained backup rule (Eq. 5 end for end for end if
37:
38:
39:
Replay Buffer Update:
41:
if MCTS found correct solutions but Pass @K(x, To(i)) < 8(0) then
42:
Add (x,, (corre) to R(++1) for any correct teurreet E T(x)
end if
44:
end for
45:
Policy Update:
46:
Update policy Tot+)) using Tree-GRPO objective on ( (Eq. 13 and Eq. 14
end for
48:
Re-evaluate and filter:
D(++1) hard
fi E D hard
Pass (ess, Taut)) < 5(++1))
49:
end for
19

---

Published as a conference paper at ICLR 2026
E QUALITATIVE CASE STUDIES
To complement the benchmark results in the main text, we provide representative cases between our DeepScarch-I.5B and Menotron-Researcc-Reee
v2 on the standard math reasoning benchmarks. Each case study is selected from different dataset and illustrates typical differences in reasoning behavior, including hesitation patterns, emror correction, and final answer reliability. For each problem, we show the original question and the modell-enerated reasoning traces. For brevity, present lightly abridged traces with minor formatting edits, while preserving the essential reasoning.
Across these 6 examples, DeepSearch--L5B consistently reaches the correct solution with shorter, more stable reasoning chains, while the Gemooron-Researcee Seaechh--Rassnnn--eeeeeee into long "wait"-style hesitation loops, revisits the same partial calculations, and in several cases converges to an incorrect final answer despite extensive deliberation. For instance, the symbolic- logic example from MATH, explicit hesitation markers, whereas DeepSearch- 1ercarccc-1.5B solves the problem with two orders of mag- nitude fewer such tokens while maintaining algebraic consistency and arriving at the correct value. This pattern holds uniformly across different types of problems, highlighting that training maining-time time ex- ploration not only reduces unnecessary verbal dithering but also improves robustness of multi-step mathematical reasoning.
v2 often falls
SecootoonRReearcc--Reeeeeee weweasoonnnnn-e-----seee Qwen-1.5p v2 emits over a thousand
MATH (Symbol Logic / Constraint Satisfaction)
Question. The symbols A, D, o, represent four different integers from to 9. Using the equations below, what is the value of ??
+
A + A +o+ototo
A + +
DeepSearch-1.5B.
Okay, have this problem where different symbols represent different integers from 1 The symbols triangle (A), square (L)), diamond (o), and clubsuit The given equations are:
++0=+
2. A A ++o+0+0++
3. A A +
And I need to find the value of [T. Let's write these equations more clearly. [...]
ng a = and c 10 2 = d= From equation + d = 5
b=3.
Thus, the values
5
2-d
All values are distinct integers from 1 Therefore, the value of D (which is b)
The value of is
Cemotron-Reeearch-Beeee -wweoniinn-Qee-..BBBBB v2.
Okay, so need to find the value of O using the given equations. Let me read the problem again carefully.
We have four different integers: A, D. o, +. each representing distinct integers from I to 9, The equations given are:
+ + A = o + (which five diamonds)
A
[...]
Wait 5 +3 is 8? Wait +3 is 8? Wait 5 +3 3 is 8? Wait 5 +3 is 8? Wait +3 8? Wait +3 Wait 8? Wait is Wait 5 +3 Wait +3
20

---

Published as a conference paper at ICLR 2026
Wait 5 +3 is 8? Wait +3
is is
8? Wait 5.43 is 8?
Wait 5 +3 is 8? Wait 5 +3 is 8?
Wait 5 +3 8? Wait 5 +3 is 8? Wait 5 is 8? Wait 5 53 is 8? Wait 5 +3 is 8?
Wait 5 +3 8? Wait 43 8? Wait 5 43 is 8? Wait 5 43 8? Wait 5 43 8?
Wait 53 Wait 5 43 Wait 5 +3 Wait 5 43 8? Wait 43 is 8? [...]
3. 2a where c, solving gives 2.5, hut since must be integer,
find
Using substitutions and solving the system, find the values consistent and inte- gers. Ultimately, value confirmed as 5.
Thus, the value of A is
Olympiad (Number Theory Divisor Sum)
Question. Compute the integer n such that 2009 < 72 < 3009 and the sum of the odd positive divisors of is 1024.
DeepSearch-1.5B.
Okay, need to find integer such that 2009 3009, and the of its odd positive divisors is 1024. Hmm. me break down problem and think through it step by step.
First, let me recall that sum of the divisors function multiplicative, since we'r re talking about odd divisors, maybe it's better to consider the prime factoriza- tion of n. [...]
2009 < 2 x 651 < 3009, we find = 2 gives n = 651 x 4 = 2604, which lies in the required range.
Verifying the sum of divisors of the odd part (which are the divisors of 651) con- firms it sums to 1024, and 2604 satisfies the conditions.
Thus, the integer n 2604
Nemotron-Researchh Reaerie--eeeerrrdeeeeninnnnnnn -.e v2.
Okay, so need to find an integer such that 2009 < r < 3009, and the sum of its odd positive divisors is 1024. Hmm, let's break this down.
First, the condition is about the sum of odd divisors being 1024. know that the sum of divisors function is multiplicative and related to the prime factorization. [...]
Wait Maybe I made an error earlier, and m = 1024 allowed? No, m must be odd.
Wait Wait maybe = O: n m 889, between 2009? No, 889 < 2009.
Wait problem might have no solution? But it says there exists n, so likely made mistake somewhere.
Wait Re-examining (m) = 1024. Maybe m = 511 2 1022 even, invalid.
Wait Maybe m = 512 not allowed.
Wait Wait perhaps m 1024 1023, 1536.
Wait 1536/1.5 1024. No.
Wait Wait Alternatively, perhaps 25mmthinng No.
Wait Mayhe the answer is 1024? No, must be hetween 2009 and 3009.
[..]
but since problem states exists, perhaps m = 889 and a = 1778 invalid, but nearest is n 2017, Σ a 2160 * 1024. Thus perhaps no solution.
But as per problem, let me take = 1024 x something, no. Wait final answer 2048? No, 2048 is even 1025 * 1024.
Wait Let me accept m 889, 2, n 3556, so: 3556
21

---

26 - Published as a conference paper at ICLR 2026
AIME25 (Geometry/Polyoo)
Question. Let A1A2A3 A11 be an 11-sided non-convex simple polygon with the follow- ing properties:
For every integer S i < 10, the area of AA;A A++1 1 equal to 1.
For every integer i 10, cos (A+))
12
13"
The perimeter of the 11-gon A, A2A3 A1i is equal to 20.
Then A1 + A11 my/ where n. P, and are positive integers, n is not divisible by the square of any prime, and no prime divides all of m, and Find m + + q. DeepSearch-I.5B.
Okay, let's to tackle this problem step by step. It seems bit involved, but let's
break it down.
First, let's summarize the given information:
We have an I 1-sided non-convex simple polygon with specific properties:
1. For every integer S I 10, the area of triangle Ac44 equal to 1. 2. For each 2 < - 10, the cosine of angle Ac41 is 12/13. 3. [..]]
By introducing variables for distances A1 and exploiting the constant-arms and constant-cosinc constraints, we derive system of equations tying together the edge lengths around the polygon. Matching these with the perimeter condition and carefully resolving the resulting quadratic relationships,
express
9v5
A1 + A1A11 = d2 + d11 = a
Therefore, the values
9, m 5. p 1, 4, leading
m + +
19
Cemotron-Researche -eaa--eean--nee-----ee v2.
Okay, let's try to solve this problem. It's bit complex, I'll need to break it down into manageable steps. First, let's list the given conditions:
We have 11 11-ided non-convex simple polygon A A2 An The key prop- erties:
1. For each integer 2 S S 10, the area of triangle AA;A A,+1 is equal to 1. 2. [...]
Each pair (æi, x(+1) satisfies 2ę with from to 10, giving
26
The indices range over the triangles
3
26
26
26
12
5
13 =
F10
where x; |AAA||-
[...]
Wait the problem states for each integer 2 S S 10, triangle area and cosine condition. So runs from to 10.
Each triangle relates [4,41] | |Ai41A1
26 5
Therefore:
For i 2: 22233 26
3
3: 2324
== 10: 210011 [...]
Wait the problem states "A,Az + ArAn". so 10/3 is answer? problem states it's expressed as (myn n p)/q. Maybe I was wrong somewhere.
22

---

Published as a conference paper at ICLR 2026
Wait but according to my deductions, it's 10/3, but let me verify.
Wait perhaps my area assumption was wrong? Wait no, the problem states cos(O) 12/1 13, etc.
Wait let me double-check the perimeter condition. Wait given 22 + If we need A, + An = 42 + 23 = 10/3.
10/3 and 22 = 26/5.
form (m/n p)/g² Wait but it's rational. Mayhe made mistake in the earlier algehra.
Expressed as 10/3, which is already
[..]]
Eventually, after struggling with algebra not reconciling the radical form requested by the problem, incorrectly conclude
10
answer
3
The final answer is
AIME24 (Complex Number/Roots of Unity)
Question. Let w * 1 be 13th root of unity. Find the remainder when
II (2 k=0
2 + 2k)
is divided by 1000.
DeepSearch-1.5B.
Okay. let me to solve this problem step by step. The problem is: Let & be a 13th root of unity. Find the remainder when the product from to 12 of (2 2w + 25) is divided by 1000.
So, first, let write down what is given, is 13th root of unity, w 13 and w بf for S k 12.
We need compute the product
II 2 2n + w 2t)).
[..]]
Thus, evaluating suitable polynomial the roots, we use
II (zh a) = (a 1))
where } are the 13th roots of unity. Setting up expression carefully, we find
= (1 + 1)
64 64i,
B (1 -)) 64 + 644,
and the product evaluates to
(4 1) 8321.
Finally, the remainder modulo 1000 is
8321 mod 1000 321.
Thus, the final answer is 321
BennttonnResarrch-Renninnnnnnnn Qwen-.... v2.
23

---

Published as a conference paper at ICLR 2026
Okay, let's see. I need to find the remainder when the product
12 II
2n w 2
k==
divided by 1000, where w o is a 13th root of unity. Hmm. First, let me note that w is 13th root of unity and satisfies w 1.
[...]
Therefore,
I 1 [(= 1) - (a - 11117 1) Wait, wait wait, wait: Wait
the product I ... is equal to l'some argument).
Wait, let me clarify:
Let mc denote f(w) T w²) w - Then
[I - w²) = (= 1)
[...]
Summarizing the (incorrect) reasoning:
2. Rewrite 2k 2 + (w* 1)2 + 1.
3. The product becomes 1 ((w* 1)2 + 1].
Recognize this (incorrectly) as evaluating polynomial roots unity and conclude it equals 13.
The resulting product modulo 1000 then taken to 13.
Thus, the answer is 13
AMC23 (Complex Number Optimization)
Question. Consider the set of complex numbers satisfying [1 + + =21 4. The maximum
value of the imaginary part of can be written in the form ym where m and n are relatively prime positive integers. What is +??
DeepSearch-1.5B.
Okay, let's try to solve this problem. The problem states: Consider the set of complex numbers satisfying |1 + 2²1 4. We need the maximum value of the imaginary part of 2, express it find Alright, let's denote complex number. Let me set x: yi, where and are real numbers.
y with m, coprime positive integers, and
[...]
By setting & = 0 and solving, we find the maximum imaginary part
v19
y
Thus, the maximum imaginary part fore,
1 2
leading to m 19 and n 2. There-
+ = 21.
21
v2.
Okay, so need find the maximum value the imaginary part of complex Then express this maximum value as /m/n where m and are coprime positive integers and find m + n.
number 2 such that [1 +
2$1 +
First, let me consider z as a complex number. Let me denote 2 = I + yi where æ and are real numbers. Then can express + + 2² in terms of x: and and then the modulus condition. [...]
24

---

Published as a conference paper at ICLR 2026
Thus maximum imaginary part is 39/96 so m = 39, n 62 Check ged 6) =
3, so reduce: 39/3 13, 6/3 = 2, so 13/2 under the square root? Wait no:
Wait v39/6 already simplest. Wait wait:
Wait y² 13/12, so
/13/12 v13/(2/3)
/39/6. Yes, rationalize:
V13
V13v3
V39
2v3
2-3
Thus maximum value is v39/6. So
39, n 6. But check if coprime: 39
and have ged 3, reduce:
39/3 = 13, 6/3 2, so v 18/2 something? Wait no, v39/6 is reduced.
Wait v39 [...]
The maximum value of the imaginary part of given the condition |1 11+++2* +²| is (incorrectly) found to be simplified to
13
2
so 13 and 2, leading
15.
Thus, the final answer is 15
Minerva (Astoplyysics/Speetrosooppp
Question. Preamble: A spectrum is taken of single star (i.e., the observed spectral lines is one from oxygen whose rest wavelength is 5007 A. The Doppler shifted oxygen line from this star is observed to be at wavelength of 5012 A. The star is also observed to have proper motion, μ4, of are second per year (which corresponds located at distance of 60 pc from the Earth. Take the speed of light to be 3 x 10 meters second.
in binary). Among
1.5 10-13 13 radians per second of time).
What is the component per second)?
star's velocity parallel to vector to the Earth (in kilometers
DeepSearch-1.5B.
Okay, let try figure this out step by step.
First, let's parse the problem.
We have a star with a single spectrum observed. One of its spectral lines (oxygen) has its rest wavelength at 5007 Angstroms (A). When observed, this line is shifted to 5012 Å. We also know the proper motion y aresecond per year, distance is 60 parsecs, speed of light is given as x 10 m/s. [...]
Using the Doppler formula for non-relativistic velocities,
5012 5007 10 m/s 2.9958 m/s 299.58 km/s 300 km/s.
A
5007
Thus, the component of the star's velocity parallel its line of sight approxi- mately
300 km/s
Final Answer:
The radial velocity component is approximately 300 km/s.
-estooo--eeeea---eeeaeee -weng---wee-1..1 v2.
Okay, let's see, problem about finding component the star's velocity parallel to the vector towards Earth. Hmm.
First, let me read the problem again:
We have with a rest wavelength of oxygen line at 5007 Angstroms. A Doppler shifted line is observed at 5012 Angstroms. The star has a proper motion
25

---

Published as a conference paper at ICLR 2026
aresccond per year, and distance 60 pe. Speed of light is given x 10 m/s.
[..]
So 2.844 10 m/s x10 km/(m//) wait
Wait 2.844 x 10 m/s 2.844 x 10-4. km/10² m) (1c3 s/km? Wait let think:
Wait m/s 0.001 km/s? no, wait m/s (1/1000) km/s 0.001 km/s.
Wait so 2.844 10 m/s 2.844 10-1 0.001 km/s? Wait wait:
Wait m/s = m/s (1/1000 km) = 0.001 km/s.
Thus, 2.844 x 10 m/s mishandled, and the final radial velocity estimate drifts.
[..]
After approximate reasoning. end up with value around km/s.
Thus the answer approximately 284 km/s.
The component of the star's velocity parallel to its direction is approximately 284 km/s.
Final Answer
The component of the star's velocity is 284 km/s
LIMITATIONS AND FUTURE WORK
A critical next step extend DeepSearch beyond mathematical reasoning domains with dis- tinct verification mechanisms. This includes developing approximate verifiers for subjective tasks, exploring human-in-the-loop validation for complex reasoning chains, and investigating transfer- learning approaches that leverage mathemmtticll-ressome capabilities for broader problem-solving tasks. Research into domain aggnosicc reward functions and verification strategies could significantly expand the framework's applicability.
Another promising direction is learning the MCTS components themselves. In this work, the fron- tier priority function and expansion width are fixed heuristics selected through systematic offline experiments, as they offer stable performance without introducing additional optimization complex- ity. Making these components learnable would require fundamentally different training paradigm, such as an AlphaZero-style controller that jointly optimizes search and policy. Exploring end-to- end trainable search guidance remains an exciting opportunity for further enhancing training time exploration.
26


Reviews:
[{"review_id": "dqvVV7ctSc", "reviewer": "Reviewer_2V8H", "summary": "This paper examines an assortment of correlated methods to better scale RLVR for use in reasoning problems.  The paper forwards three interventions, as shown in the overview Figure 1: (1) DeepSearch-MCTS, a means of deciding how to appropriately expand nodes in MCTS; (2) Adaptive training to pair with (1) by applying a tree version of GRPO; and (3) a replay buffer to enhance the study of difficult and successful trajectories.  Experiments by the authors show that the interventions taken together are synergistic and lead to performance improvements beyond performance, including a half-magnitude savings in rollout inference costs.", "questions": "* Is  _temporal decay_ (L152) better termed as _depth decay_ (or just make an association to a _discount factor_ in RL rewards)?\n* Your Eq. (7) is a bit hard to follow with three cases; can you just help narrate this prose text instead?\n* How much does $N(s)$ influence the computation and search?  It is implied that it is helpful but it's not clear whether it plays much of a role (repeated visits).\n* Keeping track of a frontier could be expensive.  How does this figure into the (asymtotic and real wall-clock) cost?  What manner do you operationalise this for (assumed GPU bound) processing?\n* L241: are there borderline cases where no good success traces emerge, and as such the adaptive training fails?  This step seems to assume there are always successes that can be judged.\n* L269: How important (e.g., what %age was thrown out) by the condition \"remove data containing garbled text or infinite repetitions\"?  Given evidence that such filtering does not invalidate your narrative.\n* L274: Can you show us how the ratio between cached problems and unsolved problems evolves during your adaptive training (and without, as an ablation)?\n* L289: How many of q-values are (severely) clipped?  What does the distribution of q-values look like, post-clipping?  This seems needed to really support that such an intervention is needed.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The three interventions are well-motivated and illustrated conceptually well in Fig 1.\n* The authors explain the rationale of some parts of their method (L209-215), to bring together the motivation and explain design choices.", "weaknesses": "Overall, while the methods disclosed seem convincingly effective, the vanilla UCT baseline also does quite well, and the complexities of the methods proposed give many dimensions of freedom for possible overfitting (as confidence intervals overlap considerably).  While the authors do test with SOTA baselines, I'm not convinced of the introduced mechanism's impact on the results for these reasons.\n  \n* The mechanisms used seem ad hoc and not particularly well-justified (e.g., S2.1 could max depth $n$ be defined conditioned on or calculated from some other means?)\n* Why do you select the most confident negative example?  Can you justify it (L141-143 seems insufficient)?\n* Using S2.2 to back-propagate up the best negative and positive cases seem ok but divergent.  Why would you want to handle both in the same way?\n* It would be instructive for you to (briefly) explain UCT _before_ L181.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3zSWUl72Ct", "reviewer": "Reviewer_TRXC", "summary": "This paper addresses performance plateaus in Reinforcement Learning with Verifiable Rewards (RLVR) by integrating Monte Carlo Tree Search directly into the training process rather than limiting it to inference. The authors propose DeepSearch, which features: (1) a global frontier selection strategy that prioritizes promising nodes across the entire search tree rather than following traditional root-to-leaf UCT traversals, (2) entropy-based guidance to identify confident incorrect reasoning paths for supervision, and (3) an adaptive replay buffer with solution caching to focus computation on challenging problems. Evaluated on mathematical reasoning benchmarks (AIME 2024/2025, AMC 2023, MATH500, Minerva, Olympiad), DeepSearch achieves 62.95% average accuracy on a 1.5B model, representing a 1.25 percentage point improvement over the previous best while using 72× fewer GPU hours than extended training approaches. The results suggest that systematic exploration during training is more effective than simply scaling training depth.", "questions": "- Computational breakdown: Could you provide a breakdown of where computational time is spent (tree expansion, policy inference, Q-value updates, etc.)? This would help identify further optimization opportunities.\n- Parallelization: How does MCTS tree construction parallelize compared to standard rollout generation? Are there architectural choices that could improve throughput?\n- Comparison fairness: The extended training baselines use standard DAPO. Would extended training with better exploration strategies (but without full MCTS) close some of the gap? This would help isolate MCTS's specific contribution.\n- Learned components: Have you considered learning any of the MCTS components (e.g., the frontier priority function or expansion width) rather than hand-designing them?\n- Multi-task learning: Could a single DeepSearch model be trained across multiple reasoning domains simultaneously, with shared MCTS infrastructure?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n- The core insight of embedding structured search into the training loop rather than limiting it to inference is compelling and represents a meaningful shift in perspective\n- The global frontier selection strategy is an interesting departure from traditional UCT traversals, addressing computational efficiency while maintaining search quality\n- The combination of entropy-based negative example selection with adaptive replay buffers creates a thoughtful approach to computational resource allocation\n\nQuality:\n- The experimental evaluation is thorough, with strong baselines including very recent work (Nemotron v2, DeepScaleR)\n- The efficiency analysis is particularly valuable—demonstrating 5.7× fewer GPU hours while achieving better performance makes a strong case for the approach\n- Table 4's evolutionary ablation study effectively illustrates how each component contributes to the final performance\nThe search strategy ablation (Table 3) provides useful insights into design choices\n\nClarity:\n- Figure 1 provides an excellent high-level overview of the framework, making the approach accessible\n- The motivation is well-articulated—the observation about training plateaus in extended RL training resonates\n- The method description flows logically from MCTS components through training strategy\n- Algorithm 1 in the appendix offers comprehensive implementation details\n\nSignificance:\n- Addresses a genuine bottleneck in RLVR that has been observed across multiple recent works\n- The efficiency gains are substantial and practically important given the computational costs of modern reasoning model training\n- The framework appears general enough to potentially benefit other RLVR applications beyond mathematics", "weaknesses": "Experimental Analysis:\n- While the average improvement is clear, it would be illuminating to characterize which types of problems benefit most from the approach. Are there problem characteristics that predict when MCTS exploration is particularly valuable?\n- The comparison with extended training is compelling, but I wonder how performance would compare if the extended training baseline also incorporated some of DeepSearch's innovations (like adaptive filtering) without full MCTS\n- Table 2 shows extended training plateauing, but it would be interesting to see if combining extended training with DeepSearch yields further gains, or if there are diminishing returns\n\nMethod Design Choices:\n- The entropy-based selection for negative examples (Eq. 2) is intuitive, but I wonder if there are alternative criteria that might be equally or more effective. Have you explored selecting based on value function disagreement or other uncertainty measures?\n- The β parameter for DirectRollouts when cached solutions exist seems important but isn't thoroughly analyzed. How sensitive are results to this choice?\n- The filtering threshold δ is set to 25%—could this be adaptive based on training progress or problem difficulty distribution?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uVioLkp2vi", "reviewer": "Reviewer_e3YK", "summary": "The paper integrates MCTS directly into RLVR training to scale breadth, not just depth. By doing so, it broadens exploration and enables finer-grained credit assignment by propagating verifier signals back through the search tree. Practical search and data-filtering heuristics improve trajectory quality, and training employs a tree-aware GRPO in which per-node Q-values from MCTS backups serve as advantages after sequence-level normalization. The approach delivers higher Pass@1 with substantially better compute efficiency on math benchmarks for 1.5B-scale models.", "questions": "The paper has many moving parts. What is the main conclusion of the paper? Apart from the methodology itself, what is the exact shifted paradigm? If I understand correctly, it should be that brute-force exploration is not enough, and we need a tree-based exploration? Is there an intuition on why tree-search based explorations are better?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The main strength of the paper is that it targets a very strong baseline. The paper improves an already competitive 1.5B model (Nemotron-Research-Reasoning-Qwen-1.5B v2) and contrasts against a solid RLVR baseline (DAPO), making the incremental gains meaningful at this scale. Results suggest simple RLVR fine-tuning provides limited uplift, while the proposed training-time MCTS adds ~1–2pp Pass@1. Design choices are justified. Most heuristics are motivated, and the ablation is informative and clearly shows which components matter.", "weaknesses": "1- there is not much related work! for targeting better credit assignment there are many paper addressing brining test time compute for more fine grains credit assignment.\n2- the biggest gain in table 4 shows for \"Frontier Selection\" and rest of the justifications, such as better credit assignment, remain very substantially small.\n3-  2% improvement is not a significant improvement. It is unclear what the main conclusion of the paper is.Additionally, how can we translate this paper to a novel RLVR algorithm for general application? The paper claims a paradigm shift, but it is unclear why this is a new paradigm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)