Enhancing Language Model Reasoning with Structured Multi-Level Modeling
Authors: Siheng Xiong
Ali Payani
Faramarz Fekri
Venue: ICLR 2026 Poster (2026)
Keywords: Chain-of-Thought reasoning
Direct Preference Optimization
Process supervision
Twisted Sequential Monte Carlo
Large language models
Primary area: foundation or frontier models, including LLMs

Abstract: Inference-time scaling enhances a model’s reasoning by extending its chain-of-thought (CoT). However, existing approaches typically rely on a single policy trained with outcome-reward reinforcement learning (RL), which often suffers from long-horizon plan failures where the implicit plan drifts from valid strategies, especially for small LMs with limited capacity. To address this, we propose Multi-Level Reasoning (MLR), which reformulates long-CoT generation as a two-level stochastic process. A high-level planner generates structured step descriptors specifying both the reasoning mode and the semantic subgoal. The low-level executor then produces detailed reasoning conditioned on these descriptors, forming an alternating plan--execute loop. To maintain scalability, we adopt a minimal design where the base model serves as the low-level policy and a lightweight LoRA module implements the high-level policy. For training, we observe that outcome-reward RL provides sparse and delayed feedback for long trajectories (e.g., several thousand tokens), hindering credit assignment. We therefore introduce iterative Step-DPO, a process-level preference optimization scheme that leverages Twisted Sequential Monte Carlo (TSMC) to provide scalable stepwise supervision. This yields more effective training, improved stability, and higher accuracy. Extensive experiments on challenging math, science, and logical reasoning benchmarks show that, under the same reduced data budget (10% SFT and 5% preference relative to the DeepSeek-R1 distillation setup), MLR outperforms both SFT-based distillation and strong RL/preference-optimization baselines across multiple base models and tasks. Moreover, MLR exhibits slower performance degradation on long-horizon reasoning, demonstrating stronger robustness under extended CoT generation.

Published as a conference paper at ICLR 2026
ENHANCING LANGUAGE MODEL REASONING WITH STRUCTURED MULTI-LEVEL MODELING
Siheng Xiong', Ali Payani", Faramarz Fekri!
Georgia Institute of Technology sxiong 50gggeecghccch. edu apayani@cisco.co com fekrišece. gatech. edu
Cisco Research
ABSTRACT
Inference-time scaling enhances model's reasoning by extending its chain-of- thought (CoT). However, existing approaches typically rely a single policy trained with outaome-reward reinforcement learning (RI.), which often suffers from long-horizon plan failures where the implicit plan drifts from valid strategies, especially for small LMs with limited capacity. To address this, we propose Multi- Level Reasoning (MLR), which reformulates long-CoT generation stochastic process. A high-level planner generates structured step descriptors specifying both the reasoning mode and the semantic subgoal. The low-level executor then produces detailed reasoning conditioned on these descriptors, forming alternating plan-execute loop. To maintain scalability, we adopt minimal design where the base model serves as the low-level policy and lightweight LoRA module implements the high-level policy. For training, we observe that outcome- reward RL provides sparse and delayed feedback for long trajectories (e.g., several thousand tokens), hindering credit assignment. We therefore introduce iterative Step-DPO, process-level preference optimization scheme that leverages Twisted Sequential Monte Carlo (TSMC) to provide scalable stepwise supervision. This yields more effective training, improved stability, and higher accuracy. Extensive experiments on challenging math, science, and logical reasoning benchmarks show that, under the same reduced data budget (10% SFT and 5% preference relative the DeepSeek-RI distillation setup), MIR outperforms both SFT-based distillation and strong RI (pprefrrence-optiiizatii tasks. Moreover, MLR exhibits slower performance degradation on long- long-horizon reasoning, demonstrating stronger robustness under extended CoT generation
two-level
baselines across multiple base models and
Qwen2.5-1.588 Strtiratttt
Qwer2.5-1..5 2-122.5-.5B-hstrrtt
R1-0keil-Qwen-1
0.5
RL- Q.ae----
Qwee2.5--155 Meee2.5-15B-MM
Quee2 5-15505RP0
0.4
0wan2.55 1.58 GAPO
Dwen225-1.58--..
Passola
0.2
0.1
Liama1.1 1-88--netruct
Liama331 28 Instruct
0.8
Lamaã. 1-88 Instruct
Rl-Dioti--lamm-ll
RL Distil Liama 86
RL Distil Lamo 88
0.8
Llamaa. L-B--RFF
1-8-331--
Uaaaa.11 118--RPP
Liama.11
Llama3.1 MLR
3.6
1
0.4
0.2-
0.2
6
10
6
10
Problems
Number Chained Probiemss
(a) MATH500
(b) AIME24
(c) Mixture
Figure 1: LLng- accuracy MATH500, AIME24, and mixture benchmark (MATH500) / AIME24
GPQA/ BoardGameQAA. Top: Qwen-2.5-1 21..2-11..55 15B; bottom: 1.amm-331--8. Prohlems concatenated in the
prompt to simulate long horizons; ML.R degrades more slowly than baselines (see Section for details).
Code and data available httpt:tt///////t /ttppp://gitte comp http:/xxinggiiiiaa wwwgg///giiiggiiiiie..eeg

---

Published as a conference paper at ICLR 2026
INTRODUCTION
OpenAT's ol series [OpenA] 2024 introduce inference-time scaling by allocating more computation to the reasoning process, often realized as longer Chain-of- Thought (CoT) (Wei et al.) 2022) gener- ation. This design improves performance on complex reasoning tasks compared non-reasoning models. Building on this direction, DeepSeek (Guo et al. 2025) proposes a large-scale reinforcemen learning (RL) pipeline that directly incentivizes the generation of long CoTs through policy opti- mization. Despite their effectiveness, approaches that rely on single-policy long CoT's for RL face important limitations, including long-horizon plan failures and the inherent challenges of RL with sparse outcome r These issues are especially pronounced for small LMs with long CoT' due to their limited capacity.
In reinforcement learning, long-horizon policy learning remains fundamental challenge due to difficult credit assignment. To address this, hierarchical reinforcement learning (HRL) introduces temporal abstraction by learning high- and low-level policies that operate over different time scales (Sutton et al. 1999). While HRL has proven effective in domains such as robotics (Nachum et al. 2018| et al. 2019), its application LM presents several challenges: 1) Scalability. Modeling multiple policies, especially when implemented as separate LMs, incurs additional compute. Naive multi-agent setups introduce substantial coordination and communication overhead (Guo al.
2024b). 2) Flexibility. The plan-then-execute structure (Wang et al., 2023a Xu et al., (203)) is brittle LM-based reasoning tasks where new information or execution failures may require mid-course corrections. Thus, it is desirable to allow the high-level plan to evolve dynamically based on the ongoing progress of the low-level execution process. To address these challenges, we propose a multi-level reasoning (MLR) strategy, where the model alternates between generating step-level descriptor and its corresponding detailed content, to enable efficient multi-policy modeling and dynamic plan adaptation.
We further introduce an iterative Step-DPO pipeline for long-CoT training to improve stability and performance. Existing RL fine-tuning frameworks struggle to obtain effective process-level supervision (Guo et al. 2025). First, evaluating the correctness of intermediate steps is difficult. Automated annotation using LLMs (Guo et al. 2024a) can be noisy, while manual annotation (Lightman et al. 2023) is prohibitively expensive at scale. Second, introducing a separate process reward model (PRM) adds complexity. It is vulnerable to reward hacking (Gao al. (023), requires substantial training data, and complicates the pipeline by necessitating repeated retraining. To overcome these limitations, we repurpose Twisted Sequential Monte Carlo (TSMC) (Doucet et al., 2001; Whiteley & Lee 2014; Heng et al., 2020) to construct step-level preferences for Step-DPO (Lai al. 2024). In the LM-based reasoning setting, TSMC assigns importance weights that correct distribution mismatch (and incorporate terminal correctness). thereby upweighting partial trajectories whose continuations are more consistent with reaching a correct outcome. We then define process preferences between two candidate continuations at the same step by comparing their incremental log-weights. This formulation has two advantages: (i) taking log-increments converts multiplicative weight updates into an additive form, improving numerical stability; and (ii) it matches the pairwise preference objective used in Step-Dre -PO. Empirically, this yields stable step-level preferences and improves long-CoT C training efficiency and performance.
We summarize our key contributions as follows:
We propose Multi-Level Reasoning (MLR) address the limitations of single-policy long- CoT approaches, such as long-horizon plan failures. MLR decomposes reasoning into alternating high-level step descriptors and low-level detailed content, enabling structured abstraction, dynamic plan adjustment, and more reliable long-horizon reasoning.
We repurpose Twisted Sequential Monte Carlo (TSMC) to provide process-level preferences for Step-DPO training. This eliminates the need for a separate process reward model, while supplying reliable and informative supervision throughout long trajectories.
"See Section for a detailed discussion. NNote that HRI. differs from prompting-basee CoT decomposition: it treats high- and low-level actions as separate distributions with distinct objectives and temporal scopes.
2

---

Published as a conference paper at ICLR 2026
We perform extensive experiments challenging benchmarks in math, science, and log- ical reasoning. Results show that our approach consistently outperforms both SFT-based distillation and strong RL/peeerreee-opiiizzaionn baselines under the same data budget.
INFERENCE-TIME SCALING VIA LONG CHAIN-OF-THOUGHT
Formulation. Given query reasoning model generates a CoT c before producing the final response a, where C, f a are all sequences of tokens, i.c., c(]), c[4)). To improve performance, these models allocate more computation to reasoning, often realized as generating longer e with behaviors such as exploration, celffverification, and reflection. The generation of long CoT's follows the standard autoregressive modeling: the probability of each token c[1] depends on its preceding tokens (e[1 I 1)), which enables the factorization of the joint likelihood of the entire sequence as:
po(c[1: L))
II po (cl4 c[1 1)).
Note that, for notational simplicity, we omit the conditioning on in Eq. [ and the following derivations. Training the model pa involves maximizing the likelihood of each token conditioned on its prefix, i.e., optimizing po(c[[] c[1 1)) over the training data.
Post-training. Guo et al. 2025) detail how they incentivize the long CoT generation from a base model through large-scale RL without relying on SFT. Specifically, they employ GRPO guided by rule-based outcome reward. For each query q. GRPO samples group of outputs {or 02. oa} from the old policy The where each output is composed of a CoT followed by the final response, i.c. 0: = [cs, a.). and then optimizes the policy He by maximizing the corresponding objective.
Discussion on the weakness of single-policy long CoT. The above approach using single-policy long CoT enables inference-time scaling with LMs, but introduces several issues:
1) Long-horizon plan failures. In single-polics long CoT generation, the same policy is responsible for both planning and execution. Without guidance structure, errors accumulate and cause the implicit plan to drift away from any valid strategy (see examples in Section C
2) Long-horizon RL with sparse outcome reward. Long CoT's involve thousands of token-level actions before receiving reward, which hinders credit assignment. As shown in Figure 2 these trajectories can be extremely long, with errors occurring at widely varying positions, which undermines the effectivenees of silcome-based RL. Moreover, Figure shows that latency and memory usage grow rapidly with trajectory length, while outcome- based supervision requires the entire trajectory to finish before feedback provided. Consequently, optimization can be slow and unstable, especially early in training when correct trajectories are rare.
E1-Distill-Laamal MATH500
RI-Distill-Lamaa-8 on MATH500
20.0
15.0
Percentage (%))
20
Percentage
12.5
15
10.0
7.5
10-
5.0
2.5
0.0
0
5
10
15
20
25
30
20
40
60
Number Generated Tokens (K)
Relative Position of First Error (%6
(a) Trajectory length distribution.
(b) First error position distribution.
Figure 2: The CoT trajectories can be long, and the position of the first error varies widely across trajectories, making outcome-only RL fine-tuning less informative. Statistics in (b) are computed from 100 trajectories with incorrect final answers, where the first error was manually identified.
3

---

Published as a conference paper at ICLR 2026
Distill-Lianme MATH500
R1 BL-DDitil--laam-BBB on MATH500
ODM
1400
1200
Average Reward
Lateno (9)
1000
800
600
0.4
400
0.2
200
0.0
128
256
512
1k
2k
4k
8k
4k
16K
Maximum Response Length
Maximum Response Length
(a) Single PPO step latency.
Average outcome reward.
Figure 3: Ouwcome-reward RL becomes increasingly costly for long trajectories: (a) PPO step latency grows with the maximum generation length, and (b) feedback is only available after the full trajectory completes. All measurements use Verl (Sheng et al. 2024) with vLLM (Kwon et al. 2023) on single A100 node (batch size 32).
METHODOLOGY
3.1 ENHANCING INFERENCE-TIME SCALING WITH MULTI-LEVEL REASONING
Reformulation MIR. We reconcettualii inference-ting time scaling by shifting from "single-policy" long CoT to "multi-level reasoning" strategy (Figure 4 Here, the reasoning process is organized hierarchically, capturing both high-level abstractions d and low-level details Formally, the overall reasoning chain is represented at two levels: d = (d(2) d(d)) and (c(1) ... (..,,(())
c(())), where M denotes the number of reasoning steps, d(m) is the descriptor of step and ((T)) represents the
corresponding detailed content. The autoregressive likelihood can be factorized hierarchically as follows:
M pe (d) = II p0 (a(m) ((1mm-1) c(1mm-1)), Pe (c) = II pb (clm) (d(mm)) c(1mm-1)) m=1
M
(2)
m=
where ((m) denotes compressed summary of the detailed content clm) We also experimented with removing the previous descriptors di d1)=m-1) from Equation (2). but found that including them
improves performance and facilitates training. The inference procedure summarized in Algorithm
Architecture. Figure illustrates the architecture used implement MLR strategy. The model
alternates between a high-level policy that produces step descriptors and a low-level policy that
Question: The sot points $ox.y215 that satisty $2x By line. The set points $os.,215 that satisty $6x 42$ another line. Find the angle these degrees.
High-level Abstraction (284 tokens)
Low-level Details (2477 tokens)
Problem Understanding: Icontify the geametric principle
Problem Understanding: Identify the geometric principle
The vacters.
their
to find between lines 3D first,
to diraction
Calculation: Finding direction vectors
Celculation: Finding direction vectors
Starting with the first line: 2x 3y- =z. Lei me parse this equstion. Okay, direction vector Sttextb/((jj
Direction vector of the firet line is (3. -6)
Verification: Varitling the using another mothad Both linas pass through the origin and intersoct there... degrees.
Verification: Verifling
onother mathad
But wait, just to be thorough, maybe should check the lines are Therefore, the between the
Figure 4: Illustration of MLR. MLR augments single-polic policy reasoning with high-level policy that generates step descriptors (reasoning mode and subgoal), providing structured guidance to reduce long-horizon plan drift and facilitate more effective learning.
4

---

Published as a conference paper at ICLR 2026
High-level -evellleel Abstraction
High-level
d
Low Low-level Details
Summarization
Low-level Base model
Figure 5:
Overview of the architecture. The model alternates between generating high-level descrip- tors and the corresponding low-level content. Ablations motivating the design are in Section B
generates the corresponding detailed content. The low-level policy is implemented with the base LM, which conditions on sequence of prior descriptors and detailed contents, together with the current descriptor, to generate the next detailed content. The high-level policy is implemented with a lightweight LoRA module (Hu et al. 2022), which conditions on previous descriptors and summaries to produce the next descriptor. Since descriptors much shorter than full reasoning content, this component remains compact and computationally efficient. The design rationale behind this architecture well as ablation studies are provided Section B Additionally, fine-tune an independent, lightweight LLM for summarization, which is shared across different base models.
3.2
ITERATIVE STEP-DPO WITH PROCESS-LEVES PREFERENCES
To train our model effectively, introduce iterative Step-DPO pipeline that performs stepwise preference optimization with periodic data refresh.
Supervised fine-tuning. We collect long-CoT traces from DeepSeek-RI (Guo et al. 2025) and use DeepSeek-V3 V3 (Liu et al. 2024) to segment each trace into steps and annotate each step with descriptor via in-context learning. We further distill each step's detailed reasoning into a compact summary that preserves the planning-relevant state. This yields aligned triples of descriptors, detailed contents, and summaries. We first full fine-tune the base LM on low-level contents, then freeze and train LoRA module on high-level descriptors. We also train lightweight summarizer the summary data and keep fixed during iterative Step-DPO. See Section 4 for implementation details and Section B for design choices and ablations.
Multi-level Step-DPO. Following Lai al. (2024) optimize the MLR framework with the following stepwise preference loss Dstop-DPo:
M Σ m=1
Po Poly(m) (m) [f(m))
Po(y!m) xz(m))
-RE F (attm) m//)) (m)) Dedt M
log a
3 log
log
(3)
(m) Pree (m())
Prof (y (m)) mm(m))
where (m)))) (m) (mm)) is preference triple at step m. During optimization, we maintain low-level policy (the base LM) and a high-level policy (the LoRA adapter). For lowelevel preference pairs, we disable the LoRA adapter and update only the base LM parameters with ((d()mm) (1=m-1)), (m) dmi)): for high-level pairs, freeze the base LM and update only the LoRA parameters with (mm))= c(1mm-1)), mm)) dm))..
Multi-level update schemes. A key challenge in jointly optimizing the two policies lies in designing an effective update scheme. We adopt an interleaved strategy: mini-batches of high-level and low- level examples alternated, allowing the planner and executor to be trained jointly while preserving modularity. We compare this update scheme with alternatives in Section B
Iterative SPP for online optimization. Motivated by the benefits of on-policy data sampling in RI, we adopt an iterative StPp-DPO framework for improved optimization. Specifically, in the I-th
5

---

Published as a conference paper at ICLR 2026
iteration, we use the current policies to sample preference pairs create the preference data D ()()) pref
We then obtain the next-iteration policies by minimizing C Dep-DPO
Po(y+ (m) xo(m))
log Po(y (m)
1
[2( y/m hmm Der perr
y
M
m==1
log o
3 log
PB
(1) (m) [((m)
(++
Po (v=
(0) (mn) Im(m))
The procedure is summarized in Algorithm 2 More implementation details are provided in Section B
Process preference modeling. A key component of our iterative Step-DPO pipeline is the process- level supervision for both the high-level descriptors d(m) and the low-level detailed contents c(T)). Consider the full reasoning trajectory after prefix (m)) future tokens 1m-1: = (d(m+1), c(m+1).
d(M), ),),, generated by rollout policy Preu- The survival probability of ((m)) is
g(x(m))= == Bim-Lar-Paallaat
[R(x(m), Tm++1:M]]
(5)
where R(x(m), Tm+11ad) E {0, }} indicates whether the final answer correct.
Given estimate of the survival probability construct preference data using utility defined as the increment in log-survivability:
Uly(m)) log ((m log (()(m))
(6)
where the survivability is clipped as ğ = clip(ğ, E, -- =) with & = 0.001 for numerical stability. Intuitively, U (y(m)) quantifies how the selected candidate changes the probability of eventual success relative to the preceding prefix. Then impose the condition that the utility difference satisfies the following:
=)
U(
(ty (m))
log g (m) (f()) log (m)(m)))))))))) , 8,
where the margin threshold 8 ensures the reliability of the preference data.
Twisted Sequential Monte Carlo. A key challenge of the above approach is computational cost: estimating future solvability by naive Monte-Carlo lookahead would require rumning the base model many times for each prefix. To address this, we adopt Twisted Sequential Monte Carlo (TSMC) to obtain efficient and reliable process-level signals. Importantly, our objective only requires pairwise comparison between candidate steps under the same prefix (i.e., generating accurate step-level preferences), rather than a fully calibrated estimate of the absolute survivability probability.
Given a prefix, the k-th particle at step 1 has state 2E (m-1). We first sample a candidate step (m) Vk ~ Protl (- Tk (m-1)) and form the updated state o = [ark
(m-1)
(m)) Its importance weight
Vi
updated
(()) (m-)
(m)
2
(8)
with incremental weight
Po (z (m)
(m-
Om (2( (m))
(m) Gm (m(())
(m)
(m-1))
(m
Om (2 (m-1))
Proll (3t
Here (-(m) is the step-m importance weight with w((0) for m < M and G ( (a) correr (1r( i.e., final answer correctness. pe denotes the base model, Proll is the rollout policy, and om is learned survivability critic at step m. When Proll is close to pe, the importance correction term pa/pe Pa/Proll and the twisting ratio dm/ /0m become mild. In this regime, the incremental weight is dominated by the task potential G. (.), and the resulting estimator provides low-variance signal that is sufficient for ranking candidate steps and constructing step-level preference pairs. Finally, the survivability estimate is given by:
= 1. The potential function Gm is defined
Gm (m)(
K
Mr
gr (x))
K
M
II j=m+1
OF
(10)
where K represents the number of particles, and M2 denotes the termination step of particle k. In our use case, Ox (-) is primarily used as a relative measure of future solvability to compare candidate steps under the same prefix.

---

Published as a conference paper at ICLR 2026
1.58 avg@1
1.56 a
8B avg@4
MATH500
-+- 1.5B avg@4
88 avg@1
86 avg
1000
Est. survival acc.
MATH500
Rollout time (s)
750
8B
1.5B
0.96
0.94
500
0.92
250
20
10 15
20
25
30
Prefix
Prefix step
(a) Accuracy of future-solvability estimation.
(b) Rollout time for batch size 64.
Figure 6: Monte Carlo rollout analysis. (a) Future-solvability estimation accuracy is similar be- Tween R1-Distill-LLaMMAA and R1-Distill-Qwen-113 (b) Runtime increases with model size, and RR1-Distill-LLaMMA- is substantially slower than R R1-Distill-Qwen-11 1.55- See Section for implementation details.
To keep rollouts efficient while limiting distribution mismatch, we fine-tune small LM on the same low-level SFT data and use Proll- This is motivated by (i) closer rollout distribution, (ii) the fact that preference construction only requires correct relative ranking, and (iii) substantially lower rollout cost. As shown in Figure 6 the small model provides comparable preference quality while being much faster. We further use an easy/hard split: for hard benchmarks where small model lags far behind the base LM, instead roll out with the base model (AIME24 and GPQA in practice).
4 EXPERIMENTS
Datasets. We evaluate on math (MATH500 (Hendrycks et al. 2021). AIME24 (MAA 2024)) science (GPQA-Diamond (Rein al. 2023)), and logical reasoning
Hard (aazemi ctt al. 2023)); dataset overview and split protocols are provided in Section A For training, we construct multi-level dataset and divide into two disjoint parts: SFT data and iterative preference data for Step-DPO. The SFT set contains about 80K examples produced using the multi-level decomposition
procedure described in Section 3.2 We reserve an additional 10K training prompts for Step-DPO preference sampling. Details of dataset construction are provided in Section B
Implementation details. We full fine-tun tune three base LMs (Qwen-2.5-1.5B Yang et al. 2024a), Qwen-2.5-MATH-1 (Yang et al.) 2024b), and LLaMA-3.1 1-BB (Grattatiori et al. 2024)) low- level SFT data, then freeze them and train a lightweight LoRA adapter as the high-level policy. We also fine-tune Qwen2.5-0.55 5-2.5---.BBBB.. I.StuInst YYngg et al.. 2024a) models. Iterative Step-DPO is implemented with TRL von Werra et al.) (020). Each round samples approximately 3K prompts; per prompt we select 4 steps and generate M candidates, scored by Equation (6). We use a Qwen-2.5-1.5 1.5BBB SFT model as the rollout policy for MATH500 and BoardGame QA, and the same base model for AIME24 and GPQA: we use
summarizer, shared across base
8 rollouts per prefix (see Figure 6 and section 4 for analysis and parameter studies). We form one preference pair per prefix by taking the largest utility gap and retaining it only the margin exceeds 0.4. Updates use batch size 32 for E = 4 epochs with 3 = 0.1, and continuations are capped at 1,192 tokens. Additional details are in Section
Baselines. We compare against the following baselines: (i) the base model and the instruction-tunee
model: (ii) outcome-rewaad RL applied directly to the base model (SimpleRL (Zeng et al. 2025
Level
Level
Summary
Percentage (96)
10
5
500
1000
1500
2000
2000
4000
6000
8000
25
50
100
125
150
Figure 7: Length distributions for high-level trajectories, low-level trajectories, and distilled sum- maries in SFT dataset. Trajectories longer than 8,192 tokens are discarded.
7

---

Published as a conference paper at ICLR 2026
Table 1: Overall performance across benchmarks. The best performance
using same base
model is in bold. Asterisks (*) denote available results obtained from the official reports.
Math
Science
Logic
Method
MATH500 Pass@1
AIMF24
GPQA (Dimnad)
BoardGameQA Pass
Avg. Pass @
@
Cons@32
Pass@1
Qwen-2.5-1.5B
Base
29.6 = 0s
0.0 =00
0.0 0.0
21.2 11
+5.0 41.0
21.5
Instruct
54.6
3.3 15
10.0
25.2
31.0
SimpleRL
59.0"
6.7
4,2"
DeepSeek-RI-DDe
83.9
28.9*
43.3
33.8
40.0
47.7
Plan-and-Soll
60.8 1
2.0 1.4
6,2 0.44
20.1 116
44,6 116
31.9
SFT + DPO
76.5
12.0
21.6 u.7
27.6
42.0
SFT Step-DPO
81.4
24.0
36.5100
19
56.2
47.8
SFT + GRPO
82.1 12
25.2 1:22
36.0 0:
30.2
56.0 1.6
48.4
MIR (SFT only)
62.0 12
1.
13
26.0 2:
46,4
35.8
MLR
86.1 =
31.2 1.0
47.4 + 04
37.6
1.7
54.2
Qwen-2.5-MATH-72
Base
52.0
2.0 =
5.0
20.5
33.0
26.9
Instruct
82.1
16.7 1.
34.0 1:4
27.8 +
44.5 144
42.8
SimpleRI.
80.2*
40.0'
24.0"
DeepScck-RI-Disilll
92.8"
55.5"
74
49.1'
42.4
60.0
Plan-and-Solve
85.6 00
18.2
34.9
28.4
52.1 155
46.1
SFT + DPO
53.0
36.0
54.5
53.4
SFT Step-DPO
88.5 009
48
70.5
48.0
16
60.3
SFT + GRPO
89.7 1.0
46.5 119
+.5
46.0 18
57.5
59.9
ML. (SFT only)
86.3 = 1.
22.4 11
40.5 0.
34.6
54.8 14.8111
49.5
MLR
58.8 1
80.5 04
16
66.2
Llama-3, 1-8B
0.0
0.0
2.
4.3
Instruct
51.9 122
6.7 1.8
13.3
22.7 1.06
40.0 1..
30.3
SimpleRL
23.0"
0.0"
DeepSeek-RI-nn RI-istill
89.1"
50.4"
70.0
49.0'
46.0 1.38
58.6
Plan-und-Solve
12.4.111
12.3 118
24.1
31.0
47.2.13
38.2
SET DPO
74.1
32.4
uu
44.0
56.0
51.6
SFT + Step-DPO
82.4
42.6
61.2
19.2
62.1 1
59.1
SFT + GRPO
86.5 1A
42.0
61.0
47.0
60.0
MLR (SFT only)
63.8
20.2 20
u4
36.2
48.5 1.8
42.2
MLR
91.5
53.2
73.3
52.8
67.0
66.1
(iii) SFT-based distillation using vanilla long CoT's (R1-Distill (Guo et al.
2025 (iv) planning-style
prompting (Plan-and- -aau-aed--oovee (Wang et al. 2023a ); (v) preference-optimizzan
baselines trained
on our SFT initialization (DPO (Rafailov et al. 2023) and Step-DPO (Lai al. 2024)); and (vi)
outcome-reward RL applied to the SFT initialization (GRPO (Shao et al. (024g).
All baselines that train ourselves (DPO, Step-DPO, GRPO, and Man-and-Solvve use exactly the same training data budget; see Section B for details. Results for external baselines (Instruct, SimpleRL, and R1-Distill) are included strong reference points. During evaluation, decoding for Base and Instruct. For all other methods (including ours), we follow Guo et al. (2025) and sampling-bar based decoding with temperature 0.6 and top-p 0.95 generate 8 responses per @ 1, and additionally cons @ 32 for AIME24.
greedy
prompt. We report pass@
Empirical results. We present representative outputs in Section C and error analysis Section C Table summarizes overall performance, and Figure shows how MLR evolves across training
MATH500
AIME24
GPQA
BoardGameQA
1.0
Selected Pair Agr.
0.9
0.8
1.5B
8B
0.6
4
16
8
16
8
16
1
4
Rollout count
Rollout count
Rollout count
Rollout count
Figure 8: Agreement of selected preference pairs versus rollout count K on (a) MATH500, (b) AIME24, (c) GPQA, and (d) BoardGameQA. For each K, report the fraction of rollout-sclected pairs whose preference direction matches an 8B reference using rollouts, We use the base model as proll for AIME24 GPQA.
8

---

Published as a conference paper at ICLR 2026
Table 2: Ablation results using Qwen-2.5-1.5B,
high- level SFT ablation, all methods use the same low-level SFT. For the hierarchical- ablation. all variants share the same trained model. Our approach is highlighted in bold.
MATH500
BoardGameQA
AIMEZ4
Average
GPQA
MATH500
AIME24
Avg. Pass
Method
@ass
Pass@1
Cons@32
Ablation of high-level SFT strategies
60
SFT (low)+ + LoRA (high)
62.0 12
8,9 144
13.3 14
35.5
LoRA (high)
56.4 15
4.1 +1
9.2 407
30.3
SFT (high)
59.8
6,5
11.0
33.2
Ablation of hierarchical levels
20
High-level + Low-level
86.1
312
47.4
58.7
SFT
RFT-1 RFT-2
RFT-3
RFT-4
High-level only
80.0 i
18.4 20
30.5 8:
49.2
Figure 9: Performance of MLR over differ- ent training stages.
Lowelevel only
27.1 11
Table 3: Ablation results for the core components of MLR using LLaMA-3.I-8B.
Math
Science
Logic
Method
MATH500
AIME24
GPQA (Dimmad)
BourdGameQA (Bud))
Avg. Pass@
Pass @
Pass@1
Cons @
Puss@
Puss @
Ours
91.5 1.3
$3.2 20
73.3 0.4
52.8 1.5
67.0 1.4
66.1
DPO-only
78.2
38.1 1
57.0
46.0
59.0
55.3
Low-level policy Step-DPO
82.4 1.
22
61.2 a5
49.2 1.
62.1 + 1.4
59.1
Low-level policy DPO
74.1 1:
1
52.0 0:66
44,0
56.0 a
$1.6
SFT-only
20.2 n.
36.7
36.2
stages. We compare instruction tuning, SFT distillation (vanilla long CoTs), preference optimization (DPO/Step-DDe) DPO),, outcome-reward RL (GRPO). Plan-and-Solver nolve,, and MLR. Across benchmarks, MLR consistently outperforms all baselines, with its structured design enabling more effective reasoning complex, hong-horizon tasks. In addition, the iterative step-DPO procedure yields substantial gains over the SFT model. We also report response-length statisties: high-level trajectories about 10-20% the length of low-level ones (Figure 11
Parameter studies. We vary the rollout count K and compare utilities from the 1.5B rollout policy against a stronger 8B reference using 16 rollouts (Figure As expected, increasing K reduces estimator variance, though at the cost of higher computation. To mitigate this overhead, introduce a margin threshold when selecting preference pairs. We further measure the agreement of the selected preference pairs function of K, defined the fraction whose preference direction agrees with the base model using 16 rollouts (Figure B). Finally, we report model performance across training stages under different values of K (Figure 12). Our default setting attains comparable final accuracy with lower rollout cost. Implementation details are provided in Section
Ablation studies. We ablate key components of MLR. We compare five configurations: (i) the full method, (ii) MLR trained with DPO instead of Step-DPO, (iii) low-level only with Step-DPO, (iv) low-level only with DPO, and (v) MLR with SFT only. Table summarizes the results, which show that both multi-level modeling and step-level preferences contribute performance. Figure further reports preference accuracy over training. We additionally ablate the high-level SFT component (Table 2). evaluating two alternatives: (i) applying LoRA the original base model and (ii) full- parameter SFT. A detailed discussion and implementation details are provided in Section B We also ablate the hierarchical structure (Table comparing (i) high-level-only and (ii) low-level-only variants. Further analysis and implementation details appear in Section C. Across all ablations, our full strategy yields the strongest performance.
DPP
Lowp.evilSttpDPP
Only
MLR
Avg. resp. lenget [K]
7.8
Low-level
High-level
MATH500 AJME24
Avg. Pass@1 (96)
Fref. acc. (86)
80
3.7
30
K=4
K=8
2.2
25
K-16
60
50
100
150
200
250
300
0.4
0.7
0.3
Training steps
MATH500
AIME24
GPQA
Board
SFT
RFT-1
RFT-2
RFT:3
RFT-1
Figure 10: Preference accuracy.
Figure 11: Avg. resp. length.
Figure 12: Effect of in MLR.

---

Published as a conference paper at ICLR 2026
5 RELATED WORK
JFT-based reasoning distillation. Reasoning models (OpenAl 2024 Qwen- Team 2024 Guo
et al. 2025) have motivated extensive interest in replicating strong long-Coll performance. Several follow-up efforts primarily rely on SFT-based distillation, training smaller models to imitate long-CoT traces generated by stronger reasoning models (HuggingFace 2025 Muennighoff et al., 2025). This line of work emphasizes data collection and curation to obtain high-quality reasoning trajectories, and has shown promising results in math and coding domains. However, SFT optimizes next-token imitation and thus passively follows demonstrations, providing limited exploration and limited credit assignment for long trajectories, where early mistakes can be amplified downstream.
Ouwcode RL for reasoning. To go beyond imitation, recent RL-based pipelines optimize LMs using outcome rewards, with methods broadly falling into PPO-, GRPO-, and REINFORCE. based families (Schulman et al., 2017 Shao et al. 2024 Williams 1992). PPO variants improve value learning and stability for long CoT's (e.g.. value pretraining, modified advantage estima- tion) (Yuan et al., 2025: Yue et al. 2025). GRPO-style methods remove the critic and refine objectives to improve stability at scale (Yu et al. 2025). REIN- FORCE variants reduce variance via KL. penalties and centralized rewards (Hu 2025 Kimi-TTem
et al. 2025). Across these methods, reward design typically combines outcome correctness with
heuristic constraints such format compliance and length control (Zhang et al. 2025a), and training
often benefits from sampling heuristics such as curriculum-style difficulty scheduling (Zhang et al.
2025b) and rejection/sample filtering (Yu et al. 2025 He al., 2025). Despite these advances, most pipelines still rely primarily on sparse outcome rewards, which remain weakly informative for long-horizon credit assignment.
Process supervision and preference optimization. Process-level supervision has been explored via process reward models (PRMs), which attempt to assign intermediate rewards to reasoning steps (Lightman et al., 20233 Wang et al., 2023b; Xiong et al., 2025). In practice, PRMs face several explicitly defining fine-grained steps is non-trivial; verifying intermediate correctness reliably is difficult; and training/maintainingg separate reward model introduces additional cost and risks (c.g., reward hacking) (Guo et al.) 2025 Xiong et al. 2026). Preference optimization provides an alternative explicit reward modeling. DPO learns from offlin,, trajectory-level preference pairs (Rafailov et al. 2023), while Step-DPO constructs step-wise preferences to better support long-chain reasoning (Lai et al. 2024). Online AI feedback can further enable preference-basee updates beyond static datasets (Guo et al. 2024a). However, scaling to long CoTs remains challenging; constructing reliable step-level preferences often depends on strong teacher models verifiers, which is costly and can be noisy on harder problems. This motivates scalable mechanisms for generating hep-wise preferences without heavy reliance on external teachers. In this paper, we introduce a scalable TSMC--based approach to produce step-wise preferences for long-horizon supervision.
challenges:
Planning for long-horizon reasoning. Beyond training signals, long-horizon reasoning with a single policy can suffer from plan drift and cascading failures. Planning-style prompting methods often follow plan-then-execute paradigm, generating full plan upfront and executing it as written (Xu al. 2023 Wang al. (023a)) which can propagate early errors when subtasks fail or new information emerges. In contrast, our work proposes multi-level reasoning (MLR), where a learned planner adapts its plan based execution feedback, enabling mid-course revisions and improved robustness.
CONCLUSION
We introduced Multi-Level Reasoning (MLR), two-level formulation for long-CoT generation that alternates between high-level step descriptors and low-level detailed reasoning, enabling scalable multi-policy modeling and adaptive planning over long horizons. To provide process-level training signals without a separate reward model, we proposed an iterative Step-DPO pipeline with TSMC- based preference construction. Across math, science, and logical reasoning benchmarks, MLR consistently improves strong distillation, outcome-reward RL. and stepwise preference baselines, and degrades more slowly under long-horizon evaluation. Future work will explore extending MLR to tool-augmented settings, where external tools knowledge are integrated into the reasoning process.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported in part by DARPA SciFy program, Award No.HRO01 125C0302, and CISCO Systems, Inc.
REFERENCES
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint vrXiv:2307.0869/ 2023.
Arnaud Doucet, Nando De Freitas, Neil James Gordon, et al. Sequential Monte Carlo methods in practice, volume Springer, 2001.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning. pp. 10835-10866 PMLR, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407.22 17783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint wrXiv:2501.12944 2025.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, al. Direct language model alignment from online ai feedback. arXiv preprint 2rXiv:2402.04792 2024a.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: survey of progress and challenges. arXiv preprint arXiv:2402.01660, 2024b.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11990 11956, 2019.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.22332, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
Jeremy Heng, Adrian N Bishop, George Deligiannidis, and Araad Doucet. Controlled sequential monte carlo. The Annals of Statistics, 48(5):2904-29) 2929,, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03266: 2025.
HuggingFace. Open rl: fully open reproduction of deepseek-rl, January 2025. URL https //ogthhub.commmuugggggaaaaa.c
Mchran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandram. Boardgameqa: A dataset for natural language reasoning with contradictory information. Advances in Neural Information Processing Systems, 36:39052-39074, 2023.
11

---

Published as a conference paper at ICLR 2026
Kimi-Team, Angang Du, Bofei Gao, Bowei Xing. Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan. Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng. Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang. Jie Zhao, Jin Zhang, Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma. Shupeng Wei, Sihan Cao, Siying Huang. Tao Jiang. Weihao Gao. Weimin Xiong, Weiran He, Weixiao Huang. Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang. Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang. Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with 1lms, 2025. URL https: /tttps://arxivv org/ abs 25 2011 12599
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Liammin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jía. Step-dpo: Step- wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.1880 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
MAA.
American American Invitational Mathematics Examination URL
invitational
mathematics examination
In Febru- httpps//mmat org/ //a competitions
aime 2024,
AIME
ary
2024.
americant invitationala hatttaatiionnl-mmammmttatttattt -mathheatiicss exam nat on- aime
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL
Ofir Nachum, Shixiang Shane Gu. Honglak Lec, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
OpenAl. Learning to reason with language models, September 2024. URL https::/ com/ coom/index. http://nexeeearnnn---iee reason-wr wet-wwithhlmmmm mmm//
Qwen-Team, Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53744 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311. 1...:2221..1022,,, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X.vv1707.06... 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033300 2024.
12

---

Published as a conference paper at ICLR 2026
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024.
Richard S Sutton, Doina Precup. and Satinder Singh. Between mdps and cemi-mmps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1 112(1-2):18 22:1121111111111 1999.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lam- bert, Shengyi Huang. Kashif Rasul, and Quentin Gallouédec. TRL: Transformers Reinforcement Learning, 2020. httpstss: http:::///qqttuut /qpthttttttu///gg/ http:////oo/gggggcceeeeeeeeeeeee
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thoughh reasoning by large language models. arXiv preprint arXiv:2305. 04091, 2023a.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce iims step-by-stey without human annotations. arXiv preprint arXiv:23/2.0893: 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 0124824-24883 2022.
Nick Whiteley and Anthony Lee. Twisted particle filters. 2014.
Ronald Williams. Simple statistical gradient- algorithms for connectionist reinforcement learning. Mach. Learn., 8(3-4):229-255 May 1992. ISSN 0885-6125, doi: 10.1007/BF00999266.
URL https: orgg/oogg1000 1007 BF 3F00999699
Siheng Xiong, Ali Payani, Yuan Yang. and Faramarz Fekri. Deliberate reasoning in language models structure- aware planning with accurate world model. In Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp.
31900-31931, 2025.
Siheng Xiong, Oguzhan Gungordu, Blair Johnson, James Kerce, and Faramarz Fekri. Scaling search-augmented Ilm reasoning via adaptive information control. arXiv preprint arXiv:2602.01662, 2026.
Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu. Rewoo: Decoupling reasoning from observations for efficient augmented language models. arXiv preprint arXiv:2305.1833 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprini arXiv:2412.15115 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-mate technical report: Toward mathematical expert model via self- iaff-mmprovemntt, 2024b. URL htlltttps///aratt org/ abs 22099 12122
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint 2rXx: 2025.
Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind ppo's collapse in long-cot? value optimization holds the secret. arXiv preprint arXiv:2503.01499, 2025.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, Tian Fan, Zhengyin Du, Xiangpeng Wei, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang. Yonghui Wu, and Lin Yan. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks, 2025. UtW.hhccss//aaaaio.or////ab/bb/
13

---

Published as a conference paper at ICLR 2026
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892 2025.
Chong Zhang, Yue Deng, Xiang Lin, Bin Wang, Dianwen Ng, Hai Ye, Xingxuan Li, Yao Xiao, Zhanfeng Mo, Qi Zhang, et al. 100 days after deepseek-rl: A survey on replication studies and more directions for reasoning language models. arXiv preprint arXiv:2505.0055// 2025a.
Xiaojiang Zhang, Jinghui Wang. Zifei Cheng, Wenhao Zhuang. Zheng Lin, Minglei Zhang, Shaojie Wang. Yinghan Cui, Chao Wang, Junyi Peng. Shimiao Jiang, Shiqi Kuang, Shouyu Yin, Chaohang Wen, Haotian Zhang, Bin Chen, and Bing Yu. Srpo: A cross-domain implementation of large-scale reinforcement learning Ilm, 2025b.
org/ abs 1 14286
14

---

Published as a conference paper at ICLR 2026
A DATASET OVERVIEW
In this section, summarize dataset statistics and split protocols for all benchmarks used in study: MATH500 (Hendrycks et al.) 2021 Lightman et al.) 2023) and AIME24 (MAA 2024) for
math reasoning, EPQA-Diamont (Rein et al. [203] for science reasoning, and BoardGame Hard
(Kazemi et al.
2023 for logical reasoning.
For each benchmark, we form training pool and an evaluation set. The training pool is split into two disjoint subsets for supervised fine-tuning (SFT) and iterative preference learning. Evaluation examples are strictly excluded from SFT, preference construction, and rollouts, and are used only for final evaluation.
The MATH dataset (Hendryeks et al. 202T contains 7,500 training and 5,000 test problems.
MATH500 is 500-problem subset of the MATH test set introduced by Lightman et al. (2023). We evaluate on MATH500 and add the remaining MATH test problems to the training pool; the MATH500 subset is excluded from SFT, preference learning, and rollouts.
The AIME dataset (MAA) 2024) is based on the American Invitational Mathematics Examination (AIME), a high-level mathematics contest administered by the Mathematical Association of America (MAA). We use a public AIME problem collection (1983-2024 where each year typically contains 15 problems (before 2000) and 30 problems (afterwards). We use AIME 2024 as the evaluation set (AIME24), and prior years as the training pool.
GPQA (Rein et al. 2023) is multiple-choice QA dataset of difficult expert-written questions in biology, physics, and chemistry. It includes three subsets: Main (448 questions), Diamond (198 questions), and Extended (546 questions). We evaluate the Diamond subset and combine the remaining subsets as the training pool. To prevent leakage, we remove any training examples that overlap with the evaluation set.
BoardGameQA (Kazemi et al., 2023 is a claim verification dataset with three labels (proved, d sproved,, unknown) that requires reasoning with conflicting information under preferences over rules. The dataset is divided into 15 subsets based on reasoning depth and levels of conflict or distractors, each with separate train/validation/lest splits. We define BoardGameQA--Bl as the union of test splits from five challenging subsets: Main-depth3, Difficul(Contiie--dptt2. HighConflict- depth2, KnowledgeHeavy-dergg depth2, and ManyDistractors-dephh yielding 500 evaluation examples. All remaining data from the benchmark are used as the training pool.
B IMPLEMENTATION DETAILS
Dataset construction. We consider four publicly available reasoning benchmarks: MATH500, AIME24, GPQA-Diamond, and HoardGameeA-Hard. For each benchmark, we form training pool and an evaluation set following Section A The training pool is split into two disjoint subsets for supervised fine-tuning (SFT) and iterative preference learning (60% Evaluation examples are strictly excluded from SFT, preference construction, and rollouts, and are used only for final evaluation. In particular, for MATH we evaluate on MATH500 and add the remaining MATH test problems to the training pool, while excluding the MATH500 subset from all training components (SFT, preference learning. and rollouts).
To increase the diversity of training prompts for AIME and GPQA, we additionally generate synthetic problems using GPT-4o. For each seed problem, we sample multiple candidate questions and retain only those whose final answers are consistent across GPT-4o and DeepSeek-Ri, i.e.. both models solve the problem and produce the same final answer. We also remove any synthetic problems that overlap with the evaluation sets to prevent leakage. After constructing the training pools, the resulting SFT subsets contain 7.2K MATH, 4.0K AIME, 4.0K GPQA, and 5.0K BoardGameQA problems, and the preference-learning subsets contain 4.8K, 1.5K, 1.5K, and 2.0K problems, respectively.
All methods that we train ourselves (DPO, Step-DPO, GRPO, Haan-and-Solle MLR, and all ablations in Table 3) use exactly the same data splits and budgets, ensuring a fair comparison. Results for external baselines (Instruct, SimpleRL, and DeepSeek- RI--tiitilll)l,, marked with an asterisk in Table [
https:/ htt::://wwwwwwm. http://wwwww.g.l... http://ll...omaaatttteeeeaaae/ee http://hhwiiiieeeaaabbbooooooo me-erobl em- set- 1983-2024
15

---

Published as a conference paper at ICLR 2026
copied from their official reports and may rely on different training corpora; include them as reference points.
Specifically, in the SFT phase, we generate multiple trajectories for each training problem by sampling four solutions from DeepSeek-RI, yielding approximately SOK filtered trajectories in total. Each accepted trajectory is then segmented into step-level chunks using DeepSeek-V. V3 via in-context learning, with each step annotated by a step descriptor. To enable multi-level reasoning, we further distill each step into a concise high-level summary, again using DeepSeek-V3 with in-context learning. All prompt templates used in this pipeline are provided in Section C length distributions across low-level trajectories, high-level trajectories, and summaries.
Figure presents the resulting
Summarization. We train separate summarization model to produce the compressed summary ((m) for each low-level step clm²) which serves as the planner's conditioning context in Equation ()
The summarizer is Qwen2.5-00.5B-Insttee schedule with linear warmup) and peak learning rate 1 x 10-5. and shared across all base models. We
fine-tuned with full-parameter SFT using Adam W (cosine
keep the summarizer frozen during iterative Step-DPO. The prompt template is provided in Section C
and the summary length distribution is shown in Figure 7
Supervised fine-tuning. We first fine-tune the base LM on low-level trajectories, then freeze and train a parameter-efficient LoRA adapter for high-level planning. This design is motivated by two considerations. First, low-level trajectories are substantially longer and require modeling fine-grained token-level dependencies, for which full-parameter updates are beneficial. Second, high-level trajectories are shorter and more abstract, making them well suited for adapter tuning while reducing interference with the low-level executor. Training the planner top of fixed executor also reduces distribution shift between the planner's conditioning context and the executor's 's behavior. From an optimization standpoint, LoRA benefits from a stronger backbone after low-level SFT and constrains the number of trainable parameters, which can mitigate overfitting on high-level data. Operationally, the final system consists of single base model plus small LoRA adapter (less than 2% additional parameters), keeping deployment overhead low.
Hyperparameters. We train each base model on 80K multi-level examples using the AdamW optimizer with a cosine learning-rate schedule and linear warm-up (5% of total steps). The batch size is set to 256, with a peak learning rate of x 10-5, and we truncate sequences to 8, 8,192 tokens. Training is performed for 3 epochs. For the base LM (low-level policy), the AdamW optimizer is used with a peak learning rate of x 10-5, while the high-level LoRA module (r = 16, a 32, k-proj, v-proj. o-proj], no bias) is trained with higher rate of 1 x 10 (dropout=0.1) 1) to allow faster adaptation. We further verified that adding MLP projections (up proj. down proj, gate proj) yields only marginal improvements while substantially increasing the number of trainable parameters.
morget.modulss((ppprr
Ablation study. To further validate our design choices, we compare against two alternative training strategies for the high-level policy, while keeping the low-level training unchanged. This is important because low-level modeling requires full-parameter updates due to its longer and more complex reasoning trajectories; LoRA is insufficient for this component. We consider:
(i) LoRA on the original (non-SFT) base model: We directly apply LoRA tuning on the unfined-tuncd Qwen-2.5-1.5BB base model using only high-level trajectories.
Base: Qwen-2.5-1.5B
LoRA: 16, 32, target modules=[q proj. k proj, v proj, p.rooo] no bias.
Optimization: AdamW with cosine learning rate schedule and linear warmup, peak learning rate of x 10-4 and a LoRA dropout of 0.1.
(ii) Full- Full-parameter SFT on high-level trajectories: We train separate base model using full SFT on only legh- trajectories.
Base: Qwen- -0
Optimization: Adam with a cosine learning-rat rate schedule and linear warmup, a peak learning rate of 1 x 10-5
16

---

Published as a conference paper at ICLR 2026
We evaluate both variants on MATH500 and AIME24. Table summarizes the results. Our default configuration (full SFT on low-level trajectories followed by LoRA tuning on high-level abstractions) achieves the highest accuracy. particularly the harder AIME tasks that require deeper multi-step planning. We also observe that applying LoRA on top of the SFT-enhanced base model substantially eases optimization and mitigates the overfitting issues that arise when fully fine-tuning a separate base model using only high-level trajectories.
Monte Carlo rollout analysis We analyze Monte Carlo rollout behavior using R RI-Disiil-LLaMA 8B and RI- RI-Distill-QU Peistil--wee--.Stt Hidden CoTs are segmented into steps using \n),n. Estimation accuracy measures the fraction of prefixes for which rollouts correctly determine whether the prefix can still lead to correct final solution.
For each partial trajectory, we assign a ground truth survival label E {0, 1 Carlo lookahead with the base model: y if at least one rollout from the prefix reaches a correct final answer (i.e., the prefix is survivable), and y 0 otherwise.
using extensive Monte
Using the fast rollout model, we draw continuations from each prefix and compute the estimated survivability
1
(k) comeec
JK-
K
We then predict a survival label
11
9K > O,
DK
0, otherwise.
A prediction is correct when ýK- y. Estimated survival accuracy for given K is the proportion of prefixes where this prediction matches ground truth.
We present the results in Figure 6 E1-Distill-LLaMM.. similar accuracy across prefix steps. As expected, runtime scales with model size: RI-Distill Diistill-LLaMMA 8B is substantially slower than R1-Distill-Qwer 1.5BB. All measurements are obtained a single A100 GPU using vLL.M.
and R1 RI-Distill-Qwen--.3 exhibit highly
Iterative Step-DPO. The reference model for KI. regularization in Step-DPO is the corre- sponding SFT checkpoint. We perform 4 train- ing rounds, with approximately 3K prompts per round, as described in the main text. For each prompt, we sample N candidate continu- ations using temperature 0.7 and top-p 0.9, and truncate continuations at 8, 8,192 tokens to match the SFT context length and keep attention computation within our memory budget. Step- wise preference pairs are constructed from these candidates and stored in replay buffer of size 50K; once the buffer is full, older entries are discarded in FIFO order. We optimize the policy with A (learning rate 1 10-5 weight decay 0.1, 31 = 0.9, 32 = 0.95), a global batch size of 256 preference pairs. gradient clipping with maximum norm of 1.0, DPO temperature = 0.1, and KI. coefficient λKI. 0.02. At each round, we perform one epoch of updates over the current replay buffer. During optimization, we maintain a low-level policy (the base LM) and a high-level policy (the LoRA adapter). For low-level preference pairs, we disable the LoRA adapter and update only the base LM parameters; for high-level pairs, we freeze the base LM and update only the LoRA parameters. Mini-batches of interleaved within each round, so that the executor and planner
Algorithm 1: Multi-Level Inference
Inputs:
query g, high-level policy new low-level policy THz, summarizer Tési
2 Hyperparameter: max steps M;
3 m t 1:
4 while m S M do
d(m) Tox (d]
5
q, d((mm--)) ((((1mm-))) c(m) ~ (c d(1mm)) c(1:m-1));
7 c((m) t- Teg (d(m), c(m));;
8 if StopCriterrin (m))))) c(m)) then
break;
10 return (d(lmm) ((1m));
m - m 1;
low- and high-level examples are optimized jointly while remaining modular. To improve sample efficiency, we apply a dynamic dropout strategy that filters "easy" prefixes, i.e., prefixes for which all candidates induce the same
17

---

Published as a conference paper at ICLR 2026
Algorithm 2: Iterative Step-DPO
Inputs: Low-level policy T high-level policy To,; Reference models T "reft rollout policy Woll; RL prompts DRL. Hyperparams: rounds T, prompts per round N, sample steps per prompt M,, rollout count K, epochs E.
H. Fast
3 for 1 to T
Sample prompts fg, 1im1 C DRLL
Initialize buffers ((t)
pref-L -- 0. D(t)
poof-H f 0:
foreach q do (9)
7
(prefix H prefix ( -- GENERATEPREFIXES (t) (6)
( (o "0g g):
Randomly select a subset of steps M (size M,) for evaluation;
foreach .M do
10
((t.m)
pref-L - COLLECTPAIR
TT prefix
(1)
(9) [m]):
D (((.m)
pref-ll + COLLECTPAIR ( (Tyy prefix H [m]]:
11
D(0)
pref-L + (()) U D (t.m)
12
pref-L.
((1)
pref-11 < D(t))
prefi U U.D(tera)
pref-1l
ift 1 then
1t-1), 1) Fee f (t-1);
15
Ref f- TH
TBH
16
for c to E do
STEPDPOUPDATE
(8 t Tefi D ( ):
(t)
(1) H D(1)
STEPDPOUPDATE Bn "re pref-H): (T).
return Tog : "0
utility; the dropout rate increases linearly from 0.1 0.9 over training. All experiments conducted on 4x A100 GPUs (80GB) with bf16 precision.
Step-DPO update schemes. We compare the proposed update scheme against alternatives under a matched training budget (same number of prompts, candidates, and optimization steps). In the planner-only variant, freeze the SFT base LM and apply Step-DPO updates only to the high- LoRA adapter for all preference pairs, thereby testing whether adapting the planner alone is sufficient the executor has been trained. In round-based variant, first Step-DPO for rounds updating only the low-level policy (LoRA disabled), and then for two rounds updating only the high-level LoRA (base L.M frozen), mirroring coarse low-then-high schedule in the iterative phase. Empirically, our joint modular scheme, which interleaves low-level and high-level updates while restricting each preference type to its corresponding module, achieves the best overall performance on MATH500 and AIME24, suggesting that simultaneously refining the executor and planner, while keeping their parameter updates disentangled, more effective than tuning either component in isolation.
DPO baseline. To isolate the effect of step-wise supervision, we train standard outcome-level DPO baseline on the same prompt pool and with the same rollout configuration as Step-DPO. The reference model for KL regularization is the corresponding SFT checkpoint, and we run training rounds with approximately 3K prompts per round. For each prompt, we sample N = 4 candidate continuations using temperature 0.7 and top-p = 0.9, truncating each continuation at 8, 192 tokens to match the SFT context length. Preference pairs are constructed at the trajectory level: we assign each candidate a scalar utility based on its final solution correctness and form DPO pairs from these outcome-level utilities, ignoring intermediate prefixes. The resulting preference pairs are stored in replay buffer size 50K with FIFO eviction, and we perform one epoch of DPO updates over the buffer per round. We optimize single policy (no hierarchical separation) with AdamW (learning rate 1 x 10 weight decay 0.1, 31 = 0.9, 32 0.95), using global batch size of 256 preference pairs, gradient clipping with maximum norm of 1.0, DPO temperature 0.1, and KI. coefficient
18

---

Published as a conference paper at ICLR 2026
Qwen2.5-1.58 SFT MATH accuracy during training
Qwen2.5-1.58 SFT AIME accuracy during training
0.80
W
0.25
0.75
0.20
Accuracy
Accuracy
0.15
0.65
0.10
100
200
300
400
500
100
200
300
400
500
Steps
Steps
Figure 13: Validation accuracy of Qwen2.5-1.55 SFT during GRPO training. For each question, we sample 8 responses and calculate the overall average accuracy to ensure stable evaluation.
ÅKL = 0.02. All experiments are conducted on 4x A100 GPUs (80GB) with bf16 precision under matched training budget to Step-DPO.
GRPO baseline. To compare MLR with standard single-policy preference-optimizzen method, we train GRPO baseline on the same prompt pool as Step-DPO. We first construct a single-policy SFT checkpoint by fine-tuning Qwen-2.5-1.5B on the processed low-level trajectories in which all step descriptors are removed from both inputs and targets, using the same optimizer, schedule, and token budget our low-level SFT. Starting from this checkpoint, keeping a frozen copy of the SFT model as the reference policy. We implement the baseline using the Verl (Sheng al., 2024) framework and vLLMM (Kwon et al. 2023) as the rollout backend. The actor and reference are both initialized from the same SFT checkpoin, For each prompt, we sample groups of N 4 candidate continuations with temperature 0.7 and top-p 0.9, cap the maximum response length at 4,096 tokens to respect GPU memory limits, and assign rule-based outcome reward of 1 if the final answer is correct and otherwise. We optimize the actor with Adam (learning rate 5 x 10-?, weight decay 0.1) under KL-penalty objective with coefficient AKL 0.02, using b 6 precision, gradient checkpointing, and FlashAttentin (Dao 2023) on 4x A100 GPUs (80GB). We train for 4 epochs, using global batch size of 32 (PPO mini-batch sizes 16, micro-batch sizes 2, respectively). and evaluate every 100 steps on the held-out validation split, selecting the checkpoint with the best validation @ass@1.
apply full-parameter GRPO,
visualize the validation accuracy of Qwen2.5-1.5B SFT during GRPO training in Figure 13 For each question, we sample 8 responses and report the average accuracy to obtain a stable estimate. The evolution of the average response length during GRPO is shown Figure 14 and the final evaluation results summarized in Table [ Compared with strategy, GRPO is less efficient for
Qwen22 5 5-1.8 SFT average response length during training
2200
Average boogth per resperoe
1800
1600
1400
100
200
300
400
500
Steps
Figure 14: The average response length of Qwen2.5-1.5B SFT the training set during GRPO.
19

---

Published as a conference paper at ICLR 2026
long-horizon reasoning: outcome rewards are (i) sparse: for long trajectories, single scalar signal is often insufficient to localize errors; and (ii) computationally expensive: generating full rollouts requires substantial memory and compute. When starting from fine-tuned models with long CoTs, these costs limit our ability to apply GRPO to larger base models. In contrast, our iterative Step-DPO procedure is easier implement and control, and provides more practical alternative for finetuning long-horizon reasoning policies.
We repeat the above protocol on a 7B model. We first obtain a single-policy SFT checkpoint by fine- tuning 1ieen-255-MATH-177- on the same processed low-level trajectories (step descriptors removed from imputs and targets). Starting from this checkpoint, we run full-parameter GRPO with frozen copy of the SFT model as the reference. Training uses FSDP sharding, bf16 precision, activation checkpointing, and FlashAttention on AxA100 (80GB), with a global batch size of 32 implemented as micro-batch of 1 per GPU and 4 gradient-accumulltion steps. Rollouts are generated with vLLM: for each prompt we sample N candidates (temperature 0.7, top-p=0.9) and cap the maximum response length at 4,096 tokens. We optimize with AdamW (learning rate 5 x 10-?, weight decay 0.1) under a KL penalty with coefficient 0k020.02, and evaluate every 100 steps, selecting the checkpoint with the best validation Pass@1. 1.
We follow the same procedure with -lama-3.1-8B. A single-policy SFT checkpoint is first obtained on the same low-level trajectories, after which we apply full- aall-arammttr GRPO using a frozen reference initialized from the SFT checkpoint. We train with FSDP, b 6, activation checkpointing, and FlashAttention 8x A100 (80GB), using global batch size of 32 realized steps. Rollouts use vLLM with N= candidates per prompt
micro-bbtch of 1
per GPU and 4 gradient-accumulatee (temperature 0.7, top-p=0.9) and a 4,096-token cap. The optimizer, KL objective, evaluation cadence, and model selection criteria are identical to the 7B setting. For additional memory headroom, the frozen reference is sharded; when necessary. we load the reference 8-bit for forward-only KL without changing any other hyperparameters.
Plan-and-Solve baseline. We compare against Plan-and-Solve (Wang et al. 2023a), which first proposes a concise, global plan and then executes the solution conditioned on that plan. The example prompt is given in Section C For a fair comparison, we use the same backbone as our method and fine-tune two LoRA heads on top of a planner (Problem - Plan) and an executor (Problem + Plan + Solution). At inference we follow the standard two-pass Plan-and-Solvv pipeline: Pass-1 generates the plan; Pass-2 solves the problem conditioned on that plan.
Training data creation. Using the same training set as our method, we prompt strong teacher model (DeepSeek-Va) V3.2) to produce corresponding trajectories. We filter trajectories by final-answer correctness and basic format checks. We match the total number of accepted trajectories to method (80K) to ensure a fair comparison.
Training configuration. Unless otherwise noted, we freeze the backbone and train LoRA adapters with identical hyperparameters for planner and executor.
Backbone: Qwen-2.5-1.5B. LoRA: = 16, a 32, grggeemmddule==(qpprj k_proj. v-proj. o-proj]. no bias. Optimization: AdamW, cosine decay with 3% warm-up, learning rate 1 x 10-4.
Backbone: 2wwm-2.5--AATH-7B Same LoRA configuration. Same optimization configura- tion except for learning rate x 5
Backbone: 1lama-3.1-8B. Same LoRA configuration. Same optimization configuration
except for learning rate 5 x 10
Results. Table I summarizes performance. Because Plan-and- here trained only with SFT, we compare it against MLR (SFT-only). Across all three backbones, our method outperforms Plan- and-Solve, with the largest margins on the harder benchmarks (AIME, GPQA). We observe that Plan-and-Solve often implicitly assumes all subtasks succeed initially planned; errors in early steps can propagate, and the executor may partially deviate from the plan. contrast, our approach learns better planner that can adapt its plans based execution signals, enabling revisions rather than committing to a fixed blueprint. This adaptive coupling between planner and executor yields more stable long-horizon reasoning than prompting plan upfront and executing it verbatim.
20

---

Published as a conference paper at ICLR 2026
Evaluation. During evaluation, we use greedy decoding for both the base model and the instruction fine-tuned model to produce more coherent and consistent CoTs. For all other baselines and our method, we follow the decoding protocol in Guo et al. (2025) using sampling-bast based decoding with a temperature of 0.6 and a top-p value of 0.95 to generate 8 responses per prompt to reduce variance and repetition. For MIR, we employ single base LM for both levels and switch the high-level LoRA adapter on or off depending on the generation stage (Algorithm [. Specifically, we enable the high-level LoRA adapter to produce step descriptors (planning), and then disable the adapter to generate the corresponding low-level trajectories conditioned on these descriptors. The maximum generation length for all models is set 16,384 tokens. Performance is measured using @ass@1 1 = & Σi= Pt, where p; denotes the correctness of the i-th response. For AIME24, we also report consensus accuracy over 32 samples, denoted as @3n
C ADDITIONAL RESULTS
Examples of MLR outputs. In this section, we present additional results to further demonstrate and analyze the effectiveness of our method. We showcase representative output examples generated by MLR across different datasets (Section C). Each sample consists of two-level reasoning trajectory, comprising shared reasoning steps annotated with both a step descriptor and corresponding step content. In the high-level trajectory, the step descriptor is generated by the high-level module, while the step content is produced by the compressor, which takes the low-level content as input and outputs concise abstraction. In the low-level trajectory, the step descriptor is provided by the high-level module, and the step content is directly generated by the low-level base model.
Error analysis. To better understand the strengths and limitations of our framework, we conduct detailed error analysis. To further enhance verification and error localization, we incorporate auxiliary models (OpenAl's o1 and o1-mini) to assist in identifying potential reasoning flaws. Specifically, we first evaluate whether the auxiliary model can independently solve each task without access to the ground-truth final answer or reference solution. If the auxiliary model successfully produces the correct solution, we then use it to help analyze erroncous trajectories generated by our framework. The error analysis provided by the auxiliary model is subsequently reviewed and confirmed by human evaluators. Through this process, we identify several recurring error patterns: 1) High-level step descriptor errors: redundant branching (multiple step descriptors that pursue the same subtask), unclosed loops (steps are never marked "complete." leading to repeated revisitation), dead-end retention (contradicted or unproductive exploratory branches are retained), copy-pasted fallback (guessed answers are repeated verbatim under different step descriptors). 2) Low-level step content errors: logical misapplication (misuse of domain-specint rules or principles), contradiction tolerance (inconsistent constraints are not resolved), repetitive reasoning (redundant inference chains without new contributions), failure to propagate known facts (previously inferred information is ignored in later steps), looping filler (verbose or stalled reasoning with redundant rephrasing).
MATH500 (K=1)
MATH500
MATH500
MATH500 (K=16)
Estimated Utiley
Estimated Uniley
Extimated Uniley
Estimated Utilty
5
Reference Utility
Reference Utility
Reference Utility
Reference Utility
AIME24 (K=1)
AIME24 (K=4)
AIME24 (K=8)
AIME24 (K=16)
Estimated Utilit
Estimated Utility
Estimated Utility
Estimated Utility
0.5
0.5
0.0
0.0
-0.5
-0.5
-0.5
0.5 Reference Utility
0.0
-0.5
0.0
Reference Utility
Reference Utility
Reference Utility
Figure 15: Effect of rollout count on the reliability of utility estimates. We compare utilitics estimated by the I.5B model under K rollouts to referenec utilities by the 8B model using 16 rollouts for 100 sampled prefixes from MATH500 and AIME24, respectively.
21

---

Published as a conference paper at ICLR 2026
Parameter studies rollout count. In our iterative step-DPO, the rollout count K directly affects the quality of the preference pairs. We first examine how K influences the reliability of the utility estimates defined in Equation (6). To do so, we compare utilities estimated by the 1.5B model under various K to reference utilities produced by the 8B model with 16 rollouts, using 100 sampled prefixes from MATH500 and AIME24, respectively (Figure 15). As expected, larger K reduces estimator variance but increases computational cost. To control this overhead, we apply margin threshold 6 when selecting preference pairs which allows us use smaller К while maintaining reliability of the preference data. Next, we evaluate the agreement of selected preference pairs as function of K, defined as the fraction of pairs whose preference direction agrees with the base model using 16 rollouts (Figure 8). For each K, we generate 100 preference pairs following Equation (7) We then recompute the reference utilities of both options and check whether the chosen response has higher reference utility than its alternative. Pairs that satisfy this condition are counted as agreed, and we report the average agreement for each K. We consider both the 1.5B model and the 8B base model as rollout policy. In practice. we use the base model as the rollout policy for AIME24 and GPQA. Finally, we study model performance across training stages under different rollout counts (Figure 12). Starting from the same SFT model, we generate the same number of preference pairs for each K and all train for 4 epochs. We report performance on MATH500 and AIME24 throughout training. Overall, our setting achieves comparable accuracy while significantly reducing computational cost.
Ablations on hierarchical levels. To investigate the role of different levels, we conduct an ablation study on the hierarchical structure. We consider two variants: (1) High-level only: the high-level module directly predicts summaries without invoking the low-level module; (2) Low-level only: the low-level module is required to predict both the high-level step descriptions and the detailed reasoning without guidance from the high-level module. The evaluation protocol matches our main setting, and the results are reported in Table Our full method consistently outperforms both variants, especially on the challenging AIME24 dataset. The high-level-only variant underperforms because the planner lacks grounded execution learning, making direct summary prediction unreliable for difficult reasoning tasks. We show an erroneous example in Section C The low-level-only variant is weaker because the absence of explicit high-level guidance causes the low-level module to drift and accumulate errors as the trajectory grows longer. Overall, these results demonstrate that our two-level design yields better performance on long-horizon reasoning tasks.
MATH-500
AIMEZ4
Misture benchmarics
20K
Owen2 SB-miinek
1i:Dsttl-Qwun-... 1.58
Qwen2.5-L S8-Iniluck
RLDDstll-Qeen-1.. 1.58
Averago responca lergth
20K
R1 Ii 58
Qwen2: 5-1 5B-lmarckk
Owen2.5:1. se ERPO
20K
Owen2.5 5-5.8 GRPO
Owen2.5 5-55 GRPO Owen2 1ern2.5-1.50...
15K
Owen2.5-11.0 58 MUR
Qwth2.5-1.00 MUR
15K
L5K
Averape Co
1.K
10K-
13K:
SK
0K
Number Probioms
Number Probiems
Number Problems
AIME24
Mixture benchmarks
R1 Littl 56
Ulme--111- 1-1.111------me BeInsrrtt
25K
R1 Usmoll 36
1lama-3.1-ffl-natrree
Avurage rospparee kough
20K
R1 (isttll-lmmm 86
Ulmm-33.1 hnttrrrrrtt
12K
Uama-3. 3:1-9B-GRPO
1:K
Ulama-1.14 B-MRLL
20K
Ulama-1.1-an--
Ulma-111-40 1-88-XLR
15K
10K
SK
10k
5K
DKC
DKC
10
Number of Probiems
Number of Probiems
Number Probiems
(a) MATH500
(b) AIME24
(c) Mixture
Figure 16: Long-horizon reasoning response length on (a) MATH500, (b) AIME24, and (c) Mixture benchmarks (MATH500, AIME24, GPQA, BoardGameQA). We simulate long-horizon reasoning by concatenating multiple problems in the prompt and report average accuracy. Compared with RI-Di -Distill models, which are prone to getting lost in long trajectories and entering dead loops, MLR benefits from explicit planning and is able to solve all problems more effectively.
22

---

Published as a conference paper at ICLR 2026
Long-horizon reasoning test. To further evaluate method on long-horizon reasoning, simulate a multi-question setting by concatenating multiple problems into a single prompt. We consider three configurations: (i) multiple problems from MATH500, (ii) multiple problems from AIME24, and (iii) mixed batches drawn from MATH500, AIME24, GPQA, and BoardGimmeAA, For each configuration, treat their questions as mini-batch. The batch size varies from 1 to 10; for each size, we present all questions in the batch in one prompt and ask the LLM to solve them jointly. To mitigate cases where the model gets stuck on a single problem, we explicitly instruct it to skip question if it cannot make progress. Prompt template can be found in Section C For instruction-tungr models, we use greedy decoding. For the other baselines, for every mini-batch we sample 8 responses with temperature 0.6 and top p = 0.95, using a maximum context length of 32K tokens, reduce variance and then report accuracy averaged over the 100 problems. Note that all methods are trained only in the single-question setting, so this constitutes fully out-of-distribution evaluation. Figure ] shows the results: MLR consistently degrades more slowly than all baselines across datasets and base models, highlighting the benefits of explicit planning for long-horizon reasoning. We further report the corresponding length distributions for different scenarios in Figure 16
randomly select 100 problems from the corresponding benchmarks and
23

---

Published as a conference paper at ICLR 2026
Example: Long-horize Plan Failures (DeepSeck-Be SrepSeck-RI-Diee RI--Dstilll D--Distill-Llama--88 on MATH500)
Prompt:
What is the least positive integer multiple of 30 that can be written with only the digits 0 and
Response:
<think>>
Okay, I need find the smallest positive integer multiple of 30 that can be written using only the digits 0 and 2. Hmm, let think about how approach this.
First, remember that a multiple of 30 must be divisible by both 3 and 10. So, let's break down what that means.
Divisibility by 10: For number be divisible by 10, must end with O. That makes sense because any number ending in is a multiple of 10. So, whatever number come up with must end with O. That means the last digit is fixed as O.
Divisibility by 3: For number be divisible by 3, the of its digits must be multiple of 3. Since the number can only have digits and 2, sum of digits will just number of 2s multiplied by So, the total sum is times the number of 2s. This sum needs to be divisible by 3.
So. putting that together, need number that consists of more 2s followed by O. The number of 2s should be such that the total sum (which is times number of 2s) is divisible by 3.
Let me formalize that. Lct's say the number has 'n' number of 2s followed by a O. So, the number would be 222...20, with 'n' 2s and ending with O.
sum the digits 2n, and we need to be multiple 3. So. mod 3. Which means n must a multiple of 3 because and are coprime (they have no common divisors other than 1). So, n needs to be divisible by 3.
Therefore, the smallest number 2s needed right? Because 3 the smallest positive integer divisible by
So, if n 3, the number would be 2220. Let me check that.
2220 is multiple of 30? Let's see. 2220 divided by 74, which an integer. So, yes. 2220 is multiple of 30.
But is it the smallest?
Wait, maybe can find a smaller number with fewer digits? Let me think. The number must end with 0, so the last digit is fixed. The other digits can be 0 or 2. But if try to make smaller number, maybe with fewer digits, but still only Os and 2s.
Let me consider numbers of different lengths see they can satisfy the conditions.
Starting with the smallest possible number: 1 digit. It would have to be 0, but 0 is not positive. So, no good.
2 digits: The number would he 20. Is 20 multiple of 30? 20 divided by 30 is less than 1, so no. Also, 20 not multiple 30.
3 digits: Let's see. The possible numbers are 200, 220, and 222.
200: 200 divided by 30 is approximately 6.666, not an integer. 220 divided by 30 is about 7.333, not integer. 222 divided by 30 is 7.4, which isn't integer either. So, none of these are multiples of 30.
4 digits: Let's list possible numbers. They end with 0, the last digit is 0, and the first three digits are either O
Possible numbers: 2000, 2200, 2220, 2222, 2020, 2022, 2202, 2220, 2222, etc. Wait, but actually, the first three digits can be any combination of Os and 2s. So, all numbers of the form 2220, 2202, 2022, 0222, but 0222 is not valid number since starts with 0. So, the numbers are 2220, 2202, 2022.
Check each:
24

---

Published as a conference paper at ICLR 2026
2220: before, 2220 12200330000 74, which integer. So, that's multiple.
2202: 2202 / 30 is 73.4, which isn't integer.
2022: 2022/ / 30 67.4, which isn't integer.
So, the only 4-digit number that works 2220.
But earlier, with 3 digits, we had 2220 well. Wait, 2220 is 4-digit number, but 3-digit numbers didn't work. So, 2220 is the first one.
Wait, but hold on. Maybe there's 3-digit number that's a multiple of 30, but haven't considered all possibilities. Let me check again.
Wait, 2220 is a 4- digit number, but what about 3-digit number? 222 222, 222 divided by 30 is 7.4, which isn't integer. 2220 74. So, maybe 2220 is the first onc.
Wait, but hold on. Maybe 3-digit number with two 2s and a O. Let's see: 220, 202, 022. But 022 isn't 3- digit number, it's just 22. So, 220 220, which is 2202200330202 30 7.333, not integer. 202 is 202 /30 "= 6.733, also not integer. So, no luck.
So, in the 3-digit numbers, there's no multiple of 30 that can be formed with Os and 2s. So, moving on to 4- digit numbers.
Wait, but 2220 is 4-digit number, but that the smallest? Wait, let me think. Is there a smaller number with 4 digits, but starting with 2, and having 2s and Os, but not necessarily 3 2s? Wait, no, bocause if you have 2 2s, then 220, 202, 022. None of these are divisible by 30.
So, the only d-iitt number 2220, which 2220. wait, maybe a 5-digit number? No, 2220 already a d-digit number, which is smaller than 5-digit number. So, 2220 is smaller.
But wait, hold Let consider 2220. It's 2220, but is there smaller number with 4 digits that be formed with and Wait, 2220 2220. what about 2220, 2202, 2022, etc. Wait, none of other permutations and Os in digits are multiples of 30. So, 2220 the only one.
But wait, is 2220 the minimal? Because 2220 is a 4 4-digit number, but is there a 3-digit number that's multiple of 30? Wait, don't think so, hecause 2220 is the first multiple of 30 with only 2s and Os.
Wait, hold on. Let me think differently. Maybe instead of 2220, there's a smaller number.
Wait, 2220 is 2220, but if can find number with fewer digits, like 220, but 220 is not multiple of 30. 220 divided by 30 is 7.333, Not integer.
Wait, 202: 202/ / 30 is 6.733. Not integer. 200: 200 /0 is 6.666. Not integer. 222: 222 / /0 is 7.4. Not integer. So, none of these are multiples of 30.
Wait, maybe 2220 the first one. 2220 a 4-digit number. Is there a 3-digit number that can be formed with 2s and Os that is a multiple of 30? Let me think.
Wait, 2220 is the first one. So, 2220 is 2220. But 2220 is 2220, which is 2220. But if can write 3-digit number with 2s and Os that's multiple of 30, that would be smaller than 2220.
Wait, 2220 is 4 digits, but 222 is 3 digits. But 222 is not multiple of 30. So, no.
Wait, maybe 220 is 220. but that's 220, which is not multiple of 30. 200 is 200, which is not multiple of 30. 222 is 222, which not multiple of 30. So, yeah, no luck.
So, 2220 is the first one.
Wait, but hold Let me think again. If have number that is 200...0, but with some 2s in between. But the sum of digits must be a multiple of 3, so as we saw earlier, the number of must be a multiple of 3.
So, with 2s. we get 2220, which is 2220. But is there a way to have 2s in other positions? Like 2020, 2002, 2022, etc. But saw, those are not multiples of 30.
25

---

Published as a conference paper at ICLR 2026
Wait, 2020: sum digits 2 0++0++++00 +=+++0=44 which is multiple 3, it's not divisible by hence not divisible by 30. Similarly, 2002: 2 0++ 0 = 4, same thing. 2200: 2 + + 0= 0, same. 2220: + + +0= 6, which is multiple of 3. So, 2220 is the first one.
So, perhaps 2220 is the minimal.
Wait, but hold on. Let me check if 2220 is indeed the minimal. Maybe a number with digits but starting with 2. and with other digits as 2s and Os, but the number of 2s is more than 3? But no, because 2220 is the first where the number of 2s is 3, which is the minimal multiple of
Wait, but maybe 2220 the minimal, is there a smaller number? Let mc think.
Wait, 2220 is 2220, which is 4 digit number. Is there 3-digit number with 2s and Os that is multiple of 30? As we saw, 2220 is the first one. So, 2220 the minimal.
Wait, but hold on. Maybe can have a number like 2200, which is 2200, but the sum is 4, which isn't a multiple of 3. So, not divisible by 3. So, no.
Wait, 2202: sum is 2 = which multiple of 3, but 2202 divided by 30 is 73.4, which isn't integer. So, not multiple.
Similarly, 2022: sum is 2 0 + + = 6, which is multiple of 3, but 2022 / is 67.4, not integer. So, no.
2002: sum 4, not multiple 3.
2220: sum is 6, which is multiple of 3, and 2220 /:00//011 is 74, which is integer. So, that works.
Therefore, 2220 is the minimal number.
Wait, but hold on. Let me think if there's a smaller number.
Wait, 2220 is 2220, which is 4-digit number. Is there 3-digit number that be formed with 2s and Os, which is multiple 30? As we saw, 2220 the first one, so no, there's no smaller number.
Wait, but hold on again. Let me think about 2220. Is the minimal, or is there smaller number with 4 digits but starting with 2, and having 2s and Os, but arranged in different way?
Wait, for example, 2020: sum 4, which not multiple so not divisible by 2002: same. 2200: same. So, no.
So. 2220 the minimal.
Wait, but hold on, let me think of another approach. Maybe think of numbers base 10, with digits only 0 and 2, and divisible by 30.
So, the number must end with 0, and the of digits must multiple of 3.
So, as we saw, the number must have a number of 2s equal to 3, 6, 9, etc.
Therefore, the minimal number is 2220, which is digits.
Wait, 2220 is 4 digits. So, 2220 2220, which is digits, and the first one.
But wait, 2220 is 4-digit number. Is there 3-digit number? 2220 is 2220, which is 2220. But 222 is digits, but 222 not multiple of 30.
Wait, so 2220 is the minimal.
Wait, but I just thought of another way. Maybe if we can have number with 0 the end, and two 2s, but also have another 2 somewhere the middle. But the sum of digits must be multiple of 2 0 4, which isn't multiple 3. So, that's not good. So, 2220 is minimal.
Wait. but 2220 is 4-digit number, but 2220 is 2220, which is 2220, which is 2220. Hmm, but 2220 is the minimal.
26

---

Published as a conference paper at ICLR 2026
Wait, hold on, but is 2220 the minimal? Let me think of 20220, but that's 5-digit number, which larger than 2220, so not helpful.
Wait, 2220 is 2220, which is 4 digits, and it's the minimal.
Wait, but hold on, maybe 20220 S-digit number, but 2220 smaller.
Wait, so 2220 is 2220, which is 4 digits, and it's smaller than 20220.
Therefore, 2220 is the minimal.
Wait, but now, hold on, another thought. Maybe number with 2220 is 2220, but is there number with 2220 in a different arrangement? For example, 20220 is 5-digit number, but 2220 is 4 digits, 2220 is smaller.
Therefore, 2220 the minimal.
Wait, but let me just confirm once again. 2220 2220, which is 2220. Let me see if any number with fewer digits be formed.
Wait, 2220 4 digits. 222 3 digits, but 222 not multiple 30. 2220 is digits, and multiple 30.
Wait, 2220 is the minimal.
Wait, but wait, 20220 is 5-digit number, which 20220, which is 20220. 2220 smaller.
Wait, but hold on, 2220 is 4-digit number, which is 2220, so 2220 is smaller than 20220, which is 5-digit number.
Therefore, 2220 the minimal.
Wait, but I just thought of another approach. Maybe think of the number as starting with 2, followed by 2s and Os.
So, for example, 2220, 2202, 2022, 2002, 2220, etc. as we saw, only 2220 a multiple of 30.
Wait, but 2220 is 4-iigit number, which is 2220. 2220 is 2220, which is 2220, it's 4-digit number.
Wait, but let me think the number 2220. the minimal?
Wait, 2220 is 2220, which is 2220. So, 2220 is multiple of 30, but is there smaller number?
Wait, 2220 2220, which 2220, which is 2220, which is 2220.
Wait, but 2220 is 2220, which is 2220, so is that the minimal? Or there number with fewer digits?
Wait, 2220 is 4 digits. 222 3 digits. 222 is not multiple of 30. So, 2220 is the minimal.
Wait, but wait, another idea: 2220 is 2220, which is 2220, which is 2220, which is 2220.
Wait, but if can have number with 2s and Os, but with leading 2s and trailing Os, but in hetween, perhaps 2s can be placed in different positions.
Wait, but as we saw, the sum of digits must be multiple of 3, so the number 2s must be a multiple of 3.
Therefore, the minimal number 2220, which is 2220.
Wait, so 2220 is digits, and it's the minimal. So, 2220 is the answer.
Wait, but wait, just thought of something else. What if the number is 20220? That's 5 digits, but it's 20220, which is 20220. 2220 is 2220, which is smaller. So, 2220 smaller.
Wait, but 2220 is 2220, which is a 4-iigit number. 20220 is 20220, which is a 5-digit number, so 2220 is smaller.
27

---

Published as a conference paper at ICLR 2026
Therefore, 2220 is the minimal.
Wait, but I'm confused because 2220 is 2220, which is 2220. So, that's a 4-digit number, and it's the minimal.
Wait, but hold on, let me just confirm if 2220 is indeed the minimal.
Wait, 2220 2220, which 2220. me check 2220 a multiple of 30, 2220 divided by 30 74, which is an integer. So, yes, it is.
So. 2220 the minimal.
Wait, but hold on, 2220 2220, which is 2220, which a 4-digit number.
But, is there smaller number?
Wait, 2220 2220, which 2220. But, 2220 2220, which 2220. So, it's 2220, which 2220. So, 2220 the minimal.
Wait, but wait, 2220 2220, which is 2220, which is 4-digit number. So, 2220 is 2220, which is 2220.
Wait, so that's the minimal.
Wait, but wait, hold on, 2220 is 2220, which is 2220, which is 2220. So, it's 2220.
Wait, 2220 minimal.
Wait, but hold on, another thought. Let me think of numbers with 2s and Os, but in different positions, but the sum of digits must be multiple of 3.
So, 2220 2220, which is 2220, sum 6, which is multiple
Wait, but if have number like 20220, which is 5 digits, sum is 6, which is multiple of 3, but 20220 is 20220, which is larger than 2220.
So, 2220 the minimal.
Wait, but hold on, another idea. Maybe 20220 is 20220, which is 20220, which is 5- number, which is 20220, which is larger than 2220, which 4-digit.
So. 2220 smaller.
Wait, 2220 is the minimal.
Wait, but, wait, let me think the number 2220. It's 2220, which is 2220, which 2220. So, 2220 digit number.
But, 2220 2220, which is 2220, that's 2220.
Wait, so, yeah, 2220 is the minimal.
Wait, but, hold on, I just thought of another number. Let's say 2220 is 2220, which is 2220. But, 2220 is 2220, which is 2220. So, is there smaller number?
Wait, 2220 is 2220, which 2220, which is 2220, which is 2220. So, 2220 is 2220, which is 2220. So, 2220 is 2220, which is 2220.
Wait, so, that's the minimal.
Wait, so, conclusion: 2220 is the minimal.
Wait, hut hold on, just thought of something else. Maybe number like 2220 is 2220, but 2220 2220. which 2220, which is 2220, which 2220.
Wait. so, yeah, 2220 is 2220, which is 2220, which is 2220.
28

---

Published as a conference paper at ICLR 2026
Wait, so, 2220 is 2220, which is 2220, which is 2220.
Wait, so, yeah, 2220 is 2220, which 2220.
Wait, so, that's 2220, which is 2220, which is 2220.
Wait, but, hold on, let me think again.
Wait, 2220 is 2220, which is 2220, which is 2220.
Wait, but 2220 is 2220, which is 2220.
Wait, so, yeah, 2220 is 2220, which is 2220.
Wait, so, 2220 is 2220, which 2220.
Wait, so, that's 2220, which is 2220.
Wait, so, yeah, 2220 is 2220.
Prompt: Boundary Detection
[SYSTEM PROMPT]
You reasoning- trajecccott segmenter.
Decide whether two adjacent paragraphs belong to the SAME subgoal.
Return JSON only.
[USER PROMPT]
Task: Determine if two adjacent paragraphs belong to the same subgoal.
Rules:
YES if both paragraphs pursue the immediate objective.
NO the next paragraph introduces a new objective/stage (e.g., starts a new computation, introduces a new intermediate target, or shifts to verification/linalzaaii
Self-correction without changing the objective => YES.
Problem: {problem)
Context before the current paragraph: (prev.context
Current paragraph: (cur_paragraph}
Next paragraph: |arx-_aarag))hh)
Output JSON schema:
"decision": "YES" or "NO", "confidence"; 0.0 to 1.0, "reason":
Prompt: High-level Abstraction Labelling
[SYSTEM PROMPT]
You are high-level abstraction labeler.
Given merged reasoning chunk, output exactly one reasoning mode, one subgoal, and one summary.
Use neutral factual style and avoid personal pronouns.
29

---

Published as a conference paper at ICLR 2026
Return JSON only.
[USER PROMPT]
Task: Label this merged chunk as ONE subgoal.
Reasoning mode candidates:
GroblemUnderstandding restate the goal, identify givens/unknowns, clarify constraints.
Planning: outline substeps, decompose, decide the next objective (without executing).
Recall: state relevant defintiow/theorem//mmaaaaaa needed for the solution.
Derivation: derive non-trivial intermediate claim/lemma (not direct computation).
Calculation: perform aigchr//arihmmtii/symoolll manipulation or explicit computation.
CascAnalysis: explore alternatives or split into cases without abandoning the current direction.
Verification: check correcness/consisteneyy (plug-in, sanity check, dimensionnbounds, cross-check,
ErrorCorecttio: fix detected mistake while keeping the overall approach.
Backtracking: explicitly abandon/revise prior hasrracccsssumpptionn and switch direction.
Synthesis: combine intermediate results to reach the final conclusion.
Finalization: present the final answer or finalize formatting.
Other
Constraints:
English.
Subgoal: action object style, <= 30 words.
Avoid repeating prior subgoal names (paraphrase if needed).
the chunk mainly corrects prior step, set subgoal to: Refine <prior subgoal>> (paraphrase needed). Summary: neutral and factual; avoid personal pronouns wwWe/Yoo/They///n user); preserve key intermediate results and decisions.
Problem: [problem]
Previous steps: (bistory-text]
Merged chunk: [merged text)
Output JSON schema:
"reasoning. mode": "..". "subgoal": "...".
"summary": ".."
Prompt: MLR (high-level)
[SYSTEM PROMPT]
You high-level planner. Given the problem and completed history, generate reasoning mode and next subgoal.
Return only two required lines DONE). Do not provide low-level solution details.
[USER PROMPT]
Instruction:
Generate the next reasoning mode and next subgoal.
Rules:
Output exactly lines in this format:
reasoning mode: <one label>
nubt_sub <text>>
II the problem is already solved, output exactly: DONE
reasoning mode must be one of: PrroblemUnderstanding Planning. Recall, Derivation, Calculation, Anaelss, Verification, FrrorCorrection, Backtracking, Synthesis, Finalization, Other)
30

---

Published as a conference paper at ICLR 2026
subgoll must be action object style, <= 30 words not repeat previous subgoals; unavoidable, paraphrase Keep minimal and actionable
Problem: {problem)
Completed history: [history _text)
Continue:
Prompt: MLR (low-level)
[SYSTEM PROMPT]
You are a low-level executor.
Focus strictly on the given reasoning mode and the current subgoal. Do not start future subgoals.
Output only the reasoning content (no headings, labels).
End your output with single line: <END STEP>
[USER PROMPT]
Continue solving the problem. Only complete the current subgoal.
Problem: {problem}
Existing reasoning: [reasoning prefix)
Reasoning mode for this step: (reasoning-mode)
Current subgoal: {subgoal}
Continue:
Prompt: MLR (summarization)
[SYSTEM PROMPT]
You are step summarizer hierarchical planning.
Return only the summary text (no headings, no bullets, no extra text).
[USER PROMPT]
Summarize the current step into neutral and factual for high-level planning.
Keep: key intermediate results/values, assumptions/constraints,, method choice, and any corrections. Drop: long derivations, repetitive explanations, and formatting. Do propose steps reasoning. No personal pronoums.
Problem: [problem]
Completed history: (history_text)
Current step: Mode: (reasoning mode] Subgoal: (subgoal}
Content: tstcp_text]
Summary:
31

---

Published as a conference paper at ICLR 2026
Example: MLR Output (high-level) MATH500
Problem:
The set of points (ar, y, 2) that satisfy 2r 3y is line.
The set of points (2, y, 2) that satisfy 6x yy 42 another line.
Find the angle between these lines, in degroes.
Solution:
Mode: SomblemmUnderstandde
Subgoal: Identify principle for angle between 3D lines
Summary:
The angle between two lines in 3D space determined the angle between their direction vectors.
Mode:
Calculation
Subgoal: Compute direction vector of line
Summary:
Direction vector of the first line is (3, 2, -6).
Mode:
Calculation
Subgoal: Compute direction vector of line 2
Summary:
Direction vector is (2, -2.
Mode:
Calculation
Subgoal: Compute angle via dot product Summary:
Direction vectors: (3, -6)
(2, -12. -3). -24 =8
Dot product: 3+2 + 2+(-12) + (-6)+(-3)
Since the dot product is zero, the vectors are perpendicular. Therefore, the angle between the lines is 90 degrees.
Mode: Verification
Subgoal: Verify perpendicularity and angle
Summary:
Both lines pass through the origin and intersect there.
Their direction vectors are perpendicular (dot product zero).
Therefore, the angle between the two lines is 90 degrees.
Mode: Finalization
Subgoal: State final angle
Summary:
Thus, the final answer is 90
Example: MLR Output (low-level) MATH550
Problem:
The set of points (z, y, 2) that satisfy 2r 3y -z is a line.
The set of points (r., y, 2) that satisfy 6ur - 42 is another line.
Find the angle between these lines, in degrees.
Solution:
Mode: ProblemUndee Understanding
Subgoal: Identify principle for angle between 3D lines
Content:
Okay, so I need to find the angle between two lines 3D space. The lines are given by the equations 2x 3y -2 and 6x -y = -42. Hmm, let me recall how to approach this. think the angle between two lines is determined the angle between their direction vectors. So first, need to find the direction vectors of each line.
Mode: Calculation
Subgoal: Compute direction vector of line
Content:
32

---

Published as a conference paper at ICLR 2026
Starting with the first line: 2x 3y Let parse this equation. It's saying that 2x, and all equal each other. Let mc denote this common value as some parameter, say t. So, By t, - Then, solving for x, y, z in terms t:
x= 1/2, 1/3, z=
So the parametric equations for first line are:
x= (1/2),
y -1/3),,
z.-t.
Therefore, the direction vector of the first line is given by the coefficients of which is (1/2, 1/3, -1). But make it easier, maybe can multiply by to eliminate
denominators. Multiplying each component by gives (3, 2, -6), So the direction vector is (3, 2, -6). Let me check that: starting from 2x = 3y = -z, I set = 6x, then 3y = 6x => 2x, and -Z = 6x => -6x. So the direction vector is (x. 2) = (x. 2x, -6x), which in the direction of (1, -6). Wait, but earlier got (3, 2, -6). Hmm, maybe made mistake there. Let me clarify.
2x = then let me set 2x = k. Then 3y = y= k/3, and =z=k=>=====--- So k/2, k/3, =-.
So the direction vector is (k/2, k/3, -k). To make this a direction vector. we can take k = 6 (the least common multiple of denominators 2 and 3). Then = 2, -6. So the direction vector (3, 2. -6). So that's correct. Okay, so direction vector (3, -6).
Mode:
Calculation
Subgoal: Compute direction vector of line
Content:
Now second line: 6x = = Similarly, let's set m. Then y= and -4z = =-m/4. So x = m/6, y= -m, z= m//4. Therefore, the direction vector (m/6, -m, -m/4). Again, to eliminate denominators, multiply by 12 (the least common multiple of and 4). Then x 2 == So the factor
common factor. Let's see, -22 -3. The greatest common divisor so maybe that's simplest form. Alternatvely, we can divide by 1, so the direction vector is (2, -12, Alternatively, maybe we can write (2, -12,-3) or simplify by dividing by a common factor. Let's check: and -12 have a common factor of 2, but -3 doesn't. So maybe (2, -2, -3) is the simplest. Alternatively, if we factor out a it's still the same. So direction vector is (2, -22--3) -3).
direction vector is 12. -3). Alternatively, simplifying further,
Mode: Calculation
Subgoal: Compute angle via dot product
Content:
Now, find the angle between the lines, we need the angle between their direction vectors and The formula for the angle between two vectors is:
cos (v //(1vv/] ul)
So first, let's compute the dot product and u. is (3, -6), (2, -12, -3).
Dot product = (3)() (2)(-12)+ (-6)(-3) 6 -24 18 is -18. -18 +18 is O. Wait, the dot product is zero? That would mean the vectors are perpendicular, the angle between them is 90 degrees. But that seems surprising. Let doublc-check the calculations.
u-u (3)()) + (2) + (-6) 6 18.
6 -- -s -18 +18 is 0. Yep, that's correct. So the dot product is indeed zero, which implies that the vectors are orthogonal. Therefore, angle between them 90 degrees. Hmm, that the answer? 90 degrees? That seems straightforward, but let me confirm once again.
Alternatiely, mayhe I made mistake in determining the direction vectors. Let me check the direction again.
First line: 2x = 3y = -2. Let me parametrize it differently. Let's set = k. Then 3y by=k=>> y= =k3, and = k=> 2 = -k. So = k/2, 1/3, z = -k. So direction vector is (k/2, I/3, -k). To eliminate fractions, multiply
by 6: (3, 2. -6). Correct.
33

---

Published as a conference paper at ICLR 2026
Second line: 6x -y -4. Let's 6x m. Then -y= -=-m, and -4z => z= =m/4 So 16, y = -m, Z -m/4, Direction vector (m/6, -m, -m/4). Multiply by 12: (2, -12, -3). Correct.
So direction vectors are (3, 2, -6) and (2, -12 -3). Their dot product is 3+2 + 2=x (-6)a(-3)=
18 So yes, they are perpendicular. Therefore, the angle between the lines 90 degrees.
Mode: Verification
Subgoal: Verify perpendieularity and angle
Content:
But wait, just be thorough, maybe I should check if the lines indeed skew lines intersecting lines. If they intersect, then the angle between them is still defined as the angle between their direction vectors. If they are skew lines, the angle is still the same. So regardless, the angle between them is determined by their direction vectors.
Alternatively, maybe I can check are parallel or intersecting, they are skew. But since their direction vectors are not scalar multiples, they are not parallel, so they are skew lines. But the angle between them is still the angle between their direction vectors.
lines are coplanar or not, since they are space, unless they
Alternatively, mayhe the problem is designed such that the angle is degrees, me verify once more.
Alternatively, maybe I can check with another method. Let me compute the cross product of the direction vectors find the sine of the angle, but since the dot product is zero, the angle must be 90 degrees.
Alternatively, if the dot product is zero, the vectors are perpendicular, so the angle degrees.
Alternatively, maybe the problem is designed to have right angle, but need to confirm.
Alternatively. maybe made a mistake in direction vectors. Let me check again.
First line: 2x = 3y = -z. Let me take point on the line. Let me choose x=0. Then By = =zz Let me set y=0, then z=0. So the point (0,0,0) is the line. Then, direction vector is (1/2, 1/3, found earlier. So direction vector (1/2, 1/3, -1).
take t=1, the point (1/2, -1). direction vector (1/2, 1/3, -1). Multiplying by gives (3, 2, -6). Correct.
Second line: 6x = = Let me take x=O. Then -y= point (0,0,0)
Let set 7=0, then y=0. So the this line as well. Wait, both lines pass through the origin? Then they intersect at the origin. if they intersect, then they are coplanar, but 3D space, two lines that intersect are coplanar, but the angle between them is still determined by their direction vectors. However, if they intersect, the angle between them is the angle between their direction vectors, which is 90 degrees. So even if they intersect, the angle between them is 90 degrees. So that's consistent.
But wait, both lines pass through the origin, and they intersect at the origin, but their direction vectors are perpendicular, then the angle between them is 90 degrees. So that's correct.
Alternatively, mayhe I can check with another point. Let's take point on the first line, For example, take x=1. Then 2=1=3y => y=3/2, and -z=3w1 => z=-3.
point (1. 3/2, -3). Similarly, take x=1 the second line. Then 6=1=-y=>> y=-6, and -42= -6 => 2=1.5. So the point (1, -6, 1.5). Now, check if these two points are connected by a line that is perpendicular to both direction vectors. But maybe that's overcomplicating. Since the direction vectors perpendicular, the angle is 90 degrees.
Alternatively, maybe I can check the angle using another formula. The angle between two lines given by the angle between their direction vectors. Since the dot product is zero, the angle is 90 degrees. So think that' s correct.
Therefore, the angle between the two lines degrees.
Mode: Finalization
Subgoal: State final angle
Content:
Thus. the final answer is 90
34

---

Published as a conference paper at ICLR 2026
Example: MLR Output (high-level) AIME24
Problem:
Find the number of triples of nonnegative inlegers b) satisfying b 300 and b²b + a&e + u + &e + da + e²6 = 6, 000, 000.
Solution:
Mode: ProblemUnnderstanding
Subgoal: Restate constraints and rewrite expression
Summary:
Triples (a, b, c) of nonnegative integers must satisfy:
a b c 300
-a + a b'a b aaa c+b 6, 000,00000000 000
sum b + ae + baa + b a c can be rewritten as ab (a + e) (a + b).
Mode:
Verification
Subgoal: Verify factoring into grouped terms
Summary:
Factoring terms: abb + 2²c a² c)
bba bec (2
caa + eb bb)
So, the expression simplifies to ab B(CC) (
Mode:
Calculation
Subgoal: Substituting the sum constraint
Summary:
Expression after substitution:
a² (3000 - a) + 6 (300 b) + 23000 - c)
Main equation:
a² (300 a) + 62 (300 - b) + 2300 - c) = 6, 00.000, 000
Mode:
Calculation
Suhgoal: Expanding and simplifying the equation
Summary:
Expanded expression:
300(a² + b² + 2²) (a + b² =²)== 000, 000
This relates to the sum a + ++c c 300.
Mode: Derivation
Subgoal: Expressing in terms symmetric sums Summary:
S1 = + + == 300
S2 ab be
S3 abc
a² 63 + è² S12 -252
83 + b3
+c² S13 351 353
Substituting into the equation:
300(S12 -222) (513 3S1S2 353) 6, 000, 000
Mode: Calculation
Subgoal: Plugging in the known value and simplifying
Summary:
First term: 27, 000, 000 600S2
Second term: 27, 000,000000000 000 + 90052
333
Combine: 27, 000, 000 60055
27, 000, 000 + 90052
353 = 6, 00000,000 000
Simplify: 300S2 353 000,0,0000 000
Divide by 3: 100S2
S3 22,000,00
35

---

Published as a conference paper at ICLR 2026
Mode: Planning
Subgoal: Analyzing the new equation and possible values
Summary:
Equation: 100.5; S3 2, 000, 000
Given: S + + 300, S2 ab + bc ca, S3 abc
Need: Integer solutions for nonnegative a, b, with a c 300 and 100S2 S3 = 2, 000, 000.
Approach: Consider cases with symmetry (e.g.. two variables equal or all equal) to find possible solutions.
Mode: CaseAnalysis
Suhgoal: Checking the case where all variables equal
Summary:
If a = = 100, then + b c 300, S2 = 30, 000, S3 1, 000, 000, and 100S; S3 2, 000, 000,
which satisfies the condition.
So. (100, 100, 100) is solution, and since all equal, there is only such triple.
Mode: CascAnalysis
Subgoal: Checking the case where two variables are equal
Summary: Case: a= f
c = 300
$2 = -2--3 600a
S3 223000 2a)
Equation:
a"3 - -300a 2 + 30,000a - 1,000,000= =0
Factor: - 100) = = 100 is the only solution.
Conclusion: The only solution
case is bsabb=e c 100; no other solutions when two variables are equal.
Mode:
CaseAnalysis
Subgoal: Considering the case where all variables distinct
Summary:
All variables distinct:
Given S = 300, 1 1005; - -33 2, 000, 000, and S3 = 1005; 2, 000,000 000.
Since S3 abc 2 0, S3 2 20, 000.
Maximum S2 30, 000, minimum S2
So, variables distinct, S, must between 20,000 30.000, and S, 100S; 2, 000, 000. The original equation reduces to 300S; 35, 6, 000, 000, confirming the earlier result.
Mode: Derivation
Subgoal: Factoring the equation for further simplification
Summary:
Given 100S2 - S3 = 2,000,000 and S1 300, S3 = 100S2 2,000,000, and S3 abc, so
abc 100(ab + bc ca) 2.000,000.
Rearranged and factored.
(a - 100)(b - 100)(c 100) = 0.
Mode:
Derivation
Subgoal: Interpreting the factored equation and finding solutions
Summary:
At least one of a, b. or must be 100.
The other variables sum 200 (since b + 300).
nonnegative integer triples (a, b, e) where one 100 and the other two sum 200 solutions.
This condition is equivalent to the original equation, so all such triples satisfy both equations.
Mode: Calculation
Subgoal: Counting the number of valid triples
Summary:
Case I: == 100 + 201 triples
Case 2: b = 100 = 201 triples
Case 3: c 100 2011 triples
36

---

Published as a conference paper at ICLR 2026
Triple (100,100,100) is counted times (once in each case).
Using inclusion-exclusion:
Total triples = 3x
601.
Mode: Verification
Subgoal: Verifying the counting with small example
Summary:
Listed all possible triples: (1,0,2), (1.1.1), (1.2.0), (0.1.2), (2.1.),, (0.2.1), (2.0.1) - total of 7 triples.
Using inclusion-exclusion. 3 x 3
- --3+1=1
Conclusion: The formula works for this small example.
Mode: Finalization
Subgoal: Extract the final answer
Summary:
Thus, the final answer is 601
Example: MLR Output (low-level) AIME24
Problem:
Find the number triples nonnegative integers (a, c) satisfying a + b + 300 a²b c b3a &e d' ebb 6, 000, 000.
Solution:
Mode: Problemunnderstadding
Subgoal: Restate constraints and rewrite expression
Content:
Okay, so the user is trying to solve problem involving finding triples (a, b, c) of nonnegative integers that satisfy two conditions: a + b 300 and a²2b a"2c b'2a b'2c c"2a c"2b 6,000,000, They started by considering factoring the expression and noticed that it can be rewritten 22b c) b'2(a c'2(a b).
Mode: Verification
Subgoal: Verify factoring into grouped terms
Content:
First, need to verify their initial approach is correct. Let's check expression:
Original expression: a"2b + a"2c b"2a b c"2a e*
Factoring terms:
a'2b + 3bbb+a22c::00 a
b'2a b
c'2a + c22b=
So yes, the expression
to
b). That's a good start.
Mode: Calculation
Subgoal: Substituting the constraint
Content:
Next, since a + b c 300, we can substitute (b + e) with (300 - a), similarly for the others. Let's apply that
substitution:
Expression becomes: 22(3300 +) c22
So the equation is: a a02(3000 - bb2 c - c) 6,000,000
Mode: Calculation
Subgoal: Expanding and simplifying equation
Content:
Now, the user might need to find to relate this to the sum + 6 c = 300. Perhaps expanding the terms or looking for symmetry could help. Let's consider expanding each term:
37

---

Published as a conference paper at ICLR 2026
For each variable (where is or c), the term x'2 x) 300x x'3. So the entire expression becomes:
300(a 2 +3 6,0000000
So have:
300(a + b22 + (3 (
Mode:
Derivation
Suhgoal: Expressing in terms symmetric
Content:
Now, we also know that a + b + c 300. Let's denote $1 = 3300, S2 ab be + ca, and S3 abc. We can express a22 b c22 in terms of S1 and S2:
a'2 + +:2 +
2(ab + ca) S1"2 2S2
Similarly, the sum of cubes can be expressed using the identity:
a"3 b"3 c"3 (a b e) 3(a b c)(ab be ca) 3abc -113-31182 +S3
Substituting these into the equation:
300(S1 282) (S1 3S1S2 3S3) 6,000,000
Mode:
Calculation
Subgoal: Plugging in the known value and simplifying Content:
Plugging in S1 300:
300(300'2 - 2S2) (300'3 90009 + 353) 6,000,000
Let's compute each term step by step:
First term: 300=(300'2 282) -,00
Second term: -3300'3 90000000222 -00 -002
Combining both terms:
27,000,000 - 600S2 + 900S2 3S3 6,000,000
Simplify:
(27,000,000- -27,000,000) + (-600S2 900S2) -3S3 = 6,000,000
Which simplifies
300S2 353 = 6, 000, 000
Divide both sides by 3:
100S2 = 2,000,00
Mode: Planning
Subgoal: Analyzing the equation and possible values
Content:
So now we have an equation involving S2 and S3: 100S2 - S3 2,000,000
But we also know that S1 300. S2 ab be + ca, and S3 abc. So need another relation between $2 and S3. However, with three variables and equations ($1, $2, S3), might need find possible integer solutions.
Alternatively. perhaps can consider that a, b. are nonnegative integers summing to 300, and we need to find the aumber of triples such that 100S2 S3 2.000,000.
38

---

Published as a conference paper at ICLR 2026
This seems complicated. Maybe there's symmetry or specific values that satisfy this equation. Let's consider possible cases where two variables are equal, or all three are equal.
Mode: CaseAnalysis
Subgoal: Checking the case where all variables equal
Content:
First, check if all three are equal: a a=b = c. Then a b c= =a= 300 + a 100. So (100, 100, 100). Let's check if this satisfies the second equation.
Compute $2 ab be 3+100+ (00 30,000
S3 = 100'3 = 1,000,000
Then 100S2 S3 100=30,000 1,000,000 3,000,000 1,000,000 2,000,000. Hey, that works! So (100, 100, 100) is solution. But problem asks the number of triples, so this one solution.
But there might be others. Let's check if permutations of this count, but since all are equal, there's 's only one such triple.
Mode: CaseAnalysis
Subgoal: Checking the case where two variables are equal Content:
Now, consider cases where variables equal. Let's say a of Then = 2a 300 300
Now, compute S2 and $3:
$2 a: 2ac a 2a(300 =a2 600a 4a"2 -3a"2 600a
S3 = a 22c 232000 2a)
Now plug into 100S2 $3 2,000,000:
100s(-3a'2 600a) a22(300 2,) 2,000,00
Compute each term:
100a(-3a'2 600a) -300a"2 + 60,000a
Subtract a 2220000 2a) -300a2 2 + 2a"3
So total equation:
-300a"2 60,000a - 300a 2a"3 2,000000
2,000,000
Combine like terms:
2a"3 600a'2 + 60,000a 2,000,000
Divide both sides by 2:
a'3 300a 2 + 30,000a 1,000,000
Now, we need to solve this cubic equation for integer where must be nonnegative integer such that 300 2a 0
150.
Let's try possible integer roots. Possible rational roots are factors of 1,000,000 divided by factors I (leading coefficient). So possible roots are divisors of 1,000,000, Let's test a = 100:
100'3 300= 100 30,000: 100 1,000.000 1,000,000 3,000,000 3,000.000 1,000,000 0. 100 root. That corresponds to the case where 100, c 100, which we already considered.
Factor out (a 100):
39

---

Published as a conference paper at ICLR 2026
Using polynomial division synthetic division:
Divide a"3 300a 2 + 30,000a 1,000,000 by (a - 100).
Using synthetic division:
100 -30 30,000 -1,000,000
100 20,000 1,000,000
-00 10,000 0 So the cubic factors as (a 100)(a"2 200a 10,000)
Now solve 200a 10,000
Discriminant D = 40,000 40,000 0. a (200)/2 100. Thus, the cubic factors as (a 100)"3. 3. So the only solution is 100. Therefore, in the case where two variables equal, the only solution is when all three equal. So other solutions in this case.
Mode: CaseAnalysis
Subgoal: Considering the case where all variables are distinct Content:
Next. consider the case where all three variables are distinct. Let's see if there are any solutions where a, b, are different.
This might be more complex. Let's think if there's way parameterize the variables Vieta's formulas. Since we have S1 300. S2, and S3 related by 100S2 S3 2,000,000, perhaps we express S3 in terms 100S2 2,000,000.
But we also know that for nonnegative integers b, c, S3 abe must be nonnegative. So 100S2 2,000,000 0 > $2 2 20,000.
But $2 = ab
ca. Given that = 300, what's the maximum and minimum possible $2?
The maximum $2 occurs when two variables are large possible. For example, if one variable is 0, then
$2 = ab where b 300, which is maximized when b 150, giving $2 1500150- 22,500. If all variables are equal. S2 = 3+100= Wait, but earlier when we considered two variables equal, S2 was -3a"2 + 600a, which for a=100 gives S2 =30,000. So S2 can range depending on the distribution.
30,000. Wait, but b 100, S2 3=100= 30,000.
Wait, actually, can be calculated + + c)²2 (a"2+ 222/2, but maybe better think in terms of maximizing/mmnniiizing S2 given a + + 300.
The maximum of S2 occurs when two variables are as large possible, which would be when one variable is and the other =3w100~ 100 30,000, which larger. So actually. the maximum occurs when variables are equal. Because S2 ab + be + ca is maximized when the variables are equal. by the inequality that ab + be + ca S (a + +b++ + 3, which for a b e 300 gives S2 < 300 / 30,000, achieved when a = b 100.
to 300. Then $2 (150) 22,500. Wait, but earlier when all three equal, $2
Similarly, the minimum from to 30,000.
when one variable is 300 and the others 0, giving S2 0. So S2 ranges
But in our case, S2 must be at least 20,000 because S3 = 100S2 2,000,000 2 > S2 20,000. So S2 is between 20,000 and 30,000.
But how does this help us? We need to find triples (a, b, c) such that S2 between 20,000 and 30,000, and S3 100S2 - 2,000,000. Additionally, since a, b, are nonnegative integers summing to 300, we can model this solving for b, with these constraints.
Alternatively, perhaps there's substitution or another identity can used. Let's revisit the original equation:
a'2b 2c b12a b"2c c"2a cº2b 6,000.000
40

---

Published as a conference paper at ICLR 2026
Notice that this can be written as ab(a bc(b+ Wait, but that's similar the earlier expression. Altematively, factor differently;
=ab(a b) bo(b + c) ca(c a)
since + b 300, we can write a = 300 c, etc. So:
ab(300 c) bc(300 a) ca(300 b)
Which expands
300ab - abc + 300hc abc + 300ca abc
330((b be 3abc
So the equation becomes:
300S2 3S3 6,000,000
Which simplifies to 100S2 S3 2,000,000, which matches what we had earlier.
Mode: Derivation
Subgoal: Factoring the equation for further simplification
Content:
So we have 100S2 S3 2,000,000, and S1 300. So S3 100S2 - 2,000,000.
But S3 = abc, abe (00 be ca) 2,000,000.
This seems like complicated equation to solve for integers a, b, 2 with + 300.
But perhaps there's way to parameterize this. Let's consider that abe 100(ab be ca) - 2,000,000.
Rearranging:
abc - 100ab - 100hc 100ca = -2,000,000
Add 10000(a c) both sides:
abe 100ab 100bc 100ca + 10000(a + =) -22000,00000 10000+300
Compute the right-hand side:
-2,000,000 + 3,000,000 = 1,000,000
Now, the lent-hand side can be factored:
100ab 100be 100ca 10000a + 10000b 10000c
Factor terms:
abe (-100ab 100hc 100ca) 10000(a
Notice that this resembles (a - 100)(b 100)(c 100) expanded. Let's check:
Expand (a 100)(b 100)(c 100):
abe 100ab 100ac 100be 10,000a + 10,000b 10,000c 1,000,000
Compare to our hend- side:
Left-hand after adding 10000(a c):
abe 100aab -100be--0i -100ca + 10,000a 10,000b + 10,000c
41

---

Published as a conference paper at ICLR 2026
So the difference between this and the expanded (a 1100pb 1000)(c 100000000(000000 100) is the constant term -1,000,000. Therefore:
(a 100)(b - 100) 100) = abc -100ab 0000000-000000000000000000000000 00000000000000000000000000000000
But left-hand side
abc 1000ab 1000e - 100ca + +10,000a +10,000 (1000000000000000000000000---000 1,000,000
Therefore, the equation becomes:
1a-1000 -100) -100) + 1,000,000 1,000,000
So:
11000 -000(c -100)
Mode:
Derivation
Subgoal: Interpreting the factored equation and finding solutions
Content:
Ah! This simplifies things. Therefore (a 100) 100)(c 100) must zero. Which means at least one of the factors must be zero. Therefore, at least one of a, h, must he equal to 100.
So the solutions all triples (a. b, c) where at least of (since + 300).
100, and the other two variables
Therefore, the problem reduces to finding all nonnegative integer triples (a, b, c) such that one of them is 100, and the other two sum to 200. However, we also need to ensure that the original equation holds, but since derived that this condition is equivalent the original equation, all such triples will satisfy both equations.
Mode: Calculation
Subgoal: Counting the number of valid triples
Content:
Now, let's count number of such triples.
Case 1: a 100. Then b c 200. The number of nonnegative integer solutions for (b, c) is 201 (from h=0 to b=200). However, since triples ordered, each solution here corresponds to unique triple (100, b, c).
Case 2: 100. Similarly, a c 200. Number solutions: 201.
Case 3: = 100. Similarly, + = 200. Number of solutions: 201.
However, we have subtract the overlaps where two variables are 100. For example, a 100 and b 100, then = 100, which is counted in all three cases. There are three such overlapping triples: 10,,0000100, etc., but since it's the same triple, it's only counted once in each case. Wait, actually, when we count the three cases, the triple (100,100, 100) is included in all three cases. So total overlaps
Each triple where two variables are 100 is counted two cases. For example, (100,100,100 counted in all three cases. Wait, no, if = 100, b = 100, then c = 100, so it's counted once in each case. Wait, no, when a = 100, b can be 100, which gives three times in total (once in each casc). Similarly, any triple where exactly counted twice. Wait, but if two variables all three being 100. So there are no triples where exactly two variables are 100 because that forces third to also be 100. Therefore, the only overlap is the triple (100,100,100), which is counted three times.
100. Similarly for the other cases. So the triple 1100,100,1 is counted
variables are 100 would be 100, say 100 and 100, then 100, which is the same
Therefore, using
Total triples = 3=201 - 3+1 1+1 = 603 =0
Mode: Verification
Subgoal: Verifying the counting with small example
Content:
But let's check with small example. Suppose we have variables summing to 3 instead of 300, and we want to count triples where one variable is 1. Then:
42

---

where the Published as a conference paper at ICLR 2026
Case I: a= + (1.0,2), (1,1,),, (1,2.0)
Case 2: b= (0.1,) (1,1,1), (2.1,0)
Case 3: c=1, (0.2.1), (1,1, (2.0.1)
Total listed triples: (1,0,2), (1,1,1,, (1,2,0), (0,1.2), (2,1,0), (0,2,1), (2,0,1). That's triples. Using inclusion- exclusion:
3+3 3
1=9 +
Correct. So the formula works.
Mode:
Finalization
Subgoal: Extract the final answer
Content:
Thus, the final answer is
601
Example: MLR Output (high-level) GPQA (Diamond)
Problem:
is treated with sodium hydride, followed by benzyl bromide, forming product 1. is treated with p-toluenesulfony/ hydrazide in the presence of catalytic amount of HCL, forming product 2 is treated at low temperature with n-butyllithium, followed by aqucous ammonium chloride, forming product 3. 3 is stirred with Pd/C under hydrogen atmosphere, forming product 4. What is structure of product 4?
Choices:
A. ((---sprpplyyaaeeeyeehhhyyyyyyyr
(3 he-ispprgyyyyoooeeyyy/eenaallln
C. N https//wwwmyy..ooooooooo///s//en
D. Hollayyyyyyynyyy...nnnnooiooooor
Solution:
Mode: Problemt Understandns
Subgoal: Understanding the problem and outlining the approach
Summary:
Need determine structure of product 4 reaction sequence.
The starting material is
'The cyclohexanone ring has hydroxymethyl group at position 3.
At position 5, there is prop-1-en-2-yl group, which is CH2=C(CH3)- methyl group attached to double bond).
The structure a cyclohexanone ring with these two substituents at positions and S.
Mode: Derivation
Subgoal: Analyzing the first reaction: NaH and benzyl bromide
Summary:
NaH deprotonates the hydroxymethyl group to form an alkoxide, which then reacts with benzyl bromide to (-ppppp--em-22yyyyyyllheeeaaae hydroxymethyl group is converted to benzyloxymethyl group.
form henzyl ether. Product is
Mode: Derivation
Subgoal: Analyzing the second reaction: p-toluenesulfonyl hydrazide and HCI Summary:
Product reacts with p-tolucnesulfony hydrazide (TsNHNH2) and catalytic HCI to form tosylhydrazzoe.
The ketone group (cyclohcxanone) reacts with the hydrazide, losing water and forming hydrazone. Product
2 the tosylhydrazoone of the ketone, with all other substituents (including prop-1-en-2-y1 unchanged. The structure is N N (weeeeeeoooloymelll---------
wetthyylenzeeesllloooyyyraiaii
Mode: Derivation
Subgoal: Analyzing the third reaction: n-butyllichium and aqueous ammonium
Summary:
Product 2 is treated with l-tttuuli and aqueous ammonium chloride (Shapiro reaction), converting the
ketone-derived hydrazone into alkene. The cyclohexanone ring becomes cyclohexene ring with double
43

---

Published as a conference paper at ICLR 2026
bond between CI and C2. Product is cyclohexene with benzyloxymethyl group at C3 and isopropenyl group C5.
Mode: Derivation
Subgoal: Analyzing the fourth reaction: hydrogenation with Pd/C and H2 Summary:
Hydrogenation Pd/C
saturates double bonds. The cyclohexene ring becomes a cyclohexane ring.
The isopropenyl group CCH22-C(CH3V) is reduced to isopropyl group (CH.)))(CH.3))))
Product 4: cyclohexane ring with benzyloxymethyl group at C3 and isopropyl group C5. The ring fully saturated; other changes occur.
Mode: Synthesis
Subgoal: Recapping the sequence and mapping substituent positions
Summary:
Starting material: cyclohexan-1
Position 3: hydroxymethyl + benzyloxymethyl (after NaH/benzyl bromide)
Position I: ketone + tosylhydrazoon - double bond (after Shapiro reaction)
Ring: eyclohexene (double bond between 1 and 2)
Position 5: prop-1-en--2yy (CHH-CCCCE3))))
Final product:
Mode: Verification
Subgoal: Evaluating the effect of hydrogenolysis the benzyl ether
Summary:
The benzyloxymethyl group should remain after hydrogenation with Pd/C, as benzyl ethers are generally stable under these conditions. Therefore, product should have benzyloxymethy group at position and isopropyl at position cyclohexane. None of the options match this structure.
Mode: Verification
Subgoal: Re Re-examining the possibility of benzyl ether cleavage under hydrogenation Summary:
Ketone is position 1 on the cyclohexane ring.
Substituents at positions (benzyloxymethy)) and (isopropyl) are two positions apart.
After Shapiro elimination, double bond forms between positions and
Hydrogenation restores the single hond; substituents remain at positions 3 and 5. In product cyclohexane ring has henzyloxymethyl position 3 and isopropyl at position 5. This arrangement does match any of the answer choices.
Mode: Verification
Subgoal: Considering the effect of hydrogenolysis on the benzyl ether group Summary:
Step alkylation benzyloxymethyl--omm
Step 2: tosylhydrazone formation at ketone-correct.
Step 3: Shapiro reaction forms alkene between positions and 2-correct.
Step 4: hydrogenation saturates both the 1-2 alkene and isopropenyl group to isopropyl. (olooyymettyy----ipooooytoee None options exactly match this structure.
Final product:
Mode: Backtracking
Subgoal: Realizing benzyl ether cleavage under hydrogenation conditions Summary:
Starting material: hatptooyymttttt--------o--------
Tosylhydrazone forms at position 1 (the ketone).
Elimination forms an alkene between positions and 2.
Substituents 3 (hydroxymethyl) and (prop-1-en-2-11 remain.
Hydrogenation reduces the double bond (1.2) and the prop-l-en-2-yl group
Final product: cyclohexanc ring with substituents at (hydroxymethy)) and (isopropyl).
Question: how to name this compound.
Mode: Recall
Subgoal: Applying IUPAC nomenclature and lowest locant rule Summary:
44

---

Published as a conference paper at ICLR 2026
After renumbering to give the substituents the lowest possible numbers, the hydroxymethyl group is at position and the isopropyl position The correct IUPAC name 1-thydroxymeehhy)-3- isorrpyylyclloeexnne also named as ((-soppopylyycohhexy/mmttaaa Therefore, option B correct.
Mode:
Finalization
Subgoal: Final conclusion and selection
Summary:
Numbering changes to give the lowest possible numbers, repositioning the substituents as and instead of and 5.
Correct answer is
Example: MLR Output (low-level) GPQA (Diamond)
Problem:
is treated with sodium hydride, followed by benzyl bromide, forming product 1. is treated with p-toluenesulfony hydrazide in the presence of catalytic amount of HCI, forming product 2. 2 is treated at low temperature with n n-butyllithium, followed by aqueous ammonium chloride, forming product 3. 3 is stirred with Pd/C under hydrogen atmosphere, forming product 4. What is structure of product 4?
Q:
Choices:
A. ((@--ssspppyyyaaeeyyyyttttayyyyr
B. (H-iorrooyyychhhssyyy/eennnnnnnn
C. N'
D. 3
Solution:
Mode:
ProblemUndee undersaadinn
Subgoal: Understanding the problem and outlining the approach
Content:
Okay, I need figure the structure of product 4 in this reaction sequence. Let start by breaking down each step carefully.
First, the starting material is The cyclohexanone ring has hydroxymethyl group position 3 and mrop-l-en-2-y- group (which is an allyl group attached at the middle carbon, it's actually a methyl group attached double bond, right? So like CH2-C(CH2)- mayhe?
Let's draw that out.
Wait, 1rop-1-en-2-11 would be CH2=C(CH3)-, right? So the substituent is methyl group attached to double bond. So the eyclohexanone ring has that position 5 and the hydroxymethyl at position 3.
Mode: Derivation
Subgoal: Analyzing the first reaction: NaH and benzyl bromide
Content:
Then, first step treatment with sodium hydride, followed by benzyl bromide. Sodium hydride strong base, so mayhe deprotonates something. The hydroxymethyl group has an -OH. Deprotonating that would give an alkoxide, which is a good nucleophile. Then henzyl bromide would act as an alkylating agent. So the benzyl group (Bn) would be attached to the oxygen, making henzyl ether. So product would be 3-( (euzylooymettty))-5-(pppo 1eae---een-yy-- (-a-roopp--------yyyllleeaaaa-aa becomes benzyloxymethyl after alkylation.
www.yyyyyooeeaa--ooooo That makes because the hydroxymethyl
Mode: Derivation
Subgoal: Analyzing the second reaction: p-toluenesulfonyl hydrazide and HCI
Content:
Next, product is treated with p-tolucnesulfonyl hydrazide (TsNHNH2) with catalytic HCL. That's typically used to form tosylhydrazone. The ketone group (cyclohexanone) would react with the hydrazide. The mechanism is that the ketone attacks the hydrazide, losing water and forming hydrazone. So product 2 would be the tosylhydrazone of the ketone. So the cyclohexanone becomes cyclohexylidene group linked to the drrazid So structure would be beeethyyeeeeeesslooooooyyy
Wait, but maybe the prop-1-en-2-11 2-ylin is still present? The reaction is only the ketone, so the other substituents remain.
45

---

Published as a conference paper at ICLR 2026
Mode: Derivation
Subgoal: Analyzing the third reaction: n--butyllichimm and aqucous ammonium
Content:
Third step: product 2 is treated with n-butyllithium at low temperature, then aqueous ammonium chloride. This sounds like the Shapiro reaction. The Shapiro reaction uses tosylhydrazones treated with strong base like n-Bul to form alkenes via elimination. The mechanism involves deprotonation and elimination of the cosylhydrazide group, forming a double bond. So the cyclohexanone ring, which had the hydrazone, would become a cyclohexene ring?
Wait, but the starting material already has an allyl group (prop-1-on-2-yl).
Wait, maybe the Shapiro reaction here is converting the ketone-derived hydrazone into an alkene. But the original ketone is at position I pycllhhxann--ooe So after Shapiro, that would form a double bond between CI and C2? So the cyclohexanone becomes cyclohexene with the substituents. So product 3 would be 3-( -- -elee---lyyycloeeeeeeeeee C1 and C2?
the position the double bond would be between
Wait, maybe need to think about regiochemistry here. The Shapiro reaction typically gives less substituted alkenes because it's concerted elimination. But in the eyclohexane ring, the original ketone is at position 1. So when eliminating, the hydrazone is at C1, and the adjacent hydrogen at C2. So climination would form a double hond between C1 and C2. But the ring already has substituents at C3 (benryloxymehhy) and C5 (prop -1-en-2-yl) yl). So after elimination, the ring becomes a cyclohexene with double bond between CI and C2. So product 3 is 1.2 e2cccocohel with substituents C3 and C5. But the prop-l-en-2-y another substituent.
Wait, but maybe the prop-1-en-2-yl is an isopropenyl group (CH2=C(CH3)-). So product would have cyclohexene ring (double bond between 1 and 2), a benzyloxymethyl group at 3, and an isopropenyl group at 5. Then, when treated with n-BuLi, but after that, aqueous ammonium chloride. Wait, the Shapiro reaction usually forms alkenes via deprotonation and elimination. But maybe this case, after forming the hydrazone, treatment with n-Buli leads to deprotonation and elimination of the tosylhydrazide, giving alkene. product 3 would be the cyclohexene derivative with those substituents.
Mode: Derivation
Subgoal: Analyzing the fourth reaction: hydrogenation with Pd/C and H2
Content:
Then, product 3 is stirred with Pd/C under hydrogen. Hydrogenation. So any double honds would be hydrogenated. The cyclohexene ring's double bond (from the Shapiro reaction) would be hydrogenated single bond. Also, the mrop-1-en-2-y- group (isopropenyl, CH2=C(CH3)- so that's vinyl group with methyl substituent on the double bond) would be hydrogenated, making it prop-2-yl group (isopropyl group).
Wait, prop-1-en-2-yl is CH2=C(CH3)~, hydrogenation would make CH2CH(CH3), So the isopropenyl becomes isopropyl. Also, the eyclohexene from the Shapiro reaction (double bond between and 2) would become cyclohexane again. So product 4 would have a eyclohexane ring with substituents: at C3, benzyloxymethyl:: at C5, isopropyl.
But wait, after hydrogenation, the ring fully saturated. Also, the hydroxymethyl group 3 is benzyloxymethyl. But after hydrogenation, is there any other change?
Mode: Synthesis
Subgoal: Recapping the sequence and mapping substituent positions
Content:
Wait, let's recap: Starting material; cyclohexan-1-ome (so position 1 the ketone), Substituents: at 3, hy hydroxymethyl; at 5, prop-1-en-2-y- (CH2=C(CH3)-).
Step 1: deprotonate hydroxymethyl's OH with NaH. then benzyl bromide gives benzyloxymethyl position
Step 2: form tosylhydrazooe at the ketone (position 1). So now, the hydrazone at position 1. Step 3: Shapiro reaction (n-BuLi) eliminates the tosylhydrazida forming a double bond between position and 2. So the ring becomes cyclohexene (positions and 2 double bond). So now, the substituents are at 3t
benzyloxymethyl, 5 (CH2= and the ring has double bond between I and 2. Step 4: hydrogenation with Pd/C and H2. The double honds in the molecule (both the eyclohexene ring's 1,2-double bond and the prop-1-en-2-yl group's double bond) would be hydrogenated. So the cyclohexene hecomes cyclohexane (single bond between and 2), and the prop-1-en-2-1 becomes isopropyl (CH(CH3)22) So the substituent at position 5 becomes isopropyl. So product 4 would be (b(ennyyloxymelly))>> isopropyleyclohexane.
46

---

Published as a conference paper at ICLR 2026
Wait, but cyclohexane is a six-membered ring. The substituents are positions 3 and But then, after hydrogenation, the ring is fully saturated. Also, the hydroxymethyl was at 3, which became benzyloxymethyl. But after hydrogenation, is there any other change?
Mode: Verification
Subgoal: Evaluating the effect hydrogenolysis on the benzyl ether
Content:
Wait, but in the final hydrogenation step. maybe the benzyloxymethyl group is hydrogenated? But benzyl ethers are generally stable under hydrogenation unless there's a catalyst that cleaves ethers. But Pd/C usually doesn't cleave ethers. So the benzyloxymethy/ should remain. So product should have a henzyloxymethyl group at position and isopropyl at position 5 on cyclohexane. But none the options have that.
Mode: Verification
Subgoal: Re Re-examining the possibility of benzyl ether cleavage under hydrogenation
Content:
Wait, maybe messed up the numbering. Let me try to visualize the cyclohexane ring. Let's imagine the ketone at position 1. Then positions 2. 3, 4, 5, 6 go around the ring. So if the ketone is at position 1, then substituents at and would be two positions apart. After elimination (Shapiro), the double bond is between I and 2. Then hydrogenation would bring back the single bond. So the substituents at and 5 are in the same positions. So in product 4, the cyclohexane ring has benzyloxymethyl group at position 3 and an isopropyl group at position 5. But the answer choices don't have that.
Mode: Verification
Subgoal: Considering the effect of hydrogenolysii on the benzyl ether group
Content:
Wait, but let me double-check the steps again. Maybe made a mistake. Step I: alkylation of hydroxymethyl benzyloxymethyl. Correct. Step 2: formation of tosylhydrazone at the ketone. Correct. Step 3: Shapiro reaction with n-BuL.. The Shapiro reaction involves deprotonation adjacent the hydrazone, leading elimination the losylhydrazide and formation of an alkene. So the ketone (position 1) becomes a double bond between positions and 2. Step 4: hydrogenation of the alkene (positions 1-2) and the isopropenyl group (position 5) to isopropyl. So product 4 would have cyclohexane ring with substituents: at position 3, benzyloxymettyyl; at position 5, isopropyl. So the structure is 3-(benzyloxymeethyl) (aaponnlllyynttlyyss---oooyyyyyo 5-iiopropyllccclheeame.. But looking at options, none them exactly match.
Mode: Backtracking
Subgoal: Realizing henzyl ether cleavage under hydrogenation conditions
Content:
Wait, maybe messed up the positions during the Shapiro reaction. Let's think again. The starting material is (-pp)))--pppoo-----------
3
When we form the tosylhydrazone at position (the ketone). then eliminate to form the alkene between positions I and 2. Now, the ring has a double bond between and 2. The substituents 3 and 5 are still in their original positions. Then, hydrogenation reduces the double bond between 1 and 2 and the prop-1-en-2-yl group So after hydrogenation, the ring is cyclohcxanc with substituents at (hydroxymethyl) and 5 (isopropyl). But how is this named?
Mode: Recall
Subgoal: Applying TUPAC nomenclature and lowest locant rule
Content:
Alternatively, maybe the numbering changes when the ketone is removed. Originally, the ketone is at position 1. After converting it to CH2 via hydrogenation. maybe the numbering starts from the hydroxymethyl group position 1. In TUPAC nomenclature, the numbering done give the substituents the lowest possible numbers. So if the hydroxymethyl is position 1 and the isopropyl position 3, that would be lower than if they were at 3 and So the new numbering would be: I: hydroxymethyl (original 3) 2: CH2 (original 4) 3: isopropyl (original 5) 4: CH2 (original 6) 5: CH2 (original 1) 6: CH2 (original 2)
Wait, this might not be accurate. Let me think of the cyclohexane as chair. If the original positions I (top 2, (bottom), 5, 6. After removing the ketone (position 1 becomes CH2), and having substituents 3 and 5. If we renumber to give the lowest numbers, we need to choose a direction where the substituents are at lower positions. If we start numbering from the hydroxymethyl (original 3) as position 1, then the isopropyl is at original 5, which would be position 3 in the new numbering (since it's two positions away in the ring). This gives substituents positions 1 and which is lower than 3 and 5. Therefore, the correct IUPAC name would be Because the methanol group (hyrroxymethyl) is considered as the substituent at position 1, and the isopropyl at position 3. Thus, option B is correct.
---mmmmtttyy--ssooopppoopo lcyloohexane,, which is named as
47

---

Published as a conference paper at ICLR 2026
Mode:
Finalization
Subgoal: Final conclusion and answer selection
Content:
Ah, so the key here is that the numbering changes to give the lowest possible numbers, which would
reposition the substituents and instead of and 5. Therefore, the correct answer B
Example: MLR Output (high-level)- BoardGameQA (Hard)
Problem:
A few players are playing boardgame. The current state of the game is as follows. The dachshund has foothall with radius of 15 inches. And the rules of the game are follows. Rule1: If the dachshund has football that fits 36.6 33.1 40.7 inches box, then the dachshund brings oil tank for the worm. Rule2: at least one animal brings an oil tank for the worm, then the stork does not dance with butterfly. Based on the game state and the rules and preferences, does the stork dance with the butterfly?
Solution:
Mode: Problem ProblemUnderssee Understanddi
Subgoal: Recap the problem and rules Summary:
The dachshund has football with radius of inches. football fits 36.6 33.1 x 40.7 inches box.
Mode: Planning
Subgoal: Identify the key condition check
Summary:
Key condition: the dachshund's foothall in the hox, Rule applics (dachshund brings oil tank), triggering Rule (stork can't dance). If the football doesn't fit, Rule doesn't apply (no oil tank), so Rule 2 might not be triggered, allowing the stork to dance.
Mode: Derivation
Subgoal: Analyze the football's dimensions and assumptions Summary:
The football has radius of inches.
Assuming is sphere, the diameter is 30 inches.
The problem uses radius", so the foothall treated as sphere.
Mode: Calculation
Subgoal: Compare football's size to box Summary:
Diameter of football: 30 inches.
Box dimensions: 36.6 33.1 1. x 40.7 inches.
30 36.6 (length), 30 33.1 (width), 40.7 (height).
football fits the box in dimensions.
Mode: Derivation
Subgoal: Apply Rule based the fit
Summary:
Diameter (30 inches) less than all box dimensions, so the football fits. By Rule 1, the dachshund brings an oil tank for the worm.
Mode: Derivation
Subgoal: Apply Rule and conclude
Summary:
Applying Rule 2: since at least one animal (the dachshund) brings an oil tank, the stork does not dance with the butterfly.
Mode: Finalization
Subgoal: Extract the final answer Summary:
Thus, final answer disproved
48

---

Published as a conference paper at ICLR 2026
Example: MLR Output (low-level) BoardGameeA (Hard)
Problem:
few players playing boardgame. The current state of the game is follows. The dachshund has football with radius of inches. And the rules of the game are follows. RuleI: If dachshund has football that fits a 36.6 x 33.1 x 40.7 inches box, then the dachshund brings an oil tank for the worm. Rule2: If at least one animal brings an oil tank for the worm, then the stork does not dance with the butterfly. Based on the game state and the rules and preferences, does the stork dance with the butterfly?
Solution:
Mode: Problem GobblemUnderstarrne Understanding
Subgoal: Recap the problem and rules
Content:
Okay, let's try to figure out whether the stork dances with the butterfly based on the given rules and the current game state.
First, let's recap the information provided. The dachshund has football with radius of 15 inches. The rules are:
If the dachshund's football fits in 36.6 33.1 x 40.7 inches box, then the dachshund brings an oil tank for the worm.
at least one animal brings an tank for the worm, then stork does not dance with the butterfly. The question is whether the stork dances with the butterfly. To determine that, we need to check if any animal (in this case, possibly the dachshund) brings an oil tank for the worm. If they do, then by Rule 2, the stork won't dance. If not, then maybe the stork does dance.
Mode: Planning
Subgoal: Identify the key condition to check
Content:
So. the key here is whether the dachshund's football fits into the specified box. If it docs, then Rule applics, and the dachshund brings
tank, which triggers Rule 2. preventing stork from dancing. the football doesn't fit, then Rule doesn't apply, so the dachshund doesn't bring the tank, Rule might not be triggered, allowing the stork to dance.
Mode: Derivation
Subgoal: Analyze football's dimensions and assumptions
Content:
Now, let's focus on the foothall's dimensions. The foothall has a radius of 15 inches. Wait, a football is typically clongated shape, like prolate spheroid, but maybe in this game, it's considered sphere? The problem mentions a radius, which term used for spheres. it's a sphere with radius inches, then the diameter would 30 inches (since diameter is twice the radius). However, if it's an American football shape, the dimensions would be different. But since the problem specifies a radius, think we can assume it's a sphere here.
Mode: Calculation
Subgoal: Compare the football's size to the box Content:
So, diameter of the football is 30 inches. Now, the box has dimensions 36.6 33.1 40.7 inches. To fit the sphere inside the box, the sphere must fit in all three dimensions, That the diameter of the sphere must he less than or equal each the box's dimensions.
Checking each dimension:
Length: 36.6 inches. The diameter is 30, which less than 36.6. So okay. Width: 33.1 inches. 30 is less than 33. Okay.
Height: 40.7 inches. 30 is less than 40.7. Okay.
Mode:
Derivation
Subgoal: Apply Rule based on
Content:
Since the diameter (30 inches) is less than all the box's dimensions, the football should fit inside the box. Therefore, according Rule 1, the dachshund brings oil tank for the worm.
Mode: Derivation
49

---

Published as a conference paper at ICLR 2026
Subgoal: Apply Rule and conclude Content:
Then, applying Rule 2: since at least one animal (the dachshund) brings an oil tank, the stork does not dance with the butterfly.
Therefore, the answer that the stork does not dance with the butterfly.
Mode: Finalization
Subgoal: Extract the final answer
Content:
Thus, answer disproved
Prompt: Long- horizon Reasoning Test
You are given multiple problems:
[Problem 1}
(Problem 2}
[Problem N}
Solve these problems in order. Please reason step by step for each problem, and at the end provide **all final answers in a single ISON listæs, where each answer is wrapped in "booxe({] If you cannot solve problem write (boxed/N/A) for that entry.
Output format:
ison
M (anseerr:: '\ boxed(aasseer))..... (boocc//aserr)....""xoooaaaa/ass "\/boxed(annsweNN)
Example: MLR (high-level only) Output
Problem: Define
p
2im1 Ee
and q q= -= +
Find way write
kE (r+kya in terms of and g.
Solution:
Mode: ProblemUnndersanddin
Subgoal: Understanding the problem
Summary:
The goal to express
Yumm (i=f) \uum (k=l)" \frac(1
terms of \(p' \) and \(q (). The current stage involves understanding problem and identifying key components to work with.
Mode: Recall
Subgoal: Recall the definitions of \(p and \(qq)) !)
Summary:
\(p= \um (0mm_kk=)"""""" [k=l)" (1) \) and \( (k=mmm(==)""aaam (frac(1) (1)(())))33) \) are defined as the sums of the reciprocals of squares and cubes, respectively.
Mode: Planning
Subgoal: Identify the desired expression
Summary:
The desired expression is:
/
50

---

Published as a conference paper at ICLR 2026
sjummji=1] (sum_(k=1) ((ikk))
This is the sum of the reciprocals of the fourth powers of the sums of \( \) and \(k V) from \(j= V to (
j= >inftyy \) and from \(k= I V to \(k= \infty V.
Mode: Planning
Subgoal: Consider symmetry and possible simplifications Summary:
The expression \(ssumm (j=I]"iinft sum-(k=1)"VVa iiinfty (frae(1) ((+kk)))))))) involves summing the
reciprocals of the fourth powers of the of \.V) and ((k).. The symmetry between (()) and \(k)
suggests potential simplifications transformations. The current stage is focusing on identifying the key components and considering symmetry to simplify the expression.
Mode: Planning
Subgoal: Attempt express the in terms of \(p \ and \(q V) Summary:
The expression ((ssum_ -j= iiinfyy /sum | iinfty {frac (((+k)444)) involves summing the reciprocals of the fourth powers of the sums of VO') ) and (K))). The current stage is attempting to express this sum in terms of p ) and \(q V. which defined ((ssmm(k=11"" iinfty \frac{ (k2))) FV) and sum. (k"33)
[k=1] }x iiffty frae [1) (k33) (33). () respectively. The goal is find relationship between these sums and the
given expression.
Mode: Recall
Subgoal: Recall known results for symmetric sums Summary:
(sum_(k=1 Minnfy ((rkk())(())))))))) involves summing the reciprocals of the fourth powers of the sums of VoV) and (kk)). The goal is to express this sum in terms of V(P
The expression )( [j=1
1) and \(g)), where \(p= \sum (k=1) }(infty fraee 1 (K22)) and V<= \su sum_(k=1] (<=e)"((inn "inifty \frac [K
'3) D. The current stage is exploring known results for symmetric sums and potential simplifications.
Mode: Derivation
Subgoal: Attempt to simplify using known results Summary:
(sum_(k=1] iinfty (G+kk)) involves summing the reciprocals of the fourth powers the sums v(() and ((k)). The current stage is attempting to simplify this expression using known results for symmetric sums, focusing on possible relationships with (p)) and (
expression ((ssum(j=11]"
q)).
Mode: Planning
Subgoal: Consider using known results for symmetric sums
Summary:
The expression \ ( \sum_[k=1] iinfty \frae 1) ((+k))44)) is being considered in terms of \(P)) and ((l)), where Xp \sum (k=1) "infft (frac11)(K2) and \gg = (k=m \frac{1)(t
"3).. The goal is to find relationship between Mp)) and ((qh) and the given sum.
Mode: Recall
Subgoal: Recall the identity for the of reciprocals of squares
Summary:
The identity for the sum of reciprocals of squares is:
Ysum (k=) (1ac(11kk22) =p
where \(p= /um (kii)" (1ka)1))k2)) )).
Mode: Recall
Subgoal: Recall the identity for the of reciprocals of cubes
Summary:
The identity for the sum reciprocals of cubes
\sum (k=1) (rac(1))(k3)) =q
where
\(9= /sum (inffy (frac[ (1k33) 1))
Mode: Recall
Subgoal: Recall the identity for the of reciprocals of fourth powers
51

---

Published as a conference paper at ICLR 2026
Summary:
The identity for the sum reciprocals of fourth powers is:
M
sum [sum-(==1> (infyy (HK44) = {frac(p)2)
+ \frac(1H44g
where \(p |sum (knit" [frac[1] \) and \(q== (ssmm((=]] (k=1)"iifffi (1aak1))(33)) V).
Mode:
Derivation
Subgoal: Apply the identity to the given sum
Summary:
The given sum \(ssum iiinfyy (sum_[k=1]))innn] (((kk())(++))4))) can be expressed in of (
pl) and ((gh) using the identity:
sum [j=I]"[infty ssum. [k=1)
\frac (I (frac (p) (2) + /frac (1)
Mode: Verification
Subgoal: Verify the result with specific example Summary:
The given sum \(sum__fii (innn jnifftty (sum_(k=1)) ((itk))) can be expressed in terms of (
pl) and (ql) using the identity:
(uum (j=l}"(inftt \sum (k=l) (((kk()) (frac (2) + [1]{44
where \pp (/summ(==])"inm (1)(K)) and ((= (sum_ (1rac(1) (I (K3)))
Mode: CascAnalysis
Subgoal: Alternative approach using generating functions
Summary:
sum ((.uum (j=I)"(intty {summ (k=1)(k=)))iimm (1rac(1) ((++))4) can be expressed terms (pp))
and ((1)) as:
M
sum (j=1}"iintt \sum (k=1)"(iit ((rac[1((++)44) = (p] + (fra [1){4)
where
Vpp = (oum__(=l)""infiiii ((aac11)(k2))))) and \q= \sum_(k==|) (fra (k3)))
Mode: Verification
Suhgoal: Review the result for potential simplifications
Summary:
The sum ((suum(j=l)) (i=1]"(initt iiifty (k=1)"==))"(imnn (1) can be expressed in terms (pp))
and ((g1) as:
M
sum [imm(jji]""nnine (sum_(k=1|""intt [frac[1 (pac + (1 [)44
where V(p = (/=mmmmmmmmmm (1a and \(q \sum_ (infy ( (k3)
Mode: Finalization
Subgoal: Finalize the answer
Summary:
The sum (j=1] (sum_(k=1] ((+k)4)) can be expressed in terms of (p!))
and ((41) as:
(p) sum {i=l)" \sum (k=1kkk=))iiiii ( \frac (1){44
where Vpp
(frac (K2()))) and \(q (summ (frac (1) (K3)))
1
Thus, the final answer 2 7
Example: Plan-and-Solve Output
Q: œ, y and z positive real numbers that satisfy following system of equations:
loga
52

---

Published as a conference paper at ICLR 2026
1oga
log2
Then the value [log] (=ny"=""")) y²= where mn and are relatively prime positive integers. Find m n.
A: Let's first understand the problem, extract relevant variables and their corresponding numerals, and make a complete plan. Then, let's carry out the plan, calculate intermediate variables (pay attention to correct numerical calculation and commonsense), solve the problem step by step, and show the answer.
Output:
Given:
1oga
loga
loga
The value of [logg(>" is where m and n relatively prime positive integers.
Plan:
Convert each logarithmic equation its exponential form.
Use properties of exponents logarithms simplify expressions.
3. Combine like terms and solve for the values of i. y. and z.
Substitute these values into expression [log,(z) (a""(("""""""" >²==
5. Simplify this expression find (ogg(""y" 2²)]
6. Express it as fraction a with m and n being relatively prime.
Find m + nt.
Calculation:
#### Step 1: Convert each logarithmic equation its exponential form
From the given equations, we have:
æ 2²/²yz yz
y 2//²zz xz
2 2-/42y
#### Step 2: Use properties of exponents and logarithms to simplify the expressions
can rewrite each equation using the property exponents:
æ Zyz
y /2z=
2/ry
#*## Step 3: Solve for the values of x, y, and
Multiply all three equations together:
(V2yz) (2zz)( //xy)) (v2) (2))z²)
This simplifies to: 22222 (2(22)
Since (22) = 21/241/3-1/41 1000:/0--10200+++0011 2(6+443//12 213/12 we get: 2²222² 2213/2
Taking the square root both sides: zyz 23/24
Now, substitute hack find individual values:
V2yz
y
2/rz
53

---

Published as a conference paper at ICLR 2026
V2ry
#iiii!t Step 4: Substitute these values into the expression [1og2(0"y* (x²)²²²))
Substitute x 22y into x² x²== x' y²²²² (x=)) = 23 ====""²²²""" 41102=")00 )00² =41y220 20)
So.
1o "²="" = 4()²=0)
Using the earlier result zyz
213/21, we know:
228/24
228/24
1/8/24
22/224 6
Simplifying further:
233/24
213/24
213/24
213/24
223/24 7
213/24
V2ys
2/2
/Iws
/2yz
2/
Finally:
| 2² = 4()²20)= =
228/24
V2yz
18/24 <7
=
222/24
V2ys
120/24
4
223//24
V2yz
7
2218/24
8irz
Therefore:
|log2(2' (x y²² 14g
Thus:
|log2 (x "ee"y"2" y²²²)|===
### Answer:
= n1
2
The tinal answer 3
54


Reviews:
[{"review_id": "6Ni6dBrOel", "reviewer": "Reviewer_bbdw", "summary": "The authors propose Multi-Level Reasoning to enhance LLM reasoning with structured multi-level modeling. Specifically, a high-level planner first generates step descriptors as the roadmap, and then the low-level executor generates the detailed content based on the high-level planner, with the help of the online step-DPO algorithm to provide step-wise supervision. The proposed model achieves good performance on both math and science reasoning tasks based on different backbone models.", "questions": "Refer to the weakness part.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-\tOverall, the idea is sound, and the proposed method achieved good results.\n-\tThe authors have conducted ablation tests to verify its essential components.", "weaknesses": "-\tIn Section 3.2, the authors first fine-tune based on the low-level content via full SFT, and then learn from the high-level content via LoRA. The order of training, as well as the usage of full SFT/LoRA on different high-/low- level data, should be discussed (better with detailed experimental results).\n-\tIn the main experiment, although the authors have implemented their method on diverse backbone models, the compared baselines are not strong enough. Other popular preference optimization methods (e.g., RL methods such as GRPO) should be compared.\n-\tIn the Related work, there are some step-wise preference optimization methods or those that also adopt certain forms of overall planners, which should be discussed and even compared. The authors are suggested to highlight the technical novelty of the proposed model compared to existing efforts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FqkosSMyYR", "reviewer": "Reviewer_yNnM", "summary": "This paper proposes Multi-Level Reasoning (MLR), a framework that addresses limitations in single-policy long chain-of-thought (CoT) approaches for language model reasoning. MLR decomposes reasoning into alternating high-level step descriptors (abstract sub-goals) and low-level detailed content. The framework uses the base model as the low-level policy and a lightweight LoRA module for the high-level policy. To enable efficient training, the authors introduce online Step-DPO, which leverages Twisted Sequential Monte Carlo (TSMC) to provide process-level preference signals without requiring a separate reward model. Experiments on MATH500, AIME24, GPQA-diamond, and BoardGameQA-hard demonstrate improvements over both distillation baselines (DeepSeek-R1-Distill) and outcome-reward RL methods across three base models.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel application of HRL concepts to reasoning tasks with a practical, efficient design (frozen base + LoRA)\n- TSMC-based process preferences are creative and avoid the overhead of training separate process reward models\n- Strong empirical results: MLR consistently outperforms DeepSeek-R1-Distill despite using only 10% SFT data and 5% preference pairs\n- Addresses a timely problem as reasoning models gain prominence", "weaknesses": "- 各个组件的创新性有限 - 主要贡献在于其组合方式而非从根本上采用新的技术", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZTLFgT29BX", "reviewer": "Reviewer_7Cso", "summary": "This paper proposes Multi-Level Reasoning (MLR), an SFT+DPO framework that addresses two critical limitations of single-policy long CoT approaches: CoT derailment and sparse reward guideline. MLR decomposes the reasoning process into a hierarchical structure with high-level step descriptors generated by a lightweight LoRA module and low-level detailed content produced by the base model, operating through an alternating plan-execute loop. The framework employs SFT to initialize both policies on multi-level decomposed data, followed by online Step-DPO that leverages Twisted Sequential Monte Carlo (TSMC) to generate process-level preferences for stepwise supervision without requiring a separate process reward model. Finally, the authors provide empirical results and ablation studies to show its effectiveness.", "questions": "1. I am wondering what the sources of your training dataset and prompts are, as this information is important not only for reproducibility but also for ensuring a fair comparison.\n\n2. The summarization step is underspecified. What model performs this? How long are summaries? \n\n3. What's the definition of \"Estimated survival accuracy\" in Figure 5 (a)?\n\n4. When computing the utilities (Equations (3) and (4)) in process preference modeling, do you sample only once or perform multiple sampling rounds to compute the average score? I guess that sampling only once could introduce too much randomness.\n\n5. Authors said they fine-tune a small LM on the low-level SFT data for \"fast rollout\", and this small model achieves comparable estimation accuracy while being much faster. How small is it? The paper lacks relevant details.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The framework is clear and easy to understand, and the paper is generally well-written.\n\n2. The empirical experience is extensive. Authors test the proposed method in several areas (math, science and logic) with average scores from multi-individual runs to show a stable improvement. The ablation study in Table 2 clearly demonstrates the effectiveness of each component in MLR.", "weaknesses": "1. One motivation, \"CoT derailment\" is unclear and hard to follow. \n\n2. To demonstrate the effectiveness of their approach, the authors compare MLR with several baselines. However, I think this comparison is somewhat unconvincing, as the models are trained on different datasets. It would be more meaningful to evaluate the methods on the same dataset (important) while varying only the training strategies — for example, comparing SFT+GRPO (baseline) and MLR (your proposed framework) under identical data conditions.\n\n3. The process of summarization from high-level abstraction to low-level details might also be an important factor, but the paper does not explore this aspect in depth. (Only said \"including them improves performance and facilitates training\" in the paper)\n\n4. The choice of K=4 rollouts in TSMC appears somewhat arbitrary, and the paper does not provide any preliminary analysis to justify this setting. I believe that an ablation study would be highly valuable for demonstrating the robustness of the framework in real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "htWVZWy4Kx", "reviewer": "Reviewer_fiGr", "summary": "The paper introduces Multi-Level Reasoning (MLR), a framework for improving long-horizon reasoning in language models by decomposing the chain-of-thought (CoT) into high-level abstract descriptors and low-level detailed contents. A lightweight LoRA-based high-level planner alternates with the base LM as the low-level executor, forming an iterative plan–execute loop.\n\nTo address the inefficiency of outcome-reward RL for long trajectories, the authors propose Online Step-DPO, a process-level preference optimization method that uses Twisted Sequential Monte Carlo (TSMC) to compute step-wise survivability utilities. This enables dense, on-policy supervision without training a separate process reward model.\n\nExperiments on math (MATH500, AIME24), science (GPQA-Diamond), and logical reasoning (BoardGameQA-Hard) benchmarks show that MLR outperforms distillation and RL baselines such as DeepSeek-R1-Distill and SimpleRL, often with substantially fewer preference pairs (5%) and SFT data (10%).", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-motivated. Alternating plan-execute loop allows dynamic adjustment and improves interpretability.\n\n2. The experiments demonstrate consistent gains across several benchmarks.", "weaknesses": "1. The writing can be further improved for clarity and completeness. For example, in the abstract, abbreviations such as RL (reinforcement learning) and LM (language model) should be defined upon their first appearance.\n\n2. The method contribution of the paper appears limited. However, I appreciate that the authors conduct sufficient ablation studies, which provide evidence for the empirical effectiveness of the proposed approach.\n\n3. It would be interesting to see how the model performs when only the high-level reasoning outputs are generated during inference, without the detailed low-level reasoning steps. This could offer insights into the interpretability and abstraction capability of the high-level policy.\n\n4. The paper should clarify the seed datasets used to generate the CoT samples. In addition, the comparisons could be made fairer by including baselines that directly apply SFT and RL on the same datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)