TSLM: Tree-Structured Language Modeling for Divergent Thinking
Authors: Doyoung Kim
JaeHyeok Doo
Minjoon Seo
Venue: ICLR 2026 Poster (2026)
Keywords: language models
reasoning
planning
Supervised Learning
Inference-time scaling
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Language models generate reasoning sequentially, preventing them from decoupling irrelevant exploration paths during search. We introduce Tree-Structured Language Modeling (TSLM), which uses special tokens to encode branching structure, enabling models to generate and selectively expand multiple search paths within a single generation process. By training on complete search trees including both successful and failed attempts, TSLM learns to internalize systematic exploration without redundant recomputation of shared prefixes. TSLM achieves 100\% accuracy on Game of 24 (vs. 17\% sequential baseline), robust extrapolation to 20×20 grids (91.5\% vs. 42.7\% for Tree-of-Thought), and superior inference efficiency by avoiding the multiple independent forward passes required by external search methods. These results suggest a new paradigm of inference-time scaling for robust reasoning, demonstrating that supervised learning on complete tree-structured traces provides an efficient alternative for developing systematic exploration capabilities in language models.

Published as a conference paper at ICLR 2026
TSLM: TREE-STRUCTURED LANGUAGE MODELING FOR DIVERGENT THINKING
Doyoung Kim
Jaehyeok Doo Minjoon Seo
KKAIST AI, NYY
doyoungkim@nyu..ed
ABSTRACT
Language models generate reasoning sequentially, preventing them from decou- pling irrelevant exploration paths during search. We introduce Tree-Structured Language Modeling (TSLM), which uses special tokens to encode branching struc- ture, enabling models to generate and selectively expand multiple search paths within single generation process. By training on complete search trees including both successful and failed attempts, TSLM learns to internalize systematic explo- ration without redundant recomputation of shared prefixes. TSLM achieves robust performance and superior inference efficiency by avoiding the multiple indepen- dent forward passes required by external search methods. These results suggest new paradigm of inference-time scaling for robust reasoning, demonstrating that supervised learning on complete tree-structured traces provides an efficient alternative for developing systematic exploration capabilities in language models.
INTRODUCTION
Complex reasoning often requíres exploring multiple solution paths before converging an answer. Consider solving the Game 24 with numbers [8, 4, 3, 6]: systematic approach would branch out +44 8 and other combina- tions simultaneously, rather than committing to a single path early. However, current language models generate solutions sequentially, making it difficult to systematically explore alternatives or recover from early mistakes.
(a) Sequential thinking: purely sequential ap- proach that traverses one deterministic path.
(h) Tree-structured think-
Recent reasoning models like ol OpenAl 2024)
ing: A branching explo- ration that expands multi- ple possibilities simultane- ously.
and DeepSeek-RI DeepSeek-Nn Al et al. (2025) have shown impressive capabilities through ex- tended reasoning traces, but still fundamentally operate sequential generators. While they may internally consider multiple options, they cannot explicitly decouple parallel branches within their generation process. External meth- ods like Thee-of-Thought Yao et al. (2023) ad- dress this through post-hoe search, but require multiple independent model calls and external orchestration.
Figure Sequential vs. Pree-Structured Reason- ing. (a) Sequential approaches commit to single paths, limiting exploration of alternatives. (b) Tree-structured approaches systematically explore multiple possibilities. enabling recovery from mistakes and comprehensive solution space coverage. TSLM bridges this gap by teaching language models to generate tree-structured explorations natively.
The key challenge is that models cannot coher- ently construct diverse exploration branches. Se- quential models collapse all reasoning into one path. External search methods like Tree-of-Thought sample multiple trajectories independently, but this produces fragmented distribution where search
Denotes equal contribution.

---

Published as a conference paper at ICLR 2026
sample is drawn without coordination, leading to redundant overlap in some regions while missing critical branches in others. The model has no mechanism to systematically reconstruct the complete search space.
TSLM addresses this through cokerent tree generation: instead of sampling multiple independent trajectories, the model generates the complete branching structure in forward pass using special tokens ([SEP), [FAIL], [COAL]) ) to mark viable paths, dead ends, and goals. During training, models learn systematically construct diverse branches for each node. During inference, the model generates this coherent tree structure, then selectively "stitches" each branch into isolated contexts for expansion, maintaining the complete search topology rather than hoping parallel samples will cover it.
This approach enables models learn the complete reasoning process by including both successful paths and failed attempts, leading to more robust problem-solving capabilities. Our experimental results demonstrate that this internalized search strategy not only outperforms sequential modeling baselines but also exhibits remarkable parameter efficiency and emergent capabilities that arise naturally from learning complete exploration patterns rather than just final answers.
1.1
OUR CONTRIBUTIONS
This paper makes the following contributions:
We introduce Phee-Structured Language Modeling (TSLM), token-based serialization framework that enables sequential language model learn tree-structured reasoning through supervised learning cohesive search traces
TSLM demonstrate consistent performance gains across diverse tasks: 100% vs. 17% Game of 24, robust extrapolation to unseen, larger environments (91.5% 42.7%
for Tree-of-Thought 20 x 20 Gridworld environments), and enhanced performance on open-ended reasoning
We uncover emergent capabilities including systematic identification of unsolvable problems and robust extrapolation beyond training complexity
We demonstrate a new test-time scaling paradigm: rather than parallel sampling inde- pendent trajectories, TSLM scales by exploring branches within single coherent tree, showing better scaling efficiency and higher convergence accuracy
Our findings challenge the granted assumption that we need inference-time scaling (either via reinforcement learning or scaffolding) is necessary for developing robust reasoning capabilities in language models. Instead, we demonstrate that properly structured supervised learning of tree- formatted reasoning traces may provide a more direct and efficient path toward enhanced reasoning and planning. We provide detailed answers to common questions in Appendix $A implementation here
We also release
2 BACKGROUND
Contemporary language models generate tokens sequentially, modeling p(y x) - Th p(ue
x, )<t) Brown et al. (2020). For problems requiring exploration of multiple solution paths, this sequential approach has limitations: (1) linear commitment to single paths, (2) error propagation, (3) redundant computation when multiple solutions are needed, and (4) inability to systematically explore alternatives parallel.
Current multi-path reasoning approaches rely on inference-time scaling methods, either for post-hoc sampling or policy gradient update from self rollouts. Tree-of-Thoughts Yao et al. (2023) samples multiple candidates at each step using external search algorithms, but faces exponential computational costs. Reasoning traces generated from autoregressive models like ol OpenAl (2024) are due to the number of rollouts for gradient update. but remain constrained by linear generation and may produce redundant information Chen et al. (2025). We provide a more extensive related works Appendix
2

---

Published as a conference paper at ICLR 2026
3 TREE-STRUCTURED LANGUAGE MODELING (TSLM)
We introduce Tree-Structured Language Modeling (TSLM) framework natively incorporate divergent exploration for natural language generation. TSLM differs from standard sequential language modeling by generating multiple possible next actions or statements and linking them into coherent tree structure.
3.1 MODELING MULTIPLE NEXT ACTIONS
Let be the current state or partial solution. In sequential language model, we predict a single next action a from & and transition to = T(s,,), By contrast, TSLM, we represent multiple possible successors:
( = (T), ... ,((,,, ax))
where each a; denotes distinct branch and k is the branching factor. TSLM learns to expand into these i successors within single forward pass, retaining the relationships among them rather than generating them independently. This branching representation is fundamentally different from sampling independent trajectories. Each a, generated conditionally on the previous actions a1...., di-1 at the same node, ensuring systematic coverage of the action space rather than redundant sampling from a marginalized distribution.
3.2 ENCODING AND DECODING WITH TREE STRUCTURE
To enable standard transformer architectures to learn tree-structured reasoning, we develop a serial- ization scheme that encodes complete search trees into linear sequences. This approach allows to train language models on tree data while preserving the branching structure.
Token-Based Tree Serialization. We introduce special tokens that TSLM learns encode tree structure:
[SEP]: Indicates viable action that can be further expanded
[FAIL]: Indicates non-viable action (dead end)
[GOAL]]: Marks the desired goal state
[BOS] and [EOS]: Mark sequence boundaries of child expansions for each node
This serialization captures both successful paths and unsuccessful explorations, teaching the model the complete search process rather than just final answers. detailed worked example showing the complete serialization format is provided in Appendix C
Training Procedure. During training, we apply standard language modeling loss to the entire serialized sequence:
L log po (ee,
where y: includes both reasoning content and structural tokens. This standard cross entropy loss has standard training signal. Unlike sequential modeling where each token depends only its prefix, TSLM tokens depend on tree structure. For example, [SEP] after action a; is conditioned on whether T(s, a.) is expandable and whether it differs from a, -. This structural conditioning teaches the model to:
1. Generate multiple actions at each decision point
2. Assign appropriate viability markers ([SEP), [FAIL), or [GOAL])
3. Structure the exploration systematically
3

---

Published as a conference paper at ICLR 2026
Exarple see
Train instnces
BOB
SEP
EdB
BDC
SEP
SEP
Bo5
SEP
EOS
Infermmoa
Gamention
BOB
SDP
ED6
Stitking
++] Bos
SEP
EoB
Figure 2: TSLM (Tree-Structured Language Model) is a language model designed for hierarchical exploration in sequence generation tasks. Beginning with initial state (e.g., "A") and progressing toward a goal state (e.g.. "T"), TSLM constructs a tree structure where nodes represent states and branches signify possible paths. During training, the model serializes the tree into linear sequences using special tokens to separate branches and mark the start/end of sequence, allowing it to learn structured expansions effectively. During inference, TSLM generates multiple branching actions to explore diverse sequences (e.g., expanding from "C" to "G" and "H")) These branches independently expanded in parallel using a stitching process, enabling broad exploration toward the goal while efficiently pruning unwanted paths.
Inference Procedure.
During inference, TSLM reconstructs the tree structure:
1. Generate the next reasoning step with multiple candidate actions
2. Parse structural tokens to identify viable branches ([SEP])
3. Fork different branches into independent sequences for parallel expansion
4. Recursively expand each viable state until finding solution ([GOAL]) exhausting options
This approach enables systematic exploration. While TSLM forks divergent branches into different sequences (involving multiple calls), the construction of the whole search tree is coherent and internalized, avoiding the redundant sampling inherent in
3.3 FORMAL TRAINING OBJECTIVE
For search tree T with nodes N, let (s::) denote the path from root node 8;. Each node 8; generates a token sequence Ye [a,, m,] where a, the action description and m; [[SEP], [FAIL], [GOAL] is the structural marker.
The context for node s; consists of:
ctx(s:)
U ye,
U
Vok
(yEn(s;)
S& Esibliings((() k<i
where siblings(s:) are nodes sharing the same parent, and A < indicates siblings generated before 8; (enforcing left-to-right ordering within each branching factor).
The training objective becomes:
C
ΣΣ>
log p(ysat ctx(s,,
W
S.EN
This formulation makes explicit that each node conditions (1) its ancestral path and (2) previously generated siblings, but not on unrelated subtrees. This selective conditioning is what enables context decoupling.
4

---

Published as a conference paper at ICLR 2026
In practice, serialize the tree using depth-first traversal. converting it into a linear sequence that transformers can process. Crucially, while the input is linear, the conditional dependencies are dictated by tree topology: node 8; depends only ctx(s;) (its ancestors and prior siblings), not all tokens that precede it in the serialized sequence. This structured conditioning is what distinguishes TSLM from simply concatenating all exploration paths sequentially.
SEARCH TREE SUPERVISION
4.1 TRAINING ON STRUCTURED TASKS
For structured tasks with predefined search trees, can directly train TSLM to learn and reproduce the tree structure. Formally, let E T be a task with a solution of a finite depth, T(x, a) be a transition function that maps acatinaction pairs to new states, and A(s) be finite action space that defines valid actions at state s. Since these components are explicitly defined, we can generate the complete search tree and employ TSLM to predict the branching structure (T(s, a) search tree. By directly imitating the predefined tree expansions, TSLM guides the model to faithfully reproduce structured exploration patterns. Examples include board games and planning problems with well-definee rules.
A()) at each state in the
4.2 SEARCH TREE SUPERVISION FOR OPEN-ENDED REASONING TASKS
While structured tasks have predefined search trees that TSLM can directly learn from, most real- world tasks lack explicit tree structures, providing only correct answers or gold trajectories. this paper, adopt simple bootstrapping method construct synthetic training trees by combining model-generated explorations with known solutions; For each training instance, we deploy the Tree- of- Thoughts sampling Yao al. (2023) from supervision language model to generate pseudo search tree. The process involves:
Sampling a set of candidate actions each state using beam search
2. Building a tree structure by propagating these actions forward
3. Incorporating known gold trajectories as high-priority branches
4. Ordering remaining branches using a reward function R(s, a)
5. Deduplicating redundant paths while preserving the tree structure
Algorithm details this procedure. Our approach ensures each training tree contains least one valid solution while exploring diverse alternatives. The reward-based ordering helps prioritize promising actions, while deduplication prevents redundant search. We adopt the original RAP et al. (2023) reward function to refine exploration by prioritizing promising branches
EXPERIMENTAL RESULTS
5.1 BASELINES FOR COMPARISON
Baseline Model Architecture
We compare TSLM to the following baselines:
Sequence Cloning (SC): A standard sequential modeling that clones single linear sequence of gold Thoughtf-Thought (language modeling similar to GPT-3 (Brown et al. 2020) or Ilama 3 (Touvron et al. 2023)).
Procedure Cloning (PC): A sequential modeling that clones Chain-of- Thouugh reasonings the entire search trace in a single linear sequence (ol-like reasoning models, trained with systematic supervision( (iim et al. 2024: Yang al. 2022)).
'Caveat here is that this is not the best method to train TSLM. For example, we can use supervision from better reasoning language models using techniques like ReJump (Zeng et al. (2025). Also we can use self-traine training with external search methods like SoS (Gandhi et al. 2024). We rather focus on the concept of TSLM itself rather than high-level training techniques.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Guided Search Tree Bootstrapping
Input: Task T, transition T,, reward R, branch factor k; supervision model
Data =
fort e T do
Initialize queue = [s].
Gold trajectory 8o 81 $........ 8n
while $ queue do tmp queue.pop(()
for = 1 to k do tmpqueue = 0
if = and tmp fsp..... Sn then
a, a a²mmmp) {Add gold action) Emmqueeue.add((((m) a:))
else
04 ma((ppp)
if aa $ {aI...., a;-1) then
umpqueeueadd(((((mm a;))
{Deduplication)
end if end if end for tmpqueue = aR( ((mppuuee))Soo (Sort by reward)
queue+ tmpqueue end while Dat Dat (ppe.)ppend(qu) end for
GRPO: A reinforcement learning approach to incentivize reasoning trace during post- training (ol-like reasoning models, trained with GRPO Shao et al. 2024) objectives, group size = 8)
Tree-of- Treo-of-Thought (ToT): Scaling the number of inference of SC model during test-time using external search algorithms Yao al. 2023) with beam search across multiple reasoning paths.
To ensure rigorous comparison, we evaluate ToT using pass 100 metric (success if any of the first 100 terminal nodes is correct), while all other methods (SC, PC, GRPO, TSLM) are evaluated using pass@1 (single attempt). We chose ToT@ @ 100 because it represents the convergence point of the search tree in our experiments, effectively serving as an upper bound for sampling- based scaffolding methods. Thus, comparing TSLM (pass @ 1) against ToT (pass @ 100) provides highly conservative estimate of TSLM's relative performance.
We test models using greedy decoding for sequential methods, and breadth of 5, temperature of 0.3 across ToT experiments. For TSLM inference, we explore different tree traversal and solution selection strategies. Unless otherwise specified, we use Breadth-First Search (BFS) as the default algorithm to systematically explore the generated tree structure until finding a successful solution. (We analyze implications of BFS versus alternative search strategies such as Depth-First Search (DFS) in Appendix $E.1) For each expansion during the inference, we select the first t = 5 actions generated and deduplicate them with exact matching.
To evaluate Tree-Structured Language Modeling (TSLM), we conduct experiments on both structured and unstructured tasks. Our experiments use Llamm-3-888 Grattafiori al. (2024) base experiments unless specified, comparing TSLM against sequential language modeling baselines. Also, We aim to compare architectural differences rather than scaling effects, using modest training data (less than 10K instances per task) for post-training. We have two task scenarios: Structured Planning Tasks, which are tasks with predefined search trees, while Open-ended Reasoning Tasks are tasks with sundefined solution spaces. Structured planning tasks include Game of 24 and Textualized Gridworld, while Open-ended reasoning tasks include ProntoQA and GSMSK. Refer to Appendix | for more details regarding each task and supervision examples.

---

Published as a conference paper at ICLR 2026
pass @
pass 100
Task
SC
PC
GRPO
TSLM
ToT
F.2
Game of 24
17.0%
47. 47.0%
15.0%
100%
32.0%
Gridworld (i.d)
78.2%
99.7%
24.0%
100%
95.0%
Gridworld (o.o.d)
33.0%
81.1%
6.0%
91.5% %
42.7%
ProntoQA
99.7%
97.5%
99.8%
100%
100%
GSM8K
55.8%
55.9%
60.8%
61.6%
62.3% %
Table I: Success rates across different tasks methods. Note that ToT results are reported as pass 100 (success if any of 100 attempts is correct), while all other methods are pass@) this disadvantage. TSLM matches or outperforms ToT on most tasks. The Gridworld results show both in-domain (10x10) and scaling (20x20) performance, highlighting TSLM's robustness when complexity scales beyond training boundaries.
mass@1, 1. Despite
5.2
BASE RESULTS
Table summarizes our experimental findings across tasks, revealing key insights about different reasoning approaches. TSLM consistently outperforms sequential models, achieving perfect accuracy on structured tasks. Most notably, the Gridworld scaling results reveal a striking limitation of Tree-of- Thought: while ToT achieves excellent in-domain performance (95.0% on 10x10 grids), it suffers catastrophic degradation when complexity scales (dropping to 42.7% on 20x20 grids).
This ToT scaling failure particularly surprising given its strong in-domain performance and sophisticated external search mechanisms. In contrast, TSLM maintains robust performance across the complexity boundary (100% + 91.5%), demonstrating that internalized search procedures generalize better than external search algorithms. For open-ended tasks, ToT shows comparable performance on GSM8K (62.3% 61. 61.6%) ) but this advantage disappears when systematic exploration is needed, as evidenced by Game of 24 results where ToT performs no better than basic sequential methods (17.0%).
5.3
FAIR COMPARISON: TEST-TIME SCALING
A eritical distinction: TSLM introduces fundamentally different test-time scaling paradigm. Tradi- tional methods scale by generating k independent trajectories through parallel sampling (ToT) or sequential enumeration. Each trajectory recomputes shared prefixes redundantly. TSLM scales by exploring k candidates within a single, coherent search tree generated by the model in one forward pass. The model explicitly constructs the branching structure, then we select which branches to verify.
We define test-time scaling for each method:
TSLM: Generates coherent tree structure, then verifies the first & terminal states (ending with either [FAIL] or [GOAL]) from Breadth-First Search traversal. Crucially, the candidates come from a single construction, not & independent samplings.
Procedure Cloning (PC): Verifies the first k terminal states from the sequential search trace.
To rigorously compare the effectiveness internalized search (TSLM) versus external scaffolding (ToT), we evaluate performance under matched candidate budgets. This comparison is crucial because TSLM's "test-time scaling" differs fundamentally from standard parallel sampling. ToT or standard scaffolding, scaling & implies generating k independent, redundant traces. In TSLM, scaling k means exploring more branches within a single, cohesive search tree structure generated by the model. This internalized scaling allows for more efficient resource allocation, the model can prioritize promising branches without regenerating common prefixes or exploring completely disjoint paths. Figure 3a presents the results GSM8K
scale the number of candidates k.
TSLM consistently outperforms ToT and PC across all budgets. Notably, TSLM with just a single candidate (61.3%) nearly matches ToT's converged performance (62.3%), and TSLM saturates at

---

Published as a conference paper at ICLR 2026
significantly higher accuracy (67.2%). This confirms that TSLM's training objective successfully internalizes a superior search tree compared to the parallel sampling in ToT.
5.4 COMPUTATIONAL EFFICIENCY ANALYSIS
An immediate followup advantage of TSLM is its inference efficiency. Unlike external scaffolding methods that require multiple independent model calls, TSLM generates the complete search tree in a single forward pass (or a few passes if using beam search decoding), sharing computation for common prefixes.
We compared the average wall-clock time required to reach solution convergence across methods. As shown in Figure 33b TSLM is significantly faster than ToT. ToT suffers from time explosion due redundant sampling and lack of computation sharing. TSLM is also faster than Procedure Cloning (PC) because TSLM's tree structure allows the model ignore irrelevant subtrees during inference, whereas PC must process the entire linearized sequence of the search trace.
65
100
60
10
55
2
3
5
6
7
8
9
10
10
(a) Mean accuracy (%,
(b) Avg. inference time (sec per instance, 4
Figure 3: Comparison of (a) mean accuracy and (b) average inference time when scaling the number of candidates across different methods GSM8K. Methods: TSLM (o). PC (A), and SC (0, convergence at ToT@ 100: -). All the cache control during the inference time is done by vLLM (Kwon 2023 v.0.666.
ANALYSIS
Beyond core performance gains, TSLM exhibits several remarkable capabilities that distinguish it from traditional approaches.
6.1 IDENTIFYING UNSOLVABLE CASES
One challenging aspect for language models is avoiding hallucination on problems that have no valid solution. Sequential models trained SC, PC, GRPO implicitly learn generate answers within their training distribution, which can be problematic when faced with unsolvable cases. For example. 2, and cannot generate 24 through any sequence of
in the Game of 24 task, the numbers arithmetic operations. Since sequential models have not been trained on examples with no solution, they tend hallucinate and generate invalid answers.
TSLM, however, demonstrates unique capability to identify unsolvable cases. Quantitatively, across 100 unsolvable Game of 24 instances, TSLM correctly identified 97 cases by terminating without solution. In contrast, baseline methods (SC, PC, GRPO) failed to identify any unsolvable cases, instead hallucinating invalid solutions (There are unintentional instances where PC correctly refused, only because the model generated excessively long traces that failed to terminate properly due to skewed search tree generation). This suggests the tree-structured exploration enables TSLM to systematically explore the full solution space and recognize when valid path exists. Unlike sequential models that are pressured to always generate some answer, TSLM's cohesive search tree allows to confidently determine and declare when problem is unsolvable.
We position this capability as addressing a fundamental limitation sequential language models. Hallucination remains major problem in language modeling, with theoretical work suggesting it's
8

---

Published as a conference paper at ICLR 2026
inevitable in the next-token prediction paradigm (Xu et al. 2025) Frontier labs like OpenAl attempt to address this through post- rostitrannng to modify refusal behavior (Kalai et al. 2025). but the mechanisms remain unclear and heavily rely RL approaches.
SC
PC
TSLM (ours)
33.0%
81.1%
91.5%
Table Qualitative comparison of model extrapolation capabilities on Gridworlds of varying sizes (maximum 20 20). Each heatmap shows performance different grid dimensions (x.y), with darker colors indicating better performance. The red box indicates the boundary of training data. Overall accuracy shown above each plot.
6.2 QUALITATIVE ANLYSIS ON EXTRAPOLATION BEYOND TRAINING DATA
We qualitatively show how TSLM demonstrates strong extrapolation capabilities when tested on larger grid sizes than seen during training for each grid size (Table 2 This suggests TSLM's structured tree representation learns generalizable navigation patterns rather than memorizing specific contigurations.
6.3 TEST-TIME SCALING: BEYOND PARALLEL SAMPLING
Beyond the quantitative improvements shown in $5.3 TSLM's test-time scaling reveals qualitative shift in how models can control exploration. Traditional scaling methods treat the model fixed sampler: increasing simply draws more independent samples from the same learned distribution. TSLM scaling is fundamentally different: the model explicitly generates branching structure, enabling adaptive control over which regions of the search space to explore. This transforms scaling from "sample more" "explore strategically."
The robustness of TSLM's internalized search extends across different base models. Appendix F.2 demonstrates that TSLM maintains consistent improvements baselines regardless of the Qwen-2.5-7B), showing that the benefits
underlying architecture (llmma-3-88, stem from structured exploration rather than model-specific quirks.
More importantly, TSLM enables flexible search strategies unavailable to parallel sampling methods. By generating explicit tree structure, we can choose between breadth-first search (prioritizing solution optimality) and depth-first search (prioritizing model confidence), or implement adaptive strategies that expand high-reward branches more aggressively. As shown in Appendix $E.1] DFS achieves 63.1 1 1% top-1 accuracy (vs BFS 61 11.3) by immediately following high-confidence actions, while BFS provides better coverage for top-k exploration. This controllability impossible with independent sampling, where each trajectory commits to full path without coordinating exploration prioritics.
Finally, TSLM's scaling extends beyond the training branching factor. TSLM's scaling extends beyond the training branching factor (Appendix $E.3). ( that models trained with k=5 successfully extrapolate to k=10 at inference time, improving convergence from 67. 67.2% to 71.1%. The model generalizes its learned exploration strategy to broader search, unlike parallel sampling which simply draws more independent samples from the same distribution.
7 CONCLUSION
We introduce Tree-Structured Language Modeling (TSLM), which enables language models to generate complete search trees within single generation process using token-based serialization.

---

Published as a conference paper at ICLR 2026
TSLM achieves superior performance across structured planning and open-ended reasoning tasks: 100% accuracy on Game of 24 (vs. 17% for baselines), robust extrapolation to larger environments (76.5% vs. 26% for Sree-of-Thought!
Our work challenges the prevailing assumption that reasoning capabilities require reinforcement learning or inference-time search orchestration. By training on complete tree-structured traces rather than single solution paths, supervised learning can internalize systematic exploration strategies. Key advantages over existing approaches include: (1) coherent tree generation vs fragmented parallel sampling, (2) selective context decoupling enabling efficient branch exploration, (3) emergent capabilities like unsolvable problem detection, and (4) new test-time scaling paradigm that explores within structured trees rather than across independent trajectories.
REFERENCES
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. Proceedings of the AAAI
Conference on Artificial Intelligence, 38116) March 2024. ISSN 2159 5399. doi: 10. v38.16..9726.. htRt:t/t://at doi www.o0.../0...........oo 609 aaai wtw..a.....11111.2.....
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901,202
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang. Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like Ilms, 2025. URL https:
//arxiv.or org/ www./ag////////oooo///....o..aae 2212241222222 21187
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025. Utt/tttss://aaxii wwtp:///wwwrgaa////////// 17161
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue. Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. I. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kcxin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu. Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yumfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou. Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. 7.. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang,
10

---

Published as a conference paper at ICLR 2026
Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL htt.htpss//aaxl...rrraaaaa//////
12948
Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training, 2024.
URL
orgg abs/ /tops///22333 22309 17179
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and
Noah D. Goodman. Stream of search (sos): Learning to search in language, 2024. htthts
//a //arxivvorg/aal 2.404 03683
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang. Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olamo, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla. Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaclle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuming Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido,
Plawiak,
11

---

Published as a conference paper at ICLR 2026
Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang. Guna Lakshminarayanan Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Iliss Leontiadis, Irina-Flena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang. Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal. Nick Egebo, Nicolas Usunier, Nikhil Mehta. Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Rughotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang. Shuqiang Zhang. Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad. Sumit Gupta, Summer Deng. Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury. Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang. Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiaojian WU, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniy Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. htthhttp:: wwt.aaaavvv.......a org/ 2tt.b///277227222
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model, 2023. URL org/ abs//bb//// 2305. 14992
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. Why language models hallucinate, 2025. URL /tttps:://axii...rrrrrrstss abt.//a..///../.e/// 04664
Doyoung Kim, Jongwon Lee, Jinho Park, and Minjoon Seo. How language models extrapolate outside the training data: A case study in textualized gridworld, 2024. URL htt.:httts/:/iiiiii.
15275
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Liammin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023.
12

---

Published as a conference paper at ICLR 2026
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' AAttumee Igor Babuschkin, Xinyun Chen, Po-Sen Huang. Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-leve code generation with alphacode. Science, 378(6 1092-1097 December 2022. ISSN 1095-9203, doi: 10.1126/science.abe ablg 158.. URL http: //xx http:/xxxiooooooooooooooooo.oooo
wtt.:/oo..............1..... /ttp:////..kkaoaoooooo////////// http://///ciieennncc1l1.... abq1158
doi wwt.:oggg/.....
OpenAl. Openai ol system card. preprint, 2024.
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang. Y. K. Li, Y. Wu, and Daya Guo. Deepseckmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL htttp:://axxi..orrtt abs//
24C2. 03300
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 022302..397,,. 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistencyy improves chain of thought reasoning in language models, 2023.
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: inevitallee An innate limitation of large language models, 2025.
Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Chain of thought imitation with procedure cloning, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
Yuchen Zeng, Shuibai Zhang, Wonjun Kang, Shutong Wu, Lynnix Zou, Ying Fan, Heeju Kim, Ziqian Lin, Jungtack Kim, Hyung 1l Koo, Dimitris Papailiopoulos, and Kangwook Lee, Rejump: A tree-jump representation for analyzing and improving Ilm reasoning, 2025. URL https
//arxiv.orq/abs/251..
Andy Zhou, Kai Yan, Michal Mhapentooh-Rottmaall Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406, 2023.
A FREQUENTLY ASKED QUESTIONS
Q1: What do these results mean for reasoning model development more broadly? Our results challenge three prevailing assumptions. First, TSLM (67.2%) outperforms ToT (Yao et al. 2023) (62.3%) despite ToT using 100 independent samples, revealing that coherent tree generation beats fragmented parallel sampling. The bottleneck not inference compute but systematic search space construction. Second, ToT's catastrophic failure on 20x20 Gridworld (42.7%) versus TSLM's extrapolation (91.5%) shows external search does not generalize when complexity scales; the search procedure must be internalized during training. Third, TSLM achieves perfect Game of 24 accuracy via supervised learning while GRPO (Shao et al., 2024) achieves only 15%, challenging the "SFT Memorizes, RL Generalizes" paradigm (Chu et al. 2025). When supervision includes complete exploration traces rather than just answers, supervised learning can acquire systematic reasoning capabilities typically associated with RI. methods.
13

---

Published as a conference paper at ICLR 2026
Q2: How does TSLM differ fundamentally from existing approaches (ToT, standard CoT, GRPO)? TSLM generates coherent tree structures within a single model forward pass, unlike ToT (independent parallel sampling), standard CoT (single linear path), or GRPO (RL-based trace generation). The key is selective context decoupling: TSL.M conditions each branch only on relevant ancestors and siblings, not the entire exploration history. This enables both efficient training (learning from complete trees) and efficient inference (avoiding redundant prefix computation).
Q3: Can TSLM be applied to more complex real-world tasks? While our current experiments
focus on relatively constrained tasks, the principles TSLM can extend to more complex domains. For tasks with well-defined structure (like code generation or game playing), direct application is straightforward. For more open-ended tasks, our bootstrapping approach provides a foundation for constructing synthetic structrud training data. Future work should explore applications to more diverse and complex reasoning domains.
Q4: Why compare only with ToT and not advanced methods like RAP, LATS, or Reflexion?
These methods represent online planning (external feedback loops during inference), while TSLM represents offline planning (internalizing search during training). ToT is the representative baseline for external scaffolding without environment interaction. Methods like LATS et al. 2023) (Saliiion(Shinnaa
and Reflexion Shinn et al.) 2023) are orthogonal and could potentially enhance TSLM, but address different problems (online refinement native tree generation). Note that ToT @ 100 evaluation upper-bounde related sampling methods: mathematically E P(((P))))) < F ((TTT@)))) and E| Weeeff-oosistenc(@fff
F[P(TTT@b")) since tree expansion has the same expectation as
parallel sampling
Q5: Does TSLM work on truly open-ended tasks? TSLM relies on the ability to construct training trees, which requires verifiable solutions (even only for a subset of data). For truly open- ended domains with subjective correctness (e.g., creative writing) or no clear verification, constructing the training signal challenging. This is limitation of the current bootstrapping approach, though the core principle of structured exploration may still apply if suitable reward signals can be defined.
B RELATED WORK
External Augree-A Language Models. major line of work augments language models with external search algorithms during inference. Tree-of-Thoughh Yao et al. (2023) applies breadth- first and depth-first search externally, sampling multiple reasoning paths with external evaluation. Graph-of-Thouge [Besta et al. (2024) extends this to general graph structures. Monte Carlo Tree Search (MCTS) approaches include RAP (Reasoning via Planning) Hao et al. (003), which employs MCTS with world models, and LATS (Language Agent Tree Search) Zhou al. (2023), which combines MCTS with reflection mechanisms. TS-LLM Feng et al. (024)) and AlphaCode-styyle approaches Li al. (2022) integrate MCTS with language generation. These methods achieve strong performance but require multiple model invocations and external orchestration, limiting efficiency and integration with model training.
Multi-Path Generation and Reasoning. Various approaches explore multiple reasoning paths without structured search. Self-consistency decoding Wang et al. (2023) generates multiple indepen- dent reasoning paths and selects the most consistent answer. Ensemble methods combine predictions from multiple reasoning chains through diverse beam search and nucleus sampling variants. Recent models like [OpenA]| (2024) and DeepSeek-R1 DeepSSek-AA et al.] (2025) generate extended reasoning traces, but remain fundamentally sequential and may include redundant computation Chen jet al. (2025). These approaches explore multiple paths but typically generate them independently without capturing structural relationships. unlike TSLM's coherent tree structures.
Learning- Laaseing-Based Reasoning Enhancement. Reinforcement learning has emerged as dominant paradigm for improving reasoning capabilities. DeepSeek Math Shao et al. (2024) applies GRPO for mathematical reasoning, while other work uses RL from human feedback for instruction following. Actor-Critic methods and policy gradient approaches have shown promise in multi-step reasoning and mathematical problem solving. Algorithmic reasoning approaches train models to imitate procedures
14

---

Published as a conference paper at ICLR 2026
like sorting, graph traversal, and dynamic programming Kim al. (2024) Yang al. (2022) learning to execute classical algorithms step-by-step. However, these methods either require complex RL training or focus on single algorithmic traces rather than dynamic exploration strategies. TSLM demonstrates that carefully structured supervised learning can achieve comparable performance without RL's complexity.
C TSLM SERIALIZATION EXAMPLE
This section provides detailed worked example showing how TSLM serializes tree structures for the Game of 24 task.
Problem Setup. Consider Game of 24 problem with numbers [4, 5, 6, 10]. The goal is to find arithmetic operations that result in 24.
Complete Tree Serialization.
The tree structure is serialized as:
Input: [SEP]
15 [SEP]
Output:
[BOS]
15
[GOAL]
15 -6 [FAIL] [EOS]
This format captures:
Multiple candidate actions at each step
Viability markers ( ([SEP] for expandable, [FAIL]] for dead ends)
Step boundaries ( ([BOS), [EOS])
Both successful and unsuccessful exploration paths
D LIMITATIONS AND FUTURE WORK
Computational Overhead. Training complete trees requires processing all nodes rather than single paths, increasing computational cost by factor of the average tree size. Future work should explore efficient training strategies like tree-aware attention caching and sparse gradient updates for selective subtree optimization.
Supervision Quality Dependence. Synthetic tree generation quality depends heavily on the super- vision model's capabilities. For instance, Llama-3-8B produces poor-quality trees for GSM8K, while Moma-3--B-Inntruuee works well. Models must match the target reasoning format (some generate Python code instead of step-by-step reasoning, others include extraneous dialogue). Better automated methods for supervision model selection and tree quality validation are needed.
Open-Ended Domains. TSLM requires verifiable solution paths construct training trees. In truly open-ended domains with subjective correctness (creative writing, open dialogue), defining branch viability becomes challenging. Adapting TSLM such domains may require preference learning or soft verification signals rather than binary correctness.
E DETAILED INFERENCE-TIME SCALING ANALYSIS
BFS VS DFS: EXPLORING SEARCH STRATEGIES
Within TSLM's inference framework, the choice between Breadth-First Search (BFS) and Depth-First Search (DFS) reveals fundamental differences in exploration priorities. We analyze these search strategies when TSLM's branching is guided by preference-based ordering.
15

---

Published as a conference paper at ICLR 2026
Theoretical Analysis. Consider two distinct solution paths A d1,..., On and B b1, ... dm in preference-ordeee search tree generated by TSLM.
BFS employs dual prioritization strategy: it first considers path length (traversing A before B if n < m), then uses preference signals to break ties. When paths have equal length (n = m), BFS selects based the first differing action's reward. If R(ax) R(bx)) the earliest divergence point k BFS favors path A. This means BES prioritizes solution optimality first, using learned preferences resolve ties.
In contrast, DFS operates purely on learned preference ordering, disregarding path length con- siderations. For the same paths A and B, DFS immediately follows the higher-reward action any divergence point k, regardless of whether this leads to longer solution paths. This approach means DF' prioritizes learned preferences consistently, potentially sacrificing optimality for high-confidence actions.
Empirical Results. Figure shows that DFS achieves better top-1 accuracy by finding preferred solutions first, but slightly underperforms BFS for top-3 top-6 candidates since it does not prioritize optimality. However, both methods converge to the same accuracy as they traverse the complete tree. This validates our theoretical analysis that DFS provides faster convergence to high-confidence solutions while BFS offers better overall exploration coverage.
66
62
2
4
6
8
10
Figure 4: Comparison of mean accuracy (%) over number of candidates for different search strategies
(BFS: DFS ))
E.2 BASE MODEL PERFORMANCE IMPACT
We investigate whether the base model's performance influences TSLM's effectiveness by comparing performance across different models with different GSM8K performance (Llama-3-88. Llama-3-8B- Instruct, and Qwen-2.5-7B). TSLM consistently outperforms baseline methods across both model variants, with improvements remaining robust across architectures. This indicates that TSLM's structured exploration provides consistent benefits independent of the base model.
E.3 BRANCHING FACTOR ANALYSIS
The branching factor l: in search tree generation (Algorithm 1 controls the maximum number of candidates expanded at each node. initially set k to match supervision breadth, but natural question arises: Can TSLM effectively scale to larger branching factors?
In principle, if we traverse the complete search tree, a larger should yield better accuracy by exploring more candidates. However, when limited to examining just the first few terminated candidates, increasing the branching factor could potentially add noise that degrades performance.
We investigate this empirically by comparing TSLM performance between k 5 and & 10 branching factors. As shown in Figure extrapolating to the larger branching factor yields consistent improvements, with convergence rate increasing from 67.2% 71.1%. This enhanced performance indicates TSLM can effectively leverage broader exploration without being overwhelmed by the expanded search space.
16

---

Published as a conference paper at ICLR 2026
90
80
70
60
Llama-3-8B
Llama--8B-Insstn
Qwen-2.5-7B
Base
SC Il P TSLM
Figure 5: GSM8K accuracy comparison across different base models and methods.
70
65
60
2
4
6
8
10
Figure Comparison of TSLM performance for the base branching factors (o) vs. extrapolated
k 10 (+). Extrapolating the branching factor improves accuracy in scalable manner.
TASK DETAILS AND EXAMPLES
Task
# Depth(Train)
# Depth(Test)
# Train Instance
# #Test Instance
Evaluation Metric
Game of 24
1.2K
100
Equation Validation
Gridworld
10 x 10
10 10
10K
1.5K
Exact Matching
Gridworld (o.o.d)
10 10
20 x 20
10K
1.5K
Exact Matching
ProntoQA GSMSK
1-5
1-5
4.5K
450
Exact Matching
2-9
2-11
7.5K
1.3K
Answer Matching
Table 3: General task settings
F.1 GENERAL TASK SETUP DETAILS
Table Billustrates the general task settings for structured and open-ended tasks. Depth refers the number of reasoning steps.
F.2 TASK SCENARIO I: STRUCTURED PLANNING
We first evaluate TSLM and baselines two different structured planning tasks (Left in Figure
77.. These tasks involve predefined search trees, enabling us to evaluate how well models reproduce algorithmic patterns:
17

---

Published as a conference paper at ICLR 2026
Game 24
Textualized Gridworld
ProntoQA
GSMIK
8,4.3,0
Tue of fase: Sarn is not warm boooded
30 mila and rntume 30 mile mph. Whatis her average speed?
22. 6
7,8.8
San's canevses
6b-3
speed
sked kTW
12,
12 2
15.6
San not hwrbhrtoue
Soot lt. rostinal
100m
ocassp- 2000
Sain warr-bincced
Agt: Agg
Amswer Puli
rpend:
Precefindd Starth Space
Open onded Reasening
Figure 7: (Left) Structured planning tasks with predefined search spaces (c.g., Game of 24, Textualized Gridworld) where success is measured by the model's ability to reproduce algorithmic search patterns. (Right) General open-ended reasoning tasks (e.g., ProntoQA, GSM8K) requiring adaptive exploration of undefined search spaces.
Game of 24 Given four numbers, the task is to obtain 24 with basic arithmetic operations
(+.-a x, +). For example, with inputs 8, 4, 3, and 6, a valid solution is (8 + 4) (6 + 3) 24 Yao et al. (2023). Each action selects two operands and an operator, while states represent partial expressions. Since there may be multiple paths that reaches the answer (e.g. (8 4) x (6 + 3) and (6+3)) x (4 + 8)), we check if the final expression equals 24, We train each method on 1.2K instances and test on 100 instances.
Textualized Gridworld A text-based navigation task where agent navigates grid using cardinal directions (up, down, left, right) while avoiding obstacles (Kim et al. 2024). Given a start position (typically bottom left) and goal position (typically top right), the agent must find valid path with shortest length. For instance, in a 3x2 grid with a pit at (1,1), the optimal solution is (right, right, up). Actions are single moves (up/down//lettrrihht)) and states are grid coordinates. We ensure all the environments to have unique shortest paths. We use 10K training instances with maximum grid size 10x10 and I.5K test instances up to the same maximum size, evaluating exact path matching.
F.3
TASK SCENARIO 2: OPEN-ENDED REASONING TASKS
We next evaluate TSLM and baselines on two open-ended reasoning tasks (Right on Figure 7 requiring exploration of undefined solution spaces. We use the synthetic tree generation method from $42 for training.
ProntoQA ProntoQA Saparov He (2023) is a logical reasoning dataset where models verify statement truth given premises. For example, given premise "Every jompus is not small. Each impus is small. Each jompus is a dumpus. Alex is a jompus." and query "True or false: Alex is not small?", the solution path to verify the query should be "Alex is jompus. Every jompus is not small. Alex is not small. So the answer is True."
Actions are logical deduction steps and states are accumulated facts. With unique valid deduction paths, we verify whether the generated path is correct. We use 4.5K training and 450 test instances. For TSLM/PC, we use tree generation of beam search (k=5, temp=0.3) with Llama-3-8B as a base model for supervision.
GSM8K GSMSK Cobbe et al. [2021 is mathematical word problem dataset requiring multi-step reasoning. For example, for given problem "James runs 3 sprints 3 times a week. Each sprint is 60 meters. How many total meters per week?", one valid solution would be "3 sprints 3 times sprints per week. 9 sprints x 60 meters = 540 meters. The answer is 540."
Actions are solution steps and states are partial solutions. Since multiple valid solution paths can reach the correct answer, we just validate whether the answer is correct. We use 7.5K training instances and 1.3K test instances. For TSLM/PC, we use tree generation of beam search (k=5, temp=0.1) with Mamm--3-8--Instruce as a base model for supervision.
F.4 TASK EXAMPLES
18

---

Published as a conference paper at ICLR 2026
Gridworld SC Instance:
USER: You are given a rectangular gridworld, where you can move up, down, left, or right as long as each of your x, y coordinate is within 0 to the x, y size of the grid. If you move up. your y coordinate increases by 1. If you move down, your y coordinate decreases by 1. If you move left, your x coordinate decreases by 1. If you move right, your x coordinate increases by 1.
You will interact with the girdworld environment to reach the goal state, while avoiding the pit and the wall. You cannot move through the wall move outside the grid. If you fall into the pit, you lose. If you reach the goal, you win. For each of your turn, you will be given the possible moves.
You should respond your move with either one of 'up'. 'down', 'left', or 'right'.
ASSISTANT: OK
USER: Grid is from (7, 7) to (18, 13). Goal: (18, 13) Current: (7, 7) The pit is at (16, 9),
(17, 7), (14, 8), (11, 9), and (16, 11). The wall is at (11, 7), (15, 10), (18, 12), (9, 9), (8, 7), (17, 10), (9, 12), (12, 7), (10, 13), (15, 12), (17, 9), (9, 11), (17, 12), (8, 9), (13, 11), (10, 9), (12, 9). (11, 1I), (12, 12), (16, 10), (15, 8), (9, 7), (8, 11), (7, 13), (13, 13), and (13, 10).
ASSISTANT:
Step 1
up
(7, 8)SP)
Step 2
(7. []EPP]
Step 3
up
(7, [0)SEP
4
up
(1)(SEP]
Gridworld PC Instance:
USER: You are given a rectangular gridworld, where you can move up, down, left, or right as long as each of your x, y coordinate is within 0 to the x, y size of the grid. you move up. your y coordinate increases by 1. If you move down, your y coordinate decreases by I. If you move left, your x coordinate decreases by 1. If you move right, your coordinate increases by 1.
You will interact with the girdworld environment to reach the goal state, while avoiding the pit and the wall. You cannot move through the wall move outside the grid. If you fall into the pit, you lose. If you reach the goal, you win. For each of your turn, you will be given the possible moves.
You should respond your move with either one of up', 'down', 'left', or 'right'.
ASSISTANT: OK
USER: Grid is from (7, 7) to (18, 13). Goal: (18, 13) Current: (7, 7) The pit is at (16, 9),
(17, 7), (14, 8), (11, 9), and (16, 11). The wall is at (11, 7), (15, 10), (18, 12), (9, 9), (8. 7), (17, 10), (9, 12). (12, 7). (10, 13), (15, 12), (17, 9), (9, 11), (17, 12), (8, 9), (13, 11). (10, 9),
(12, 9), (11, 11), (12, 12), (16, 10), (15, 8), (9, 7). (8, 11), (7. 13), (13, 13), and (13, 10).
ASSISTANT:
Step
up
(7, {)EEP]
down
(7, [[AIL]
left
(6. [)FAIL]
right
(8. [)[AIL]
Step 2
19

---

Published as a conference paper at ICLR 2026
up
(7, 9][EP]
down
(7, []
left
(6, [)FAIL]
right
(8, 8)SEP]
Gridworld TSLM Instance:
USER: You are given a rectangular gridworld, where you move up, down, left, right as long as each of your x, y coordinate is within 0 to the x, size the grid. you move up. your y coordinate increases by 1. If you move down, your y coordinate decreases by 1. If you move left, your x coordinate decreases by 1. you move right, your coordinate increases by 1.
You will interact with the girdworld environment to reach the goal state, while avoiding the pit and the wall. You cannot move through the wall or move outside the grid. If you fall into the pit, you lose. If you reach the goal, you win. For each of your turn, you will be given the possible moves.
You should respond your move with either one of "up", 'down', 'left', 'right'.
ASSISTANT: OK
USER: Grid is from (7, 7) to (18, 13). Goal: (18, 13) Current: (7, 7) The pit is at (16, 9),
(17, 7), (14, 8), (11, 9), and (16, 11). The wall at (11, 7), (15, 10), (18, 12), (9, 9), (8, 7), (17, 10), (9, 12), (12, 7). (10, 13), (15, 12), (17, 9), (9, 11), (17, 12), 12), (8, 9), (13, 11), (10, 9), (12, 9), (11, 11), (12, 12), (16, 10), (15, 8), (9, 7), (8, 11), (7, 13), (13, 13), and (13, 10).
ASSISTANT:
Step 1
up
(7, 8)SEP]
Step 2
(7, [SEP]
Step 3
up
(7, 10)[SP
Step 4
up
(7, (DISSP)
down
(7, [[FAIL]
left
(6, [0)[AIL]
right
(8, 10][SEP]
Game24 SC Instance:
USER: Input: 4, 5, 6, 10
ASSISTANT: Step
4+ 5 9[SEP]
Step 2
6+ 10 = 16[SE]]
Game24 PC Instance:
USER: Input: 4, 5, 6, 10
ASSISTANT: Step
4+ = 9[SEP]
5-5 -[[EPP]
4* = 20[SEP]
4/5 4/5[SEP]
20

---

Published as a conference paper at ICLR 2026
6+ 10[SEP]
4-6= [
4*6= 24[SEP]
4/6= 2/3[SEP]
4+ 1010- 14[SEP]
1-10=-6[SEB
4 * 4010= 40[SEP]
4/10= = 2/S[SEP]
4 9[SEP]
5-4= [[SEP]
* 4= 20[SEP]
5/4 5/4[SEP)
6 ]
6 = 1 [[EPP]
5 * = 30[SEP]
5/6 5/6[SEP]
5+ = ]5[SEP]
5-10 [5[SEP]
5 10 50[SEP]
5/ 1/2[SEP]
6 ]
6-4 2[SHP]
6* 6*4= 24[SEP]
6/4 3
5+ 1][SEP)
6-5= [[SEP]
6*5 = 30[SEP]
6/5 6/5[SEP]
6+ 10 16[SEP]
6- = 4|SSPP]
* 10 60[SEP]
6/ 10 = [/P
10 4 14[SEP]
10 4 = 6[SEP]
4 40[SEP]
10 / = 5/2[SEP]
+ 5 = [
10-5 5[SEP]
10 5 = 50[SEP]
10/5 = 2[SEP]
10 60+6= 16[SEP]
10-6 4[SEP]
10 6 60[SEP]
10/ 5/][SE]]]FAIL]
Step 2
6+ 10 16[SEP]
6 10 = 4[SEP]
* 60[SEP]
6/ 10- = 3/5[SEP]
6+9= ]
-39==3
6*9= ]
//9=2/33[SE))
10 16[SEP]
6 (0-6=41SEP)
10 6 6= 60[SEP]
10/6 = 5/3[SEP]
= 1+99=19[SEP)
10 9= [[SEP]
21

---

Published as a conference paper at ICLR 2026
10 $ 9- 90[SEP]
10/9= = 10/9[SEP]
6+6= 15[SEP]
1-6=31SEP
9 6 54[SEP]
9/6 3/2[SEP]
9+ 10 19[SEP]
9- 10 = [[[EP]
* 90[SEP]
9/ 10 = 9/10[SEP][FAIL]
6+ = ]6[SEP]
6- 10 = 4[SEP]
6* 10 60[SEP]
6/ 10 = 3/5[SEP]
6+-1 = SISEP]
6--1 7[SEP]
6 6 = -6[SEP]
6/-1 -6[SEP]
10 6 = 16[SEP]
10 6 4[SEP]
10 * 6 60[SEP]
10 6 = 5/3[SEP]
10 + 1 = 9[SEP]
-1 = ]
10 * -1 10[SEP]
10/ = 10[SEP]
1 5[SEP]
- ]
-1 * 6 = ]6[SEP]
-1 -1/6= -1/[[SEP]
-1 10 9[SEP]
-1-10 = [
-1 * #10 ]0[SEP]
-1/10 = 1/10[SEP]]FAIL]
6+ 10 = ]6[SEP]
6- 10 = 4[SEP]
6 * [0[SEP]
6/ 10 = 3/
60 = ]
2- = ]
6* = = 120[SEP]
6/20 = 3/10[SEP]
10 = 16[SEP]
10 4[SEP]
10 * 6 60[SEP]
10 = 5/3[SEP]
10 + 20 = 30[SEP]
10 20 |
10 * 20 200[SEP]
10/20 1/2[SEP]
20 6 26[SEP]
20-6 14[SEP]
20 * 6 120[SEP]
/ = 10/3[SEP]
20 10 = 30[SEP]
20 10 10[SEP]
20 * 10 200[SEP]
20 10 [[SEP][FAIL]
22

---

Published as a conference paper at ICLR 2026
Game24 TSLM Instance:
USER: Input: 4, 5, 6, 10
ASSISTANT: Step
4+ 5 9[SEP]
Step 2
6+ 10 = 16[SEP]
6 10 4[[SEP]
6 * 10 60[SEP]
6/10- = 3/5[SEP]
-+ 15[SEP]
6-9=-3[SER)
=*9= =
6/9= 2/3[SEP]
10 = 16[SEP]
6 40-E=)
10 6 60[SEP]
10 = 5/3[SEP]
10 + [9[)
10 9 I[SEP]
10 * 90[SEP]
10/9 = 10/9[SEP]
9+6= 15[SEP]
9- 3[SEP]
9*6= = 54[SEP]
9/6 3/2[SEP]
9+ =0 = 19[SEP]
9- 10 [[[EP]]
9 = 90[SEP]
9/ 10 9/10[SEP][FAIL]
ProntoQA SC Instance:
USER: Arthropods are small. Insects are not eight-legged. Lepidopterans are insects. Invertebrates are animals. Every spider eight-legged. Insects are arthropods. Every butterfly is a lepidopteran. Animals are multicellular. Arthropods are invertebrates. Sam is butterfly.
True or false: Sam is not eight-legged.
ASSISTANT:
Step 1
Sam a bStterfly([SPP)
Step 2
Every butterfly is ISpidopteran.(SEP
Step 3
Sam is a lepidopteran. [SEP]
Step 4
Lepidopterans are insects.[SEP]
Step 5
Sam is an iSsec
Step 6
Insects not eight-legged./SEP)
Step 7
Sam is not (Sght
Step 8
The answer true.[GOAL]
с Arthropods are small. Insects are not eight-legged. Lepi- dopterans are insects. Invertebrates are animals. Every spider is eight-legged. Insects are arthropods. Every butterfly a lepidopteran. Animals are multicellular. Arthropods are invertebrates. Sam is butterfly. True or false: Sam not eight-legged.
ProntoQA PC Instance:
ASSISTANT:
23

---

Published as a conference paper at ICLR 2026
Step
Sam a butterfly.[SEP]
Step 2
Every butterfly is Sppidopterann[SEPP
Step 3
Sam is a Iepidopteeaan
Step 4
Lepidopterans are insects.[SEP]
Step 5
Sam is an insect.[SEP
Step
Insects are not (ight-legged.((EP)
Insects are arthropods.(SEN
Step 7
Sam not sight-legged.[SEF Sam is arthropod.[GOOA]]
ProntoQA TSLM Instance:
USER: Arthropods are small. Insects are not eight-legged. Lepidopterans are insects. Invertebrates are animals. Every spider eight-legged. Insects are arthropods. Every butterfly is a lepidopteran. Animals are multicellular. Arthropods are invertebrates. Sam is butterfly.
True or false: Sam is not eight-legged.
ASSISTANT:
Step 1
Sam a [Stte
Step 2
Every butterfly is Iepidopteran.(SEP
Step 3
Sam is a lepidopteran.(SEE)
Step 4
Lepidopterans are insects.[SEP]
Step 5
Sam is an insect.[SEP]
Step 6
Insects not eight-legged.(SEP)
Step 7
Sam is not eight-legged.
Sam an arthropod.[GOAA]
GSMSK SC Instance:
USER: Kylar went to the store to buy glasses for his new apartment. One glass costs $5, but every second glass costs only 60% of the price. Kylar wants to buy 16 glasses. How much does he need pay for them?
ASSISTANT:
Step
Every second glass costs x 0.6 $ << 5
0.6 = >>> 3.[SEP]
Step 2
So for 16 glasses, Kylar needs to pay 3 $ << 16/2 2
>> for every second
glass.[GOAL_]
GSMSK PC Instance:
USER: robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?
ASSISTANT:
Step 1
It takes 2 * 0.5 =<<< < 2 * 0.5 =0 bolt of white fiber.[SEP
The robe takes 2 bolts of blue fiber.|SEPI
Step 2
So it takes 2 =<< 2 = >>> 3 bolts in total.[SEP]
24

---

Published as a conference paper at ICLR 2026
It also takes half as much white fiber, which means takes bolt of white fiber (since half of 2 is 1){SEP]
Step 3
The answer is 3.GGOAL]
GSM8K TSLM Instance:
USER:
Kylar went to the store to buy glasses for his new apartment. One glass costs $5, but every second glass costs only 60% of the price. Kylar wants to buy 16 glasses. How much does he need pay for them?
ASSISTANT:
Step
Every second glass costs 5 * 0.6 << 5 0.6 = 3.PEPP]
Step 2
So for 16 glasses, Kylar needs to pay 3 8 $
16/2 >> for every second
glass.[SEP]
So for glasses, it would cost 8 24 dollars. For the remaining glasses, it would cost 8 x 5 = 40 dollars. So in total, it would cost 24 + 40 = 64 dollars|[OOA]
G USE OF LARGE LANGUAGE MODELS
We acknowledge the use of large language models to assist in the preparation of this manuscript. Specifically:
Writing assistance. Large language models were used to aid in polishing and refining the writing throughout the paper, including improving clarity, grammar, and expression of technical concepts.
Related work discovery. Large language models were employed for retrieval and discovery tasks, particularly in identifying and organizing relevant related work and ensuring comprehensive coverage of the literature.
All technical contributions, experimental design, implementation, analysis, and conclusions presented in this work are the original work of the authors. The use of LLMs was limited to editorial assistance and literature search support, and did not influence the core scientific contributions or findings reported in this paper.
H ACKNOWLEDGEMENS
This work partly supported by Institute of Information communications Technology Plan- ning Evaluation (IIP)) grant funded by the Korea government (MSIT) (No.RS-2019) 211919777 Artificial Intelligence Graduate School Program (KAIST), 10%; No.RS-2021-1121200 2068,, Artificial Intelligence Innovation Hub, 10%; RS-2024-00398 15, Research on the reliability and coherence of outcomes produced by Generative AI, 20%; No. 1o.2022-0-001 13, Developing Sustainable Collab- orative Multi-modal Lifelong Learning Framework, 20%; RS--2222-11222266 Comprehensive Video Understanding and Generation with Knowledge-basee Deep Logic Neural Network, 20%; Development of Cybersecurity Specialized RAG-based sLLM Model for Suppressing Gen-AI Malfunctions and Construction of Publicly Demonstration Platform) and the InnoCORE program of the Ministry of Science and ICT(N10250155).
25


Reviews:
[{"review_id": "b0AlnmyloG", "reviewer": "Reviewer_cbMw", "summary": "This paper introduces Tree-Structured Language Modeling (TSLM), enabling language models to perform tree-structured reasoning natively within a single forward generation process.\nThe thought branching structures are shown into the token sequence through special markers.\nSerialized tree traces that capture both successful and failed reasoning paths are used to train standard transformers.\nExperiments are conducted on structured planning tasks and open-ended reasoning tasks.", "questions": "What is the time cost of the proposed method?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Tree structured reasoning is reflected into token sequences and only one model call is required. This is different from multiple independent model calls used by Tree-of-Thought.\n2. The token-based serialization approach is practical for achieving tree-structured generation using LLMs.\n3. The experiments span both structured and open-ended reasoning tasks.", "weaknesses": "1. The time cost comparison with methods such as Tree-of-Thought is not presented.\n2. As shown in Table 1, the performance of TSLM is not as good as ToT in GSM8K.\n3. The token-based tree serialization relies on search trees gotten from other reasoning methods. As such, the performance is also limited by the other reasoning methods.\n4. There are some more advanced reasoning methods such as Graph-of-Thought and Everything of Thoughts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vCnEvjOm14", "reviewer": "Reviewer_fgNv", "summary": "LLMs can generate sequential solutions but are unable to systematically explore multiple possibilities. The paper introduce a framework to serialize tree exploration into linear sequences for training to enable tree-structured reasoning. The method achieves better performance than baselines on most tasks.", "questions": "Does the training data for the TSLM method include the 5 tasks that are also used for evaluation?\n\nCan you also try letting baseline methods train on the same training set, by extracting correct path from the tree. Basically, TSLM receives additional training signal and eliminating that factor will make the experiment results more solid.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is generally clearly written.\n\nThe proposed tree-structured language modeling paradigm is interesting.", "weaknesses": "The method is helpful on tasks where systematic exploration is needed, but I think more open-ended and real-world tasks are more important. As indicated in Table 1, the proposed method has no benefit in these scenarios. The bolded number in the GSM8k row should be the ToT one, by the way.\n\nThe experiment detailed configuration for the ToT baseline is not mentioned. Since the proposed method requires much higher inference-time compute, it is unclear how much the advantage will diminish\tif you let all methods use the same budget (say, the total tokens).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o6TFf2fZEo", "reviewer": "Reviewer_BGCK", "summary": "The paper proposes TSLM, a training approach that teaches an LLM to internally construct and traverse a tree of reasoning within a single generation, rather than relying on external multi-pass search at inference time. The authors position TSLM as more efficient and easier to train and deploy than external search pipelines, and as a simpler alternative to RL approaches such as GRPO for improving multi-step reasoning and planning. Empirically the paper claims that TSLM improves extrapolation on structured reasoning tasks and helps smaller models approach the performance of larger ones, while also benefiting large models.\n\nThe paper situates itself against external search methods, and also cites multi-path generation like self-consistency and recent long-reasoning models. The narrative is consistent with the goal of moving reasoning structure into the model rather than the inference loop. However, there is a relevant gap regarding diversity-driven reasoning frameworks that target breadth before convergence and explicitly reduce redundant reflections with task-agnostic memory.", "questions": "(1) Under a fixed budget of model calls and total tokens, how does TSLM compare to ToT or self-consistency, and to a diversity-driven method with a task-agnostic memory bank such as DoT? \n\n(2) What happens if you linearize the tree to a CoT trace with the same token budget, or if you randomize branching order?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "TSLM aims at deployment simplicity. Eliminating external orchestration and multiple model calls is attractive for production where latency and engineering complexity matter. The authors take care to compare conceptually with ToT and with RL, explaining why a supervised approach can be more stable and economical. The stated effect that TSLM helps smaller models close the gap with larger ones is promising for cost-sensitive settings. \n\nThe conceptual boundary is easy to communicate. External search explores a tree outside the model, TSLM learns to write the tree inside the generation. That is a clean story, and it could generalize to code or games where branching structure is natural.", "weaknesses": "Related work coverage is incomplete on diversity-first inference and redundancy reduction with memory, which is a very proximate thread to the authors’ efficiency and breadth claims. The current submission neither cites nor contrasts with these works (e.g., Lingam et al., ICLR 2025), which could mislead readers about the frontier on exploration efficiency. This needs correction. \n\nEvidence granularity is thin in the visible draft. The claims about improved extrapolation, small-to-large bridging, and efficiency would be more convincing with matched-budget comparisons against ToT, self-consistency, and diversity-driven methods, for example controlling node expansions or total tokens, plus wall-clock latency. The draft does not show whether TSLM outperforms a well tuned inference-time breadth strategy at the same compute.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HJYEfiahyA", "reviewer": "Reviewer_Aewk", "summary": "The paper proposes Tree-Structured Language Modeling (TSLM), a simple but distinctive way to let a standard transformer natively generate and traverse a search tree within a single sequence. The core idea is to serialize a tree into a linear token stream using special markers, so that teacher traces include both successful and failed branches. Training is standard LM loss on these serialized traces; inference reconstructs the tree and explores it until a solution or exhaustion. The paper reports results across Game of 24, Textualized Gridworld, ProntoQA, and GSM8K. On structured tasks, TSLM is very strong, shows rapid adaptation with few samples on ProntoQA, and claims parameter efficiency. For open-ended math (GSM8K), TSLM is competitive. Overall, TSLM is positioned as a purely supervised alternative to RL-style “reasoning” methods that require external search or test-time orchestration.", "questions": "Q1. What were the exact ToT settings (beam width, evaluation heuristic, depth/budget) per task? Did you evaluate Self-Consistency (sample-and-vote) or GSM8K verifier selection? If so, how do they compare under equal token budgets?\n\nQ2: The reported scaling performance (“68% for 0.5B vs 19–26% for 7B baselines”) requires clarification: What is the exact metric and task (Gridworld 20×20 only?), and are these single-seed results? Please provide confidence intervals.\n\nQ3: The paper provides a qualitative example. Across a large set, how often does TSLM correctly abstain on unsolvable Game of 24 instances?\n\nQ4: How well does TSLM handle branches in open-ended problem domains, where the branching structure is unclear or multiple valid solution paths exist without a single “correct” branch?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The token-based tree serialization for end-to-end supervised training of internal exploration is a neat, low-friction idea. It avoids extra modules (verifiers or value functions) and RL, and while LM-based search is well-studied, representing divergent branches natively within a single generation is an original framing with distinct trade-offs.\n\nS2: Strong performance on Game of 24 and Gridworld demonstrates the method’s core promise. The BFS vs. DFS analysis in Appendix G provides insight into how traversal strategies interact with learned preferences.\n\nS3. No architectural changes are required; the approach works with standard SFT and any base LM (e.g., Llama-3, Qwen). Training and inference procedures are clear, with many worked examples provided, supporting reproducibility.", "weaknesses": "W1: ToT is configured minimally, and comparisons omit Self-Consistency and verifier-based selection, which are strong baselines on GSM8K, as well as natural tree-search comparators like TS-LLM, LATS, and RAP. Without these, it is difficult to judge when TSLM’s internalized tree search provides meaningful benefits.\n\n\nW2: TSLM trains over all tree nodes O(N⋅L)), and inference produces long serialized trees. Claims of efficiency are not quantitatively compared against ToT’s extra model calls or SC/self-consistency multi-sample decoding.\n\nW3: The tasks are well-suited to structured search-tree reasoning but do not represent open-ended domains such as large-scale software engineering, real-world system design, multi-agent planning, or long-horizon code generation. Claims that strong performance on structured tasks will generalize to these domains remain untested and speculative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)