DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search
Authors: Murong Yue
Wenlin Yao
Haitao Mi
Dian Yu
Ziyu Yao
Dong Yu
Venue: ICLR 2025 Poster (2025)
Keywords: large language model
reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a method to dynamically select an optimal trajectory of reasoning actions tailored to the specific characteristics of questions and LLMs.

Abstract: Enhancing the capability of large language models (LLMs) in reasoning has gained significant attention in recent years. Previous studies have demonstrated the effectiveness of various prompting strategies in aiding LLMs in reasoning (called "reasoning actions"), such as step-by-step thinking, reflecting before answering, solving with programs, and their combinations. However, these approaches often applied static, predefined reasoning actions uniformly to all questions, without considering the specific characteristics of each question or the capability of the task-solving LLM. In this paper, we propose DOTS, an approach enabling LLMs to reason Dynamically via Optimal reasoning Trajectories Search, tailored to the specific characteristics of each question and the inherent capability of the task-solving LLM. 
Our approach involves three key steps: i) defining atomic reasoning action modules that can be composed into various reasoning action trajectories; ii) searching for the optimal action trajectory for each training question through iterative exploration and evaluation for the specific task-solving LLM; and iii) using the collected optimal trajectories to train an LLM to plan for the reasoning trajectories of unseen questions. In particular, we propose two learning paradigms, i.e., fine-tuning an external LLM as a planner to guide the task-solving LLM, or directly fine-tuning the task-solving LLM with an internalized capability for reasoning actions planning. Our experiments across eight reasoning tasks show that our method consistently outperforms static reasoning techniques and the vanilla instruction tuning approach. Further analysis reveals that our method enables LLMs to adjust their computation based on problem complexity, allocating deeper thinking and reasoning to harder problems.

Published as a conference paper at ICLR 2025
DOTS: LEARNING TO REASON DYNAMICALLY IN LLMS VIA OPTIMAL REASONING TRAJECTORIES
SEARCH
Murong Yue Wenlin Y
Haitao Mi
Dian Yu
Ziyu Yao
Dong Yu
"Gorgg Mason University Tencent AI Lab, Bellevue
{myue, ziyuyao (gm... [wenlinyao, ha taomi,
yudiaaa yudian, dyu]eglobal tencent.
ABSTRACT
Enhancing the capability of large language models (LLMS)) in reasoning has gained significant attention in recent years. Previous studies have demonstrated the effectiveness of various prompting strategies in aiding LLMs in reasoning (called "reasoning actions"), such as step-by-step thinking, reflecting before an- swering, solving with programs, and their combinations. However, these ap- proaches often applied static, predefined reasoning actions uniformly to all ques- tions, without considering the specific characteristics of each question the pability of the task-solving LLM. In this paper, we propose DOTS, an approach enabling LLMs reason Dynamically via Optimal reasoning Trajectories Search, tailored to the specific characteristics of each question and the inherent capability of the task-solving LLM. Our approach involves three key steps: i) defining atomic reasoning action modules that can be composed into various reasoning action tra- jectories; ii) searching for the optimal action trajectory for each training question through iterative exploration and evaluation for the specific task-solving LLM; and iii) using the collected optimal trajectories to train an LLM to plan for the reasoning trajectories of unseen questions. In particular, we propose two learning paradigms, i.e., fine-tuning an external LLM as a planner to guide the task-solving LLM, or directly fine-tuni sanittunng the task-solving LLM with an internalized capabil- ity for reasoning actions planning. Our experiments across eight reasoning tasks show that our method consistently outperforms static reasoning techniques and the vanilla instruction tuning approach. Further analysis reveals that our method enables LLMs to adjust their computation based on problem complexity, allocat- ing deeper thinking and reasoning to harder problems. Our code is available at attps://qithis
com wwwp://wwwwwoogoo/eoooooooo
INTRODUCTION
Large Language Models (LLMMs) have demonstrated remarkable performance in solving complex reasoning tasks (Rae et al. |202 Lewkowycz et al. 2022 Zhong et al.) (023), such math reason- ing (Imani et al. 2023 Ahn et al. 2024)) symbolic reasoning (Kojima et al. 2022), and common- sense reasoning (Krause & Stolzenburg 2023 Zhao al., 2024). The dominant approaches to eliciting reasoning capability in LLMs mainly fall into two categories, i.e., instruction tuning and prompt engineering. Instruction tuning (Wang et al. 2022) collects question-answer pairs about the reasoning task and employs supervised fine-tuning to optimize an LLM for better reasoning perfor- mance (Yue al., 2024 Tang et al. (024)), with recent effort focusing on improving the scale and the quality of the fine- tunin data (Luo et al. 2023 Peng et al. [023 Yue et al. 2023 2024] Chan et al., 2024). Prompt engineering instead aims design better prompts elicit the reasoning capability of an LLM without updating its parameters. The Chain-of-1 -hoouhhhh (CoT) approach (Wei et al. 2022
Kojima et al.) (222)) prompts an LLM to answer the reasoning question step by step natural lan- guage, and program-aide aided approaches (Chen et al. [202 Gao al.) (023)) prompt the LLM to write executable code and leverage an interpreter to execute code for obtaining the final result. Besides,
Work done during Murong Yuc's intersship at Tencent AT Lah.

---

Published as a conference paper at ICLR 2025
LLM, Tunable
(a) Vanilla Instruction Tuning'SFT
(d) Internalized Planner
Tuning (Dors)
Query
Query
Dynamic Reasoning Actions
Query
Query
Static Reasoning Actions
LLM
LLM
Query 1
Query 2
Rowriting Decomposition
EMPTY
Reasoning Trajectory Explanation) Reasoning
Reasoning Process
Fixed Prompt
Answer
Process Answer
CoT
PoT
(b) Prompt Engineering
External Planner
Tuning (DOTs)
Result X
Result
Query Reasoning
Self-Verification
EMPTY
Format
Query
Query
LLM
LLM
Result
Result
LLM
Roasoning Trajectory (w/ Expl.)
Reasoning Process Answer
Reasoning
Process Answer
LLM Frozen
Figure 1: A comparison of different paradigms of LLM reasoning. Unlike prior approaches with predefined, static reasoning actions, DOTS dynamically plans for the optimal reasoning trajectory per each question and the specific task-solving LLM (LLM,). In particular, DOTS encompasses two inference setups, i.c., external planner tuning (c) and internalized planner tuning (d), depending on whether to introduce an external LLM planner (LLMp) or to internalize the trajectory planning capability into the same solver LLM (LLM,).
@ tunable; @ frozen)
prompting the LLM to decompose the question before answering it (Radhakrishnan al., 2023 Zhou et al. (202),, or to verify the solution before returning it as the final answer (Madaan et al.
2024). has also been proven effective in specific reasoning tasks.
However, both types of approaches suffer from a critical limitation, i.e., being unable to dynamically decide the best reasoning strategies.
Instruction-tunge LLMs are constrained by the reasoning formats in their training data (e.g., CoT (Luo et al., )023)), ) as illustrated by GPT-4o case in Appendix B FireAct (Chen et al.) (023a) shows that fine-tuned LLMs can switch between reasoning strategies like CoT and ReAct, but our experiments reveal that training with varied trajectories alone is insufficient for optimal results.
On the other hand, current prompt engineering approaches assume predefined prompting strategies and uniformly apply the same to every question. However, different types of questions are better suited to different reasoning strategies (Zhao et al., 2023), and the effectiveness of a prompting approach also depends on the inherent capability of the task-solving LLM (e.g.. LLMs pre-trained on code data are better at programming-atin -aided reasoning). Consequently, the same prompt may not be equally effective for every question and every LLM.
In this paper, present DOTS, approach empowering LLMs to actively select optimal reason- ing actions for given questions and the task-solving LLM (Figure []. We begin by constructing atomic reasoning action modules, which are composed generate multiple potential reasoning ac- tion trajectories. Then we collect the training data by searching for an optimal (in terms of both its success rate and the number of reasoning actions needed) action trajectory through numerous explorations and evaluations. This optimal trajectory is tailored to the specific task-solving LLM. Subsequently, we employ supervised fine-tuning to train an LLM in determining the optimal rea- soning action trajectory. We implement this approach in two distinct setups: (1) For closed-source or computationally costly task-solving I.Mss, we fine-tune smaller L.LM as an external planner to predict optimal reasoning actions for the task-solving LLM; (2) For open-source and small-size LLMs, we fine-tune the task-solving LLM itself to plan on the reasoning actions to take before solv- ing the reasoning task, internalizing the autonomous planning capability directly into the LLM. This dual approach allows for flexible application across different LLM accessibility constraints.
Our experimental results demonstrate the efficacy of our proposed method in enhancing the reason- ing capabilities of LLMs. We conducted extensive evaluations across multiple LLMs (GPT-4o-minl) Llamma370B- Ins-turt and Llama3-88 Bamma-88-innstnue 8B-instruct (Dubey et al. 2024g) and diverse of reasoning
2

---

Published as a conference paper at ICLR 2025
tasks, encompassing in-distribution, few-shot, and out-of-distribution scenarios. The results reveal that DoTs consistently outperforms static prompt engineering techniques and vanilla instruction tuning methods across various reasoning challenges. Through comprchensive ablation study, validate the significance of each component in our methodology. Moreover, our analysis of rea- soning action distributions highlights that method can adapt to the specific characteristics of reasoning questions and the inherent capability of task-solving LLMs. We further confirm that our method incurs minimal additional financial costs. Lastly, showcase that LLMs can naturally de- velop the capacity to allocate more computational resources to complex problems through process of exploration and learning, without explicit guidance.
2 DOTS: LEARNING TO REASON DYNAMICALLY
2.1 OVERVIEW
Our goal is to enable L.MSs to select the most effective reasoning actions autonomously. Denote L.M,, the task-solving LLM,, Q the input query, p as the reasoning action trajectory path, E as the explanation for trajectory, and as the reasoning process leading to the final answer y. Our approach encompasses two setups during the inference stage (Figure D
External Planner Tuning This setup is designed for scenarios where the solver (I.LM,) is closed-source LLM or is computationally costly to train. As depicted in Figure (c), we train an external planner, denoted as LLMp, to determine the optimal reasoning actions:
(I) where 8p is the parameters of LLMp. We empirically found that training the planner to explain its trajectory selection (E) helps its learning. Upon obtaining reasoning actions, the solver LLM, parameterized by 0, then proceeds to generate the reasoning process R and the final answer
(E,p) LLMp(QQ:pp
(R, y) = LLM, p: 0x)
Internalized Planner Tuning This setup is designed for solving LLMs (LLM)) that are open-source and small-size. In this case, we propose internalize the trajectory planning capability into the task-solving LLM by training to simultaneously learn to plan and learn to perform the reasoning task. As shown in Figure (d), the final answer is obtained by:
(E,p, R, = LQ:,(Q:0.)
overview DOTS's learning process presented in Figure consisting of three key steps: (i) Defining atomic reasoning modules: We define several atomic reasoning modules, each represent- ing a distinct reasoning action, (ii) Searching for optimal action trajectories: We conduct explo- rations and evaluation of various reasoning paths to identify optimal reasoning actions for questions in the training data, and (iii) Fine-tuning LLMs to plan for optimal reasoning trajectories: We fine-tune LLMs to autonomously plan the reasoning action trajectory under the two aforementioned setups. In what follows, we elaborate on each step.
Table 1: Prompt engineering methods with different reasoning actions. Our method could dynami- cally select reasoning actions among all of them.
Analysis Layer
Solution Layer
Verification Layer Verification
Prompting Method
Rewriting Decomposition
NL Program
CoT
Wei et al.
2022
X
PoT
Chen et 2022
iZhou al.
R&R Deng et
2023
Self- MMidann et
2024
X
X
/
Self- Verification Weng ct
2022
X
PromptAgent (Wang et al. 2025
Dors (ours)
2.2 DEFINING ATOMIC REASONING ACTIONS MODULES
Prior studies have validated the effectiveness of various reasoning strategies (Table D. We build on top of them and categorize the existing strategies as reasoning actions across three layers:
Analysis Layer Actions in this layer enable the LLM to analyze the input query before attempting to solve it, including (1) Query rewritings reformulating the query to enhance comprehen- sion, and (2) Query decomposi i breaking down the initial question into multiple, more manageable sub-questions. We denote the action taken in this layer as A.-
3

---

Published as a conference paper at ICLR 2025
Searching Optimal Reasoning Actions Trajectory (given LLMg Query: How many integers between and 200 are multiples both and but not of either or 7?
Rewriting
Decomposition
EMPTY
Rewriting
Decomposition
EMPTY
Analysis Layer
CoT
Solution Layer
Sclf-Verification
EMPTY
Selt-Verfficttion
EMPTY
Verification Layer
Constructed Training Data
X
Optimal Reasoning Trajectory. Pol Scr- Vertication
Reasoning
write Python codes
question: "pytoon
Let's verily the answer by formulsting a verlication queelion...
EExplaaation: The problom can be oitidenly savadd using programming to liarate thouugh the range.. The answer easy to verty. Verfication ensures that the programmatic Esolution aligns with problem's logic conditins...
Use GPT-4o explain why
trajectory is optimal for this question
Query
Query
Fine-Tuning the Planner LLM
LLMp
LLM,
Explanation, Trajectory
Explanation, Trajectory,
Reasoning Process, Answer
(a) External Planner Tuning
(b) Internalized Planner Tuning
The training process of DOTS, including searching for the optimal reasoning trajectories for questions in the training set and fine-tuning the internalizeddexttrmaall planner LLM.
Figure 2:
Solution Layer Actions in this layer consider variants in the reasoning format. Prior works showed that different queries are better solved following different reasoning processes (Zhao al., 2023). In our work, we consider the most commonly adopted formats, i.c., (1) CoT: solving the question step-by-step in natural language, and (2) PoT: addressing the question through code gen- eration. We denote the action chosen in this layer as A,,
Verification Layer Finally, the verification layer is responsible for checking the correctness of the proposed solution. It is particularly useful for problems where verification is significantly eas- set
ier than solving the problem itself, e.g., the Game of 24 (Yao et al., 2024). Therefore, S-lf-Veri f cat on action module in this layer. If this module determines that the reasoning process from the solution layer is incorrect, the LLM will revert the solution layer to reattempt to solve the problem. During this reattempt, the LLM is provided with both the initial answer and the feedback from the verifier explaining why the initial answer was incorrect. The process continues until the verifier confirms that the answer is correct or the pre-defined maximum number of iterations for self- serfiicciiia is reached. We denote the action taken in this layer as Ag.
We observe that introducing excessive reasoning actions can lead to increased latency, and even sometimes result in incorrect answers. To mitigate this issue, we introduce an Empty action in both the analysis and the verification layers, allowing the LLM to bypass these two layers when dealing with simple questions. Detailed prompts for each module are provided in Appendix D
2.3 SEARCHING FOR OPTIMAL REASONING ACTION TRAJECTORIES
To teach the external/internalized planner to plan for the optimal reasoning trajectory, we start by constructing training data containing questions and their optimal action trajectories for the specific task-solving I.M.. We obtain this by iteratively searching all possible reasoning trajectories for each question, including exploring the current paths and pruning paths that are unlikely to be optimal. The task-solving LLM is used during this search process generate answers to make the reasoning trajectory align with their intrinsic ability to perform different reasoning actions effectively.
This searching process is shown in Algorithm Given the query and ground-truth answer sourced from the training data, the process runs iteratively. In each iteration, the algorithm considers either the full set of candidate trajectories (for iteration & 1) the current best subset (for iteration > 1). Each candidate trajectory is executed for Nevat times with a non zero temperature to obtain more reliable evaluation of its success rate. We then sort the current subset of trajectories by its
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Searching for the Optimal Reasoning Action Trajectory
Require: Input query Q and ground true answer y", solver L.LM,, max iteration K, number of evaluations Nevat, and number of candidate trajectories to retain in each iteration N1, N2..... Nx-
Ensure: Optimal action trajectory path p" for query Q and solver L.M.a.
1: Initialized candidate trajectory set P <- {(A.. Empty- A, CoT, A. = Empty),
2: Initialize the record of the accumulated success rate of each candidate trajectory: R {p: 0p:00ppP:: pe
1 K 4: for all p E P de
3: for iteration
5: Execute the trajectory against Q and LLM, for Neval times with non-zcro temperature and
obtain average success rate rp (compared to the true answer y');
6:
Update the accumulated success rate of p:: RRpp) + Rlpl- (k-1) (1--kk-----a--
Nevattry Nenst;
Nevai
7:
end for
Sort P first by the accumulated success rate R(p] and then the trajectory length |p| in ascending order; 9: Reset 4- top N₂ trajectories in P.
8:
end for Returnp" c arg m Rlp].
11:
success rate accumulated from the past k iterations and then the trajectory length encourage shorter trajectory (which is thus computationally more efficient). Only the top .ix candidates will be retained and rolled over to the next iteration of the assessment. In practice, we opt for smaller Nevat and run the search for multiple iterations, as opposed to finishing the search with a larger Neval in one iteration, as the latter incurs much larger cost (Nevel X (Pol with a large Neval vs. Nevat x (|Pol + N1 + ....... NK-1) with a small Newnt in our algorithm).
In the process of validating various trajectories for each question, we exclude instances where any trajectory solves the query all fail to do so, as they do not contribute to the planner L.M''s trajectory planning learning. After identifying the best reasoning trajectory, we leverage GPT-4o to verbally explain why the trajectory is optimal. Our prompt is shown in Appendix [ This process applied to all instances in the training data, giving us tuples of query Q, ground true answer v" optimal trajectory p', and its explanation E. For internalized planner tuning, we collect the reasoning process R when running the solver LLM, following the optimal trajectory p".
2.4 LEARNING TO PLAN FOR OPTIMAL REASONING TRAJECTORIES
Having obtained the optimal trajectories, we then use supervised fine-tuning with cross-entropy loss to train the planner LLM to predict optimal trajectories for input questions and the specific solver LLM. For external planner tuning. a lightweight LLM, is trained to predict a concatenation of the explanation and the optimal trajectory (Eq []: for internalized planner tuning, the solver LLM, is trained to predict the explanation, the optimal trajectory, the reasoning process collected from LLM,, itself, and the true answer y" (F)
3 EXPERIMENT
3.1 EXPERIMENTAL SETUP
Datasets We evaluate the effectiveness of method across multiple datasets and various reason- ing tasks. Based on the distribution of the training and testing data, we divide the evaluation into three settings as shown in Table In-distribution setting evaluates the model that resembles what it has seen during training. Few-shot setting aims to evaluate whether our proposed method can effectively learn from a small amount labeled data. In the real world, it is often difficult to obtain large amounts of in-domain training data across different tasks, but a small number of cases can be annotated. Out-of-distribution (OOD) setting further evaluates whether the model can handle scenarios it was not explicitly trained for, testing its ability to gen- eralize beyond the training set. For the training data, we use the MATH (Hendrycks et al. 202T) training set. For the few-shot learning, we select 4 examples from each category of BBH (Suzuun
Table 2: Overview of our evaluation datasets.
Dataset MATH
Distribution and Size In Distributio,, Many shot
Task Type
math mixture numerical scientific
Game of 24
In Distribution, Few shot
TheoremQA
Deepmiind Math
math
MML.U-pr
scientific
Out-of-Distributtion
StralegyQA
common sense multi-hop
DROP
5

---

Published as a conference paper at ICLR 2025
Table 3: Accuracy (%) of the external planner tuning in-distribution and few-shot datasets. The reasoning format & represents language, and P means program.
Reasoning MATH BBH Game 24 TheoremQA
Method
Tuning
Average
Format
External Planner: Llama 1aama-3-8BB 3-8B-Instruch. Solver:
Alama-3-70B-Insttee
CoT
50.4
72.7
27.5
27.4
44.5
LTM
x
L
50.1
73.8
24,9
28.8
44.4
PA
/
L
52.5
72.9
26.8
28.8
45.3
PoT
P 54.7
65.8
63.9
31.1
53.9
Self-refine
C.
55.9
71.4
30.8
56.6
DOTS: External
C.
P
57.7
77.3
67.7
31.2
58.5
External Planner: Llama-3-mm 3-----------me Innttruttt; Solver: GPT Go-4ommini
CoT
C
70.2
80.3
27.7
38.9
54.2
LTM
72.2
79.4
25.5
36.4
53.3
PA
L
73.5
81.1
26.7
38.9
55.1
PoT
X
P
67.2
73.9
61.4
35.8
59.6
Self-refine
C. P
73.7
74.8
68.7
34.6
63.0
DOTS: External
C. P.P.
75.4
84.2
65.2
41.4
66.5
al., 2022) as composed of 27 diverse tasks resulting 108 examples in total, examples
from Game of 24 (ao et 2024), and 4 examples from TheoremQA (Chen et al. 2023b) datasets. the MATH dataset for the in-distribution
For the test data, we evaluate the model on the test setting and on the test sets or hold-out sets of BBH, Game of 24, and TheoremQA for the few-shot learning setting. For the OOD evaluation, we test each approach's 's generalization ability on Deep- mind Math (Saxton et al. 2019). MMLU-pro (Wang et al. 2024), strategyQA (Geva et al. 2021 and DROP (Dua et al.) 2019). All evaluations (unless specified) were conducted when prompting the solver LLMs in zero shot. For answer evaluation, we use the simple-eva P for MATH, standard evaluation for Game of 24 (Yao et al. 2024). and exact string matching for the others.
Training Setup For external planner tuning, we utilize Llama-3-8B -----nstturtt our planner and GPT-4o-mini and Llama-700 Saamma-70B-Insee 70B-Instruct as task-solving LLMs. Experiments of internalized planner tuning were conducted with -amaa------nnttuue For more details, refer to Appendix A
3.2 BASELINES
We include the following highly related baselines in our experiments. (1) CoT (Wei et al. 2022) prompts an LLM to answer step-by-step; (2) PoT (Chen al. 2022) prompts an LLM to generate Python code and execute the code to get the final answer; (3) Least-to-most (LTM) (Zhou et al.)
2023) prompts an LLM to first decompose the question into multiple sub-questions before solving it; (4) Self-refine (Madaan et al. 2024) prompts an LLM to generate the answer and verify and refine answer by the LLMM itself. Madaan et al. (0024) used PoT solving math questions, therefore we follow their setting to use PoT in generating the initial answer; (5) PromptAgent (PA) (Wang al. 2023) searches for better prompt for the specific task based on its training data; this baseline is im- plemented with the default hyperparameter setting; and (6) Vanilla Supervised Fine-Tuunin (Vanilla SFT) uses GPT-4o to generate the CoT reasoning process for questions in the training datasets and then fine-tune the solver LLM to predict the generated reasoning process and the ground-truth an- swer; this baseline is fine-tuned using the same hyperparameter setting as our internalized planner tuning. The training data for PA, Vanilla SFT, and DOTS are from the same source.
3.3 EXTERNAL PLANNER TUNING RESULTS
Table 3 presents the results of using the external planner, which suggest that:
External planner tuning outperforms other methods on the in-domain task Our method achieves 57.7% accuracy with Llama-3-70b-Instrue and 75.4% accuracy with GPT-4o-mini on MATH, achieving significant improvement than baselines. This suggests that DoTS is robust across different LLMs and it can significantly enhance the LLM''s s zero-shot reasoning ability. The improve- ment from DOTS remains consistent as the solver LLM's capabilities increase, indicating DOTs has long-term value even as LLMs continue to improve rapidly.
The external planner can learn the appropriate action trajectory with only a few training examples. On the BBH, DOTS achieves improvements of 3.5% and 3.1 % over the best static
outga///oocccccclaaaaaaaatttttbn https://githhbe
evai

---

Published as a conference paper at ICLR 2025
methods when using Llama-3-70B- Ins-rnstt and GPT-4o-ming respectively. In the Game of 24 and TheoremQA, DOTS also shows slight improvements or performs similarly to the best static method. This indicates that even small number of cases can help the LLM learn the optimal strategy for the given task. Besides, DOTS demonstrates greater stability across various datasets. Our flexible action trajectory selection demonstrates its advantages datasets requiring diverse reasoning actions, such as BBH as shown in Appendix [ Conversely, the Game of 24 features a uniform question type, where the predefined static method self-refine is sufficient. While the self-refine excels on Game of 24, it significantly lags behind on other datasets. This reflects the external planner's 's ability to effectively select the appropriate action trajectory, leading to more robust performance even across tasks with varying reasoning demands.
3.4 INTERNALIZED PLANNER TUNING RESULTS
Table presents the results our internalized planner tuning, where we observed:
Internalized planner tuning demonstrates superior performance DOTS outperforms existing methods on average, including prompt engineering methods and vanilla SFT. Notably, our approach surpasses self-refine in the Game of 24, different observation than the experiments with exter- nal planner (Table 3). We attribute this performance boost to our joint optimization of the trajectory planning and problem-solving processes. Unlike external planner tuning which only updates the external planner (LLM,), internalized planner tuning enables the task-solving LLM to simultane- ously learn trajectory planning and accurate reasoning process generation. This highlights that the internalized planner tuning effectively further enhances performance.
Searching for the optimal reasoning action trajectory helps enhance the utilization of training data Compared to vanilla SFT, our method consistently shows performance improvements across all datasets, notably achieving an 8.7% increase on BBH. This suggests that, instead of training with a question and step-by-step reasoning process pair, our approach of searching for an optimal action trajectory and generating the corresponding reasoning process construct training data is superior. This finding indicates that our search methodology could effectively enhance the utilization of training data for reasoning tasks without the need for additional human annotations.
Table 4: Internal planner tuning performance on in-distribution and few-shot datasets.
Reasoning format
Method
Tuning
MATH BBH Game of 24
TheoremQA
Average
Solver:
CoT
X
E
29.6
48.9
12.7
14.8
26.3
LTM
x
C
29.5
50.3
14.4
15.2
27.4
PA
C
31.0
47.2
11.8
15.1
26.3
PoT
P
25.3
44.6
16.8
16.7
25.9
Self-refine
X
C.
P
28.7
46.6
17.0
15.3
30.1
Vanilla SFT
C
33.9
61.0
18.5
14.8
33,6
DOTS: Internalized
C.
P
34.4
69.7
21.9
16.1
35,5
3.5 DUTTOR--DSTRIBBUTION EXPERIMENTAL RESULTS
Our method consistently generalizes well across diverse OOD challenges As shown in Table DOTS maintains high accuracy across different datasets and models. In contrast, static methods of- ten fluctuate significantly in performance. For instance, despite static methods like CoT showing slight advantage on MML.-Pro and StrategyQA over DOTS using the Llama-3-70B- 3-70B-Inntruu model, they experience sharp decline on DeepMind Math. This pattern of fluctuations can be observed in other methods as well, where some excel on individual tasks but fail to maintain strong perfor- mance. In contrast, DOTS continues deliver consistently high accuracy across various models and datasets. The stability of our method is attributed to its ability to dynamically select appropriate reasoning trajectories. The results indicate that DOTs is better suited to meet the demands of diverse tasks, demonstrating stronger robustness and generalization, making a more reliable and adaptable approach for handling wide variety of OOD challenges.
3.6 ABLATION STUDY
In this section, we perform the ablation study and assess the effectiveness of each component of method: (1) Without Searching: To demonstrate the effectiveness of searching for the optimal action trajectory, we test the performance of the LLM tuned with a randomly selected action trajec- tory; (2) Without Explanation: To understand if training the planner generate explanation for

---

Published as a conference paper at ICLR 2025
Table 5: Accuracy (%) out-of-distribution (OOD) tasks.
Method
DeepMind-Math MMLU-pro StrategyQA DROP
Average
External Planner: Finefuned Llama-3-8B- Solver: Llama-3-705 700tst-r
CoT
54.6
60.6
81.3
66.1
65.66
LTM
55.6
60.9
81.9
64.3
65.6
PA
58.1
54.2
80.3
58.7
62.8
PoT
73.0
57.3
74.8
62.8
66.9
Self-refine
73.9
59.5
77.8
64.8
69.0
DOTS: External
74.1
59.4
80.3
66.3
70.0
External Planner: Finetuned -lama-3-8B--Istruccl Solver: GPT4o-mini
CoT
80.2
61.7
78.8
65.8
71.6
LTM
61.4
80.9
64.5
PA
82.2
48.1
78.3
67.0
68.9
PoT
87.7
57.1
77.9
72.4
73.7
Self-refine
85.9
58.3
77.2
72.3
73.4
DOTS: External
87.6
61.5
78.8
73.8
75.4
Solver: Finetuned Llama-3-aa I-st-Insttuut
CoT
28.3
37.2
72.7
52.9
47.8
LTM
30.9
38.6
70.7
55.2
48.9
PA
29.3
34.5
69.7
51.6
46.3
PoT
48.1
37.3
63.9
44.6
Self-refine
44.9
33.1
65.3
47.1
47.6
Vanilla SFT
39.6
40.3
71.8
49.0
50.2
DOTS: Internalized
55.3
39.7
68.2
48.8
53.0
Table 6: Ablation Study
MATH
BBH
Game24
TheoremQA GPT-4o-mini
Average
External Planner: 1amma-3-8-IIntruune Solver:
DOTS: External
75.4
84.2
65.2
41.4
66.5
-wo Searching 69.2
78.6
28.9
40.2
$4.2
-wo Explanation
68.2
81.3
57.4
36.4
60.8
Internalized Planner & Solver:
3-8B-Instruct
DOTS: Internalized
34.4
69.7
21.9
16.1
35.5
-woo Searching
31.4
55.8
19.6
15.1
30.5
-w/o Explanation 33.8
65.8
18.6
15.7
33.4
the optimal reasoning trajectory is helpful, we test DOTs's performance when the planner trained to predict the trajectory without explanation.
The results Table 6 indicate that both optimal trajectory searching and explanation generation crucial in DOTS. For example, in the Game of 24, the planner trained without searching for the optimal trajectory did not consistently select the PoT action (which was considered the most effective for this task) in its trajectory. Additionally, planner's ability to predict optimal trajectories becomes less reliable. Incorporating explanations effectively guides the planner learn to predict suitable action trajectories for the given questions.
observe that without explanations, the
3.7 OPTIMAL TRAJECTORY ANALYSIS FOR DIFFERENT TASKS
Table shows the distribution of actions selected in the optimal trajectories by our planner on the MATH test set. The distribution suggests two key findings:
DOTS adapts to the characteristics of specific questions In mathematics, number theory prob- lems
suitable be solved with programs, the proportion of PoT is higher, while geom- etry problems are not easily represented and solved with naive Python code; as result, our planner mainly uses CoT for such problems. This indicates that DOTS tailors its action selection based on the unique characteristics of each problem type.
DOTS adapts the capability of specific task-solving LLMs As shown in Table 3 on the MATH dataset, GPT-4o-mini performs better using CoT for problem-solving, whereas Llama3- 70B-in 70B-instruct performs better using PoT. When GPT-40-minn is the task-solving LLM, our fine-tuned planner selects a higher proportion of CoT actions; when Llama3-70B 7amma3-7B--nstruct is used, PoT actions dominate. This suggests that our planner not only aware of the problem type but also adapts the reasoning action trajectory prediction based the capabilities of the task-solving L.M.M
8

---

Published as a conference paper at ICLR 2025
Table 7: Planning action distributions of DoTs over three different layers on the MATH test set.
Sub-tasks MATH
Accuracy
Analysis Layer
Solution
Veriication Veri.
Rewr.
Deco.
Empty
Col
Pol
Empty
External Planner: 3-no---------tte
Solver: GPT-4o-mini
Algebra
92.1
0.00
0.05
0.92
0.90
0.10
0.29
0.71
Prealgebra
88.6
0.03
0.01
0.31
0:21
0.79
Number Theory
81.8
0.01
0.01
0.98
0.43
0.57
0.15
0.85
Counting and Probability
76.8
0.08
0.06
0.84
0.78
0.32
0.30
0.70
Geometry
61.8
0.03
0.01
0.05
0.06
0.94
Internediate Algebra
57.1
0.05
0.02
0.93
01.85
0.15
0.44
0.56
Precalculus
52.6
0.06
0.02
0.92
0.95
0.05
0.46
0.54
External Planner: Llama-3-88h 3-8II-IIsttteeee
Solver:
3aama-3-70B--mma
Algebra
74.9
0.03
0.04
0.93
0..7
0.23
0.12
0.88
Prealgebra
74.5
0.02
0.03
0.43
0.10
0.90
Number Theory
69.9
0.01
0.01
0.98
0.32
0.68
0.13
0.87
Counting and Probability
55.4
0.04
0.02
0.94
0.59
0.41
0.11
0.89
Geometry
39.6
0.05
0.01
0.94
0.24
0.18
0.82
Precalculus
36.9
0.07
0.03
0.90
0.78
0.22
0.28
0.72
Internediate Algebra
34.6
0.03
0.01
0.72
0.28
Furthermore, we observe that question rewriting and decomposition were selected with low fre- quency. This is likely because the MATH dataset consists of precise problems that do not benefit from rewriting. Additionally, given the strong reasoning abilities of Llama3--0B-Innee Instruutt and GPT- 4o-mini, their CoT process inherently includes task decomposition, reducing the need for further planning interventions.
3.8 ADDITIONAL ANALYSES
Table External planner tuning under the few-shot setting with GPT-44-miní as the solver.
Few-shot In-context Learning Setting Our main results report the performance with zero-shot evaluation. In cases where reasoning tasks are known advance, common approach to leveraging train- ing data and improving the performance of closed-source LLMs is few-shot in- context learning (ICL), where training ex- amples are incorporated directly into the DOTS
Method CoT
MATH
BBH
TheoremQA
Average
72.3
84,2
38.2
64,9
LTM
72.7
83.4
37.3
64.5
PA
71.3
83.3
38.7
64.4
PoT
69.8
82.1
36.4
62.8
Self-refine
73.2
83.1
35.4
63.9
75.4
86.1
39.9
67.1
context. Our external planner tuning can also be utilized in this scenario seamlessly. Specifically, we can first construct few-shot ICL prompts for each potential reasoning action trajectory. Once the external planner selects the appropriate rea- soning actions, the corresponding few-shot prompt will chosen and applied. We evaluate the external planner tuning setup of DOTS, with Llama-3-8B-Instrun GPT-4o-mini being the solver LLM, in this setting. We compare our approach with the same base- lines similarly implemented in the few-shot ICL setting, where we randomly selected 8 examples from MATH, 4 examples from each category of BBH, and 4 examples from TheoremQA to form the generated by GPT-4o and manually verified for quality.
being the external planner and
prompt All few-shot demonstrations
As shown the Table DOTS continues to outperform baseline models. Interestingly, compared 3 which presents the zero-shot results, adding few-shot demonstrations static prompting methods does not lead to consistent improvement, except on the BBH dataset. This indicates that simply expanding the context with additional demonstrations does always serve as an effective way to leverage available training data. In contrast, our method demonstrates its superior ability to effectively utilize the training data.
Table 9: Avg. number of output tokens for each method (solver: Llama-3-8B-ae Instruus.
How efficient is DOTS? We compare the cost effi- ciency, measured by the average output token count, of each method (based on Llama-3-8B-Instruu Table 9 The result shows that DOTS consumes fewer tokens on average than other advanced ap- proaches and only more than COT. Advanced prompt engineering methods often introduce supplementary text to facilitate reasoning. However, not all ques-
in
Method
Avg. Output Tokens
CoT (Wei et al. (002)
263.6
LTM (Zhou et al. (2023)
436.4
Self-refine (Madaan et al. 2024)
527.6
Dors: Internalized
409.1
We excluded the "Game of 24" task because knowing the task in advance enables it to be solved with straightforward program.

---

Published as a conference paper at ICLR 2025
tions require this additional context to the same tent. By constructing training data via searching, our goal is to optimize the balance between min- imizing extraneous steps and maintaining high success rate, thereby reducing unnecessary output tokens. Our method avoids redundant reasoning actions, resulting in a more efficient system. Do we need more reasoning steps for difficult questions? Recent research suggests that LLMs can better solve difficult questions by increasing the thinking time in the inference stage (Brown et al.)
Figure 3: Average reasoning trajectory length per difficulty level on MATH for DOTS (solver: GPT-4o-mini; External
2024 OpenAl 2024). In our study, we explore the relationship between question difficulty and the av- erage reasoning action trajectory length. The trajec- tory length is determined by assigning a value of
planner: 1amma3-8B-Instruce
Action Trajectory Length
1.45
1.40
the EMPTY module and 1 to all other actions, while the question difficulty is derived from anno- tated levels on the MATH dataset. Figure pressets that harder problems demand more computational steps, resulting longer reasoning trajectories. Case analyses further reveal that our planner increases the proportion of verification steps as problem difficulty rises. This highlights an exciting fact - LLMs can learn to employ more reasoning steps for chal-
1.30
1.25
1.20
level
level 2 level
level4
lovel
Dificully Level
lenging problems through exploration, without requiring explicit expert guidance.
RELATED WORK
Prompt engineering for LLM reasoning LLMs have demonstrated remarkable proficiency in solving complex reasoning tasks (Lewkowycz et al. 2022). The Chain-of-Thought (CoT) approach, introduced by Wei et al. (2022), significantly improves performance on reasoning problems by prompting L.MSs to think step-by-step in natural language. [Cen et al. (2022) and Gao et al. (2023) proposed the Program-of- Thoughht prompting method, where code is used as intermediate rea- soning step. Advanced prompt engineering methods, such as question decomposition (Zhou et al. 2023) and self-verification (Madaan al. 2024), have also proven effective in improving reason- ing performance. Additionally, recent approaches have incorporated automatic prompt optimiza- tion based on training data. For instance, WWang al. (2023) refines prompts by analyzing error cases, and self-discovery (Zhou et al.] (024)) utilizes modular reasoning components to construct the task-adaptive prompt. However, these automated prompt optimization techniques still produce static prompts for all instances. Srriaattaaa et al. (2024) proposed instance-level prompt optimization via LLM self-refining, but it relies on expert-designed workflows and lacks active adaptation. In our method, we internalize the reasoning action selection capability into the LLM itself without an expert-designed workflow, allowing to autonomously fit both the characteristics of questions and the inherent capability of task-solving LLM.
Searching for boosting LLM reasoning Recent research suggests that incorporating searching mechanisms can significantly enhance LLM reasoning. In the inference process, Thouuhh- (ToT) (Yao et al. (2044) and Traph-of-Thoughh (GoT) (Besta et al. 2024 have been proposed to search and investigate different reasoning paths, either by leveraging the LLM itself (Yao et al.) 2024] or designing heuristic functions (Hao et al. (023) as the signal evaluate each step. More recently, Monte Carlo Tree Search (MCTS) has been introduced to assist the LLM in learning how to evaluate each step (Qi et al., 20224) Xie et al. 2024). The searching mechanism can also be used in training to collect training instances for improving LLM reasoning (Luo et al. 2024). However, all these searching methods treat each "CoT reasoning step" as the atomic component or step in searching, while we choose each reasoning action the atomic component in our case.
5 CONCLUSION
In this paper, we introduce DOTS, method that enables LLMs to autonomously think about appro- priate reasoning actions before answering questions. By defining atomic reasoning action modules, searching for optimal action trajectories, and training LLMs to plan for reasoning questions, we en- able LLMs to dynamically adapt to specific questions and their inherent capability. The flexibility of our two learning paradigms, i.e., external and internalized planner tuning. further highlights the adaptability of our method to different LLMs. Our experimental results show the effectiveness of DOTS, revealing the promise of harnessing explorations and evaluations to turn LLMs into planners for better reasoning.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00) 00/ 57,, 2024.
Lightning AL Litgpt.
2023.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gian- inazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 17682-17690, 2024.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.2 2024.
Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint vrXiv:2406.20094 2024.
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915, 2023a.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. 2022.
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: theorem-driven question answering dataset. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023b.
Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu. Rephrase and respond: Let large language models ask better questions for themselves. 2023.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning paragraphs. arXiv preprint 1eXxv:1903000100 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 /783, 2024.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference Machine Learning, pp. 10764-10799, PMLR, 2023.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant, Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361 2021.
Shibo Hao, Yi Gu, Haodi Ma. Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8154-8173, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10. 00.18653/vv120923 http:///2222mmmmmmmm.....0 URL https: / aacl aelllnnhhooggg org/2023. moanppmmaing 507
Dan Hendrycks, Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05338 2023.

---

Published as a conference paper at ICLR 2025
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-2223 22213, 2022.
Stefanie Krause and Frieder Stolzenburg. Commonsense reasoning and explainable artificial in- telligence using large language models. In European Conference on Artificial Intelligence, pp- 302-319. Springer, 2023.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo et al. Solving quantitative reasoning problems with language models, 2022. URL htlps://arziv. org/abs/2206. 14858, 2022.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprini arXiv:2308. 200822308.09583, 2023.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint (tti::20060065992 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
OpenAL. Learning to reason with 1lms, 2024.
URL https:/ com index/ Learning- to reason reason with llms/ / Accessed: 2024-10-01.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2.304.000 2.2.:2..004.33277,, 2023.
Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reason- ing makes smaller ilms stronger problem-solvers. arXiv preprint arXiv:2408.00 06955, 2024.
Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Heraandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilè Lukošiūté, al. Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint arXiv: 1768, 2023.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint /2. 2021.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical rea- soning abilities of neural models. In International Conference Learning Representations, 2019. URL htt.tttss///ppeeee///
wwt./frutftttteeo id=H1 g
Saurabh Srivastava, Chengyue Huang. Weiguo Fan, and Ziyu Yao. Instances need more care: Rewriting prompts for instances with LLMs in the loop yields better zero-shot performance. In Lei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Compu- tational Linguistics ACL 2024. pp. 6211-6232, Bangkok, Thailand and virtual meeting, August http://///22...nnnnnnallllll//l
2024. Association for Computational Linguistics. doi: 10.18653/v1/2000 findings--al..311. URL
htt.pss//ccllatthoooo...o/////// otg.://oo......22...oo.ooooeeee. 371
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint 10Xiv:2210009261,, 2022.
Zhengyang Tang. Xingxing Zhang. Benyou Wan, and Furu Wei. Mathscale; Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.022 028884, 2024.
Xinyuan Wang. Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo. Jiayou Zhang, Nebojsa Jojic, Eric P Xing, and Zhiting Hu. Promptagent: Strategic planning with language models enables expert- level prompt optimization. arXiv preprint 2310 2023.
12

---

Published as a conference paper at ICLR 2025
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabshii, Yeganeh Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705. 2, 2022.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang. Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 21224824-24337 2022.
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models better reasoners with self-veriffiation. arXiv preprint arXiv:2212.09561, 2022.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint ar 4rXiv:2405.00045/ 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint vrXiv:2309.056533 2023.
Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arXiv preprint arXiv:2405.03546 2024.
James Xu Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, and Michael Qizhe Xie. Automatic model selection with large language models for reasoning. arXiv preprint arXiv:2305. 14333, 2023.
Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36, 2024.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.0 06364, 2023.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schu- urmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Confer- ence on Learning Representations, 2023. URL http:::/tp/oottott https://ppeneevee..rtt net. forum?id- W2H7099tqfM
Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng-Tze Cheng, Quoc V Le, Ed H Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. Self-discover: Large language models self- compose reasoning structures, arXiv preprint 2xxv:24402.03220 2024.
13

---

Published as a conference paper at ICLR 2025
A TRAINING IMPLEMENTATION
In our optimal trajectory search, we set the number of iterations (K) to 2 and the number of evalu- ation times (Nevat) 4. The number of paths retained Ni set 8 and N2 3. Throughout the search, we maintain a sampling temperature of 0.4, Searching on the training datasets eventually yields 1722 for GPT-4o-mini, 1624 for 1lama-3-700 Instruntt and 2140 for (luma-3-8B-Instrunee training examples for planner tuning, respectively. We up-sample few-shot examples by factor of 8 and fine-tune the planner LLM using the LitGPT library (Al 2023). applying learning rate of 2e-5, global batch size of 64, maximum sequence length of 4096, and training epoch of 4.
B CASE STUDY
In this section, Qı is a numerical reasoning problem that can be solved with Python code easily, so our tuned LLM selected PoT to solve Qι is a question widely tested in the community where GPT-4o even tends make mistakes, but after tuning with searched trajectory data, the LLM actively chose to use code to solve the problem.
14

---

Published as a conference paper at ICLR 2025
Question Case Study
Question I:
A book with 50 pages numbered I through 50 has its pages renumbered in reverse, from 50 to I. For how many pages do both sets of page numbers share the same digit?
Answer from GPT-40-2024-08-06 (Incorrect):
To solve this problem, need to determine how many pages have the same ones digit in both the original numbering (1 to 50) and the reversed numbering (50 to 1).
First, let's understand the problem: The original numbering of the pages is from 1 to 50. The reversed numbering is from 50 to 1. meaning page becomes 50, page 2 becomes 49, and on, until page 50 becomes
We need to find pages where the ones digit of the original page number is the same as the ones digit of the reversed page number. Let's denote the original page number as n and the reversed page number as n. We are looking for such that the ones digit of n is equal to the ones digit of 51 n.
The ones digit number is the remainder when the number is divided by 10. Therefore, we need:
n= 51 (mod 10)
Since 51 = (mod 10), we have:
2n = 1 (mod 10)
Thus, multiplying both sides of the congruence by 5 gives:
n x 1 (mod 10)
This means n must have a ones digit of 5. The pages with ones digit of 5 in the original numbering are 5, 15, 25, 35, and 45.
Thus, the number of pages that share the same one's digit in both numbering systems is: 5
Answer from our internal planner tuned -lama3-8h-Instrus
can efficiently iterate through the range of page numbers to count occurrences of the same ones digit. Programming solver provides quick and reliable method to handle the counting of digits across defined range. Required Action: programming
count = 0
for original.page page in range 51): =51-original-page
renumbered.pagg # Check if the ones digit is the same if original_papaee page % 10 = renumbered page % 10: count +=
print (count)
After execution, we get:
Answer: 0
15

---

Published as a conference paper at ICLR 2025
Sub Task
Col
Pol
DOTS
object counting
94.4
98.0
98.0
temporal sequences
96.8
83.6
97.2
tracking shuffled objects five objects
95.2
78.8
96.4
word sorting
50.8
98.0
98.0
salient translation error detection
61.6
59.2
62.6
penguins in table
71.2
54.8
73.3
snarks
83.1
69.1
83.3
ruin names
81.2
80.2
81.2
web of lies
91.2
59.2
92.0
navigate
96.4
90.4
96.8
date understanding
80.8
52.8
76.4
hyperbaton
92.0
67.6
91.0
dyck languages
38.0
42.4
54.4
tracking shuffled objects three objects
99.2
89.2
99.2
formal fallacies
82.0
74.4
82.0
tracking shuffled objects seven objects
91.2
73.2
92.8
causal judgement
62.6
62.6
62.0
sports understanding
85.6
74.8
87.6
logical deduction five objects
85.6
80.4
88.8
movie recommendation
62.0
56.0
62.8
logical deduction three objects
99.2
96.4
99.2
multistep arithmetic two
98.0
100.0
100.0
boolean expressions
99.2
96.8
99.2
geometric shapes
56.4
65.6
78.8
disambiguation qa logical deduction seven objects
42.8
40.8
45.6
76.8
71.6
82.0
reasoning about colored objects Average
87.6
66.0
87.6
80.3
73.9
84.2
Table 10: Results of BBH sub-tasks.
Question 2 Case Study
Question II:
How many word 'strawberry"?
Answer from GPT-40-2024-08-06 (Incorrect): The word "strawberry" contains 2 letter "r"s.
Answer from our internal planner tuned -lama-8b-Instree can efficiently count the occurrences of a specific character (in this case, 'r') in a word. Programming_solee solver provides quick and reliable method for counting characters in text, ensuring accuracy in the result.
Required Action: programming
word = 'strawberry' count=word lower () count ('r')
print (count)
After execution, we get:
Answer: 3
C SUB-TASK PERFORMANCE OF BBH
Table 10 shows the results of BBH sub-tasks of GPT-4o-mini task-solving LLM and our tuned Llama-33 Beae-----B--itiie as the planner. From the table, we can see that our planner has learned to select appropriate actions based the task. For instance, in the word sorting task, model consistently
16

---

Published as a conference paper at ICLR 2025
code to solve the problem. Additionally, for the Dyck languages task, method outperforms both CoT and PoT. This because the task is easy to verify, and our model proactively requests GPT-4o-mini to verify the answer, thereby improving performance. The only exception is the date understanding task. Upon analysis, we found that while code could solve this type of problem using Python's "datetime" library, it often fails follow the required output format. This leads to lower accuracy in our method's s prediction when choosing writing program as the reasoning action.
D PROMPTS USED IN EXPERIMENTS
Prompt for query rewrite module
In this step, you need to reveal the Core Question with only simple sentence and useful information. The output follows the format: core question:...
Note: Please extract the question-solving information related to the problem, and list them one by one.
useful information:...
Prompt for query decomposition module
In this step, you need to reflect on the problem, and describe it in your own words. Analyze how you can decompose the problem into smaller, more manageable sub-tasks. Pay attention to small details, nuances, notes and examples in the problem description.
Prompt for CoT module
In this step, you need to think step by step with words, solve the problem and get the answer.
Prompt for PoT module
In this step, you need write Python codes to solve the query. Use the simplest and most straightforward programming methods to solve the problem. For instance, if query can be efficiently solved using a brute force method, prefer it over heuristic or more complex methods. Utilize any available and commonly-used libraries that can simplify the task or improve code maintainability, All the calculations must leverage codes. Print the results with the print() function. Before executing the program, you have no idea of the final answer. Don't show it in your comment or code. And don't 't use the plot function. In this step, start with "# Now write Python codes to answer this question and use print() to print out the result"
Prompt for self-verification module
In this step, you need to carefully verify the correctness of the previous thoughts with natural language. You need to formulate verification question (not the same question as before) based on the final answer and then verify the final answer you have. If the results are in- correct, the last line should end up with "The answer is: incorrect". Otherwise, the last line should end with "The answer is: correct"
17

---

Published as a conference paper at ICLR 2025
Prompt for explanation generation
Action Categories:
1. Understanding process: query rewriting: Rewrite the question and answer it. Decomposi- Decompose the questions into multiple subtasks to solve the sub-question. 2. Solving process: chain of thought: For step-by-step reasoning with language. programming: For programming solver. 3. Verification process: selffverification: To check the correctness of the solution.
tion:
Task Instruction: For the given question, explain why the above Required actions are nec- essary.
Example 1:
Query: Find 2 8-8 11 (mod 56). Express your answer as an integer from to 55, inclusive.
Required Action: programming. self-verification Explanation: This is a Modular arithmetic problem. The problem can be solved using straightforward python code with sympy library, particularly modular arithmetic. Besides, this type of problem is relatively easy to verify. After computing the result, one can check the calculations step by step to ensure correctness and verify that the final answer is within the given range (0 to 55 inclusive). Programming solver is more efficient and accurate for this type of calculation and the verifier ensures the correctness the result and adherence the given constraints. (multiple examples)
Query: Given Query
Required Action: Actions After Searching Explanation:
18

---

Published as a conference paper at ICLR 2025
Prompt for prior knowledge exploration module
...(test query)
Before solving the problem, take moment to: Choose the most appropriate action trajecto-
ries from: Understanding Process:
query rewriting: Suitable for: Ambiguous poorly phrased questions. Reasoning: This action ideal when the question is not clear or is open to multiple interpretations. By rewriting the question, we clarify the intent and ensure that the answer addresses the correct issue. This is particularly useful in natural language processing tasks where the original query might be vague contain errors.
-Decomposition: Suitable for: Complex questions that can be broken down into simpler, more manageable parts. Reasoning: When question involves multiple components or steps, decomposing it into subtasks can make it casier to tackle. This approach is beneficial for multi-step problems, such as those found in project management, research planning, or complex caablemmsolving scenarios where each subtask can be addressed individually before integrating the solutions.
2. Solving Process: chain of thought: SSuitable for: Questions requiring logical reasoning or step-by-step deduction. Reasoning: This method is effective for problems that benefit from structured approach, such as mathematical proofs, logical puzzles, any scenario where a sequential thought process is necessary to arrive at the correct answer. allows for clear demonstration of the reasoning path taken to solve the problem.
programming: Suitable for: Questions that can be solved algorithmically or require com- putational solutions. Reasoning: When a problem can be addressed through coding, such as data analysis, algorithm design, or simulation tasks, using a programming solver is appro- priate. This approach is particularly useful for problems that can be solved with brute force searching, where writing and executing code can efficiently produce the desired outcome. Verification Process: verifier: Suitable for: Questions where the correctness of the solution is critical. Reasoning: This action is essential when the accuracy of the solution must confirmed, such as in scientific research, engineering calculations, or any high- stakes decision-maker making process. Verification ensures that the solution is not only plausible but also correct, providing confidence in the results.
You need to first think about the question and select the best action trajectories, for exam- ple, Iprogramming, verifier] to solve it. Then proceed with solving the problem using your selected strategies. Remember to follow the question format.
E MORE BASELINES
E THREE-LAYERS STATIC PROMPTING
The baselines in experiments did not utilize all layers. To ensure a fairer comparison, in this section, we compared our method with results using all layers, as shown in Table IT] The results demonstrate that our method still outperforms all baselines.
E.2 HYBRID TRAINING
The second baseline is about hybrid training, i.c., using a mixture of training data with both CoT and PoT (Yue et al. [202)). We use the same training instance and each instance is provided with both the PoT and CoT reasoning paths. We reported the results in Table 12 The result shows that our method still outperforms the hybrid training. From the results, it can also be observed that for datasets where CoT outperforms PoT, training solely with CoT yields better performance. We analyzed several cases and found that although mixed training allows the LLM to choose between CoT and PoT dynamically, the results tend to perform random selection rather than actively analyzing the problem and selecting the most suitable path for the given question carefully because didn't 't guide the LLM learn this. This highlights the importance of searching for the optimal reasoning path before tuning the L.M..
19

---

Published as a conference paper at ICLR 2025
Methad
MATH
BBH
Game
Theorem QA
DM math
MMLU
Strategy
DROP
Average
pro
QA
Task solving LLM:
Llama3- 70b Instruct
Rewr.#CoT+Vert.
50.9
73.1
28.2
28.8
58.1
60.1
81.9
66.0
55.9
Rewr.+PoT+Veri.
55.3
67.9
30.6
58.1
79.1
62.5
Deco.+CoT+Van Veri.
51.1
74.2
29.6
29.1
58.6
60.5
81.4
65.7
56.3
Deco.+PoT+Veri..
55.8
72.5
68.1
30.8
73.9
58.6
78.9
65.6
63.0
DOTS
57.7
77.3
67.7
31.2
74.1
59.4
80.3
66.3
64.3
Task solving LLM:
GPT-4o-mint
Rewr:+Co|+Ve
71.9
79.5
32.1
36.9
80.1
61.1
80.6
68.5
63.8
Rewr.PPoT+Ven..
73.4
76.7
67.9
34.8
85.8
58.4
75.8
72.9
68.2
Deco.+CoT+Veri. Veri.
72.7
79.8
33.5
37.8
82.8
62.3
81.2
67.3
64.7
Vecr.+PoT+Veri.
73.6
78.7
35.2
86.5
59.3
76.9
68.9
DOTS
75.4
84.2
65.2
41.4
87.6
61.5
78.8
73.8
71.0
Task solving LLM: Finetuned
Rewr.#CoT+V Veri.
29.5
48.9
14.5
15.8
30.1
38.7
70.3
52.9
37.6
Rewr.PoT+Veri
27.2
48.3
17.8
14.9
46.9
35.5
64.9
47.1
37.8
Deco.+CTT Veri.
30.1
49.5
15.5
14.6
31.7
39.2
71.3
52.1
38.0
Deco.+PoT+Veri..
27.6
49.2
18.9
15.2
47.8
36.2
64.7
45.8
38.2
DOTS
34.4
69.7
21.9
16.1
55.3
39.7
68.2
48.8
44.3
Table 11: The three-layer baselines performance
SFT Method
MATH
BBH
Game24
Theorem
DM math
MMLU
Strategy
DROP
Average
QA
QA
Vanilla SFT
33.9
61.0
18.5
14.8
40.3
71.8
49.0
36.7
Hybrid SFT
32.3
53.3
18.7
15.6
49.1
36.5
65.9
46.2
39.7
DOTS
34,4
69.7
21,9
16,1
55.3
39.7
68.2
48.8
44.3
Table 12: The result of hybrid training
F DISTRIBUTION OF REASONING PATH
In this section, we show a detailed distribution of reasoning paths. We denote the reasoning paths as:
Pl: Empty, CoT, Empty
P2: Empty, PoT, Empty
P3: Empty, CoT, verification
P4: Empty, PoT, verification
P5: Question Decomposition, CoT, Empty
P6: Question Decomposition, PoT, Empty
P7: Question Decomposition, CoT, verification
P8: Question Decomposition, PoT, verification
P9: Question Rewrite, CoT. Empty
PI0: Question Rewrite, PoT, Empty
PI1: Question Rewrite, CoT, verification
P12: Question Rewrite, PoT, verification
The distribution is shown in Table 13
We can still maintain the conclusion drawn from the sec- tion 3.7 For different tasks, the planner predicts different action trajectories. For example, the planner selects PoT more frequently in Game of 24, while preferring CoT in DROP. The planner can also adapt to the LLM ability. For instance, GPT-4o-miní excels at solving problems using CoT, leading to higher selection rate of CoT compared to
20

---

Published as a conference paper at ICLR 2025
Dataset
PT
P2
P3
P4
P5
P6
P7
P8
P9
PTO
PIT
P12
Task solving LLM:
Llama3-700 Instruct
MATH
50.5
26.0
8.4
3.7
3.1
1.1
0.4
1.9
1.7
1.9
1.3
0.0
BBH
32.1
16.2
21.1
9.1
10.9
0.3
0.5
0.7
9.0
0.1
0.0
0.0
Game 24
0.2
4.3
94.1
0.0
0.4
0.3
0.0
TheoremQA
42.6
30.3
13.0
8.1
0.4
0.0
5.0
0.0
0.4
0.2
0.0
0.0
Deepmind Math
1.1
32.1
6.1
59.1
0.2
0.2
0.0
0.6
0.0
0.0
0.0
0.6
MMLU-pro
59.8
5.2
31.0
0.4
1.0
0.4
2.0
0.2
0.0
0.0
0.0
0.0
StrategyQA
62.6
4.0
20.1
2.1
10.2
0.0
1.0
0.0
0.0
0.0
0.0
0.0
DROP
82.8
0.9
16.2
0.0
0.1
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Task solving LLM:
4PT4o-mini
MATH
54.9
12.4
20.1
5.1
1.6
1.1
0.6
0.0
3.4
0.0
0.8
0.0
BBH
45.4
15.8
14.7
2.0
13.0
3.7
0.0
0.0
Game 24
0.5
1.6
0.3
97.6
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
TheoremQA
43.8
25.2
28.4
0.0
1.1
0.0
0.0
0.0
1.5
0.0
0.0
0.0
Deepmind Math
2.1
41.1
0.6
50.2
0.0
0.0
0.0
0.0
0.0
4.0
0.0
2.0
MMLU-pro
71.3
20.7
5.0
0.0
0.0
2.0
0.0
0.0
0.2
StrategyQA
68.1
7.0
19.1
0.0
5.7
0.1
0.0
0.0
DROP
98.7
1.0
0.2
0.0
0.0
0.1
0.0
0.0
0.0
0.0
0.0
0.0
Table 13:
The action trajectory distribution of all datasets
Method
MATH
BBH
Game 24
Theorem QA
DM math
MMLU
Strategy QA
DROP
Average
pro
Solver: GPT4o-mini
DOTS
75.4
84.2
65.2
41.4
74.1
59.4
80.3
66.3
68.4
w/o Searching
69.2
78.6
28.9
40.2
62.5
58.3
79.5
64.5
60.2
w/o Explanation
68.2
81.3
57.4
36.4
58.5
59.2
81.2
66.1
63.5
Solver: Llama3-8B --nnsruutttt
DOTS
34.4
69.7
21.9
16.1
55.3
39.7
68.2
48.8
44.3
w/o Searching
31.4
55.8
19.6
15.1
42.5
38.2
67.5
47.5
39.7
w/o Explanation
33.8
65.8
18.6
15.7
47.2
38.5
69.6
50.2
42.4
Table 14: Full result of ablation study
G ABLATION STUDY FULL RESULT
In this section, we provide the full ablation result of all datasets in Table 14 It reveals the effective- ness of each step of our method.
H CAN WE ADD PRIOR KNOWLEDGE IN THE PROMPT TO HELP THE LLM SELECT ACTIONS?
We designed experiment with Llama-3-8B- --nttt--BB-nnttrett, which shows that simply feeding the prior knowledge as instruction insufficient for effective planning. Specifically, we compared the inter- nalized DOTS planner with variants: (1) Vanilla Prompting-base based Planning, where off off-the- shelf Jlama--8B-Instruus model was directly prompted to select optimal actions for given question based its parametrie knowledge; and (2) Prompting-basee Planning w/ Prior Knowledge, where the model was prompted with the prior knowledge that we summarized from the solver LLM's op- timal trajectories (e.g., common rules for action selection) part of the instructions at time. The prompt is provided in D In both variants, we did not train the planner LLM. The results are shown in Table 15 Our experimental results show that feeding no prior knowledge, or simply feed- ing the prior knowledge without SFT, is insufficient for an LLM to plan effectively for reasoning tasks. We analyzed several cases and found that LLMs often failed in selecting the best trajectory. For example, one prior knowledge in the prompt is that "decomposition is useful when question involves multiple components... We find that prompt-based LLM tends to select a fixed subset of actions including decomposition. However, our search results indicate that incorporating decom- position does not provide significant benefits tasks such as Game of 24, Because decomposition appears reasonable to the LLM, is only through search and learning that the LLM know that
21

---

Published as a conference paper at ICLR 2025
Game
Theorem
Deepmind
MMLU
Method
MATH
BBH
SQA
DROP
Average
24
QA
-math
-pro
Vanilla Prompting -based Planning Prompting-base
29.5
48.7
11.9
13.3
34.9
38.0
69.2
53.0
37.3
Planning with
30.2
47.5
13.0
14.5
34.2
37.8
69.3
55.2
37.7
Prior Knowledge
DOTS
34,4
69.7
21.9
16.1
55.3
39.7
68.2
48.8
44.3
Table 15: The result of using prompt with prior knowledge
decomposition unnecessary for this type of problem. It clearly demonstrates that relying solely on naive prompt design is insufficient for enabling LLMs to learn how to reason about selecting paths effectively. The reason that enumerating all detailed and comprehensive prior knowledge in the prompt is challenging,
it is difficult account for numerous case-by-case scenarios. Our method integrates prior knowledge from explanations with the searched optimal trajectory, effee- tively teaching the model to learn how to select the optimal trajectory. In view, searching is to implicitly acquire the "prior knowledge" and training is a reliable way to make LLMs learn the "prior knowledge" determine how select reasoning actions.
22


Reviews:
[{"review_id": "IeBTpUwhin", "reviewer": "Reviewer_kuiZ", "summary": "This paper proposes DOTS, an approach enabling LLMs to reason dynamically via optimal reasoning trajectories search, tailored to the specific characteristics of each question. DOTS first defines atomic reasoning action modules, then searches for the optimal action trajectory for each training question through iterative exploration and evaluation for the specific task-solving LLM, and uses the collected trajectories to train a model to plan for the reasoning trajectories of unseen questions.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose a dynamic reasoning method which can enable the model to decide the appropriate atomic actions based on the characteristics of the input question.\n2. The authors conduct comprehensive experiments to prove the effectiveness of the proposed method, containing in distribution, few-shot, and out-of-distribution settings.\n3. The proposed method can be used on both open-source and close-source models.", "weaknesses": "1. The method proposed in this paper does not show significant improvement on out-of-distribution (OOD) tasks, and it incurs additional computational overhead compared to prompt engineering methods.\n2. The baseline for Vanilla SFT only used the training data from CoT. I believe it should be compared with baselines from other reasoning formats to demonstrate the effectiveness of the proposed method, such as using the program reasoning format and mixed training data from CoT and Program.\n3. The experiments demonstrating the effectiveness of the Search section are compared with randomly selected reasoning paths. I am curious about how the results would compare if we contrasted it with reasoning paths generated by combining non-empty actions from each layer.\n4. Using PoT prompts on non-code models may yield suboptimal results. Would combining PoT with code models improve the results of the PoT baseline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b6Kqgx47gH", "reviewer": "Reviewer_in1A", "summary": "The paper proposes a new way to learn reasoning through two learning paradigms. The paper (DOTS) proposes both a way to FT an planner LLM and in directly FT-ing the task solver.", "questions": "Q1. How many trajectories are required to improve performance substantially at FT-time? A more thorough discussion would be useful.\n\nQ2. How does the proposed approach scale to more complex problems where process rewards would be beneficial?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Adaptability of the method: The method can be used either on the planner or on the task solver.\n\n2. The method shows lower cost than non-CoT methods.\n\n3. Improved Reasoning ability: The paper demonstrates improved reasoning ability by allowing for dynamic selection for a given question, outperforming static methods as well as self refinement (most but not all scenarios).", "weaknesses": "1. The proposed method is complex and involves several steps. It is not clear if the complexity is warranted and ablation studies on the various aspects can help.\n\n2. The paper does not explore decomposition or tool use which would be crucial for complex tasks.\n\n3. The paper glosses over how the atomic trajectories are collected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kFqiLm3ZuZ", "reviewer": "Reviewer_6PPW", "summary": "The paper proposes DOTS, which can dynamically choose prompting methods to do reasoning, by dividing reasoning into analysis, solution, and verification modules, and choosing from recent popular methods for each module.\n\nThe dynamic planner can be either external or internal to incorporate various LLMs with difference sizes and openness.\n\nExperiments are conducted on in-distribution, near-distribution, and out-of-distribution settings, and zero-shot and few-shot evaluations, where DOTS outperforms most recent LLM prompting reasoning methods.", "questions": "+ (Type) Equation 2: $T$ has not been defined.\n+ (Clarification) For the \"few-shot\" settings, do you use the planner trained on MATH dataset? If so, they should be \"near-distribution + few-shot\", since those \"few-shot\" datasets are math-related.\n+ Since DOTS has marginal performance on OOD tasks (despite DOTS need additional training), what's the performance on near-distribution (e.g. the few-shot datasets in the paper) but zero-shot settings?\n+ In Table 3, 4, 5, on a few datasets DOTS is not the best among methods. Does it mean in some datasets, it's better to choose one single baseline method instead of choosing modules dynamically as in DOTS?\n+ DOTS select one reasoning method (one choice for each module). Would it be better to attempt multiple choices and choose the best one (and this would align more with \"trajectory search\")?\n+ (Minor) It would be beneficial to also include the computing efficiency of external DOTS.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "DOTS unifies many of recent LLM prompting reasoning methods, and can dynamically choose better method for each module for each data sample. It borrows the strength of various solutions in literature, and can possibly incorporate future works.\n\nThe paper contains comprehensive experimental results, on various datasets across domain, and in/near/out-of-distribution experiments. The experiments show superior performance of DOT to previous prompting methods consistently.\n\nThe paper also includes ablation studies to verify the importance of each of the breakdown modules.\n\nIt also includes analysis of preferences of DOTS on different tasks, with insightful explanations. It also includes an analysis of efficiency, showing comparable computing costs to other advanced prompting techniques.", "weaknesses": "Although DOTS has higher average performance, DOTS cannot consistently beat baselines in all tasks. This is a little surprising as DOTS should be able to be considered as superset of all baselines. On those dataset DOTS fall behind baselines, it would mean it's not necessary to do dynamical reasoning (as one can choose the baseline instead of choosing different modules for each data sample).\n\nThis phenomenon is more often in out-of-distribution settings. The average score on out-of-distribution also shows smaller margin. This means lack of training (in a domain) can lead to much worse performance of DOTS. However, as also mentioned by author, such training set may be not available, casting generalizability concern on DOTS.\n\nIt would be beneficial to show the comparison on near-distribution zero-shot settings (e.g. the few-shot datasets in the paper) to verify that only near-distribution training is needed, but the paper only includes few-shot settings on these datasets. (also see questions)\n\nTo clarify, the few-shot experiments with consistent outperformance can partly address the point, but I would expect better zero-shot performance since training is involved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k3l0y23LXY", "reviewer": "Reviewer_Npg3", "summary": "The author creates a planning prompt strategy named DOTS. It can choose the path to solve the problem via a small model (as the planner) and prompt the large model to solve the problem. The path contains 3 parts: the Analysis Layer, the Solution Layer, and the Solution Layer. The experiment shows that both external and internalized planner can help the large model to solve the problem more effectively and efficiently. The ablation study also shows that the searching and explanation in planner's plan is useful.", "questions": "1. could you present the distribution of the path the planner will choose in the experiment? (i.e. x% of the path will choose Empty + POT + self-verifications, y % of the path will choose Decomposition + COT + Empty, etc.) (This is different from Table 7 in 2 parts: 1. It shows the distribution of the 12 types of combination of 3 layers in the path, 2. It needs to contain more benchmarks besides the MATH dataset)\n\n2. Can 2 findings in Section 3.7 be generalized to other datasets?  \n\n3. (From weakness 1) The authors should add a comparison with the baseline with 3 layers (and the choice of the layer should be reasonable towards a given benchmark) to make the comparison more fair.\n\n4. From my perspective, the OOD benchmark is more important than the IND benchmark, however, the authors only show the ablation study in the IND benchmark. Why?\n\n5. The improvement of the DOTS seems to come from the explanation (if you compare the result in Table 6's \"-w/o Explanation\" line with the second to the last line in Tables 3 and 4). So, does the prior knowledge in the prompt cause the improvement? \n\n6. (minor) The author claims that \"the fine-tuned LLMs are constrained to follow the same reasoning format of the training data (e.g., CoT (Luo et al., 2023)) and lack the flexibility to adopt other reasoning strategies\" (in line 53,77-78). However, FireAct[1], shows that the fine-tuned LLMs can change the reasoning strategies (COT or ReAct) by itself without any additional prompt. Maybe training COT and POT at the same time can help the model to be more flexible. The authors should revise this point to make it more accurate.\n\n7. In Internalized setting, I'm curious that if llama3-8b is trained only in the Reasoning Process and Answer (like FireAct), what performance will be, will the trained model choose the Reasoning Process automatically? (It should also be a baseline), although I admit this can't be used in external setting.\n\n8. (minor) In Table 2, \"Few-shot\" is not a kind of Distribution, maybe 4 columns (add example number column) can be more clear.\n\n[1] FIREACT: TOWARD LANGUAGE AGENT FINE-TUNING", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. The paper is easy to read and follow. Its ideas are presented straightforwardly along with illustrations.\n\n2. It simplifies the planning prompt construction process. \n\n3. The authors study the DOTS over a wide range of benchmarks.", "weaknesses": "1. The comparison between the DOTS and other methods is not fair. The DOTS's prompt contains 3 layers, however, the baselines only contain 1/2 layer. The authors should compare the baseline with 3 layers (for example compulsory use of Decomposition + POT + self-verifications at the same time in MATH) to make the comparison more fair.\n\n2. The method is not generalizable enough. It needs to SFT a new model for each large model / new atomic action in layers / new layers. (Although the authors claim that this is an advantage to make the prompt more suitable for a given model, it is still a limitation for the method.)\n\n3. The improvement of the DOTS is not significant, especially in the OOD dataset. \n\n4. The ablation study only shows the result in the IND benchmark \n\n5. The ablation study in Table 6 does not explain the inherent reason that DOTS can improve performance, it's not enough for ICLR. (For example, can a large model have a similar performance with DOTS by only adding prior knowledge in the explanation?). (see question 3,4,6 below for more details)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)