DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search

## Abstract

Abstract Enhancing the capability of large language models (LLMs) in reasoning has gained significant attention in recent years. Previous studies have demonstrated the effectiveness of various prompting strategies in aiding LLMs in reasoning (called “reasoning actions”), such as step-by-step thinking, reflecting before answering, solving with programs, and their combinations. However, these approaches often applied static, predefined reasoning actions uniformly to all questions, without considering the specific characteristics of each question or the capability of the task-solving LLM. In this paper, we propose \method , an approach enabling LLMs to reason D ynamically via O ptimal reasoning T rajectories S earch,
tailored to the specific characteristics of each question and the inherent capability of the task-solving LLM.
Our approach involves three key steps: i) defining atomic reasoning action modules that can be composed into various reasoning action trajectories; ii) searching for the optimal action trajectory for each training question through iterative exploration and evaluation for the specific task-solving LLM; and iii) using the collected optimal trajectories to train an LLM to plan for the reasoning trajectories of unseen questions. In particular, we propose two learning paradigms, i.e., fine-tuning an external LLM as a planner to guide the task-solving LLM, or directly fine-tuning the task-solving LLM with an internalized capability for reasoning actions planning.
Our experiments across eight reasoning tasks show that our method consistently outperforms static reasoning techniques and the vanilla instruction tuning approach. Further analysis reveals that our method enables LLMs to adjust their computation based on problem complexity, allocating deeper thinking and reasoning to harder problems. Our code is available at https://github.com/MurongYue/DOTS .

## 1 Introduction

Large Language Models (LLMs) have demonstrated remarkable performance in solving complex reasoning tasks (Rae et al., ; Lewkowycz et al., ; Zhong et al., ), such as math reasoning (Imani et al., ; Ahn et al., ), symbolic reasoning (Kojima et al., ), and common-sense reasoning (Krause & Stolzenburg, ; Zhao et al., ).
The dominant approaches to eliciting reasoning capability in LLMs mainly fall into two categories, i.e., instruction tuning and prompt engineering.
Instruction tuning (Wang et al., ) collects question-answer pairs about the reasoning task and employs supervised fine-tuning to optimize an LLM for better reasoning performance (Yue et al., ; Tang et al., ), with recent effort focusing on improving the scale and the quality of the fine-tuning data (Luo et al., ; Peng et al., ; Yue et al., ; ; Chan et al., ).
Prompt engineering instead aims to design better prompts to elicit the reasoning capability of an LLM without updating its parameters.
The Chain-of-Thought (CoT) approach (Wei et al., ; Kojima et al., ) prompts an LLM to answer the reasoning question step by step in natural language, and program-aided approaches (Chen et al., ; Gao et al., ) prompt the LLM to write executable code and leverage an interpreter to execute code for obtaining the final result. Besides, prompting the LLM to decompose the question before answering it (Radhakrishnan et al., ; Zhou et al., ), or to verify the solution before returning it as the final answer (Madaan et al., ), has also been proven effective in specific reasoning tasks.

However, both types of approaches suffer from a critical limitation, i.e., being unable to *dynamically decide the best reasoning strategies*. For instruction-tuning-based approaches, the fine-tuned LLMs are constrained to follow the same reasoning format of the training data (e.g., CoT (Luo et al., )) and lack the flexibility to adopt other reasoning strategies.
An example revealing a similar weakness of GPT-4o is shown in Appendix B.
On the other hand, current prompt engineering approaches assume predefined prompting strategies and uniformly apply the same to every question. However, different types of questions are better suited to different reasoning strategies (Zhao et al., ), and the effectiveness of a prompting approach also depends on the inherent capability of the task-solving LLM (e.g., LLMs pre-trained on code data are better at programming-aided reasoning). Consequently, the same prompt may not be equally effective for every question and every LLM.

Figure: Figure 1: A comparison of different paradigms of LLM reasoning. Unlike prior approaches with predefined, static reasoning actions, \methoddynamically plans for the optimal reasoning trajectory per each question and the specific task-solving LLM ($LLM_{s}$). In particular, \methodencompasses two inference setups, i.e., external planner tuning (c) and internalized planner tuning (d), depending on whether to introduce an external LLM as a planner ($LLM_{p}$) or to internalize the trajectory planning capability into the same solver LLM ($LLM_{s}$). (: tunable; : frozen)
Refer to caption: x1.png

In this paper, we present \method, an approach empowering LLMs to actively select optimal reasoning actions for given questions and the task-solving LLM (Figure 1).
We begin by constructing atomic reasoning action modules, which are composed to generate multiple potential reasoning action trajectories. Then we collect the training data by searching for an optimal (in terms of both its success rate and the number of reasoning actions needed) action trajectory through numerous explorations and evaluations. This optimal trajectory is tailored to the specific task-solving LLM.
Subsequently, we employ supervised fine-tuning to train an LLM in determining the optimal reasoning action trajectory. We implement this approach in two distinct setups: (1) For closed-source or computationally costly task-solving LLMs, we fine-tune a smaller LLM as an external planner to predict optimal reasoning actions for the task-solving LLM; (2) For open-source and small-size LLMs, we fine-tune the task-solving LLM itself to plan on the reasoning actions to take before solving the reasoning task, internalizing the autonomous planning capability directly into the LLM. This dual approach allows for flexible application across different LLM accessibility constraints.

Our experimental results demonstrate the efficacy of our proposed method in enhancing the reasoning capabilities of LLMs. We conducted extensive evaluations across multiple LLMs (GPT-4o-mini, Llama3-70B-Instruct, and Llama3-8B-instruct (Dubey et al., )) and a diverse set of reasoning tasks, encompassing in-distribution, few-shot, and out-of-distribution scenarios. The results reveal that \methodconsistently outperforms static prompt engineering techniques and vanilla instruction tuning methods across various reasoning challenges. Through a comprehensive ablation study, we validate the significance of each component in our methodology.
Moreover, our analysis of reasoning action distributions highlights that our method can adapt to the specific characteristics of reasoning questions and the inherent capability of task-solving LLMs.
We further confirm that our method incurs minimal additional financial costs. Lastly, we showcase that LLMs can naturally develop the capacity to allocate more computational resources to complex problems through a process of exploration and learning, without explicit guidance.

## 2 \method : Learning to Reason Dynamically

### 2.1 Overview

Our goal is to enable LLMs to select the most effective reasoning actions autonomously.
Denote $LLM_{s}$ as the task-solving LLM, $Q$ as the input query, $p$ as the reasoning action trajectory path, $E$ as the explanation for a trajectory, and $R$ as the reasoning process leading to the final answer $y$. Our approach encompasses two setups during the inference stage (Figure 1):

#### External Planner Tuning

This setup is designed for scenarios where the solver ($LLM_{s}$) is a closed-source LLM or is computationally costly to train. As depicted in Figure 1 (c), we train an external planner, denoted as $LLM_{p}$, to determine the optimal reasoning actions:

$$ $(E,p)=\text{$LLM_{p}$}(Q;\theta_{p})$ (1) $$

where $\theta_{p}$ is the parameters of $LLM_{p}$. We empirically found that training the planner to explain its trajectory selection ($E$) helps its learning.
Upon obtaining reasoning actions, the solver $LLM_{s}$ parameterized by $\theta_{s}$ then proceeds to generate the reasoning process $R$ and the final answer $y$:

$$ $(R,y)=\text{$LLM_{s}$}(Q,T;\theta_{s})$ (2) $$

#### Internalized Planner Tuning

This setup is designed for task-solving LLMs ($LLM_{s}$) that are open-source and small-size. In this case, we propose to *internalize* the trajectory planning capability into the task-solving LLM by training it to simultaneously learn to plan and learn to perform the reasoning task.
As shown in Figure 1 (d), the final answer $y$ is obtained by:

$$ $(E,p,R,y)=\text{$LLM_{s}$}(Q;\theta_{s})$ (3) $$

An overview of \method’s learning process is presented in Figure 2, consisting of three key steps: (i) Defining atomic reasoning modules: We define several atomic reasoning modules, each representing a distinct reasoning action, (ii) Searching for optimal action trajectories: We conduct explorations and evaluation of various reasoning paths to identify optimal reasoning actions for questions in the training data, and (iii) Fine-tuning LLMs to plan for optimal reasoning trajectories:
We fine-tune LLMs to autonomously plan the reasoning action trajectory under the two aforementioned setups.
In what follows, we elaborate on each step.

Figure: Figure 2: The training process of \method, including searching for the optimal reasoning trajectories for questions in the training set and fine-tuning the internalized/external planner LLM.
Refer to caption: x2.png

**Table 1: Prompt engineering methods with different reasoning actions. Our method could dynamically select reasoning actions among all of them.**
| Prompting Method | Analysis Layer | Solution Layer | Verification Layer |  |  |
| --- | --- | --- | --- | --- | --- |
| Rewriting | Decomposition | NL | Program | Verification |  |
| CoT (Wei et al., ) | ✗ | ✗ | ✓ | ✗ | ✗ |
| PoT (Chen et al., ) | ✗ | ✗ | ✗ | ✓ | ✗ |
| LTM (Zhou et al., ) | ✓ | ✗ | ✓ | ✗ | ✗ |
| R&R (Deng et al., ) | ✗ | ✓ | ✓ | ✗ | ✗ |
| Self-Refine (Madaan et al., ) | ✗ | ✗ | ✓ | ✓ | ✓ |
| Self-Verification (Weng et al., ) | ✗ | ✗ | ✓ | ✗ | ✓ |
| PromptAgent (Wang et al., ) | ✓ | ✓ | ✓ | ✗ | ✗ |
| \method(ours) | ✓ | ✓ | ✓ | ✓ | ✓ |

### 2.2 Defining Atomic Reasoning Actions Modules

Prior studies have validated the effectiveness of various reasoning strategies (Table 1). We build on top of them and categorize the existing strategies as reasoning actions across three layers:

#### Analysis Layer

Actions in this layer enable the LLM to analyze the input query before attempting to solve it, including
(1) Query rewriting: reformulating the query to enhance comprehension, and (2) Query decomposition: breaking down the initial question into multiple, more manageable sub-questions. We denote the action taken in this layer as $A_{a}$.

#### Solution Layer

Actions in this layer consider variants in the reasoning format. Prior works showed that different queries are better solved following different reasoning processes (Zhao et al., ). In our work, we consider the most commonly adopted formats, i.e.,
(1) CoT: solving the question step-by-step in natural language, and (2) PoT: addressing the question through code generation. We denote the action chosen in this layer as $A_{t}$.

#### Verification Layer

Finally, the verification layer is responsible for checking the correctness of the proposed solution.
It is particularly useful for problems where verification is significantly easier than solving the problem itself, e.g., the Game of 24 (Yao et al., ).
Therefore, we set a Self-Verification action module in this layer.
If this module determines that the reasoning process from the solution layer is incorrect, the LLM will revert to the solution layer to reattempt to solve the problem.
During this reattempt, the LLM is provided with both the initial answer and the feedback from the verifier explaining why the initial answer was incorrect.
The process continues until the verifier confirms that the answer is correct or the pre-defined maximum number of iterations for self-verification is reached.
We denote the action taken in this layer as $A_{v}$.

We observe that introducing excessive reasoning actions can lead to increased latency, and even sometimes result in incorrect answers. To mitigate this issue, we introduce an Empty action in both the analysis and the verification layers, allowing the LLM to bypass these two layers when dealing with simple questions. Detailed prompts for each module are provided in Appendix D.

### 2.3 Searching for Optimal Reasoning Action Trajectories

To teach the external/internalized planner to plan for the optimal reasoning trajectory, we start by constructing training data containing questions and their optimal action trajectories for the specific task-solving LLM. We obtain this by iteratively searching all possible reasoning trajectories for each question, including exploring the current paths and pruning paths that are unlikely to be optimal. The task-solving LLM is used during this search process to generate answers to make the reasoning trajectory align with their intrinsic ability to perform different reasoning actions effectively.

Figure: Algorithm 1 Searching for the Optimal Reasoning Action Trajectory

This searching process is shown in Algorithm 1. Given the query and ground-truth answer sourced from the training data, the process runs iteratively. In each iteration, the algorithm considers either the full set of candidate trajectories (for iteration $k=1$) or the current best subset (for iteration $k>1$). Each candidate trajectory is executed for $N_{eval}$ times with a non-zero temperature to obtain a more reliable evaluation of its success rate. We then sort the current subset of trajectories by its success rate accumulated from the past $k$ iterations and then the trajectory length to encourage a shorter trajectory (which is thus computationally more efficient).
Only the top $N_{k}$ candidates will be retained and rolled over to the next iteration of the assessment. In practice, we opt for a smaller $N_{eval}$ and run the search for multiple iterations, as opposed to finishing the search with a larger $N_{eval}$ in one iteration, as the latter incurs a much larger cost ($N_{eval}\times|\mathcal{P}_{0}|$ with a large $N_{eval}$ vs. $N_{eval}\times(|\mathcal{P}_{0}|+N_{1}+\cdots+N_{K-1})$ with a small $N_{eval}$ in our algorithm).

In the process of validating various trajectories for each question, we exclude instances where *any* trajectory solves the query or *all* fail to do so, as they do not contribute to the planner LLM’s trajectory planning learning.
After identifying the best reasoning trajectory, we leverage GPT-4o to verbally explain why the trajectory is optimal.
Our prompt is shown in Appendix D.
This process is applied to all instances in the training data, giving us tuples of query $Q$, ground true answer $y^{*}$, optimal trajectory $p^{*}$, and its explanation $E$.
For internalized planner tuning, we collect the reasoning process $R$ when running the solver $LLM_{s}$ following the optimal trajectory $p^{*}$.

### 2.4 Learning to Plan for Optimal Reasoning Trajectories

Having obtained the optimal trajectories, we then use supervised fine-tuning with cross-entropy loss to train the planner LLM to predict optimal trajectories for input questions and the specific solver LLM.
For external planner tuning, a lightweight $LLM_{p}$ is trained to predict a concatenation of the explanation and the optimal trajectory (Eq 1); for internalized planner tuning, the solver $LLM_{s}$ is trained to predict the explanation, the optimal trajectory, the reasoning process collected from $LLM_{s}$ itself, and the true answer $y^{*}$ (Eq 3).

## 3 Experiment

### 3.1 Experimental Setup

**Table 2: Overview of our evaluation datasets.**
| Dataset | Distribution | Task Type |
| --- | --- | --- |
| MATH | In Distribution | math |
| BBH | Few-shot | mixture |
| Game of 24 | numerical |  |
| TheoremQA | scientific |  |
| Deepmind Math | Out-of-Distribution | math |
| MMLU-pro | scientific |  |
| StrategyQA | common sense |  |
| DROP | multi-hop |  |

#### Datasets

We evaluate the effectiveness of our method across multiple datasets and various reasoning tasks.
Based on the distribution of the training and testing data, we divide the evaluation into three settings as shown in Table 2:
In-distribution setting evaluates the model that resembles what it has seen during training.
Few-shot setting aims to evaluate whether our proposed method can effectively learn from a small amount of labeled data. In the real world, it is often difficult to obtain large amounts of in-domain training data across different tasks, but a small number of cases can be annotated.
Out-of-distribution (OOD) setting further evaluates whether the model can handle scenarios it was not explicitly trained for, testing its ability to generalize beyond the training set.
For the training data, we use the MATH (Hendrycks et al., ) training set. For the few-shot learning, we select 4 examples from each category of BBH (Suzgun et al., ) as it is composed of 27 diverse tasks,(^1^11[https://huggingface.co/datasets/lukaemon/bbh](https://huggingface.co/datasets/lukaemon/bbh)) resulting in 108 examples in total, 4 examples from Game of 24 (Yao et al., ), and 4 examples from TheoremQA (Chen et al., ) datasets.
For the test data, we evaluate the model on the test set of the MATH dataset for the in-distribution setting and on the test sets or hold-out sets of BBH, Game of 24, and TheoremQA for the few-shot learning setting. For the OOD evaluation, we test each approach’s generalization ability on Deepmind Math (Saxton et al., ), MMLU-pro (Wang et al., ), strategyQA (Geva et al., ), and DROP (Dua et al., ).
All evaluations (unless specified) were conducted when prompting the solver LLMs in zero shot.
For answer evaluation, we use the simple-eval(^2^22[https://github.com/openai/simple-evals](https://github.com/openai/simple-evals).) for MATH, a standard evaluation for Game of 24 (Yao et al., ), and exact string matching for the others.

#### Training Setup

For external planner tuning, we utilize Llama-3-8B-Instruct as our planner and GPT-4o-mini and Llama-70B-Instruct as task-solving LLMs.
Experiments of internalized planner tuning were conducted with Llama-3-8B-Instruct. For more details, refer to Appendix A.

### 3.2 Baselines

We include the following highly related baselines in our experiments.
(1) CoT (Wei et al., ) prompts an LLM to answer step-by-step; (2) PoT (Chen et al., ) prompts an LLM to generate Python code and execute the code to get the final answer; (3) Least-to-most (LTM) (Zhou et al., ) prompts an LLM to first decompose the question into multiple sub-questions before solving it;
(4) Self-refine (Madaan et al., ) prompts an LLM to generate the answer and verify and refine the answer by the LLM itself. Madaan et al. () used PoT in solving math questions, therefore we follow their setting to use PoT in generating the initial answer;
(5) PromptAgent (PA) (Wang et al., ) searches for a better prompt for the specific task based on its training data;
this baseline is implemented with the default hyperparameter setting;
and (6) Vanilla Supervised Fine-Tuning
(Vanilla SFT) uses GPT-4o to generate the CoT reasoning process for questions in the training datasets and then fine-tune the solver LLM to predict the generated reasoning process and the ground-truth answer; this baseline is fine-tuned using the same hyperparameter setting as our internalized planner tuning.
The training data for PA, Vanilla SFT, and \methodare from the same source.

### 3.3 External Planner Tuning Results

**Table 3: Accuracy (%) of the external planner tuning on in-distribution and few-shot datasets. The reasoning format $\mathcal{L}$ represents language, and $\mathcal{P}$ means program.**
| Method | Tuning | Reasoning | MATH | BBH | Game of 24 | TheoremQA | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Format |  |  |  |  |  |  |  |
| External Planner: Llama-3-8B-Instruct; Solver: Llama-3-70B-Instruct |  |  |  |  |  |  |  |
| CoT | ✗ | $\mathcal{L}$ | 50.4 | 72.7 | 27.5 | 27.4 | 44.5 |
| LTM | ✗ | $\mathcal{L}$ | 50.1 | 73.8 | 24.9 | 28.8 | 44.4 |
| PA | ✓ | $\mathcal{L}$ | 52.5 | 72.9 | 26.8 | 28.8 | 45.3 |
| PoT | ✗ | $\mathcal{P}$ | 54.7 | 65.8 | 63.9 | 31.1 | 53.9 |
| Self-refine | ✗ | $\mathcal{L}$, $\mathcal{P}$ | 55.9 | 71.4 | 68.3 | 30.8 | 56.6 |
| \method: External | ✓ | $\mathcal{L}$, $\mathcal{P}$ | 57.7 | 77.3 | 67.7 | 31.2 | 58.5 |
| External Planner: Llama-3-8B-Instruct; Solver: GPT4o-mini |  |  |  |  |  |  |  |
| CoT | ✗ | $\mathcal{L}$ | 70.2 | 80.3 | 27.7 | 38.9 | 54.2 |
| LTM | ✗ | $\mathcal{L}$ | 72.2 | 79.4 | 25.5 | 36.4 | 53.3 |
| PA | ✓ | $\mathcal{L}$ | 73.5 | 81.1 | 26.7 | 38.9 | 55.1 |
| PoT | ✗ | $\mathcal{P}$ | 67.2 | 73.9 | 61.4 | 35.8 | 59.6 |
| Self-refine | ✗ | $\mathcal{L}$, $\mathcal{P}$ | 73.7 | 74.8 | 68.7 | 34.6 | 63.0 |
| \method: External | ✓ | $\mathcal{L}$, $\mathcal{P}$ | 75.4 | 84.2 | 65.2 | 41.4 | 66.5 |

Table 3 presents the results of using the external planner, which suggest that:

#### External planner tuning outperforms other methods on the in-domain task

Our method achieves 57.7% accuracy with Llama-3-70b-Instruct and 75.4% accuracy with GPT-4o-mini on MATH, achieving significant improvement than baselines.
This suggests that \methodis robust across different LLMs and it can significantly enhance the LLM’s zero-shot reasoning ability.
The improvement from \methodremains consistent as the solver LLM’s capabilities increase, indicating \methodhas a long-term value even as LLMs continue to improve rapidly.

#### The external planner can learn the appropriate action trajectory with only a few training examples.

On the BBH, \methodachieves improvements of 3.5% and 3.1% over the best static methods when using Llama-3-70B-Instruct and GPT-4o-mini, respectively. In the Game of 24 and TheoremQA, \methodalso shows slight improvements or performs similarly to the best static method.
This indicates that even a small number of cases can help the LLM learn the optimal strategy for the given task.
Besides, \methoddemonstrates greater stability across various datasets.
Our flexible action trajectory selection demonstrates its advantages on datasets requiring diverse reasoning actions, such as BBH as shown in Appendix C.
Conversely, the Game of 24 features a uniform question type, where the predefined static method self-refine is sufficient.
While the self-refine excels on Game of 24, it significantly lags behind on other datasets.
This reflects the external planner’s ability to effectively select the appropriate action trajectory, leading to more robust performance even across tasks with varying reasoning demands.

### 3.4 Internalized Planner Tuning Results

Table 4 presents the results of our internalized planner tuning, where we observed:

#### Internalized planner tuning demonstrates superior performance

outperforms existing methods on average, including prompt engineering methods and vanilla SFT.
Notably, our approach surpasses self-refine in the Game of 24, a different observation than the experiments with an external planner (Table 3).
We attribute this performance boost to our joint optimization of the trajectory planning and problem-solving processes. Unlike external planner tuning which only updates the external planner ($LLM_{p}$), internalized planner tuning enables the task-solving LLM to simultaneously learn trajectory planning and accurate reasoning process generation.
This highlights that the internalized planner tuning effectively further enhances performance.

#### Searching for the optimal reasoning action trajectory helps enhance the utilization of training data

Compared to vanilla SFT, our method consistently shows performance improvements across all datasets, notably achieving an 8.7% increase on BBH. This suggests that, instead of training with a question and step-by-step reasoning process pair, our approach of searching for an optimal action trajectory and generating the corresponding reasoning process to construct training data is superior.
This finding indicates that our search methodology could effectively enhance the utilization of training data for reasoning tasks without the need for additional human annotations.

**Table 4: Internal planner tuning performance on in-distribution and few-shot datasets.**
| Method | Tuning | Reasoning | MATH | BBH | Game of 24 | TheoremQA | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| format |  |  |  |  |  |  |  |
| Solver: Llama-3-8B-Instruct |  |  |  |  |  |  |  |
| CoT | ✗ | $\mathcal{L}$ | 29.6 | 48.9 | 12.7 | 14.8 | 26.5 |
| LTM | ✗ | $\mathcal{L}$ | 29.5 | 50.3 | 14.4 | 15.2 | 27.4 |
| PA | ✓ | $\mathcal{L}$ | 31.0 | 47.2 | 11.8 | 15.1 | 26.3 |
| PoT | ✗ | $\mathcal{P}$ | 25.3 | 44.6 | 16.8 | 16.7 | 25.9 |
| Self-refine | ✗ | $\mathcal{L}$, $\mathcal{P}$ | 28.7 | 46.6 | 17.0 | 15.3 | 30.1 |
| Vanilla SFT | ✓ | $\mathcal{L}$ | 33.9 | 61.0 | 18.5 | 14.8 | 33.6 |
| \method: Internalized | ✓ | $\mathcal{L}$, $\mathcal{P}$ | 34.4 | 69.7 | 21.9 | 16.1 | 35.5 |

### 3.5 Out-of-Distribution Experimental Results

**Table 5: Accuracy (%) on out-of-distribution (OOD) tasks.**
| Method | DeepMind-Math | MMLU-pro | StrategyQA | DROP | Average |
| --- | --- | --- | --- | --- | --- |
| External Planner: Finetuned Llama-3-8B-Instruct; Solver: Llama-3-70B-Instruct |  |  |  |  |  |
| CoT | 54.6 | 60.6 | 81.3 | 66.1 | 65.6 |
| LTM | 55.6 | 60.9 | 81.9 | 64.3 | 65.6 |
| PA | 58.1 | 54.2 | 80.3 | 58.7 | 62.8 |
| PoT | 73.0 | 57.3 | 74.8 | 62.8 | 66.9 |
| Self-refine | 73.9 | 59.5 | 77.8 | 64.8 | 69.0 |
| \method: External | 74.1 | 59.4 | 80.3 | 66.3 | 70.0 |
| External Planner: Finetuned Llama-3-8B-Instruct; Solver: GPT4o-mini |  |  |  |  |  |
| CoT | 80.2 | 61.7 | 78.8 | 65.8 | 71.6 |
| LTM | 80.6 | 61.4 | 80.9 | 64.5 | 71.8 |
| PA | 82.2 | 48.1 | 78.3 | 67.0 | 68.9 |
| PoT | 87.7 | 57.1 | 77.9 | 72.4 | 73.7 |
| Self-refine | 85.9 | 58.3 | 77.2 | 72.3 | 73.4 |
| \method: External | 87.6 | 61.5 | 78.8 | 73.8 | 75.4 |
| Solver: Finetuned Llama-3-8B-Instruct |  |  |  |  |  |
| CoT | 28.3 | 37.2 | 72.7 | 52.9 | 47.8 |
| LTM | 30.9 | 38.6 | 70.7 | 55.2 | 48.9 |
| PA | 29.3 | 34.5 | 69.7 | 51.6 | 46.3 |
| PoT | 48.1 | 37.3 | 63.9 | 44.6 | 48.5 |
| Self-refine | 44.9 | 33.1 | 65.3 | 47.1 | 47.6 |
| Vanilla SFT | 39.6 | 40.3 | 71.8 | 49.0 | 50.2 |
| \method: Internalized | 55.3 | 39.7 | 68.2 | 48.8 | 53.0 |

#### Our method consistently generalizes well across diverse OOD challenges

As shown in Table 5, \methodmaintains high accuracy across different datasets and models. In contrast, static methods often fluctuate significantly in performance. For instance, despite static methods like CoT showing a slight advantage on MMLU-Pro and StrategyQA over \methodusing the Llama-3-70B-Instruct model, they experience a sharp decline on DeepMind Math.
This pattern of fluctuations can be observed in other methods as well, where some excel on individual tasks but fail to maintain strong performance.
In contrast, \methodcontinues to deliver consistently high accuracy across various models and datasets. The stability of our method is attributed to its ability to dynamically select appropriate reasoning trajectories. The results indicate that \methodis better suited to meet the demands of diverse tasks, demonstrating stronger robustness and generalization, making it a more reliable and adaptable approach for handling a wide variety of OOD challenges.

### 3.6 Ablation Study

**Table 6: Ablation Study**
|  | MATH | BBH | Game24 | TheoremQA | Average |
| --- | --- | --- | --- | --- | --- |
| External Planner: Llama-3-8B-Instruct; Solver: GPT-4o-mini |  |  |  |  |  |
| \method: External | 75.4 | 84.2 | 65.2 | 42.4 | 66.8 |
| -w/o Searching | 69.2 | 78.6 | 28.9 | 40.2 | 54.2 |
| -w/o Explanation | 68.2 | 81.3 | 57.4 | 36.4 | 60.8 |
| Internalized Planner & Solver: Llama-3-8B-Instruct |  |  |  |  |  |
| \method: Internalized | 34.4 | 69.7 | 21.9 | 16.1 | 35.5 |
| -w/o Searching | 31.4 | 55.8 | 19.6 | 15.1 | 30.5 |
| -w/o Explanation | 33.8 | 65.8 | 18.6 | 15.7 | 33.4 |

In this section, we perform the ablation study and assess the effectiveness of each component of our method: (1) Without Searching: To demonstrate the effectiveness of searching for the optimal action trajectory, we test the performance of the LLM tuned with a randomly selected action trajectory; (2) Without Explanation: To understand if training the planner to generate an explanation for the optimal reasoning trajectory is helpful, we test \method’s performance when the planner is trained to predict the trajectory without explanation.

The results in Table 6 indicate that both optimal trajectory searching and explanation generation are crucial in \method.
For example, in the Game of 24, the planner trained without searching for the optimal trajectory did not consistently select the PoT action (which was considered the most effective for this task) in its trajectory.
Additionally, we observe that without explanations, the planner’s ability to predict optimal trajectories becomes less reliable. Incorporating explanations effectively guides the planner to learn to predict suitable action trajectories for the given questions.

### 3.7 Optimal Trajectory Analysis for Different Tasks

Table 7 shows the distribution of actions selected in the optimal trajectories by our planner on the MATH test set. The distribution suggests two key findings:

#### \method adapts to the characteristics of specific questions

In mathematics, number theory problems are more suitable to be solved with programs, so the proportion of PoT is higher, while geometry problems are not easily represented and solved with naive Python code; as a result, our planner mainly uses CoT for such problems. This indicates that \methodtailors its action selection based on the unique characteristics of each problem type.

#### \method adapts to the capability of specific task-solving LLMs

As shown in Table 3, on the MATH dataset, GPT-4o-mini performs better using CoT for problem-solving, whereas Llama3-70B-instruct performs better using PoT.
When GPT-4o-mini is the task-solving LLM, our fine-tuned planner selects a higher proportion of CoT actions; when Llama3-70B-Instruct is used, PoT actions dominate. This suggests that our planner is not only aware of the problem type but also adapts the reasoning action trajectory prediction based on the capabilities of the task-solving LLM.

Furthermore, we observe that question rewriting and decomposition were selected with a low frequency.
This is likely because the MATH dataset consists of precise problems that do not benefit from rewriting.
Additionally, given the strong reasoning abilities of Llama3-70B-Instruct and GPT-4o-mini, their CoT process inherently includes task decomposition, reducing the need for further planning interventions.

**Table 7: Planning action distributions of \methodover three different layers on the MATH test set.**
| Sub-tasks on MATH | Accuracy (%) | Analysis Layer | Solution | Verification |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Rewr. | Deco. | Empty | CoT | PoT | Veri. | Empty |  |  |
| External Planner: Llama-3-8B-Instruct; Solver: GPT-4o-mini |  |  |  |  |  |  |  |  |
| Algebra | 92.1 | 0.03 | 0.05 | 0.92 | 0.90 | 0.10 | 0.29 | 0.71 |
| Prealgebra | 88.6 | 0.03 | 0.01 | 0.96 | 0.79 | 0.31 | 0.21 | 0.79 |
| Number Theory | 81.8 | 0.01 | 0.01 | 0.98 | 0.43 | 0.57 | 0.15 | 0.85 |
| Counting and Probability | 76.8 | 0.08 | 0.06 | 0.84 | 0.78 | 0.32 | 0.30 | 0.70 |
| Geometry | 61.8 | 0.03 | 0.01 | 0.96 | 0.95 | 0.05 | 0.06 | 0.94 |
| Intermediate Algebra | 57.1 | 0.05 | 0.02 | 0.93 | 0.85 | 0.15 | 0.44 | 0.56 |
| Precalculus | 52.6 | 0.06 | 0.02 | 0.92 | 0.95 | 0.05 | 0.46 | 0.54 |
| External Planner: Llama-3-8B-Instruct; Solver: Llama-3-70B-Instruct |  |  |  |  |  |  |  |  |
| Algebra | 74.9 | 0.03 | 0.04 | 0.93 | 0.77 | 0.23 | 0.12 | 0.88 |
| Prealgebra | 74.5 | 0.02 | 0.03 | 0.95 | 0.57 | 0.43 | 0.10 | 0.90 |
| Number Theory | 69.9 | 0.01 | 0.01 | 0.98 | 0.32 | 0.68 | 0.13 | 0.87 |
| Counting and Probability | 55.4 | 0.04 | 0.02 | 0.94 | 0.59 | 0.41 | 0.11 | 0.89 |
| Geometry | 39.6 | 0.05 | 0.01 | 0.94 | 0.76 | 0.24 | 0.18 | 0.82 |
| Precalculus | 36.9 | 0.07 | 0.03 | 0.90 | 0.78 | 0.22 | 0.28 | 0.76 |
| Intermediate Algebra | 34.6 | 0.03 | 0.01 | 0.96 | 0.72 | 0.28 | 0.20 | 0.80 |

### 3.8 Additional Analyses

#### Few-shot In-context Learning Setting

**Table 8: External planner tuning under the few-shot setting with GPT-4o-mini as the solver.**
| Method | MATH | BBH | TheoremQA | Average |
| --- | --- | --- | --- | --- |
| CoT | 72.3 | 84.2 | 38.2 | 64.9 |
| LTM | 72.7 | 83.4 | 37.3 | 64.5 |
| PA | 71.3 | 83.3 | 38.7 | 64.4 |
| PoT | 69.8 | 82.1 | 36.4 | 62.8 |
| Self-refine | 73.2 | 83.1 | 35.4 | 63.9 |
| \method | 75.4 | 86.1 | 39.9 | 67.1 |

Our main results report the performance with zero-shot evaluation. In cases where reasoning tasks are known in advance, a common approach to leveraging training data and improving the performance of closed-source LLMs is few-shot in-context learning (ICL), where training examples are incorporated directly into the context.
Our external planner tuning can also be utilized in this scenario seamlessly.
Specifically, we can first construct few-shot ICL prompts for each potential reasoning action trajectory. Once the external planner selects the appropriate reasoning actions, the corresponding few-shot prompt will be chosen and applied.
We evaluate the external planner tuning setup of \method, with Llama-3-8B-Instruct being the external planner and GPT-4o-mini being the solver LLM, in this setting.
We compare our approach with the same baselines similarly implemented in the few-shot ICL setting, where we randomly selected
8 examples from MATH, 4 examples from each category of BBH, and 4 examples from TheoremQA to form the prompt.(^3^33We excluded the “Game of 24” task because knowing the task in advance enables it to be solved with a straightforward program.) All few-shot demonstrations were generated by GPT-4o and manually verified for quality.

As shown in the Table 8, \methodcontinues to outperform baseline models. Interestingly, compared to Table 3, which presents the zero-shot results, adding few-shot demonstrations to static prompting methods does not lead to consistent improvement, except on the BBH dataset. This indicates that simply expanding the context with additional demonstrations does not always serve as an effective way to leverage available training data.
In contrast, our method demonstrates its superior ability to effectively utilize the training data.

#### How efficient is \method ?

**Table 9: Avg. number of output tokens for each method (solver: Llama-3-8B-Instruct).**
| Method | Avg. # of |
| --- | --- |
| Output Tokens |  |
| CoT (Wei et al., ) | 263.6 |
| LTM (Zhou et al., ) | 436.4 |
| Self-refine (Madaan et al., ) | 527.6 |
| \method: Internalized | 409.1 |

We compare the cost efficiency, measured by the average output token count, of each method (based on Llama-3-8B-Instruct) in Table 9.
The result shows that \methodconsumes fewer tokens on average than other advanced approaches and only more than CoT.
Advanced prompt engineering methods often introduce supplementary text to facilitate reasoning. However, not all questions require this additional context to the same extent. By constructing training data via searching, our goal is to optimize the balance between minimizing extraneous steps and maintaining a high success rate, thereby reducing unnecessary output tokens. Our method avoids redundant reasoning actions, resulting in a more efficient system.

#### Do we need more reasoning steps for difficult questions?

Figure: Figure 3: Average reasoning trajectory length per difficulty level on MATH for \method(solver: GPT-4o-mini; External planner: Llama3-8B-Instruct).
Refer to caption: x4.png

Recent research suggests that LLMs can better solve difficult questions by increasing the thinking time in the inference stage (Brown et al., ; OpenAI, ).
In our study, we explore the relationship between question difficulty and the average reasoning action trajectory length.
The trajectory length is determined by assigning a value of 0 0 to the EMPTY module and $1$ to all other actions, while the question difficulty is derived from annotated levels on the MATH dataset.
Figure 3 presents that harder problems demand more computational steps, resulting in longer reasoning trajectories. Case analyses further reveal that our planner increases the proportion of verification steps as problem difficulty rises. This highlights an exciting fact — LLMs can learn to employ more reasoning steps for challenging problems through exploration, without requiring explicit expert guidance.

## 4 Related Work

#### Prompt engineering for LLM reasoning

LLMs have demonstrated remarkable proficiency in solving complex reasoning tasks (Rae et al., ; Lewkowycz et al., ; Zhong et al., ). The Chain-of-Thought (CoT) approach, introduced by Wei et al. (), significantly improves performance on reasoning problems by prompting LLMs to think step-by-step, thereby activating their inherent reasoning capabilities (Madaan & Yazdanbakhsh, ). To further enhance LLMs’ capabilities in mathematical and symbolic reasoning, Chen et al. () and Gao et al. () proposed the Program-of-Thought prompting method, where code is used as an intermediate reasoning step.
Advanced prompt engineering methods, such as question decomposition (Zhou et al., ) and self-verification (Madaan et al., ), have also proven effective in improving reasoning performance. Additionally, recent approaches have incorporated automatic prompt optimization based on training data. For instance, Wang et al. () refines prompts by analyzing error cases, and self-discovery (Zhou et al., ) utilizes modular reasoning components to construct the task-adaptive prompt.
However, these automated prompt optimization techniques still produce static prompts for all instances.
Recently, Srivastava et al. () proposed the instance-level prompt optimization via LLM self-refining while it is still a passive expert-designed workflow and lacks the explorations and evaluations to guide the LLM to better actively adapt to the question and LLM capability.
In our method, we internalize the reasoning action selection capability into the LLM itself without an expert-designed workflow, allowing it to autonomously fit both the characteristics of questions and the inherent capability of task-solving LLM.

#### Searching for boosting LLM reasoning

Recent research suggests that incorporating searching mechanisms can significantly enhance LLM reasoning. In the inference process, Tree-of-Thought (ToT) (Yao et al., ) and Graph-of-Thought (GoT) (Besta et al., ) have been proposed to search and investigate different reasoning paths, either by leveraging the LLM itself (Yao et al., ) or designing heuristic functions (Hao et al., ) as the signal to evaluate each step. More recently, Monte Carlo Tree Search (MCTS) has been introduced to assist the LLM in learning how to evaluate each step (Qi et al., ; Xie et al., ). The searching mechanism can also be used in training to collect training instances for improving LLM reasoning (Luo et al., ). However, all these searching methods treat each “CoT reasoning step” as the atomic component or step in searching, while we choose each reasoning action as the atomic component in our case.

## 5 Conclusion

In this paper, we introduce \method, a method that enables LLMs to autonomously think about appropriate reasoning actions before answering questions. By defining atomic reasoning action modules, searching for optimal action trajectories, and training LLMs to plan for reasoning questions, we enable LLMs to dynamically adapt to specific questions and their inherent capability. The flexibility of our two learning paradigms, i.e., external and internalized planner tuning, further highlights the adaptability of our method to different LLMs. Our experimental results show the effectiveness of \method, revealing the promise of harnessing explorations and evaluations to turn LLMs into planners for better reasoning.

## Appendix A Training Implementation

In our optimal trajectory search, we set the number of iterations ($K$) to 2 and the number of evaluation times ($N_{eval}$) to 4. The number of paths retained $N_{1}$ is set to 8 and $N_{2}$ to 3. Throughout the search, we maintain a sampling temperature of 0.4.
Searching on the training datasets eventually yields 1722 for GPT-4o-mini, 1624 for Llama-3-70B-Instruct, and 2140 for Llama-3-8B-Instruct training examples for planner tuning, respectively.
We up-sample few-shot examples by a factor of 8 and fine-tune the planner LLM using the LitGPT library (AI, ), applying a learning rate of 2e-5, a global batch size of 64, a maximum sequence length of 4096, and training epoch of 4.

## Appendix B Case Study

In this section, $Q_{I}$ is a numerical reasoning problem that can be solved with Python code easily, so our tuned LLM selected PoT to solve it. $Q_{II}$ is a question widely tested in the community where GPT-4o even tends to make mistakes, but after tuning with searched trajectory data, the LLM actively chose to use code to solve the problem.

## Appendix C Sub-Task Performance of BBH

**Table 10: Results of BBH sub-tasks.**
| Sub Task | CoT | PoT | \method |
| --- | --- | --- | --- |
| object counting | 94.4 | 98.0 | 98.0 |
| temporal sequences | 96.8 | 83.6 | 97.2 |
| tracking shuffled objects five objects | 95.2 | 78.8 | 96.4 |
| word sorting | 50.8 | 98.0 | 98.0 |
| salient translation error detection | 61.6 | 59.2 | 62.6 |
| penguins in a table | 71.2 | 54.8 | 73.3 |
| snarks | 83.1 | 69.1 | 83.3 |
| ruin names | 81.2 | 80.2 | 81.2 |
| web of lies | 91.2 | 59.2 | 92.0 |
| navigate | 96.4 | 90.4 | 96.8 |
| date understanding | 80.8 | 52.8 | 76.4 |
| hyperbaton | 92.0 | 67.6 | 91.0 |
| dyck languages | 38.0 | 42.4 | 54.4 |
| tracking shuffled objects three objects | 99.2 | 89.2 | 99.2 |
| formal fallacies | 82.0 | 74.4 | 82.0 |
| tracking shuffled objects seven objects | 91.2 | 73.2 | 92.8 |
| causal judgement | 62.6 | 62.6 | 62.0 |
| sports understanding | 85.6 | 74.8 | 87.6 |
| logical deduction five objects | 85.6 | 80.4 | 88.8 |
| movie recommendation | 62.0 | 56.0 | 62.8 |
| logical deduction three objects | 99.2 | 96.4 | 99.2 |
| multistep arithmetic two | 98.0 | 100.0 | 100.0 |
| boolean expressions | 99.2 | 96.8 | 99.2 |
| geometric shapes | 56.4 | 65.6 | 78.8 |
| disambiguation qa | 42.8 | 40.8 | 45.6 |
| logical deduction seven objects | 76.8 | 71.6 | 82.0 |
| reasoning about colored objects | 87.6 | 66.0 | 87.6 |
| Average | 80.3 | 73.9 | 84.2 |

Table 10 shows the results of BBH sub-tasks of GPT-4o-mini as task-solving LLM and our tuned Llama-3-8B-instruct as the planner. From the table, we can see that our planner has learned to select appropriate actions based on the task. For instance, in the word sorting task, our model consistently uses code to solve the problem. Additionally, for the Dyck languages task, our method outperforms both CoT and PoT. This is because the task is easy to verify, and our model proactively requests GPT-4o-mini to verify the answer, thereby improving performance. The only exception is the date understanding task. Upon analysis, we found that while code could solve this type of problem using Python’s “datetime” library, it often fails to follow the required output format. This leads to lower accuracy in our method’s prediction when choosing writing a program as the reasoning action.

## Appendix D Prompts Used in Experiments

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1. The authors propose a dynamic reasoning method which can enable the model to decide the appropriate atomic actions based on the characteristics of the input question.
2. The authors conduct comprehensive experiments to prove the effectiveness of the proposed method, containing in distribution, few-shot, and out-of-distribution settings.
3. The proposed method can be used on both open-source and close-source models.
summary: This paper proposes DOTS, an approach enabling LLMs to reason dynamically via optimal reasoning trajectories search, tailored to the specific characteristics of each question. DOTS first defines atomic reasoning action modules, then searches for the optimal action trajectory for each training question through iterative exploration and evaluation for the specific task-solving LLM, and uses the collected trajectories to train a model to plan for the reasoning trajectories of unseen questions.
weaknesses: 1. The method proposed in this paper does not show significant improvement on out-of-distribution (OOD) tasks, and it incurs additional computational overhead compared to prompt engineering methods.
2. The baseline for Vanilla SFT only used the training data from CoT. I believe it should be compared with baselines from other reasoning formats to demonstrate the effectiveness of the proposed method, such as using the program reasoning format and mixed training data from CoT and Program.
3. The experiments demonstrating the effectiveness of the Search section are compared with randomly selected reasoning paths. I am curious about how the results would compare if we contrasted it with reasoning paths generated by combining non-empty actions from each layer.
4. Using PoT prompts on non-code models may yield suboptimal results. Would combining PoT with code models improve the results of the PoT baseline?

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Q1. How many trajectories are required to improve performance substantially at FT-time? A more thorough discussion would be useful.

Q2. How does the proposed approach scale to more complex problems where process rewards would be beneficial?
soundness: 3
strengths: 1. Adaptability of the method: The method can be used either on the planner or on the task solver.

2. The method shows lower cost than non-CoT methods.

3. Improved Reasoning ability: The paper demonstrates improved reasoning ability by allowing for dynamic selection for a given question, outperforming static methods as well as self refinement (most but not all scenarios).
summary: The paper proposes a new way to learn reasoning through two learning paradigms. The paper (DOTS) proposes both a way to FT an planner LLM and in directly FT-ing the task solver.
weaknesses: 1. The proposed method is complex and involves several steps. It is not clear if the complexity is warranted and ablation studies on the various aspects can help.

2. The paper does not explore decomposition or tool use which would be crucial for complex tasks.

3. The paper glosses over how the atomic trajectories are collected.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: + (Type) Equation 2: $T$ has not been defined.
+ (Clarification) For the "few-shot" settings, do you use the planner trained on MATH dataset? If so, they should be "near-distribution + few-shot", since those "few-shot" datasets are math-related.
+ Since DOTS has marginal performance on OOD tasks (despite DOTS need additional training), what's the performance on near-distribution (e.g. the few-shot datasets in the paper) but zero-shot settings?
+ In Table 3, 4, 5, on a few datasets DOTS is not the best among methods. Does it mean in some datasets, it's better to choose one single baseline method instead of choosing modules dynamically as in DOTS?
+ DOTS select one reasoning method (one choice for each module). Would it be better to attempt multiple choices and choose the best one (and this would align more with "trajectory search")?
+ (Minor) It would be beneficial to also include the computing efficiency of external DOTS.
soundness: 3
strengths: DOTS unifies many of recent LLM prompting reasoning methods, and can dynamically choose better method for each module for each data sample. It borrows the strength of various solutions in literature, and can possibly incorporate future works.

The paper contains comprehensive experimental results, on various datasets across domain, and in/near/out-of-distribution experiments. The experiments show superior performance of DOT to previous prompting methods consistently.

The paper also includes ablation studies to verify the importance of each of the breakdown modules.

It also includes analysis of preferences of DOTS on different tasks, with insightful explanations. It also includes an analysis of efficiency, showing comparable computing costs to other advanced prompting techniques.
summary: The paper proposes DOTS, which can dynamically choose prompting methods to do reasoning, by dividing reasoning into analysis, solution, and verification modules, and choosing from recent popular methods for each module.

The dynamic planner can be either external or internal to incorporate various LLMs with difference sizes and openness.

Experiments are conducted on in-distribution, near-distribution, and out-of-distribution settings, and zero-shot and few-shot evaluations, where DOTS outperforms most recent LLM prompting reasoning methods.
weaknesses: Although DOTS has higher average performance, DOTS cannot consistently beat baselines in all tasks. This is a little surprising as DOTS should be able to be considered as superset of all baselines. On those dataset DOTS fall behind baselines, it would mean it's not necessary to do dynamical reasoning (as one can choose the baseline instead of choosing different modules for each data sample).

This phenomenon is more often in out-of-distribution settings. The average score on out-of-distribution also shows smaller margin. This means lack of training (in a domain) can lead to much worse performance of DOTS. However, as also mentioned by author, such training set may be not available, casting generalizability concern on DOTS.

It would be beneficial to show the comparison on near-distribution zero-shot settings (e.g. the few-shot datasets in the paper) to verify that only near-distribution training is needed, but the paper only includes few-shot settings on these datasets. (also see questions)

To clarify, the few-shot experiments with consistent outperformance can partly address the point, but I would expect better zero-shot performance since training is involved.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. could you present the distribution of the path the planner will choose in the experiment? (i.e. x% of the path will choose Empty + POT + self-verifications, y % of the path will choose Decomposition + COT + Empty, etc.) (This is different from Table 7 in 2 parts: 1. It shows the distribution of the 12 types of combination of 3 layers in the path, 2. It needs to contain more benchmarks besides the MATH dataset)

2. Can 2 findings in Section 3.7 be generalized to other datasets?  

3. (From weakness 1) The authors should add a comparison with the baseline with 3 layers (and the choice of the layer should be reasonable towards a given benchmark) to make the comparison more fair.

4. From my perspective, the OOD benchmark is more important than the IND benchmark, however, the authors only show the ablation study in the IND benchmark. Why?

5. The improvement of the DOTS seems to come from the explanation (if you compare the result in Table 6's "-w/o Explanation" line with the second to the last line in Tables 3 and 4). So, does the prior knowledge in the prompt cause the improvement? 

6. (minor) The author claims that "the fine-tuned LLMs are constrained to follow the same reasoning format of the training data (e.g., CoT (Luo et al., 2023)) and lack the flexibility to adopt other reasoning strategies" (in line 53,77-78). However, FireAct[1], shows that the fine-tuned LLMs can change the reasoning strategies (COT or ReAct) by itself without any additional prompt. Maybe training COT and POT at the same time can help the model to be more flexible. The authors should revise this point to make it more accurate.

7. In Internalized setting, I'm curious that if llama3-8b is trained only in the Reasoning Process and Answer (like FireAct), what performance will be, will the trained model choose the Reasoning Process automatically? (It should also be a baseline), although I admit this can't be used in external setting.

8. (minor) In Table 2, "Few-shot" is not a kind of Distribution, maybe 4 columns (add example number column) can be more clear.

[1] FIREACT: TOWARD LANGUAGE AGENT FINE-TUNING
soundness: 4
strengths: 1. The paper is easy to read and follow. Its ideas are presented straightforwardly along with illustrations.

2. It simplifies the planning prompt construction process. 

3. The authors study the DOTS over a wide range of benchmarks.
summary: The author creates a planning prompt strategy named DOTS. It can choose the path to solve the problem via a small model (as the planner) and prompt the large model to solve the problem. The path contains 3 parts: the Analysis Layer, the Solution Layer, and the Solution Layer. The experiment shows that both external and internalized planner can help the large model to solve the problem more effectively and efficiently. The ablation study also shows that the searching and explanation in planner's plan is useful.
weaknesses: 1. The comparison between the DOTS and other methods is not fair. The DOTS's prompt contains 3 layers, however, the baselines only contain 1/2 layer. The authors should compare the baseline with 3 layers (for example compulsory use of Decomposition + POT + self-verifications at the same time in MATH) to make the comparison more fair.

2. The method is not generalizable enough. It needs to SFT a new model for each large model / new atomic action in layers / new layers. (Although the authors claim that this is an advantage to make the prompt more suitable for a given model, it is still a limitation for the method.)

3. The improvement of the DOTS is not significant, especially in the OOD dataset. 

4. The ablation study only shows the result in the IND benchmark 

5. The ablation study in Table 6 does not explain the inherent reason that DOTS can improve performance, it's not enough for ICLR. (For example, can a large model have a similar performance with DOTS by only adding prior knowledge in the explanation?). (see question 3,4,6 below for more details)

Meta-review: This paper proposes to enhance LLM reasoning via dynamically tailoring trajectory search. The proposed method should be generally applicable. The experiments show that its inference cost could potentially be lower than non-CoT methods. The proposed method is intuitive. However, the experiments, especially the setting could be more principled. Some of the comparison may not be completely fair to the baselines when considering the details. Its OOD improvement seems limited, considering it requires SFT before deploying the method. Thanks to some of the added new experiments, some of the concerns is alleviated.

Decision: Accept (Poster)