Title: Large Language Model Guided Tree-of-Thought
Venue: NeurIPS 2023
Accepted: no
Keywords: LLM

tree of thought

problem solving
Abstract: In this paper, we introduce the Tree-of-Thought (ToT) framework, a novel approach aimed at improving the problem-solving capabilities of auto-regressive large language models (LLMs). The ToT technique is inspired by the human mind's approach for solving complex reasoning tasks through trial and error. In this process, the human mind explores the solution space through a tree-like thought process, allowing for backtracking when necessary. To implement ToT as a software system, we augment an LLM with additional modules including a prompter agent, a checker module, a memory module, and a ToT controller. In order to solve a given problem, these modules engage in a multi-round conversation with the LLM. The memory module records the conversation and state history of the problem solving process, which allows the system to backtrack to the previous steps of the thought-process and explore other directions from there. To verify the effectiveness of the proposed technique, we implement a ToT-based solver for the Sudoku Puzzle. Experimental results show that the ToT framework can significantly increase the success rate of Sudoku puzzle solving.

Large Language Model Guided Tree-of-Thought

Abstract

In this paper, we introduce the Tree-of-Thought (ToT) framework, a novel approach aimed at improving the problem-solving capabilities of auto-regressive large language models (LLMs). The ToT technique is inspired by the human mind's approach for solving complex reasoning tasks through trial and error. In this process, the human mind explores the solution space through a tree-like thought process, allowing for backtracking when necessary. To implement ToT as a software system, we augment an LLM with additional modules including a prompter agent, a checker module, a memory module, and a ToT controller. In order to solve a given problem, these modules engage in a multi-round conversation with the LLM. The memory module records the conversation and state history of the problem solving process, which allows the system to backtrack to the previous steps of the thought-process and explore other directions from there. To verify the effectiveness of the proposed technique, we implemented a ToT-based solver for the Sudoku Puzzle. Experimental results show that the ToT framework can significantly increase the success rate of Sudoku puzzle solving. Our implementation of the ToT-based Sudoku solver is available on GitHub 1 .

1 Introduction

Self-attention based auto-regressive large language models (LLMs) such as GPT-4 have recently taken the world by storm [1, 2, 3, 4, 5, 6]. These LLMs excel at a variety of tasks that previously thought as extremely difficult or even impossible. For example, they are able to handle various logical and mathematical reasoning tasks, particularly those that entail 'short-range reasonings' necessitating only a few steps to arrive at conclusions [6, 7]. Such remarkable capabilities have even led to speculation that an early form of artificial general intelligence (AGI) may have already emerged [7]. However, today's LLMs still exhibit limitations in certain domains, especially for 'long-range' reasoning tasks, where long-term planning and solution exploration are necessary [7]. When presenting a LLMs such as GPT-4 with a challenging problem solving task, especially the so called System-2 reasoning problems [8], the model does not always succeed. Although the generated answer may be indicative of the correct direction, the derivation process frequently includes logical errors. We hypothesize that there are two main contributing factors which limits the problem solving ability of LLMs:

Lack of correctness checking : To ensure correctness, a good practice for a human solver is to carry out verification procedures at every step of the problem-solving process, thereby ensuring the credibility of the final solution. In comparison, auto-regressive language models do not explicitly perform logical correctness checks as it generates a new token based on the previous tokens. This limits the model's capacity to rectify its own mistakes. A minor error could be amplified as the model generates more tokens, thereby leading to rapid solution quality deterioration and making it difficult to recover from mistakes.

Solution generated linearly : As mentioned above, LLMs typically generate a token based on the preceding sequence of tokens without backward editing. On the contrary, when a human solver

(a) ToT search strategy.

(b) ToT software system.

root

Problem description

from the user

prompt

Prompter

control

signals

ToT

agent

controller

conversation and

node visit history

LLM

Search guided by LLM

Backtrack by ToT

Checker

intermediate

Memory

controller

response

module

solution

module

attempts to solve a problem, she might backtrack to previous steps if a derivation step is incorrect, or if she becomes stuck and is unable to make further progress towards arriving at the final answer. Fields Medal winner Terence Tao once shared his experiences solving hard math problems 2 : 'When I was a kid, I had a romanticized notion of mathematics, that hard problems were solved in Eureka moments of inspiration... With me, it's always, Let's try this. That gets me part of the way, or that doesn't work. Now let's try this. Oh, there's a little shortcut here... You work on it long enough and you happen to make progress towards a hard problem by a back door at some point. At the end, it's usually, oh, I've solved the problem.' The problem solving process as he described is a tree-like thinking process, rather than a linear chain-of-thought [9]. The limitation of linear response generation is also apparent from a computational complexity perspective. The number of computation steps an auto-regressive LLM can perform is polynomial in terms of its input length. Unless P = NP holds which contradicts the widely accepted belief, there would be problems in NP that is not solvable by auto-regressive LLMs.

Inspired by these two shortcomings of auto-regressive LLMs, we propose a novel framework which augments an LLM with several additional modules including an automatic 'prompter agent'. This framework employs a solution search strategy we call the Tree-of-Thought ( ToT 3 ) . This strategy solves a problem through a multi-round conversation between the LLM and the prompter agent. Figure 1a provides a visual description of the ToT search strategy, in which the LLM plays a crucial role in guiding the search for solutions. To make it more concrete, let us assume the problem to be solved is an instance of the Sudoku puzzle. The 'root' node represents the initial state, corresponding to when a human mind just reads through the problem description, and begins the thinking process. A blue node in the figure represents a valid partial solution, which can be used by the LLM as a basis to generate the next search step. In the context of Sudoku puzzle solving, this means presenting a partially filled Sudoku board to an LLM and letting the LLM fill in a few more cells. The rationale is that an LLM like GPT-4 has been trained on a vast amount of text corpus which includes many Sudoku puzzle solutions. Given a partially filled board, likely the LLM is able to recognize the pattern, and provide useful insights on how to proceed following the Sudoku rules. Hence, it is highly probable that a search guided by the LLM is significantly more efficient than a brute-force search. In the figure, the search steps guided by the LLM are represented by the solid arrows. However, these steps generated by the LLM are not guaranteed to be always logically correct. Thus, we introduce a 'checker module' to perform correctness checks. In Figure 1a, a gray node with an 'X' marker represents a 'dead-end', i.e. a partial solution that the checker module considers as invalid. For

Sudoku, this means the partially filled board violates the Sudoku rules. If the current node is invalid, obviously we need to return to a parent or an ancestor node in order to correct the mistake. This can be coordinated by a module called the 'ToT controller' which oversees the ToT search. With the backtracking capability, the system can regenerate the solution and thus recover from errors. In addition, even when the current node is valid, if the system remains stuck at it for too long, the ToT controller could issue a backtrack signal to explore other possible solutions. This is similar to a scenario where a human mind realizes that there is no viable path towards reaching the final solution through a particular direction, prompting her to change course and explore alternative routes. This process continues until either a full solution is found (represented by a green node in the figure), or a pre-specified maximum round of conversations is reached.

Note that while the above discussion utilized Sudoku solving as a tangible example to illustrate our main ideas, the ToT framework can potentially be applied to more general mathematical and logical reasoning tasks. For example, in the context of mathematical theorem proving, a full solution corresponds to the complete proof, encompassing a total of n derivation steps. On the other hand, a partial solution refers to a subset of these steps, specifically the initial k steps, where k is less than n . The checker verifies the logically correctness of a given partial proof. In parallel, the prompter agent and the ToT controller can offer hints and suggestions to the LLM, encouraging it to think about the subsequent proving step, or explore different directions for theorem proving when necessary.

To evaluate the effectiveness of the ToT framework, we implemented a ToT-based Sudoku puzzle solver and evaluate it on a suite of Sudoku puzzle benchmarks we created. As shown by the experimental results in Section 4.2, the ToT framework can significantly increase the success rate of Sudoku puzzle solving.

The remainder of the paper is organized as follows. Section 2 reviews the related literature and compared our approach with the most relevant works. Section 3 provides the details of the ToT system architecture. Section 4 describes our implementation of a ToT-based Sudoku puzzle solver and presents the experimental results. Finally, Section 5 discusses the limitation of the present work, and potential future extensions of the ToT framework.

2 Related Works

Developing intelligent systems that can reason has long been one of the primary goals of artificial intelligence [10, 11, 12]. Recent advancements in large language models, particularly the discovery of their emergent properties and in-context learning abilities, have opened up a new avenue for machine reasoning [6, 7, 9]. It is discovered that prompting language models using chain-of-thought and other hints can elicit them to output step-by-step solutions for mathematical and logical reasoning tasks [9, 13]. Building on these findings, recent studies have also explored the practice of sampling multiple solutions and using self-consistency or complexity-based criteria to determine the optimal response [14, 15]. Experiments were also conducted to evaluate the performance of different prompts [15]. The self-taught reasoner (STaR) [16] is a technique which asks an LLM to generate reasoning chains and drop those producing incorrect answers. Then, the model is fine-tuned with the remaining valid reasoning chains.

Despite showing high potential, these techniques often necessitate human involvement. For example, chain-of-thought style prompting techniques require carefully hand-crafted examples and is thus difficult to scale. Consequently, researchers have started to explore the possibility of automatic prompt generation. Early exploration in this domain includes AutoPrompt [17], prefix-tuning [18], and parameter-efficient prompt tuning [19]. This research direction received even more attention lately. In a recent study [20], the authors experimented with training verifiers to check if the solution provided by an LLM to an given mathematical problem is logically correct. If the trained verifier can effectively judge the LLM outputs, it would provide another avenue for prompt evaluation. Automatic prompt engineer [21] examines a method to select the best prompt from a set of model-generated candidates. The three-phase augment-prune-select method was suggested in [22]. It first generates multiple chain-of-thought candidates, which was then pruned based on whether the derived answer matches with the ground truths. Finally, a policy gradient based method was used to select the optimal combination of several rationale chains from the pool for CoT prompting.

Very recently researchers have also turned their attention to augmenting LLM with additional agents for various purposes. This is also the research field that is most relevant to our current work. AutoGPT

[23] is a program which combines GPT-4 with additional modules including an execution agent and a memory unit. It can chain together LLM 'thoughts', in order to autonomously achieve whatever goal the user sets. PromptPG [24] proposes an approach that can learn to select in-context examples from a small amount of training data via policy gradient for prompt learning. The PromptPG agent learns to find optimal in-context examples from a candidate pool, with the goal of maximizing the prediction rewards on given training examples when interacting with the GPT-3 environment. DEPS [25] is a proposal that utilizes multi-step reasoning and sub-task error correction to tackle complex tasks with long-range dependencies. By being able to provide explanations for errors in sub-tasks within a trial, DEPS exhibits remarkable performance. ReAct [26] is an approach that employs emergent properties present in LLMs, such as traces of verbal reasoning, to enable agents to reason and take action, resulting in impressive performance on different text-based benchmarks. Building on top of ReAct, Reflexion [27] is an approach that equips an agent with dynamic memory and self-reflection capabilities, improving its existing reasoning trace and ability to choose task-specific actions. To achieve complete automation, a simple but effective heuristic was designed to enable the agent to identify hallucination instances and prevent repetitive action sequences. Our proposal shares some commonalities with these approaches, for example, the use of a memory module and additional agents for automatic prompt generation. However, our approach is unique in that it introduces a ToT controller which can explicitly conduct backtracking when necessary. This not only allows the system to recover from mistakes, but potentially can also enlarge the solution search space.

3 Architecture

3.1 The Tree-of-Thought Framework

Figure 1b depicts the software system that implements the ToT Framework. As mentioned earlier, it incorporates several components which enhance the problem solving capability of the LLM, including a prompter agent , a checker module , a memory module , and a ToT controller .

The problem solving process starts with the user inputting the problem description. The prompter agent then relays the problem to the LLM, with additional prompt text which encourages the LLM to come up with an intermediate solution instead of trying to reach the full solution in a single shot. After receiving the response from the LLM, the checker module is invoked to check the validity of the intermediate solution generated. If it passes the correctness check, the intermediate solution will be parsed and stored in the memory module. Then, based on the content of the memory module, the prompter agent generates a prompt to encourage the LLM to generate the next step. Conversely, if the LLM generates an invalid intermediate solution, the ToT controller will activate the prompter to offer hints to the LLM and request it to consider again. Note that in general, a valid intermediate solution does not always leads to the correct final solution. In order to prevent getting stuck, the ToT controller constantly monitors the search process and determines whether to continue trying from the current node or backtrack to a parent or an ancestor node and explore alternative directions.

The ToT strategy can be viewed as a tree-search algorithm using an LLM as a heuristic for generating the search steps. In this setting, the LLM is only used for the 'short-range reasoning' tasks, i.e deriving the next intermediate solution, which is a type of tasks that have been shown to have a high success rate for LLMs [7]. On the other hand, by introducing the checker, the system have a higher likelihood to discover the mistakes it makes as it generates the solutions. Moreover, by allowing the system to backtrack from a valid but somewhat 'hopeless' intermediate solution, the system is able to explore a larger solution space, which enhances the 'long-range reasoning' capability of the system as a whole. The ToT framework thus combines the best of both world. Furthermore, this multi-round conversation technique increases the number of computation steps the system can perform. Thus, based on the time hierarchy theorem in computational complexity theory [28], the ToT framework can expand the range of problems that can potentially be solved compared to relying solely on a single round of conversation with an LLM.

3.2 ToT Modules

In this section we provide more details of the components of the ToT software system.

Checker Module . The checker module can either be rule-based or implemented as a deep neural network. For problems that have an explicit polynomial time algorithm for correctness checking (i.e.

problems in NP ), rule-based checkers can be implemented. Numerous important mathematical and logical problems are in this category, for example, equation solving, polynomial factoring, 3SAT, and puzzles like Sudoku. With a rule-based checker, the ToT software can be viewed as a hybrid system which allows explicitly encoding prior knowledge (e.g. the Sudoku rules) into a neural network powered system. An alternative is to train and use a neural network based classifier as the checker [20]. This is especially useful for problems where a rule-based checker is difficult to implement, e.g. checking whether a mathematical proof is correct.

Memory Module . The memory module can be used to store the entire conversation history between the LLM and the prompter agent, as well as other supplemental data useful for problem solving. The data stored can be served as the information source for the prompter agent to generate helpful hints for the LLM.

ToT Controller . The ToT controller oversees the entire ToT search. It can be implemented in a number of ways. It can be as simple as encoding two rules: 1) if the checker thinks the current partial solution is invalid, backtrack to the parent node, and 2) if the current partial solution is valid, but the ToT search tree has explored its C children and yet failed to find the final solution, then backtrack to the parent node. Here C is an pre-configured integer.

A more advanced version of the ToT controller can employ a policy network to determine the backtracking policy. The network's inputs include the recent search history comprised of the sequence of the last k +1 node visited in the search tree s i -k , .., s i -1 , s i ( k is a hyper-parameter). The network also takes in c i , a Boolean variable which indicates whether the checker module considers the current node s i is valid. We can sample from the policy to determine the next action a i :

where π t ρ represents the policy network of the ToT controller with parameters ρ . The set of candidate actions A cand includes simply staying at the current node to generate the next step, and backtracking to the parent or an ancestor node at most L levels up in the search tree where L is a hyper-parameter. Thus, we can use one-hot encoding for the actions, where backtracking j levels up is represented by a vector where only the j th position is set to 1. The action vector a and checker output c i are processed by a feed-forward network (FFN) to for deep features extraction. A linear layer with learnable parameters W 1 and b 1 is added on top of the FFN to map its output to a vector g ( a, c i ) . The latest k +1 visited nodes are concatenated into a string, and then added with position embedding (PE), and finally inputted into a self-attention model [1]. The idea is that by adding position embedding, the attention model will be able to make decisions based on the sequence of the recent node visits. A linear layer with learnable parameters W 2 and b 2 is added on top of the attention model to transform its output to a vector g ( s i , .., s i -k ) whose dimension matches with that of g ( a, c i ) . Finally, we calculate the inner-products of these two vectors, and use the softmax function to compute the probability of each action candidate:

In the above formula, '||' is the string concatenation operator. Section 3.3 will discuss the training algorithm for the ToT controller policy network.

Prompter Agent . The prompter agent gives hints to the LLM for it to generate the next search step. The most basic hint can be a generic prompt using the following template: generic _ tmpl = ' For the given problem: [problem description], we have come up with a partial solution: [partial solution summary]. Please derive the next step on top of this partial solution, and return the next step in the following JSON format {next_step: <next_step>} '. Note that the template requires the LLM to respond with a structured JSON string. This is a trick to make it easier for the checker to extract the next step from the LLM response. To create an actual prompt from this template, the prompter needs the [problem description] and the [partial solution summary] , both of which can be queried from the memory module.

Similar to the ToT controller, we can also implement the prompter agent as a policy network, which can generate prompts based on the current partial solution and the conversation history. First we

define the prompt template as follows: prompt _ tmpl = generic _ tmpl || ' Here are a few examples: [in-context learning examples] .', where || is the string concatenation operator. The variable [in context learning examples] are in-context learning examples for the problem being solved, which can be picked by the prompter policy network from a set of candidates, similar to the PromptPG approach [24]. The rationale is that given the current and recently attempted intermediate solution, some in-context examples might work better than others as hints for the next step. Given the recently visited node sequence s i -k , .., s i -1 , s i , our goal is to select l examples e i = { e 1 i , e 2 i , ..., e l i | e j i ∈ E cand } where E cand is a pool of in-context learning example candidates. The examples are selected according on a policy:

where π p θ represents the policy network of the prompter agent with parameters θ . With the set of selected examples, the prompter agent generates a prompt from the template: p i = prompter ( prompt _ tmpl, e i , s i ) , which can be fed into the LLM to obtain the next intermediate solution s i +1 = LLM ( p i ) . The neural network architecture for the prompter's policy network is similar to that of the ToT controller. The only difference is that since the in-context examples are expressed in natural language, instead of FFN, we use an attention model to process them:

The prompter policy network can be trained together with the ToT controller using multi-agent reinforcement learning methods. The training algorithm of the prompter's policy network is discussed in Section 3.3.

3.3 ToT System Training

In the previous sections, we have described the multi-agent ToT framework. This section dives into how we can train the agents, in particular, the policy networks of the ToT controller and the prompter agent. While there are many multi-agent reinforcement learning algorithms (MARL) proposed in the literature [29, 30, 31], in this work we adopt a relatively simple approach which uses a modified version of the REINFORCE algorithm [32] to train the policy networks of the ToT controller and the prompter agent directly. The more advanced MARL algorithms will be explored in the future.

First, we define a run of the ToT system as the process where a user inputs the problem description, and the ToT system attempts to solve the problem until it thinks the problem is solved, or a prespecified maximum round of conversations is reached. Next, we define the reward r of a run: if the problem is correctly solved, then r = +1 . Otherwise, if the system outputs an incorrect solution, or the maximum round of conversations is reached, then r = -1 .

The training algorithm is provided in Algorithm 1. The algorithm takes two inputs, the training data set P train , and the number of training epochs N (Line 1-2). The two policy networks π t ρ ( a i | s i , .., s i -k ) and π p θ ( e i | s i , .., s i -k ) are randomly initialized (Line 3-4). We train the two policy networks in turns, i.e. training one network with policy gradient while keeping the other fixed (Line 6). To be more specific, when the current epoch is an even number, we select the ToT controller policy π t ρ , and keep the parameters of the prompter agent fixed. Otherwise, we select the prompter agent policy π p θ and fix the ToT controller policy. Next, the algorithm updates the parameters of the selected policy network using the policy gradient method (Line 7-9). For each problem in the training data, we attempt to solve it with a ToT system run. Based on the result, we obtain the reward for that run (Line 8). The entire training algorithm runs for N epochs.

3.4 Problem Solving Using the ToT System

After the ToT system is trained, we can use it for inference, i.e. problem solving. Algorithm 2 provides the pseudo code for solving problems using the ToT system. It starts with a user inputting description of the problem (Line 1-2). The prompter module then converts the user input into a prompt (Line 3) using a prompt template for user input, for example: user _ input _ prompt = ' For the given problem: [problem description], please derive the first step, and return the step in the following JSON format {next_step: <next_step>} '.

Next, up to K rounds of conversations with the LLM are conducted for problem solving (Line 4). In each round, the LLM first produces a response for the given prompt (Line 5). Then, the checker analyzes the response, and returns a result (Line 6). The result contains the partial solution extracted from the LLM response, as well as information like whether the checker considers the solution as a valid final solution, a valid intermediate solution, or an invalid partial solution, etc. If the solution is a valid final solution, the algorithm simply returns it (Line 7-9). Otherwise, the result is stored in the memory module (Line 10). Based on the content of the memory module, the ToT controller issues control signals, e.g. backtracking for l levels, to the prompter (Line 11). Finally, based on the control signal, the prompter looks up the relevant information from the memory module, and produce the next prompt for the LLM (Line 12). If no valid final solution is found within K rounds of conversations, the algorithm return nil indicating it fails to solve the problem (Line 14).

4 Evaluation

This section provides the evaluation methodology and experimental results for our proposed ToT framework. Our evaluation focuses on the ToT-based solver for the Sudoku problem. At the first glance, Sudoku problems seem to be just brain teasers with little practical importance. However, the generalized Sudoku problem on n 2 × n 2 grids of n × n blocks is known to be NP-complete [33]. If the ToT framework can solve instances of the generalized Sudoku problem (granted that it might takes an exponential number of rounds of conversations), in principle it can handle many other mathematical

3x3 puzzles

4x4 puzzles

5x5 puzzles

0

0

.

5

1

0

.

4

0

.

2

0

.

1

0

.

9

0

.

4

0

.

5

0

.

9

0

.

5

0

.

5

1

0

.

9

0

.

8

Success rate

zs

os

fs

tot

and logical reasoning tasks. In fact, it is straightforward to re-purpose the implementation described below to solve other puzzles, such as 3SAT, 3-coloring, etc.

Below we first describe the implementation details of the solver. Then, we present the test suite used in our evaluation, as well as the experimental results.

4.1 ToT Solver for Sudoku Puzzles

The ToT-based Sudoku solver follows the generic framework described in Section 3 with some specific tweaks for the Sudoku problem. It allows a user to input a Sudoku puzzle using natural languages, for example: 'Please solve this 4x4 Sudoku puzzle [[3,*,*,2],[1,*,3,*],[*,1,*,3],[4,*,*,1]] where * represents a cell to be filled'.

We have implemented the ToT-based Sudoku solver as described in Section 4.1 in Python. We adopted a rule-based approach for the checker module since the Sudoku rules are precise and easy to check. The memory module stores the conversation history between the prompter and the LLM, as well as a search tree which maintains all the partially filled Sudoku board the LLM has generated so far. This way, when backtracking happens, the previous board configuration can be retrieved. The ToT controller in our implementation is also rule-based. It returns to the parent node in the search tree if either the current node considered invalid by the checker, or the search algorithm has explored more than 5 children of the current node. Finally the prompter agent uses a variation of the generic template mentioned above, with the [problem description] being the initial configuration of the Sudoku board input by the user, and [partial solution summary] being the partially filled board represented by the current node in the search tree. The LLM utilized in this study is the "gpt-3.5-turbo" model, which is accessible through the OpenAI API suite. The temperature parameter was set to 1 in our experiments.

4.2 Experimental Results

We have implemented four LLM-based Sudoku puzzle solvers and compared their performance: 1) zero-shot solver ( zs ) which directly posts the puzzle description to the LLM, 2) one-shot solver ( os ) which provides a chain-of-thought style step-by-step solution of a 3x3 Sudoku puzzle as an example in addition to the problem description, 3) few-shot solver ( fs ) which provides multiple examples with CoT-style solutions, and 4) our proposed Tree-of-Thought solver ( tot ). We constructed three benchmarks, comprising of ten 3x3, 4x4, and 5x5 Sudoku puzzles, respectively. The objective of a solver is to fill the n × n Sudoku grid with digits so that each row and column contain all of the digits from 1 to n ( n = 3 , 4 , 5 in our experiments).

Figure 2 compares the success rates of different LLM-based solvers across the three benchmarks. Here the term success rate refers to the fraction of problems in a benchmark set that are successfully solved by a solver. For example, if a solver is able to solve 4 out of 10 problems in the '3x3 puzzles' benchmark set, then the success rate of this solver for this benchmark set is 0.4. As expected, the zero-shot solver has the worst performance across all the three set of benchmarks. Adding CoT-style

step-by-step examples significantly boosts the success rate, especially for the 3x3 puzzles. This is expected, since one can pretty much rely on 'short-range' reasoning skills, which is a strength of the LLM models, to solve a small-sized 3x3 Sudoku puzzle, espcially when CoT-style hints are provided. However, as the puzzle size gets bigger, the success rate of the one-shot and few-shot solver dropped to around 0.5. This is because solving bigger puzzles requires trial and error, which is a capability LLMs generally lack of as discussed earlier.

In comparison, the ToT-based solver demonstrates superior performance when compared to the other solvers. For the 3x3 benchmark set, it was able to solve all the puzzles. The success rate improves by 11% compared to the second best for the two benchmark sets. For the 4x4 benchmark set, the ToT-based solver failed to find the solution for 1 out of the 10 puzzles before reaching the maximum round of conversations (which is set to 100 in our experiments). We suspect it is due to the limited capability of the rule-based ToT controller. In particular, the rule-based controller has no sense of whether the current partially-filled board can be completed without violating the Sudoku rules, which decreases the efficiency of the solution search. We expect a neural network based ToT controller will perform better, which we will verify in the future extension of this work. Despite this, the success rate of the ToT based solver is still 80% higher compared to that of the one-shot and few-shot based solvers. Finally, for the 5x5 puzzles, the ToT-based solver failed with 2 puzzles before reaching the maximum round of conversations. Nonetheless, the success rate is 60% higher compared to that of the one-shot and few-shot based solvers.

5 Discussions and Future Works

In this paper, we presented the Tree-of-Thought framework, which enhances LLMs with additional agents and memory modules, resulting in improved performance for mathematical problem-solving tasks. To evaluate the performance of this technique, we implemented a Sudoku puzzle solver based on the ToT framework. One of the limitations of the current implementation is that it utilizes a rule-based checker that contains custom logic, making it less easily adaptable to other problems. For more generic problems, for example, general mathematical and logical reasoning problems, where rule-based solution checking is difficult to implement, a future direction is to explore checkers based on neural network or other probabilistic models. Moreover, the experiments we conducted in this work also uses a rule-based ToT controller, which as we pointed out, has limited capabilities. In the future, we will implement the neural network based ToT controller which can hopefully enhance the system performance. Additionally, the policy-gradient based training algorithm proposed in this work is relatively simple and may be susceptible to training stability issues. To further optimize the ToT system, more advanced multi-agent reinforcement learning algorithms, particularly those designed for cooperative agents, could be adopted.

References

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.

Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018.

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.

OpenAI. Gpt-4 technical report, 2023.

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.

Shu Tay, Paul Ryan, and Anthony C Ryan. Systems 1 and 2 thinking processes and cognitive reflection testing in medical students. Canadian Medical Education Journal , 2016:97-103, 11 2016.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.

Larry Wos, Ross Overbeck, Ewing Lusk, and Jim Boyle. Automated Reasoning: Introduction and Applications . Prentice Hall Professional Technical Reference, 1984.

Frederick Hayes-Roth, Donald A. Waterman, and Douglas B. Lenat. Building Expert Systems . AddisonWesley Longman Publishing Co., Inc., USA, 1983.

Ronald Fagin, Joseph Y. Halpern, Yoram Moses, and Moshe Y. Vardi. Reasoning About Knowledge . MIT Press, Cambridge, MA, USA, 2003.

Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, Roman Wang, Nikhil Singh, Taylor L. Patti, Jayson Lynch, Avi Shporer, Nakul Verma, Eugene Wu, and Gilbert Strang. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences , 119(32), aug 2022.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023.

Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning, 2023.

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022.

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV au2, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts, 2020.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers, 2023.

KaShun Shum, Shizhe Diao, and Tong Zhang. Automatic prompt augmentation and selection with chain-of-thought from labeled data, 2023.

Auto-gpt: An autonomous gpt-4 experiment, 2023. https://github.com/Significant-Gravitas/ Auto-GPT .

Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning, 2023.

Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents, 2023.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023.

Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection, 2023.

Juris Hartmanis and R. Stearns. On the computational complexity of algorithms. Transactions of The American Mathematical Society - TRANS AMER MATH SOC , 117:285-285, 05 1965.

Thanh Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE Transactions on Cybernetics , PP:1-14, 03 2020.

Afshin OroojlooyJadid and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning, 2021.

Kaiqing Zhang, Zhuoran Yang, and Tamer Ba¸ sar. Multi-agent reinforcement learning: A selective overview of theories and algorithms, 2021.

Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems , NIPS'99, page 1057-1063, Cambridge, MA, USA, 1999. MIT Press.

Michael Haythorpe. Reducing the generalised sudoku problem to the hamiltonian cycle problem, 2016.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach (called Tree-of-Thought, or ToT) for boosting the problem-solving abilities of LLMs by means of backtracking in solution space. The proposed ToT architecture augments the LLM with four modules, and is broadly framed to include multiple potential implementations of those modules, including neural networks (for the prompter and controller) that could be trained by means of policy gradients. One implementation of ToT is evaluated on Sudoku, where it demonstrates significant improvement over an LLM without the ToT augmentations.\n\nSTRENGTHS:\nThe motivation is highly topical, as improving the reasoning power of LLMs is a challenge of great research interest and economic value. Most of the paper’s presentation is clear, and the experimental results seem fairly sound, as far as they go.\n\nWEAKNESSES:\nThe work is limited in four ways.\n\n1 - While the paper lays out a fairly sophisticated and general architecture, only one narrow implementation of ToT is actually tested. Specifically, the current version does not include the policy-gradient options laid out in Algorithm 1 and equations 1-4. Instead, the tested implementation uses the much simpler rule-based controller and LLM-based prompter that have no additional weights to train. This makes it impossible at present to draw conclusions regarding the effectiveness of more advanced realizations of the architecture. \n\n2 - The evaluations, limited as they are to Sudoku, do not demonstrate the generality of the approach, despite the assertion that \"in principle it can handle many other mathematical and logical reasoning tasks''. For comparison, consider the recent work of Yao et al., 2023, *Tree of Thoughts: Deliberate Problem Solving with Large Language Models*, which reports evaluations on three separate problem types:  Game of 24, Creative Writing, and 5x5 Crosswords.\n\n3 - Since humans play Sudoku in graphical rather than language-based forms, there is little reason to expect text-only LLMs to perform particularly well on Sudoku at all. And for any algorithmic puzzle (like Sudoku) for which solutions can be explicitly verified, it is unsurprising that explicit checking for valid solutions would boost the performance of an LLM alone. The more central question is how much LLMs contribute to solving such puzzles, but this question is not addressed. The *baselines* tested (zero-shot, one-shot, and few-shot) are actually just *ablations* of ToT, all of them using LLMs.\n\n4 - This paper discusses none of the prior work on Sudoku, such as *SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver* by Wang et al., 2019, which provided a Sudoku benchmark dataset that was used by Ahmed et al., 2013, in *Semantic Strengthening of Neuro-Symbolic Learning*. The *Neural Logic Machine* of Dong et al., 2019, has also been applied to Sudoku (https://github.com/ashutosh1919/neuro-symbolic-sudoku-solver), achieving a 94% success rate on 5x5 puzzles, which is significantly better than the ToT results reported in this paper. \n\nBecause of these limitations, the work’s contributions do not seem significant at this point. Nevertheless, the general architecture might prove to be significant once it is fully implemented and tested on a broader set of benchmarks for which baseline results from the literature are available for comparison.\n\nQUESTIONS:\nHow many Sudoku instances were tested to produce each bar in Figure 2? Just 10 problems each?\n\nLIMITATIONS:\nRegarding the first limitation described in the Weaknesses section of this review, the paper states \"We expect a neural network based ToT controller will perform better, which we will verify in the future extension of this work.\" However, the crucial point is that policy-gradient training is not evaluated at all, and this limitation is never stated with sufficient clarity. The other three limitations are not adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper present tree-of-thought (ToT) as a way of using LLMs to solve problems. ToT involves search + backtracking in a tree-like structure. The work demonstrates the success of this method in simplified sudoku tasks.\n\nSTRENGTHS:\nThe idea is interesting, and the Sudoku tasks are a reasonable regime for evaluating it.\n\nWEAKNESSES:\nWhile the method is interesting, the evaluation is minimal. Furthermore, while the sudoku tasks are reasonable as a single setting for evaluation, as the only setting they are quite a narrow domain. The paper also seems to be missing some valuable ablations/baselines. In more detail:\n\n1) The ToT method appears to contain many details which are not individually tested with ablations. Without testing these, it is unclear which aspect is relevant, and what we should learn from the work. \n    - For example, the paper emphasizes the benefit of the tree-structured generation, which allows the model to backtrack in its reasoning. However, if I understand correctly, none of the baseline experimental conditions include *any* form of generating multiple answers, or using the checker to verify the answer is correct. For example, rather than a tree search, the model could just run 10 complete rollouts (productions of the correct answer), and then use the checker to identify the correct one (if any). Or, the model could use the checker to decide at each step whether to accept a generation, but without backtracking further. Would one of these methods perform as well? If so, then the key feature is not the tree structure per se, but merely the possibility of generating multiple completions and then checking. In order to understand the contributions of the method, it would be necessary to see these ablations. \n    - Likewise, the ToT method appears to benefit from a prompter policy which can decide on the examples for the prompt (if I understand correctly), while the other methods don't; perhaps a prompter is all that's needed.\n    - More fundamentally, if I understand correctly (although the paper is not entirely clear on this point), the ToT method is the only one that involves training; there are many ways the one/few-shot prompts could be \"trained\", such as selecting the examples in the prompt. Ideally, the methods would be evaluated in a setting where the baselines can also benefit from training. \n\n2) The evaluation is quite minimal, in both scope and depth.\n    - The method is only tested on a single domain (simplified sudoku, without the box constraints); sudoku is a well-known puzzle, and so is likely to be well-covered in the training corpus. While that does not mean the present results are invalid, it would be useful to see a demonstration of the idea in other, less commmon domains (even other NP ones, such as Traveling salesman, are likely much rarer in the training corpus), or ideally fully novel ones.\n    - What evaluation there is only involves a handful of puzzles (10 per condition). The differences between ToT and FS in each condition would not achieve statistical significance by an exact binomial test. While the consistent benefit of ToT across conditions makes it more plausible that there is an overall positive effect, it would be useful to run a larger number of puzzles per condition in order to more accurately assess the magnitude of that effect.\n\n3) The paper could do a more thorough job of situating the present work within the existing literature with similar motivations, e.g. https://arxiv.org/abs/2208.14271 or https://proceedings.neurips.cc/paper/2021/hash/d3e2e8f631bd9336ed25b8162aef8782-Abstract.html\n\n4) The paper presents itself as though it evaluates on Sudoku, but in fact it evaluates on a simplified variant (no boxes). The present abstract, for example, seems somewhat misleading in that it never makes clear that the algorithm is tested on a smaller, simpler version of the task.\n\n5) It would be nice to see comparisons with other language models (especially smaller ones, and ones with an open training process) to understand how general the results are.\n\nQUESTIONS:\nIs there something I misunderstood about the baselines; e.g. are some of them also matched to ToT in terms of training/optimization or compute/calls to the checker?\n\nLIMITATIONS:\nYes, limitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel algorithm ToT (tree-of-thought) based on: \n- LLM (GPT 3.5 in this case)\n- checker module (which verifies solutions and partial solutions)\n- memory module\n- ToT controller that guides the search (it can be a neural network or a set of rules)\n- prompter agent (in this paper this is a policy network that selects the best in-context examples for a given tree node\n\nThe toT algorithm is tested on a challenging Sudoku task for 3 different board sizes.\n\nSTRENGTHS:\nThe approach is novel. It mixes general LLM with two neural networks, trained together. The introduction Section is good: it identifies two main limitations for using LLMs in complex problem-solving. Sudoku is a complex task that requires backtracking and a search, which makes it interesting in the context of ToT. The training of policies is an interesting idea that could be of use in other LLM-based algorithms.\n\nWEAKNESSES:\nThe biggest weakness of this paper is the small number of experiments, which also are conducted on a single task (sudoku). In the text, many different versions of ToT are discussed, however, experiments are done only for a single setup and a single task. ToT was not tested on any other task, thus we cannot know if it really generalizes at all.\n\nThere are far too few experimental results and data. What is missing:\n- How many nodes ToT needs on average to solve a given task?\n- How many steps baseline needs to solve a given task?\n- There are only 3 versions of the Sudoku: for n=3, 4, and 5. (this would be ok if there was more tasks). What happens for n >= 6? If ToT is still the best then it would be great for the method. If not, we will clearly know where is the limit for ToT. If the method is too slow for n>=6 it is important to know.\n- What is the price (or number of tokens needed) on average for a single ToT run?\n- What was the number of Sudoku puzzles used for evaluation? It is not clearly stated in the text, however, I guess it was 10 boards for each n=3,4,5. If I am wrong please correct me. If it was 10 boards then the results cannot be trusted at all. In such a case for the 0.4 success rate, the 90% confidence interval is (0.15 - 0.7), which tells nothing about the real results. Results on 10 testing boards have no scientific meaning and this is the main reason for such a low score I gave. If I am wrong (and the number is higher I will be happy to improve the score).\n-There are no error bars in Figure 2.\n\n\nAuthors claim that: \"If the ToT framework can solve instances of the generalized Sudoku [...] in principle it can handle many other mathematical and logical reasoning tasks.\". The claim that ToT should be able to handle other complex problems is based on the idea that complex tasks require a similar way of thinking  More advanced mathematical problems like automated theorem proving have their own sources of complexity (e.g. choosing the appropriate lemmas to consider or how to represent the state in a compact form, which fits to the transformer). I know that authors do not claim that for sure ToT works in such tasks, but after reading the paper it seems that the significance of the paper is built upon a promise that ToT can be easily adapted to more serious problems. Since there are no experiments to support this claim I think that the significance of this paper is limited. \n\nThere are no experiments concerning other variants of ToT: for example with neural network checker or rule-based policy. \n\nNotation in Algorithm 1 is hard to understand. I had trouble understanding which \\pi stands for ToT policy and which for prompt policy. Please consider more natural notation like \\pi_{tot} \\pi_{prompt} or similar. \n\nAlgorithm 2. The meaning of (nil) is not introduced in the algorithm, it is only later in the text. \n\nI think that the version of the text review should use specific LaTeX options: for example line numbers. It is hard for me to refer to concrete lines without them.\n\nIt makes no sense for me to describe the procedure of training ToT policy if it was not used in the experiments.\n\nQUESTIONS:\nThe most important: what was the number of test boards for each n?\n\nWhat is the hierarchy theorem? You should briefly explain it in the paper for readers who are not familiar with complexity theory.\n\nEquations (1) and (3): what is s_i precisely? How it is represented?\n\nWhat networks were trained in the experiments? It is hard to find the text.\n\nLIMITATIONS:\nIn the paper, there is no separate section for limitations (some ar ementioned in Section 5). Many missing limitations I already described in the Weaknesses Section of this review.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Tree-of-Thought (ToT) framework, a novel approach to enhance the problem-solving capabilities of large language models (LLMs). The ToT technique mimics the human mind's trial-and-error thought process, allowing LLMs to explore the solution space of complex reasoning tasks and backtrack when necessary. The paper presents an implementation of a ToT-based Sudoku puzzle solver and evaluates its effectiveness on a suite of Sudoku puzzle benchmarks. The experimental results show that the ToT framework can significantly increase the success rate of Sudoku puzzle solving. \n\nThe contributions of the paper include introducing a new approach to enhance the problem-solving capabilities of LLMs, presenting an implementation of a ToT-based Sudoku puzzle solver, and demonstrating the effectiveness of the ToT framework on a suite of Sudoku puzzle benchmarks.\n\nSTRENGTHS:\n1. The motivation for moving from linear reasoning, like Chain-of-thought, to a tree-like searching/reasoning is strong and well recognized. Considering the fundamental limitation of autoregressive generation of GPT-like LLMs, we do need more advanced reasoning/search algorithms for better decoding.\n2. The proposed method is reasonable and technically sound. The checker module echos the recent findings of self-evaluation of LLMs, and the memory module also is useful in agent-based modeling. \n3. The empirical results on the Sudoku pizzles show the effectiveness of the proposed method, esp. when the puzzle becomes harder, the performance of the proposed method is still good.\n\nWEAKNESSES:\n1. One of the biggest issues of this paper is the mismatch between the described method and the actual one used in the experiments. The paper spends lots of space talking about how the ToT controller and prompter agent can be modeled by a policy network and trained via multi-agent RL. But it never tried such formulation and training in the experiments and only presented them as some kind of future work. Without valid evidence, empirically or theoretically, the method section is largely questionable. \n2. Another issue is the novelty of this work probably is not as big as the paper claims. The formulation of multi-agent RL for controller and agent probably is overcomplicated, and I encourage the authors to think of reformulating them all as LLMs to make things easier. Also, there are many missing related works [1, 2, 3, 4] that have a similar tree search/reasoning formulation with more operational and rigorous experiments.\n3. The experiment scope is limited. The proposed method is only demonstrated in one task with the simple formulation of the controller and agent (discussed above). This isn't ok for NeurIPS papers, and we need to better figure out why and how the proposed method can or cannot be applied to more general tasks.\n\nQUESTIONS:\nI raised some questions in the weaknesses section, and there could be lots of improvement space for the authors to make and answer those questions. The following are some minor comments:\n\n1. It seems the catchy name of Tree-of-though has been popularized by another work [5], which draws far more attention than this one; I'd suggest the authors rethink the core contributions of the proposed method and position it in a different and unique way \n2. I wonder whether the authors could explicitly explain what kind of search algorithms are used in the proposed method for a better understanding of the method.\n\n[1]. Xie, Yuxi, et al. \"Decomposition enhances reasoning via self-evaluation guided decoding.\" arXiv preprint arXiv:2305.00633 (2023). \n\n[2]. Jung, Jaehun, et al. \"Maieutic prompting: Logically consistent reasoning with recursive explanations.\" arXiv preprint arXiv:2205.11822 (2022). \n\n[3]. Zhu, Xinyu, et al. \"Solving math word problem via cooperative reasoning induced language models.\" arXiv preprint arXiv:2210.16257 (2022). \n\n[4]. Hao, Shibo, et al. \"Reasoning with language model is planning with world model.\" arXiv preprint arXiv:2305.14992 (2023).\n\n[5]. Yao, Shunyu, et al. \"Tree of thoughts: Deliberate problem solving with large language models.\" arXiv preprint arXiv:2305.10601 (2023).\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a tree-of-thought (ToT) framework to improve complex reasoning and problem solving capabilities of auto-regressive language models. Specifically, motivated by how humans process thoughts with trail and error, ToT maintains a memory module, and employs a ToT controller to decide when to proceed on a thought of nodes in a tree, and when to backtrack to a previous parent node depending on a tracker. Evaluated on Sudoku 3x3, 4x4, and 5x5 puzzles, results suggest that ToT is effective compared to few-shot baselines.\n\nSTRENGTHS:\n1. This paper presents an interesting tree of thought method to enable back-tracking in auto-regressive language models. This solves one of the key limitations of LLMs. Similar to humans and inspired by system 2 reasoning, the proposed ToT structure, especially how we can employ a checker to dynamically modify and utilize memory, makes a great contribution to the field, and can inspire future work on how LLMs can be prompted, and even pre-trained.\n\nWEAKNESSES:\n1. Although the high-level idea of tree-of-thought is promising, with corresponding ToT controller, agent, and memory, the paper is only evaluated on one Sudoku task, especially when the details of evaluation (e.g., number of games evaluated, and computational cost and prompts used compared to the baselines) are not specified. This makes the evaluation results less convincing. Moreover, despite that the method sounds generalizable, there is no strong evidence on how each module in the framework should be actually implemented to be effective and demonstrated (apart from simply mentioning some future work in the end). \n2. It is not clear how each module in the ToT framework should work in details. For example, the memory module seems to be compelling where the LLM can retrieve previous configuration when backtracking, there is no explicit demonstration of how the memory is maintained, and how backtracking would work. Furthermore, the ToT controller is rule-based in the experiment, but Section 3.2 and 3.3 mostly explains how the ToT controller should be trained similar to a policy network. This makes it very confusing to judge the proposed method. I would suggest the authors to add more detailed illustrations using specific examples in the paper revision.\n\nQUESTIONS:\n1. Can you provide more evaluation details for the Sudoku setup? For example, how many games are used for evaluation, and what prompts are employed to the LLM (especially when comparing to the baselines).\n2. Why do you use a rule-based controller for backtracking? How do you derive the rules? \n3. Can you provide more details in terms of how backtracking interact with the memory?\n\nLIMITATIONS:\nThe authors only briefly mentioned the limitations in terms of implantation, but not the limitations of the method overall, such as computational cost.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}