SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dian Yu
Jeffrey Zhao
Izhak Shafran
Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role. To surmount these challenges, we introduce a new framework for language model inference, "Tree of Thoughts" (ToT), which generalizes over the popular "Chain of Thought" approach to prompting language models, and enables exploration over coherent units of text ("thoughts") that serve as intermediate steps toward problem solving. ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices. Our experiments show that ToT significantly enhances language models' problem-solving abilities on three novel tasks requiring non-trivial planning or search: Game of 24, Creative Writing, and Mini Crosswords. For instance, in Game of 24, while GPT-4 with chain-of-thought prompting only solved  $4\%$  of tasks, our method achieved a success rate of  $74\%$ . Code repo with all prompts: https://github.com/princeton-nlp/tree-of-thought-llm.
Originally designed to generate text, scaled-up versions of language models (LMs) such as GPT [25, 26, 1, 23] and PaLM [5] have been shown to be increasingly capable of performing an ever wider range of tasks requiring mathematical, symbolic, commonsense, and knowledge reasoning. It is perhaps surprising that underlying all this progress is still the original autoregressive mechanism for generating text, which makes token-level decisions one by one and in a left-to-right fashion. Is such a simple mechanism sufficient for a LM to be built toward a general problem solver? If not, what problems would challenge the current paradigm, and what should be alternative mechanisms?
The literature on human cognition provides some clues to answer these questions. Research on "dual process" models suggests that people have two modes in which they engage with decisions – a fast, automatic, unconscious mode ("System 1") and a slow, deliberate, conscious mode ("System 2") [30, 31, 16, 15]. These two modes have previously been connected to a variety of mathematical models used in machine learning. For example, research on reinforcement learning in humans and other animals has explored the circumstances under which they engage in associative "model free" learning or more deliberative "model based" planning [7]. The simple associative token-level choices of LMs are also reminiscent of "System 1", and thus might benefit from augmentation by a more deliberate "System 2" planning process that (1) maintains and explores diverse alternatives for current
Figure 1: Schematic illustrating various approaches to problem solving with LLMs. Each rectangle box represents a thought, which is a coherent language sequence that serves as an intermediate step toward problem solving. See concrete examples of how thoughts are generated, evaluated, and searched in Figures 2,4,6.
choices instead of just picking one, and (2) evaluates its current status and actively looks ahead or backtracks to make more global decisions.
To design such a planning process, we return to the origins of artificial intelligence (and cognitive science), drawing inspiration from the planning processes explored by Newell, Shaw, and Simon starting in the 1950s [21, 22]. Newell and colleagues characterized problem solving [21] as search through a combinatorial problem space, represented as a tree. We thus propose the Tree of Thoughts (ToT) framework for general problem solving with language models. As Figure 1 illustrates, while existing methods (detailed below) sample continuous language sequences for problem solving, ToT actively maintains a tree of thoughts, where each thought is a coherent language sequence that serves as an intermediate step toward problem solving (Table 1). Such a high-level semantic unit allows the LM to self-evaluate the progress different intermediate thoughts make towards solving the problem through a deliberate reasoning process that is also instantiated in language (Figures 2,4,6). This implementation of search heuristics via LM self-evaluation and deliberation is novel, as previous search heuristics are either programmed or learned. Finally, we combine this language-based capability to generate and evaluate diverse thoughts with search algorithms, such as breadth-first search (BFS) or depth-first search (DFS), which allow systematic exploration of the tree of thoughts with lookahead and backtracking.
Empirically, we propose three new problems that challenge existing LM inference methods even with the state-of-the-art language model, GPT-4 [23]: Game of 24, Creative Writing, and Crosswords (Table 1). These tasks require deductive, mathematical, commonsense, lexical reasoning abilities, and a way to incorporate systematic planning or search. We show ToT obtains superior results on all three tasks by being general and flexible enough to support different levels of thoughts, different ways to generate and evaluate thoughts, and different search algorithms that adapt to the nature of different problems. We also analyze how such choices affect model performances via systematic ablations and discuss future directions to better train and use LMs.
We first formalize some existing methods that use large language models for problem-solving, which our approach is inspired by and later compared with. We use  $p_{\theta}$  to denote a pre-trained LM with parameters  $\theta$ , and lowercase letters  $x, y, z, s, \dots$  to denote a language sequence, i.e.  $x = (x[1], \dots, x[n])$  where each  $x[i]$  is a token, so that  $p_{\theta}(x) = \prod_{i=1}^{n} p_{\theta}(x[i] | x[1 \dots i])$ . We use uppercase letters  $S, \dots$  to denote a collection of language sequences.
Input-output (IO) prompting is the most common way to turn a problem input  $x$  into output  $y$  with LM:  $y \sim p_{\theta}(y|\text{prompt}_{IO}(x))$ , where  $\text{prompt}_{IO}(x)$  wraps input  $x$  with task instructions and/or few-shot input-output examples. For simplicity, let us denote  $p_{\theta}^{\text{prompt}}(\text{output} \mid \text{input}) = p_{\theta}(\text{output} \mid \text{prompt}(\text{input}))$ , so that IO prompting can be formulated as  $y \sim p_{\theta}^{IO}(y|x)$ .
Chain-of-thought (CoT) prompting [38] was proposed to address cases where the mapping of input  $x$  to output  $y$  is non-trivial (e.g. when  $x$  is a math question and  $y$  is the final numerical answer). The key idea is to introduce a chain of thoughts  $z_{1},\dots ,z_{n}$  to bridge  $x$  and  $y$ , where each  $z_{i}$  is a coherent language sequence that serves as a meaningful intermediate step toward problem solving (e.g.  $z_{i}$  could be an intermediate equation for math QA). To solve problems with CoT, each thought  $z_{i}\sim p_{\theta}^{CoT}(z_{i}|x,z_{1\dots i - 1})$  is sampled sequentially, then the output  $y\sim p_{\theta}^{CoT}(y|x,z_{1\dots n})$ . In practice,  $[z_{1\dots n},y]\sim p_{\theta}^{CoT}(z_{1\dots n},y|x)$  is sampled as a continuous language sequence, and the decomposition of thoughts (e.g. is each  $z_{i}$  a phrase, a sentence, or a paragraph) is left ambiguous.
Self-consistency with CoT (CoT-SC) [36] is an ensemble approach that samples  $k$  i.i.d. chains of thought:  $[z_{1\dots n}^{(i)},y^{(i)}]\sim p_{\theta}^{CoT}(z_{1\dots n},y|x)(i = 1\dots k)$ , then returns the most frequent output:  $\arg \max_y\# \{i\mid y^{(i)} = y\}$ . CoT-SC improves upon CoT, because there are generally different thought processes for the same problem (e.g. different ways to prove the same theorem), and the output decision can be more faithful by exploring a richer set of thoughts. However, within each chain there is no local exploration of different thought steps, and the "most frequent" heuristic only applies when the output space is limited (e.g. multi-choice QA).
3 Tree of Thoughts: Deliberate Problem Solving with LM
A genuine problem-solving process involves the repeated use of available information to initiate exploration, which discloses, in turn, more information until a way to attain the solution is finally discovered. — Newell et al. [21]
Research on human problem-solving suggests that people search through a combinatorial problem-space – a tree where the nodes represent partial solutions, and the branches correspond to operators that modify them [21, 22]. Which branch to take is determined by heuristics that help to navigate the problem-space and guide the problem-solver towards a solution. This perspective highlights two key shortcomings of existing approaches that use LMs to solve general problems: 1) Locally, they do not explore different continuations within a thought process – the branches of the tree. 2) Globally, they do not incorporate any type of planning, lookahead, or backtracking to help evaluate these different options – the kind of heuristic-guided search that seems characteristic of human problem-solving.
To address these shortcomings, we introduce Tree of Thoughts (ToT), a paradigm that allows LMs to explore multiple reasoning paths over thoughts (Figure 1(c)). ToT frames any problem as a search over a tree, where each node is a state  $s = [x, z_1 \dots i]$  representing a partial solution with the input and the sequence of thoughts so far. A specific instantiation of ToT involves answering four questions: 1. How to decompose the intermediate process into thought steps; 2. How to generate potential thoughts from each state; 3. How to heuristically evaluate states; 4. What search algorithm to use.
1. Thought decomposition. While CoT samples thoughts coherently without explicit decomposition, ToT leverages problem properties to design and decompose intermediate thought steps. As Table 1 shows, depending on different problems, a thought could be a couple of words (Crosswords), a line of equation (Game of 24), or a whole paragraph of writing plan (Creative Writing). In general, a thought should be "small" enough so that LMs can generate promising and diverse samples (e.g. generating a whole book is usually too "big" to be coherent), yet "big" enough so that LMs can evaluate its prospect toward problem solving (e.g. generating one token is usually too "small" to evaluate).
2. Thought generator  $G(p_{\theta}, s, k)$ . Given a tree state  $s = [x, z_{1\dots i}]$ , we consider two strategies to generate  $k$  candidates for the next thought step:
(a) Sample i.i.d. thoughts from a CoT prompt (Creative Writing, Figure 4):  $z^{(j)} \sim p_{\theta}^{CoT}(z_{i+1}|s) = p_{\theta}^{CoT}(z_{i+1}|x, z_{1\dots i})$  ( $j = 1\dots k$ ). This works better when the thought space is rich (e.g. each thought is a paragraph), and i.i.d. samples lead to diversity;
(b) Propose thoughts sequentially using a "propose prompt" (Game of 24, Figure 2; Crosswords, Figure 6):  $[z^{(1)},\dots ,z^{(k)}]\sim p_{\theta}^{\text{propose}}(z_{i + 1}^{(1\dots k)}\mid s)$ . This works better when the thought space is more constrained (e.g. each thought is just a word or a line), so proposing different thoughts in the same context avoids duplication.
3. State evaluator  $V(p_{\theta}, S)$ . Given a frontier of different states, the state evaluator evaluates the progress they make towards solving the problem, serving as a heuristic for the search algorithm to determine which states to keep exploring and in which order. While heuristics are a standard approach to solving search problems, they are typically either programmed (e.g. DeepBlue [3]) or
learned (e.g. AlphaGo [29]). We propose a third alternative, by using the LM to deliberately reason about states. When applicable, such a deliberate heuristic can be more flexible than programmed rules, and more sample-efficient than learned models. Similar to the thought generator, we consider two strategies to evaluate states either independently or together:
(a) Value each state independently:  $V(p_{\theta}, S)(s) \sim p_{\theta}^{value}(v|s) \forall s \in S$ , where a value prompt reasons about the state  $s$  to generate a scalar value  $v$  (e.g. 1-10) or a classification (e.g. sure/likely/impossible) that could be heuristically turned into a value. The basis of such evaluative reasoning can vary across problems and thought steps. In this work, we explore evaluation via few lookahead simulations (e.g. quickly confirm that 5, 5, 14 can reach 24 via  $5 + 5 + 14$ , or "hot_1" can mean "inn" via filling "e" in "-") plus commonsense (e.g. 1 2 3 are too small to reach 24, or no word can start with "tzxc"). While the former might promote "good" states, the latter could help eliminate "bad" states. Such valuations do not need to be perfect, and only need to be approximately helpful for decision making.
(b) Vote across states:  $V(p_{\theta}, S)(s) = \mathbb{1}[s = s^{*}]$ , where a "good" state  $s^{*} \sim p_{\theta}^{\text{vote}}(s^{*}|S)$  is voted out based on deliberately comparing different states in  $S$  in a vote prompt. When problem success is harder to directly value (e.g. passage coherency), it is natural to to instead compare different partial solutions and vote for the most promising one. This is similar in spirit to a "step-wise" self-consistency strategy, i.e. cast "which state to explore" as a multi-choice QA, and use LM samples to vote for it.
For both strategies, we could prompt the LM multiple times to aggregate the value or vote results to trade time/resource/cost for more faithful/robust heuristics.
Algorithm 1 ToT-BFS(x,pθ,G,k,V,T,b)	Algorithm 2 ToT-DFS(s,t,pθ,G,k,V,T,vth)
Require: Input x, LM pθ, thought generator G() & size limit k, states evaluator V(), step limit T, breadth limit b. S0←{x} for t=1,...,T do S′t←{[s,z] | s∈St-1,zt∈G(pθ,s,k)} Vt←V(pθ,S′t) St←arg maxS⊂St′,|S|=b ∑s∈S Vt(s) end for return G(pθ, arg maxs∈STVT(s),1)	Require: Current state s, step t, LM pθ, thought generator G() and size limit k, states evaluator V(), step limit T, threshold vth if t>T then record output G(pθ,s,1) end if for s′∈G(pθ,s,k) do ▷ sorted candidates if V(pθ,{s'}) (s)>vthres then ▷ pruning DFS(s',t+1) end if end for
4. Search algorithm. Finally, within the ToT framework, one can plug and play different search algorithms depending on the tree structure. We explore two relatively simple search algorithms and leave more advanced ones (e.g. A* [11], MCTS [2]) for future work:
(a) Breadth-first search (BFS) (Algorithm 1) maintains a set of the  $b$  most promising states per step. This is used for Game of 24 and Creative Writing where the tree depth is limit  $(T \leq 3)$ , and initial thought steps can be evaluated and pruned to a small set  $(b \leq 5)$ .
(b) Depth-first search (DFS) (Algorithm 2) explores the most promising state first, until the final output is reached ( $t > T$ ), or the state evaluator deems it impossible to solve the problem from the current  $s$  ( $V(p_{\theta}, \{s\})(s) \leq v_{th}$  for a value threshold  $v_{th}$ ). In the latter case, the subtree from  $s$  is pruned to trade exploration for exploitation. In both cases, DFS backtracks to the parent state of  $s$  to continue exploration.
Conceptually, ToT has several benefits as a method for general problem-solving with LMs: (1) Generality. IO, CoT, CoT-SC, and self-refinement can be seen as special cases of ToT (i.e. trees of limited depth and breadth; Figure 1). (2) Modularity. The base LM, as well as the thought decomposition, generation, evaluation, and search procedures can all be varied independently. (3) Adaptability. Different problem properties, LM capabilities, and resource constraints can be accommodated. (4) Convenience. No extra training is needed, just a pre-trained LM is sufficient. The next section will show how these conceptual benefits translate to strong empirical performance in different problems.
We propose three tasks that are hard even when sampling from the state-of-the-art language model, GPT-4 [23], using standard IO prompting or chain-of-thought (CoT) prompting. We show how
Game of 24	Creative Writing	5x5 Crosswords
Input	4 numbers (4 9 10 13)	4 random sentences	10 clues (h1. presented;..)
Output	An equation to reach 24 (13-9)*(10-4)=24	A passage of 4 paragraphs ending in the 4 sentences	5x5 letters: SHOWN; WIRRA; AVAIL; ...
Thoughts	3 intermediate equations (13-9=4 (left 4,4,10); 10-4=6 (left 4,6); 4*6=24)	A short writing plan (1. Introduce a book that connects...)	Words to fill in for clues: (h1. shown; v5. naled; ... )
#ToT steps	3	1	5-10 (variable)
Table 1: Task overview. Input, output, thought examples are in blue.
deliberate search in trees of thoughts (ToT) produces better results, and more importantly, interesting and promising new ways to use language models to solve problems requiring search or planning. Unless otherwise stated, we perform experiments using a Chat Completion mode GPT-4<sup>1</sup> with a sampling temperature of 0.7.
4.1 Game of 24
Game of 24 is a mathematical reasoning challenge, where the goal is to use 4 numbers and basic arithmetic operations  $(+ - * /)$  to obtain 24. For example, given input "4 9 10 13", a solution output could be  $(10 - 4)*(13 - 9) = 24$
Figure 2: ToT in a game of 24. The LM is prompted for (a) thought generation and (b) valuation.
Task Setup. We scrape data from 4nums.com, which has 1,362 games that are sorted from easy to hard by human solving time, and use a subset of relatively hard games indexed 901-1,000 for testing. For each task, we consider the output as success if it is a valid equation that equals 24 and uses the input numbers each exactly once. We report the success rate across 100 games as the metric.
Baselines. We use a standard input-output (IO) prompt with 5 in-context examples. For chain-of-thought (CoT) prompting, we augment each input-output pair with 3 intermediate equations, each operating on two remaining numbers. For example, given input "4 9 10 13", the thoughts could be "13 - 9 = 4 (left: 4 4 10); 10 - 4 = 6 (left: 4 6); 4 * 6 = 24 (left: 24)". For each game, we sample IO and CoT prompting for 100 times for average performance. We also consider a CoT self-consistency baseline, which takes the majority output from 100 CoT samples, and an iterative-refine approach on top of an IO sample for at most 10 iterations. At each iteration, the LM is conditioned on all previous history to "reflect on your mistakes and generate a refined answer" if the output is incorrect. Note that it uses groundtruth feedback signals about equation correctness.
ToT Setup. To frame Game of 24 into ToT, it is natural to decompose the thoughts into 3 steps, each an intermediate equation. As shown in Figure 2(a), at each tree node, we exact the remaining numbers and prompt the LM to propose some possible next steps. The same "propose prompt" is used for all 3 thought steps, though it only has one example with 4 input numbers. We perform a breadth-first search (BFS) in ToT, where at each step we keep the best  $b = 5$  candidates. To perform deliberate BFS in ToT, as shown in Figure 2(b), we prompt LM to evaluate each thought candidate as "sure/maybe/impossible" with regard to reaching 24. The aim is to promote correct partial solutions that can be verdicted within few lookahead trials, and eliminate impossible partial solutions based on "too big/small" commonsense, and keep the rest "maybe". We sample values 3 times for each thought.
<sup>1</sup>Experiments were done between May 5-16, 2023.
Method	Success
IO prompt	7.3%
CoT prompt	4.0%
CoT-SC (k=100)	9.0%
ToT (ours) (b=1)	45%
ToT (ours) (b=5)	74%
IO + Refine (k=10)	27%
IO (best of 100)	33%
CoT (best of 100)	49%
Table 2: Game of 24 Results.
Figure 3: Game of 24 (a) scale analysis & (b) error analysis.
Results. As shown in Table 2, IO, CoT, and CoT-SC prompting methods perform badly on the task, achieving only  $7.3\%$ ,  $4.0\%$ , and  $9.0\%$  success rates. In contrast, ToT with a breadth of  $b = 1$  already achieves a success rate of  $45\%$ , while  $b = 5$  achieves  $74\%$ . We also consider an oracle setup for IO/CoT, by calculating the success rate using best of  $k$  samples ( $1 \leq k \leq 100$ ). To compare IO/CoT (best of  $k$ ) with ToT, we consider calculating the tree nodes visited per task in ToT across  $b = 1 \cdots 5$ , and map the 5 success rates in Figure 3(a), treating IO/CoT (best of  $k$ ) as visiting  $k$  nodes in a bandit. Not surprisingly, CoT scales better than IO, and best of 100 CoT samples achieve a success rate of  $49\%$ , but still much worse than exploring more nodes in ToT ( $b > 1$ ).
Error analysis. Figure 3(b) breaks down at which step CoT and ToT samples fail the task, i.e. the thought (in CoT) or all  $b$  thoughts (in ToT) are invalid or impossible to reach 24. Notably, around  $60\%$  of CoT samples already failed the task after generating the first step, or equivalently, the first three words (e.g. "4 + 9"). This highlights the issues with direct left-to-right decoding.
4.2 Creative writing
Next, we invent a creative writing task where the input is 4 random sentences and the output should be a coherent passage with 4 paragraphs that end in the 4 input sentences respectively. Such a task is open-ended and exploratory, and challenges creative thinking as well as high-level planning.
Task setup. We sample random sentences from randomwordgenerator.com to form 100 inputs, and there is no groundtruth passage for each input constraint. As we find that GPT-4 can follow the input constraints most of the time, we focus on evaluating passage coherency in two ways: using a GPT-4 zero-shot prompt to provide a 1-10 scalar score, or using human judgments to compare pairs of outputs from different methods. For the former, we sample 5 scores and average them for each task output, and we find these 5 scores usually consistent, with a standard deviation of around 0.56 on average across outputs. For the latter, we employ a subset of the authors in a blind study to compare the coherency of CoT vs. ToT generated passage pairs, where the order of passages is random flipped over 100 inputs.
Baselines. Given the creative nature of the task, both IO and CoT prompts are zero-shot. While the former prompts the LM to directly generate a coherent passage given input constraints, the latter prompts the LM to first make a brief plan then write the passage, i.e. the plan serves as the intermediate thought step. We generate 10 IO and CoT samples per task. We also consider an iterative-refine  $(k \leq 5)$  method on top of a random IO sample for each task, where the LM is conditioned on input constraints and the last generated passage to decide if the passage is already "perfectly coherent", and if not generate a refined one.
ToT setup. We build a ToT with depth 2 (and only 1 intermediate thought step) — the LM first generates  $k = 5$  plans and votes for the best one (Figure 4), then similarly generate  $k = 5$  passages based on the best plan then vote for the best one. Here the breadth limit  $b = 1$ , as only one choice is kept per step. A simple zero-shot vote prompt ("analyze choices below, then conclude which is most promising for the instruction") is used to sample 5 votes at both steps.
Results. Figure 5(a) shows average GPT-4 scores across 100 tasks, where ToT (7.56) is deemed to generate more coherent passages than IO (6.19) and CoT (6.93) on average. While such an automatic metric might be noisy, Figure 5(b) confirms the finding by showing that humans prefer ToT over CoT in 41 out of 100 passage pairs, while only prefer CoT over ToT in 21 (other 38 pairs are found "similarly coherent"). Lastly, iterative-refine is more effective on this natural language task, where
Figure 4: A step of deliberate search in a randomly picked Creative Writing task. Given the input, the LM samples 5 different plans, then votes 5 times to decide which plan is best. The majority choice is used to consequently write the output passage with the same sample-vote procedure.
(a) GPT-4 coherency scores
(b) Human coherency comparison
Figure 5: Creative Writing results.