SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Data. To generate data for prompting and baseline training, we follow [25] to create 2000 tasks with randomly initialized scenes and expert trajectories. There are several settings for the evaluation. Simple tasks are the tasks that only require the rearrangement of one item generated from the same distribution as the training dataset. Comp. refers to the composition of simple tasks in order to rearrange multiple objects sampled from the same distribution as the dataset (e.g., "Put plate on kitchen table and chicken inside fridge" is the composition of "put plate on kitchen table" and "put chicken inside fridge,"). The composition of tasks increases the planning horizon, making it more challenging to complete. In evaluation, we also use the Novel Simple tasks with seen items (e.g., in the dataset, we have "put one plate on the kitchen table" and "put one chicken inside the fridge," and we use "put one plate inside the fridge" and "put one chicken on the kitchen table" to evaluate; these tasks are not included in the training dataset). For compositional tasks, we include Novel Compositional tasks, with 2 or 3 primary tasks composed, denoted as  $\text{NovelComp}(2)$  and  $\text{NovelComp}(3)$  (e.g., we have "put plate on kitchen table" and "put chicken inside fridge," in dataset but their composition "Put plate on kitchen table and chicken inside fridge" is not.) We also generate scenes at a Novel Apartment for testing, where the distribution of object positions differs from the dataset.
The expert data are generated by an Oracle agent implemented in [25]. The expert has the full knowledge of the environment (hence, does not need to understand where objects are likely to be placed) and uses handcrafted heuristics for completing various tasks. It uses regression planning to search for solutions to a task. We collect the actions and observations of the expert completing the tasks in the VirtualHome simulator as the dataset. There are 10,000 trajectories in total for training the baseline. To show the capability of LLMs when using a small training set, we only select 200 instances uniformly at random from the dataset as prompt candidates for the LLM model and policy when used in a few-shot mode with no fine-tuning. We also generated 800 tasks in total for evaluation.
Evaluation. We evaluate the success rate of completing the tasks within 30 steps, while a typical task can be finished within at most 15 steps. The task is considered successful if all the requirements of object positions are satisfied. For example, given the instruction "Put one apple inside the fridge," the task is successful if any apple is in the fridge. For simplicity, we don't consider the task of rearranging a very specific object, e.g., putting the leftmost apple in the fridge.
Baselines. We evaluate several baselines to compare. UCT [21]: We use the UCT algorithm to conduct planning without commonsense knowledge and use the ground-truth reward function in simulation. We use uniform distribution as the initial belief for states of objects. It is to provide evidence that commonsense knowledge improves planning efficiency. Finetuned GPT2 policy [22]: we use the training dataset with 10000 trajectories to fine-tune GPT-2 as the planning policy. This is to show that larger pre-trained LLM without fine-tuning outperforms the smaller model fine-tuned in specific tasks. GPT3.5 Policy [18]: LLM takes as input the instructions and history of actions and the currently visible objects to generate the next action. We use the LLM as the policy only, with a few examples as prompts to interact with the environments.
This baseline demonstrates the benefits of additional information from the commonsense model and algorithmic benefits from MCTS.
Main result. The main results of the experiments are shown in Table 1, reporting the success rate of our method
Table 1: Main results: mean ± standard error of success rate (%)
Method	Seen Home
Simple	Comp.	NovelSimple	NovelComp.(2)	NovelComp.(3)
UCT [21]	0.0±0.0	0.0±0.0	0.0±0.0	0.0±0.0	0.0±0.0
finetuned GPT2 policy [22]	81.3±2.4	59.0±6.7	41.2±7.1	30.9±2.8	2.3±1.5
GPT3.5 Policy [18]	83.4±6.8	47.0±7.8	74.3±4.0	48.2±8.8	5.4±2.0
GPT3.5-MCTS (Ours)	91.4±3.3	71.2±6.2	88.1±4.3	72.6±6.9	33.6±3.1
Unseen Home
Method	Simple	Comp.	NovelSimple	NovelComp.(2)	NovelComp.(3)
UCT [21]	0.0±0.0	0.0±0.0	0.0±0.0	0.0±0.0	0.0±0.0
finetuned GPT2 policy [22]	65.5±3.4	39.9±5.2	33.4±6.4	12.8±3.9	1.1±0.9
GPT3.5 Policy [18]	74.3±5.0	43.3±4.0	67.8±4.9	54.0±3.0	6.9±2.1
GPT3.5-MCTS (Ours)	82.9±3.2	71.9±5.6	79.3±3.3	70.4±6.4	38.8±3.4
and baselines in completing the tasks in VirtualHome environments. In this result, GPT3.5-MCTS outperforms all the compared baselines, especially for unseen situations. UCT works poorly in all conditions, as the poor model and the huge search tree make the planning intractable. Thus, we focus our discussion on comparing the finetuned GPT2 policy and GPT3.5 policy. For Simple, in-distribution tasks, the planning horizon is relatively short. Finetuned GPT2 policy, GPT3.5 Policy, and our method work reasonably well, but our method still outperforms the baselines. For Novel Simple tasks, finetuned GPT2 policy works significantly worse than GPT3.5 Policy and GPT3.5-MCTS. This is because the fine-tuning of narrow tasks results in a biased distribution of the policy and compromises generalizability. GPT3.5 Policy and GPT3.5-MCTS work better due to the LLM's few-shot planning capability. GPT3.5-MCTS works better for both situations. It benefits from the MCTS' look-ahead search that explore possible states for potential outcomes in order to make reasoned decisions.
For the Compositional, in-distribution tasks, the finetuned GPT2 policy and GPT3.5 policy get significantly worse performance, while GPT3.5-MCTS works far better. The finetuned GPT2 policy is trained by behavior cloning that suffers from compounding errors. Therefore, when the planning horizon gets longer, the influence of the errors accumulates and compromises the overall performance significantly. As for GPT3.5 Policy, the longer horizon potentially introduces more errors during planning, which might not be included in the prompt examples. Without suitable guidance from prompt, we cannot guarantee the GPT3.5 Policy will carry out suitable replanning when encountering errors. MCTS encourages exploration to a certain extent of different possible actions during searching, introducing additional guidance to the GPT3.5 policy to look into other possible solutions. This is because the action selection procedure in GPT3.5-MCTS is not purely determined by GPT3.5 Policy but also by the  $Q$  value and visit counts. Thus, MCTS encourages GPT3.5 Policy to explore other possible search directions instead of excessively applying certain actions sampled by itself.
Ablation study. We conduct ablation studies to see the individual contributions of different components within the GPT3.5-MCTS framework. The No Heuristic Policy version of GPT3.5-MCTS refers to the absence of PUCT guided by the GPT3.5 Policy for action selection. Instead, it solely relies on UCT with an initial commonsense belief derived from LLM. The variant employing the Uniform State Prior utilizes a uniform prior belief regarding states, in contrast to the LLM-generated initial belief employed during the search process. Lastly, the variant operating in a Fully Observable environment aims to assess the accuracy of LLM's knowledge in modeling the world.
Table 2 shows the results of our ablation study. The outcomes obtained under the No Heuristic Policy version highlight the significance of heuristic policies in facilitating MCTS to conduct efficient searches for complex and large-scale planning tasks. Con
Table 2: Ablation Study: mean ± standard error of success rate (%)
Method	Seen Home
Simple	Comp.	NovelSimple	NovelComp.(2)	NovelComp.(3)
GPT3.5-MCTS (No Heuristic Policy)	0.0±0.0	0.0±0.0	0.0±0.0	0.0±0.0	0.0±0.0
GPT3.5-MCTS (Uniform State Prior)	3.2±1.1	0.0±0.0	1.1±0.4	0.0±0.0	0.0±0.0
GPT3.5-MCTS (Fully Observable)	94.0±2.1	80.7±3.3	94.3±2.4	78.5±4.0	34.0±4.4
GPT3.5-MCTS (Ours)	91.4±3.3	71.2±6.2	88.1±4.3	72.6±6.9	33.6±3.1
Unseen Home
Method	Simple	Comp.	NovelSimple	NovelComp.(2)	NovelComp.(3)
GPT3.5-MCTS (No Heuristic Policy)	0.0±0.0	0.0±0.0	0.0±0.0	0.0±0.0	0.0±0.0
GPT3.5-MCTS (Uniform State Prior)	1.1±0.2	0.0±0.0	0.0±0.0	0.0±0.0	0.0±0.0
GPT3.5-MCTS (Fully Observable)	85.1±5.0	77.5±3.2	82.2±3.3	76.6±3.1	37.9±2.9
GPT3.5-MCTS (Ours)	82.9±3.2	71.9±5.6	79.3±3.3	70.4±6.4	38.8±3.4
versely, the results of the Uniform State Prior row indicate that incorrect world models compromise search performance. This is because the model of the world determines the  $Q$  value. The wrong model results in an inaccurate estimation of the  $Q$  value, misleading the search process toward irrelevant locations. The Fully Observable results demonstrate that GPT3.5-MCTS with perfect knowledge of the environment only slightly outperforms its counterpart without it, implying that the commonsense knowledge of LLM regarding world modelling suffices for practical purposes.
Failure analysis. Policy, model, and translation errors are the primary causes of failures. Among these, policy errors are responsible for the majority of the failures. Oftentimes, the policy produces unreasonable behaviours that mislead the search procedure. For example, it usually outputs inadmissible actions, such as "walk to the cutleryfork" where the "cutleryfork" is not in the observation. It also produces back-and-forth behaviours, resulting in an unreasonable heuristic and slowing the search procedure. For example, when putting objects inside the microwave, it is sometimes struck by repeatedly opening and closing the microwave. For model error, the predicted positions of objects are not always correct. Since a random rollout policy is employed, incorrect object states can result in higher  $Q$ -values than correct states, leading to misguided exploration. The wrong translation also
compromises the performance as we translate the response from LLM to admissible action or object names to ensure executability. This is partly caused by the VirtualHome environments, as the policy might not understand the underlying logic of the actions in VirtualHome, such as you have to walk close to interact with the object. Thus, if the LLM outputs "open fridge" but is not close enough to the fridge, the action will be translated to other admissible actions ("open fridge" is not inside the admissible actions for this case as it is invalid due to the setting of VirtualHome).
4 LLM as a model or a policy?
When would using LLM as a model outperform using LLM as a policy, and vice versa? We propose using the minimum description length (MDL) principle, also known as Occam's Razor from the philosophy of science, to gain insights into the issue. The MDL principle suggests choosing the method that has a shorter description when both methods fit the training data well. MDL has been formalized in various ways. One formal statement (from section 7.3 of [28]) is provided here:
Theorem 4.1 (Occam's Razor). Let  $\mathcal{H}$  be a hypothesis class and let  $d: \mathcal{H} \to \{0,1\}^*$  be a prefix-free description language for  $\mathcal{H}$ . Then, for every sample size,  $m$ , every confidence parameter,  $\delta > 0$ , and every probability distribution,  $D$ , with probability greater than  $1 - \delta$  over the choice of  $S \sim D^m$  we have that,  $\forall h \in \mathcal{H}, L_D(h) \leq L_S(h) + \sqrt{(|h| + \ln(2/\delta))/2m}$  where  $L_S(h)$  is the empirical loss of  $h$  on the  $S$ ,  $L_D(h)$  is the expected loss of  $h$ , and  $|h|$  is the length of  $d(h)$ .
According to Theorem 4.1, we can bound the expected loss of a solution  $h$  by the description length  $|h|$  and the training loss  $L_{S}(h)$ . We do not know the LLM training loss for using it as a model or as a policy, but for the purpose of gaining insights, it is reasonable to assume that they are both small. In that case, the MDL principle suggests selecting between a model or policy depending on which of them has the smaller description length given the description language.
Numerous caveats should be observed when using Theorem 4.1 to gain insights into the behaviour of LLM as a model and policy. The theorem assumes that the training data is independent and identically distributed (iid), which is likely not true in practice. Nonetheless, the qualitative behaviour is often similar for non-iid data. As the training data of GPT is unknown, when comparing the description length, we also assume that the training data for each subproblem is roughly the same. In addition, using the theorem to gain insights requires major assumptions on the predictor classes  $\mathcal{H}$  for model and policy; here, we assume that  $\mathcal{H}$  is the model (or policy) class that we are analysing and assume that LLM training using powerful approximators such as transformers has similar behaviour to training using the model (or policy) class. Finally, depending on how the model and policy are used, error propagation may need to be analysed separately.
Besides the multiplication example described earlier, we discuss an air travel planning task and the VirtualHome object rearrangement task examined earlier.
4.1 Travel planning
Consider planning for air travel from a starting city to the destination city. To solve the problem through L-Model, we need to have the model – the direct flights out of each city – together with a shortest path algorithm. The model can be represented as a graph, which is likely to be sparse in the real world. For a sparse graph, an adjacency list will give a compact representation. Assuming that the total number of edges grows proportionally to the number of cities,  $O(n \log n)$  bits would be sufficient to describe a graph with  $n$  cities, with approximately  $\log n$  bits used to describe each city in the adjacency list structure. The shortest path algorithm can be described by any reasonable programming language with a constant size. The method regarding L-Policy can be represented as a 2-dimensional table, where each row and column denotes the current and destination city, and the table entry describes the next city to fly to on the shortest path from the current city to the destination. The next city in the table can be described with  $\log n$  bits. As such, with  $n$  cities in the rows and columns, there should be approximately  $n^2 \log n$  bits in total. Thus, according to MDL, the L-Model has a shorter description length and should make fewer generalization errors than the L-Policy for travel planning with sufficiently large  $n$ .
Experiment. We conducted experiments about planning for air travel from a starting city to a destination city, which we analyzed above. We utilized GPT-3.5 to generate flight paths between cities. We compare it to the GPT-3.5 model-based approach: we use GPT-3.5 to predict neighbouring
cities connected by a direct flight, which feeds into the uniform-cost search (i.e., replace node expansion by GPT-3.5 as the world model).
We use the data from the Kaggle World cities database, select 68 cities with populations exceeding 5 million in different countries and 62 middle-size cities with populations between 500 thousand and 2 million, and use the Virtual Radar Server to get the flight routes dataset as ground truth. In our tests, we sampled 400 city pairs among large cities and 400 pairs among mid-size cities, evaluating path accuracy by verifying each direct flight exists. Paths were accepted if all flights were valid, even if they extended beyond our selected source and target cities. We evaluate the methods in two settings: predicting the flight route given two large cities and two mid-size cities.
Results. The main result shown in Fig 2 suggests that the LLM model + search algorithm consistently outperforms the LLM policy, supporting our analysis<sup>2</sup>. Furthermore, the performance gaps for mid-size cities are larger than for large cities. This is consistent with the fact that there are more mid-size cities than large cities (the gap between the description lengths of  $n^2 \log n$  for policies vs  $n \log n$  for models grows with the number of cities). Note that performance decreases
Figure 2: Flight planning results.
as the path length increases for both methods. As the number of predictions required for each path increases with path length, the probability of incorrect path prediction also increases.
4.2 Object rearrangement task
Consider a house with  $n$  movable objects,  $m$  containers, and  $k$  rooms. If we use L-Model, we must describe the model and search algorithm (i.e., MCTS). The model can be represented as a sparse graph with objects, containers, and rooms as nodes, and weighted directed edges signifying a "located in" relation between the nodes, with the weights specifying the probabilities. Assume that each weight is specified with a constant number of bits. Each of the  $n$  objects can be located within the  $m + k$  containers or rooms, so each object edge would require approximately  $\log(m + k)$  bits to describe. Each of the  $m$  containers can be located in  $k$  rooms, so each container edge would require approximately  $\log(k)$  bits to describe. Assume further that the degree of each object and container node in the graph is bounded by a constant; the entire graph then requires  $O(n \log(m + k) + m \log(k)) = O((m + n) \log(m + k))$  bits to describe. The MCTS algorithm should be described by a programming language with a constant size. For the L-Policy, tasks can be designated using object-container pairs. Each object-container policy can be defined by a sequence of actions, e.g. "walk to the fridge, open the fridge," until the object is found, followed by a sequence of actions until the destination container is found. Each action takes  $O(m + k)$  bits to describe. Assuming search sequences and the size of each object-container policy are bounded by a constant. Describing the policies for all  $mn$  object-container pairs requires  $O(mn \log(m + k))$  bits.
The composed tasks provide further challenges for L-Policy. Composing tasks increases the description complexity of policies to  $O((mn)^N \log (m + k))$ , where  $N$  is the number of composed tasks if the composition is not exploited in the policies. For the L-Model, decomposition is automatically done in MCTS at the expense of more computation, whereas for the L-Policy, the LLM must learn to do the decomposition. This may make the problem of learning the decomposed policy computationally more difficult and less likely to be approximated by the LLM in practice.
This analysis indicates that the L-Model has a shorter description length than the L-Policy. According to the MDL principle, the L-Model will likely have a lower error rate than the L-Policy. However, in this case, we do not have an algorithm that is guaranteed to be efficient for solving L-Model. Instead, we use the L-Policy as a search heuristic to obtain a practically effective search algorithm. As predicted by the MDL principle, the search with the LLM-induced world model, when limited to the neighbourhood of the LLM-induced policy, provides improved performance over L-Policy.
While we have discussed problems where the L-Model outperforms the L-Policy, we would also expect the L-Policy to outperform the L-Model when the description length of policies is shorter than
2Single-edge results are omitted for mid-size cities as there are no such routes in the experiments.
the models. For example, when recommending a tourist itinerary for a city, the description length of the itinerary should be shorter than describing all the places of interest plus the reward and the length of time recommended for visiting each location. In such a case, the LLM may be able to do better in recommending an itinerary than in providing accurate information on all places of interest for planning itineraries.
In the multiplication problem, the model-based approach used a known efficient multiplication algorithm. In the air travel planning task, an efficient shortest-path algorithm is used. However, for the object rearrangement problem, we do not have an efficient search algorithm. In this case, we demonstrate another strength of LLMs – as a policy, it can be used as a heuristic for improving the efficiency of search algorithms.
Task planning has a long-standing history in the AI research community. In the early stage, many studies [1, 15, 14, 12] focused on task planning in a discrete state space with deterministic transitions. These methods are intractable for large-scale, long-horizon problems. Recently, researchers have used learning-based methods to learn planning policies directly [22] or to learn search heuristics to accelerate planning [32, 31, 41, 7, 29]. Those policies or heuristics are not generalizable to other unseen settings. Most recently, the pre-trained LLMs have been applied as few-shot policies for task planning [18, 2, 19]. However, the planning policy may not have good compositional generalizability. In the robotics community, many studies proposed that task planning should be integrated with physical-level motion planning, i.e., task and motion planning (TAMP) [11, 12, 9]. This paper focuses on large-scale task planning with partial observations and large, object-centric domains, in which classical planning is intractable and learning-based methods require massive data.
Various approaches are proposed to scale up to large-scale planning problems. Initially, the Monte Carlo method [21, 8, 6] is proposed to tackle intractable large-scale problems using random sampling in the tree search. However, further scaling up to the problem with a large domain with sparse rewards requires massive sampling. Silver et al. [32, 31] integrate MCTS with deep learning to bias the action selection and reduce sampling. The idea has been successfully applied in large-scale planning scenarios [32, 31, 7, 27]. Most recently, studies show that LLM can be a few-shot open-loop [18, 2] and closed-loop [19, 37] planning policy in the large domain, while it may suffer from hallucinations. In this paper, we show that LLMs' commonsense knowledge can guide a search algorithm, reducing the search space sufficiently for practical planning and producing reasoned results.
How to use LLMs for a planning problem? LLMs have been used as a few-shot policy for language-conditioned task planning [18, 2, 19], or a policy for multi-step reasoning problems [38, 42]. However, recent research [10] also suggests that transformer LLMs are inherently limited for solving multi-step reasoning problems. In addition, some studies try to use LLMs as heuristics [43] or transition function [13] in MCTS, boosting the performance in coding or small-scale reasoning. However, the literature has not discussed utilizing LLMs as a world model in depth. We show the benefits of using LLM to model the world, as well as using MDL analysis to decide how to use LLM for planning problems.
We use LLMs as the commonsense world model and the heuristic policy within MCTS to achieve better-reasoned decision-making for daily tasks. MCTS enables LLM to leverage its world modelling knowledge for informed reasoning and explore new combinations of actions to tackle novel tasks. LLM helps MCTS through the biased sampling of states and actions, improving its efficiency in resolving complex task-planning problems. Our analysis and empirical evidence suggest that, for certain real-world domains, if the description length of the world is substantially shorter than policy, using LLM as a model in a model-based approach is a better option than using LLM as policy.
Limitations. The runtime of LLM-MCTS is currently hindered by multiple LLM calls. The details of runtime performance are in Appendix E. While our method requires multiple LLM calls, it provides substantially improved results. There are also various ways to enhance runtime performance like using smaller LLMs like Llama [35, 36] or distilling LLM's knowledge into a smaller model [30, 16, 23]. Those are interesting avenues for future research.