System 1.x: Learning to Balance Fast and Slow Planning with Language Models
Authors: Swarnadeep Saha
Archiki Prasad
Justin Chen
Peter Hase
Elias Stengel-Eskin
Mohit Bansal
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Planning
Primary area: foundation or frontier models, including LLMs

Abstract: Language models can be used to solve long-horizon planning problems in two distinct modes. In a fast 'System-1' mode, models directly generate plans without any explicit search or backtracking, and in a slow 'System-2' mode, they plan step-by-step by explicitly searching over possible actions. System-2 planning, while typically more effective, is also computationally more expensive and often infeasible for long plans or large action spaces. Moreover, isolated System-1 or System-2 planning ignores the user's end goals and constraints (e.g., token budget), failing to provide ways for the user to control the model's behavior. To this end, we propose the System-1.x Planner, a framework for controllable planning with language models that is capable of generating hybrid plans and balancing between the two planning modes based on the difficulty of the problem at hand. System-1.x consists of (i) a controller, (ii) a System-1 Planner, and (iii) a System-2 Planner. Based on a user-specified hybridization factor x governing the degree to which the system uses System-1 vs. System-2, the controller decomposes a planning problem into subgoals, and classifies them as easy or hard to be solved by either System-1 or System-2, respectively. We fine-tune all three components on top of a single base LLM, requiring only search traces as supervision. Experiments with two diverse planning tasks -- Maze Navigation and Blocksworld -- show that our System-1.x Planner outperforms a System-1 Planner, a System-2 Planner trained to approximate A* search, and also a symbolic planner (A* search), given a state exploration budget. We also demonstrate the following key properties of our planner: (1) controllability: by adjusting the hybridization factor x (e.g., System-1.75 vs. System-1.5) we can perform more (or less) search, improving performance, (2) flexibility: by building a neuro-symbolic variant composed of a neural System-1 planner and a symbolic System-2 planner, we can take advantage of existing symbolic methods, and (3) generalizability: by learning from different search algorithms (BFS, DFS, A*), we show that our method is robust to the choice of search algorithm used for training.

Published as a conference paper at ICLR 2025
SYSTEM-1.2: LEARNING TO BALANCE FAST AND SLOW PLANNING WITH LANGUAGE MODELS
Swarnadeep Saha
Archiki Prasad
Justin Chih-Yao Chen
Peter Hase
Elias Stengel-Eskin
Mohit Bansal
UNC Chapel Hill
ABSTRACT
Language models can be used solve long-horizon planning problems in two distinct modes. In a fast 'System-1' mode, models directly generate plans without any explicit search or backtracking, and in a slow 'System-2' mode, they plan step-by-step by explicitly searching over possible actions. System-2 planning, while typically more effective, also computationally more expensive and often infeasible for long plans or large action spaces. Moreover, isolated System-1 or System-2 planning ignores the user's 's end goals and constraints (c.g.. token budget), failing to provide ways for the user to control the model's behavior. To this end, we propose the System-1.zz Planner, a framework for controllable planning with language models that is capable of generating hybrid plans and balancing between the two planning modes based on the difficulty of the problem at hand. System-1... consists of (i) controller, (ii) a System-1 Planner, and (iii) System-2 Planner. Based on a user-specified hybridization factor x governing the degree to which the system uses System-1 vs. System-2. -2, the controller decomposes a planning problem into sub-goals, and classifies them as casy or hard to be solved by either System-1 or System-2, respectively. We fine-tune all three components on top of a single I.LM, requiring only search traces as supervision. Experiments with two diverse planning tasks Maze Navigation and Blocksworld - show that our System-1.2 Planner outperforms System-1 Planner, a System-2 Planner trained to approximate A* search, and also a symbolic planner (A* * search), given a state exploration budget. We also demonstrate the following key properties of our planner: (1) controllability: by adjusting the hybridization factor x (e.g.. System-1. vs. System-1.5) we can perform more (or less) search, improving performance, (2) flexibility: by building a neuro-symbolic variant composed of a neural System-1 planner and symbolic System-2 planner, we can take advantage of existing symbolic methods, and (3) generalizability: by learning from different search algorithms (BFS, DFS, A*), we show that our method is robust to the choice of search algorithm used for training.
INTRODUCTION
key feature of intelligence both human and artificial is the ability plan. Indeed, planning has been a major topic of Al research for most of its history (Russell and Norvig, 2016), with applications ranging from navigation and robotics to manufacturing and story-telling. These days, auto-regressive Large Language Models (LLMs) are increasingly used for various long-horizon planning and reason- ing tasks (Bubeck et al., 2023; Wei et al., 2022; Yao al., 2024). Despite some initial promise, recent careful investigations have highlighted LLMs' shortcomings
classical planning tasks (Pallagani al., 2023; Valmeekam et al., 2023; 2024; Momennejad et al., 2024). Supervising models with correct plans has also led to limited success, especially for out-of-distributiin generalization (Dziri et al., 2023). Fundamentally, this issue is attributed to a lack of explicit search, backtracking, or learning from mistakes (Kambhampati et al., 2024; Bachmann and Nagarajan, 2024; Gandhi et al.,
"oode available https: //ta::///ittii com/ swa www.ubaau/ae.. /Syttmms

---

Published as a conference paper at ICLR 2025
System
System 2
System 1.x
This is casy use Systom 1 Y
Fast
Accurato
Fast
Accurate
Fast
Accurate
GD
Tha is hard System22 48
2/2
>,2
0,0
wantt 1xd
14.0 &
14,41
Starting
Obstade
States Explored (Nlid)
This casy use System 1
LLM Flanner
End Paint
State Taken
States Explored Onvalic)
Figure 1: Comparative overview of a System-1 Planner, System-2 Planner, and our hybrid and controllable System-1.a Planner. Given a Maze Navigation task, a System-1 Planner directly generates a plan as sequence of steps without search, thus making it fast but inaccurate. A System-2 Planner performs search before generating a plan and hence is more accurate but much slower. Our System- 1.. Planner generates hybrid plans, where the level of hybridization is controlled by a user-defined hyperparameter. Hybrid plans balance System-1 1 sub-plans for easier sub-goals and System-2 sub-plans for harder ones, making System- ..r fast, accurate, and controllable.
2024), all of which are essential components of standard planning systems. LLMs directly generating plans generally resemble humans' fast 'System-1" decision decision-making making (Kahneman, 201 1), characterized by quick, intuitive judgments rather than slow, deliberate planning (Kambhampati et al., 2024).
As potential remedies, slow 'Sysstmm2' -inspired LLM planning approaches have emerged that either (1) augment LLMs during inference with symbolic search algorithms (Yao et al., 2024; Sel et al., 2023; Ahn et al., 2022) or (2) directly teach LLMs to carry out search themselves (Gandhi et al., 2024; Lehnert et al., 2024). While the former approach of using external search tools can enhance test-time performance, the latter approach- training models to search has the advantage of resulting in single unified search model and also discovering better and more efficient search strategies (Gandhi et al., 2024). However, such rainiig-timme approaches also face challenges due to high computational demands and potential infeasiblity. For instance, one such recently proposed (trained) System-2 method, Searchformer, uses token sequences that verbalize entire search trajectories (Lehnert et al., 2024). Consequently, during inference, based on the action space and the plan length, these models explore a large number of states, resulting in an increased number of generated tokens. The cost of exploration is further exacerbated by the fact that Transformers (Vaswani et al., 2017) typically exhibit quadratic complexity w.r.t. the sequence length. Thus, as sequence length increases, the computational complexity grows, significantly impacting memory usage and speed, rendering many long-horizon tasks infeasible due to token limits. A more promising alternative would thus be to avoid incurring high cost on all samples and steps by training a planner with smarter, more balanced allocation of compute, i.c., one that uses System-1 for easier (sub- sub-))problems while reserving the more expensive System-2 for harder (sub-)problems based on the budget.
To this end, we propose the System- Planner, drawing inspiration from the Dual- Pral-Process Theory (Wason and Evans, 1974; Kahneman, 201 1) that argues for the co-existence of fast and slow planning in humans. The fast system, relying on expertise and quick judgments, can be cheaply and quickly applied either instinctively system is more costly, involving sequential processing and relying on working memory to formulate more deliberate plan and includes the consideration of hypothetical alternatives (Evans, 2003). Crucially, humans can switch between these two modes as determined by the task at hand (Kahneman, 2011). Based on this distinction, we develop the System-1.rr Planner, controllable framework for long-horizon planning with LLMs that is capable of generating hybrid plans interleaved with System-1 and System-2 sub-plans System-1.rr offers a promising middle ground between a fast but inaccurate System-1 Planner, used for easy problems, and an accurate but slow System-2 Planner, used for hard problems (see Fig. for an overview).
in areas where an individual has acquired expertise. The slow
System-1.ar .x is composed of three trained components: (1) a controller that decomposes planning problem into easier and harder sub-goals, (2) a System-1 Planner that solves the easier sub-goals
For example, full System-2 x mazes with plan length of requires ~ 1500 tokens average.
2

---

Published as a conference paper at ICLR 2025
without any explicit search or exploration, and (3) System-2 Planner that solves the harder sub sub-goals by deliberately searching over possible actions. One of the primary strengths of System-1.a x is its train-rima controllability - based on a user-specified hyperparameter that ultimately governs the degree of hybridization between the System-1 and 2 planners. It also offers test-fime controllability, by which an already trained controller can be adjusted towards solving more or fewer problems using System-2, providing another level of inference-time compute." Importantly, System-1.x is a fully (LLM-based) neural planner built on top of one base LLM, using supervision only from search traces and not relying on any external solvers or verifiers. Thus, it has an advantage over inference- time methods that rely on symbolic planners which may not exist for certain domains, making data-driven approaches like System-1..r self-contained and easier to apply to new domains and to scale up. At the same time, our LLMM-based System-1.zz can also be seamlessly converted into a neuro-symbolic planner by replacing the System-2 Planner with symbolic solver (c.g., A+), allowing us to integrate powerful symbolic tools when they are available.
[0, 1], we train controller
We conduct experiments on the domains of Maze Navigation and Blocksworid to show that System- 1. Planner matches and generally outperforms a System-1 Planner, a System-2 Planner, and a System-1.2 variant (without sub-goal decomposition) at all budgets by generating up to 33% more valid plans. We also report similar findings with our neuro-symbolic System-1.0z that in fact typically outperforms symbolic search, beating A by up 39% at a fixed budget of explored states and matching it at maximum budget. Next, we show the controllability of our framework by training System-1.75 Planner that, compared to a System-1.5 Planner, trades off efficiency for greater accuracy. This trend can be continued to recover the full System-2 performance. System-1.2 also generalizes different search algorithms (BFS, DFS, A*) and exhibits exploration behavior that closely resembles the corresponding algorithm it is trained on. Overall, the System- Planner introduces novel way of applying LLMs to controllable and hybrid planning. optimizing for both accuracy and efficiency. This, in turn, allows System- to outperform both System-1 and -2 at fixed budget. By intelligently allocating more resources to harder sub-goals, System-1.ar is able to save its System-2 budget for cases where it is necessary.
2 SYSTEM-1.2: CONTROLLABLE AND HYBRID PLANNING WITH LLMS
2.1 BACKGROUND AND PROBLEM SETUP
A planning problem can be modeled as a Markov Decision Process M = (S,A, R,, defined with a set of states S, a set of actions A on any given state, transition function S x A - S defining transitions between pairs of states based on an action, and a reward function R: S + R assigning a reward for reaching the goal state. Given such planning problem, start state so E S and goal state 8y E S, the objective of classical planning is to generate a plan P = (a1,..,, ... an). ),), as a sequence of n actions a, E A that helps transition from the start state to the goal state. Using this notation, we define some salient concepts in a planning problem below:
System-1 Planner. We use the term "System-1 Planner" to denote any planner that directly generates a plan sequence of actions without conducting any explicit exploration search. System-1 Planners are typically only model-based, trained to only output the final plan. Within the scope of this study, a System-1 1 Planner will specifically refer to a neural LLM-based planner.
We use this term to denote planners that do conduct explicit search before generating a plan. Unlike model-based System-1 Planners, System-2 Planners can also be symbolic (c.g., search algorithms like Depth First Search, Breadth First Search, A+), A model-based System- 2 Planner, within the scope of this study, will specifically refer to LLM trained to verbalize search trajectories (defined below) before generating a plan. Broadly, the terms System-1 1 and System-2 will refer LLM--bseed planners while symbolic algorithms will be addressed separately.
System-2 Planner.
Search Trajectory. Given a search algorithm, we define a search trajectory or a search trace as the step-by-step execution of the algorithm for an input problem. In the context of LLMs, this will mean a verbalized representation of search trajectory. Trajectories consist of the final plan, all intermediate actions explored during search, their corresponding states, their validity, etc.
FE.., can train System- 1.5 model and then add bias term to the controller test time to increase the probability of System-2, inducing it to like System-2 model (or System-1 System-1.777 775, etc.).
3

---

Published as a conference paper at ICLR 2025
Valid Plan: A plan P (a1,.... .... .....p))) is considered valid if starting from the start state so, se- quentially executing all taken actions in the plan leads to the goal state sy. or more formally, (T.....T an)...,nn11).. (0,-1),,,,))) a,,) Sg. Plan Validity is metric measuring the fraction of valid plans that reach the goal state.
#States-Explored: Given planning problem and planner (System-1, 2, or 1.2), we define #States #States-Explored (SE, in short) the set of all states (both valid well as invalid) that are visited by the planner on its way to generating the plan. In the case of an LLM planner, this corresponds to the number of states that the planner needs to verbalize for exploration (see right of Fig. 2 for an example output), and thus adds to the token cost of generation. Since a System-1 Planner generates plan directly without any additional explorations, it only explores the states that are part of its plan, i.e., Sys nt where is the length of the generated plan. A System-2 Planner, on the other hand, will perform additional state explorations before generating a plan, resulting in SE syy2 > SEsyst. Note that "States-Explores underlying search algorithm or the LLM planner learning from that algorithm's traces. For example, A", being an informed search algorithm, will typically explore fewer states than Breadth First Search. Plan Validity and #States- Explored will be our two metrics of interest, using which we will compare the performance at varying budgets of all planners (discussed in $4).
for System-2 Planner will vary based on the
Sub-goal. Finally, planning problems are typically compositional, meaning that they can be decomposed into sub-problems, which we will refer as sub-goals. Formally, given plan P between states Su and $g. sub-goal can be characterized by pair of states (s;, 8z) s.t. i, E [0. gl. S j. where 8; and Sj are intermediate states on the plan to get from Sy to Sg.
Problem Setup. Having defined the key terms above, we now describe the setup underlying our LLM-based System-1.r Planner. To build it, we assume access to two things: (1) a training dataset D of search trajectories for N planning problems, and (2) user-specified hybridization factor x E [o, 1]. Given a planning problem of initial state sp, goal state 8g. and a gold plan P, let us denote the cor-
responding gold search trajectory as T, making the training dataset D {(s Sg (()), ((")))())))))))))) >- In our experiments, the gold plans and the corresponding trajectories will be obtained using symbolic System-2 2 Planner (e.g.. A search) but more generally, the source of these trajectories could be any neural or symbolic engine capable of exploration or search. The hybridization factor r is a user-defined hyperparameter that will determine the balance between System-1 and System-2.
(6)
2.2 SYSTEM-L.2 1.2 OVERVIEW
System- .ar Planner consists of three trained components: (i) Controller for decomposing planning problems into sub-goals and classifying them as easy or hard, (ii) a System-1 Planner for fast planning of easy sub- sub-goals, and (iii) a System-2 Planner for slow planning of hard sub-goals. Fig. 2 shows an overview of our method. We train all three components by automatically generating data from the gold search traces in a way that also respects the user's constraints (specified via the hybridization factor). Below, we describe each component and how they combine to make System-1...
2.3 TRAINING DATA GENERATION FOR SYSTEM-1.2
System-1 Planner Training Data. We denote it as 13ys11((5) , p( JM where the input is natural language description of the planning problem consisting of start state so and goal state oy and the output is the corresponding plan in natural language p(() (see Appendix D for examples). For Maze Navigation, these gold plans are also optimal plans (i.e.. having the minimum length) while for Blocksworld, they may include additional explorations, making them sub-optimal times (though always valid).
System-2 Training Data. Similar to System-1, the input to System-2 Planner is also natural language description of the planning problem. However, the target output of the System-2 2 Planner is verbalized search trajectory (+) as described in $2.1 (refer to Appendix D for examples). This (0)) TO)N
yields a training dataset of Dsyx2 [(5 $9
(6)
Controller Training Data. Recall that the System-1.z Planner is designed to minimize state explorations by only performing System-2 planning as needed, ie., saving states with System-1 for easier sub-goals and reserving 2 for harder sub-goals. Hence, the role of the controller is
4

---

Published as a conference paper at ICLR 2025
Constructing Training Data from. Soarch Traces
Controlte (Easy Syslem-1 Hard 9yslem3)
Sydtem- Planner Gutput
Syskes
Syslmm Search
Centroler
Yamre Cotter 211
Uiwer Chatos
Eaplaxiny aetlen sape
(P)
Descepoted Fdbqalk purt Eu
Engientay anltes
Sear sirte 1%, 1)
chr wree saate and annee
Date Th: testis
ontee
Fast It, shatul
Supervised Fine-turing with an LLM
Systame Planner
Cake sagi Iee alale
System-1
Ploreer
System=22 Planner
Controlio
Training
Inferaace
Staniig Stats
Grdl State
Tauar Strve
Eapkend Wild Susee
Eaplore Strse
Dasy Subgoai
Hard Subggoa
Ouatapped Stores
Figure 2: Overview of our System- Planner. Left: The process for training data construction. We generate three kinds of data to fine-tune our three components: (I) System-1 data (linear chains), (2) System-2 data (search trees), and (3) controller data (consisting of sub-goals and easy/hard annotations) conditioned controller generates the sub-goals and predicts whether each is easy or hard. Then, it invokes System-1 and System-2 for easy and hard sub-goals respectively construct the final plan.
user-specified hybridization factor Right: During inference, the
two-fold: (i) decomposing a problem into sub-goals characterized by start and end state (ss, 8z). and (ii) classifying those sub-goals as easy or hard so that System-2 can be invoked only for the hard sub-goals. However, most planning domains lack gold annotations of sub-goal decompositions and their easy and hard classifications. Therefore, we propose an automatic data generation process for training our controller, as described below and more formally in Algorithm 1. The inputs to preparing the controller data are (1) the gold plans [P()) Ji=1 that will be decomposed into sub-goals, (2) the user-defined hybridization factor z, and (3) hardness function h(.,)) for (sub-)goals.
Step 1: Defining and Ranking using Hardness Function. A hardness function h(s;,)) takes a start state s; and a goal state 8j as input, and returns a scalar value estimating the hardness of the goal or the sub-goal. An example of hardness function for Maze Navigation task is the number of obstacles in a sub-maze, where a higher value indicates greater difficulty. Appendix A and Appendix B more concretely describe and compare these hardness functions for different tasks, respectively. Generally, hardness functions can be constructed for most planning domains where standard heuristics are frequently applied for various search techniques." 4 Using this hardness function between the start and the goal state (1 sg), we generate rank ordering of the training examples.
Step 2: Annotating Easy Instances as System-l-only, Intuitively, some planning problems in the train set can be solved successfully via only System-1 Planner. For such instances, no further decomposition is needed. These correspond to the first (1 - x) N easiest samples with the least I).:)). Note that this is a definition of hardness, and what considered as easy determined by the hybridization factor æ. For example, when r 0.0, all samples will be annotated as easy, making System-1.x the same as System-1.
Step 3: Annotating Hard Instances Hybrid System-1 + System-2. For the remaining x x N (hard) samples, we assume that they can be decomposed into sub-goals, each of which be assigned either System-1 or System-2. However, to ensure overall task success, the decompositions should neither be too coarse, i.e., requiring further decompositions, nor too fine- grained, making them susceptible to decomposition errors (Prasad et al., 2024). Hence, we adopt sliding window approach that optimizes for the following: find a contiguous chunk of length x n of an n-length plan to be assigned System-2, such that it corresponds to the hardest sub-goal and the sub-goals on either side of it correspond to the easiest (and hence assigned System-1). More formally, this requires us to solve the following constrained optimization problem:
kj) ar prrgmin,,oe(o.aill (Su, s.) h(su, s.) h(su, sy). s.t. |e ul X n
Given a start state so and goal state Sg. we want to find two intermediate states sy and sy between so and sy along the length of the plan, such that the hardness of that sub-goal h(s), sk) is maximized (and hence assigned System-2) while the hardness of the other two sub-goals h (0. 85) and h(SE, 8g)
httpps: :/ //wwww fast- dwwwwardwddooooooo.doo org/ /oc// Wva uator
5

---

Published as a conference paper at ICLR 2025
is minimized (and hence assigned System-1). This process results in way decomposition.
The final result is a controller training dataset Dr {(5)) .8 ). 13 1 where for each planning problem, we have annotated list of t 2.3) sub-goals {9, " j==1 and their corresponding System-1/System-2 assignments. In practice, the components we use for controller data generation (e.g., sub-goal decomposition strategy, hardness heuristic) can easily be swapped with other variants, based on the problem at hand (further explored in Appendix B).
2.4 TRAINING AND INFERENCE OF SYSTEM- 1.. PLANNER
Using the data generation process described above, we finetune a single base LLM to behave as System-1 Planner, a System-2 Planner, or a controller. During inference, we first use the controller to generate a meta-plan- a list of sub-goals and their easy versus hard assignments. After solving each with the appropriate planning model, we concatenate the sub-plans from the two planners, in order, to construct the final plan (see Fig. 2 for an example of a concatenated plan, denoted by '+').
Train- and Test-time Controllability. We design System-1.x such that there is a single point of control for training the system, i.e., the hybridization factor z, and this control lies with the end-user. Setting it adjusts the training data for the controller, i.e., what is annotated as easy versus hard, both across samples (Step 2 of the algorithm) and within samples (with sub-goals; Step 3 of the algorithm). This ultimately yields the user-specified level of hybridization. Hence, System- offers training time control of compute (e.g.. one could train a System-1.25 1.5, a 1.75 Planner). Furthermore, the controller's design also offers an additional degree of test-time control: since the controller classifies sub-goals as System-1 or 2, the end-user could also re-balance the model towards either system based on the controller confidence threshold. This means that an already trained System-1.5 Planner, for example, can be biased behave System-1.25 time. In our experiments, we explore how both train-time and test-time control impacts performance.
System-1. 1.75 at test
EXPERIMENTAL SETUP
Tasks. We evaluate System-1.xr Planner two classical planning tasks that are challenging for LLMs (Valmeekam et al., 2023; Lehnert et al., 2024): (I) Maze Navigation and (2) Blocksworld. In Maze Navigation, given 2D maze with some cells containing obstacles and four permissible actions ('left,, 'right', 'up". 'down'), ). the objective is to find a path from a start state to goal state by avoiding the obstacles. We randomly generate a balanced dataset of 4K planning problems (split into 3200/400/400 samples) with 5x5 mazes, 40% of the cells containing obstacles, and having optimal plan lengths between to Next, to test out-of-distribution (OOD) generalization to longer plans, we experiment with Blocksworld: a task of moving blocks to another goal configuration by only moving one unstacked block at a time. Following the data creation algorithm in Bohnet et al. (2024), generate problems consisting of 4-7 blocks (without repetition). From there, we create muin/validation/ess split of 3000/250/200 samples where the train and the validation split consist of samples with plan lengths 1-6 and the test split consists of samples with plan lengths 7-10.
Baselines. We compare our System-1..z Planner with (1) a System-1 Planner, (2) a System-2 Planner, and (3) System-1.r Planner without sub-goal decomposition. The latter is also hybrid planner but the controller is trained to predict an instance as either fully System-1 or fully System-2, thus allowing us evaluate the effectiveness of sub-goal decomposition of our full System-1..r model. For showing the effectiveness of our neuro-symbolic System- 1.x variant, we compare it to A' by matching their "'States-xxplores Under the umbrella of System-2 Planners, we train multiple variants that are fine-tuned with data obtained from different search algorithms. In particular, we consider two uninformed search algorithms (Breadth-First Search and Depth-First Search) and an informed search algorithm (A*). Given the superiority of A*, our primary experiments will be based on A" traces while we reserve the other search algorithms for further analysis in $4.3. Refer to Appendix A for short background on A
Evaluation Metrics. We compare all methods along two axes: plan validity and cost. As defined in 1, plan validity is given by the fraction of valid plans generated by planner while for cost, we compute the average #States-Expporee by a planner before reaching the goal. Note that in order
Note that when u 0 u = g. there will be two decompasitions rather than three.

---

Published as a conference paper at ICLR 2025
Sys-1
5ys-1.5 w/o sub-goals
Truncation
Sys-1
Sys-1.5 weo sub-goals
Truncation
Sys-2
Sys-1.5 w/ sub-guolt
w/ Test-Time Control
Sys-2
Sys-1.5 w/ sub sub-goels
.... w/ Test-Time Control
100
100
70.4
Plan Validity
Plan Validity
60
60
50.0
40
37.2
25.0
9.0
10 13.615
20
25
10
20
30
40
50
#States-Explored
#Sttatss-Eplorce
(a) Maze Navigation
(b) Blocksworld
Figure 3: Comparison of System-l.2 Planner (with and without sub-goals) with a System-1 Planner and a System-2 Planner. (a) Maze Navigation: System-1.r with sub-goals matches or generally outperforms all baselines.
(b) Blocksworld: For sub-goals leads to much higher plan validity at lower "States-Explorse
generalization, System-1.ar with
to fairly compare all systems, their #States- Explored should matched, such that we compare performance fixed budget for each system. First, since System-1 does not do any exploration, it is constant w.r.t. budget and hence has the same plan validity irrespective #States-Explored. Next, to adapt System-2 and System-1.r to lower budgets, we truncate the search when the maximum number of states allowed by the budget has been reached. This value of maximum allowed states is given by the highest possible value such that the average #States-Explored matches the desired budget. We refer to this as truncation below. Finally, to adapt System-1.z to higher budgets, we use System-1.z* test-time controllability, increasing the bias term on System-2 reasoning (cf. $2.4). As this term increases, the model uses more and more System-2 planning, i.e., x - in System- We refer to this as test-time control. We perform this matching and report plan validity for the following values of Ettatte-Exppored First, vary the maximum budget of states explored in intervals of 5 for Maze and 10 for Blocksworld, up to the number of states used by System-2 2 (the most compute-intensii system consider). We also consider two specific points in the range of #States #States-Explone Explored: match System-2's s #States- Exates-Explored to the number of states that System-1.ar uses by default. In the other direction, match System-1.z2' 1a.''s "States-Explooee to the number of states used by System-2. These points let us directly compare the two systems without any truncation or test-time control. In the main paper, present our results with plots while refer to the Appendix E for detailed tables.
Implementation Details. We choose Mistral- 2aarall-7--Isstrutt--0222 (Jiang et al., 2023) as the base L.M and fine-tune all our components with LoRA (Hu et al., 2021) with rank of 8 for a maximum of epochs and batch size of 4, resulting in three adapters for System-1, System-2. and the controller. For the purposes of $4.1, System-1.. . will specifically refer System- Planner (i.e., = 0.5, which is balanced hybrid between a System-1 Planner and System-2 Planner) and fine-tuned from A traces. Refer to Appendix A for additional implementation details e.g., our hardness functions and System-2 implementations.
4 RESULTS AND ANALYSIS
4.1 SYSTEM-L.2 OUTPERFORMS SYSTEM-1 AND SYSTEM-2
Study Setup. In this first set of experiments, compare System-1.x Planner with (1) System-1 Planner, (2) a System-2 Planner, and (3) a System-L.z Planner without sub-goal decomposition.
Results for Maze Navigation. Fig. 3(a) (ef. Table 3) shows the plan validity obtained by each planner at different #States Extttred. The values #States- Exploord are as described previously in $3. Based on this, summarize our key findings.
System-1.æ outperforms all baselines at all budgets. Without truncation or test-time control, System-1.a (xr = 0.5) uses 13.6 states average. At this number of states, it generates 70.4%
7

---

Published as a conference paper at ICLR 2025
valid plans, outperforning the System-1 Planner by 21% (48.7% + 70.4%), the System-2 Planner by 33% (37.2%- - 70.4%) and the System- 1.2 variant without sub-goal decomposition by 20% (50.0% - 70.4%). Next, to compare all methods at lower budgets (i.e., with "SStatesExppored of 5 and 10), we do state truncation (as described in $3) by limiting the search (generation) to a maximum number of states. We find that even these lower budgets, System-1.rr continues to be significantly more accurate compared to baselines, pointing to an effective allocation of resources. Finally, System-1.x's test-time control capability lets increase compute to match that of System-2 by biasing the controller and solving more sub-goals with System-2. Doing so eventually transforms a System-1.a Planner into a System-2 Planner with sub-goal decomposition. outperforming vanilla S by 3% and achieving the highest validity of 96.7% at 27.3 states. This highlights the effectiveness of our sub-goal decomposition (more details below).
System- 1. L.z benefits from sub-goal decomposition. We validate the utility of sub-goal decompo- sition from two observations. First, the full System- outperforms the System-1.r variant that does not perform sub-gou decomposition. Second, System-2 also benefits from sub-goal decom- position as shown by the maximum validity of 96.7% which is achieved via test-time control and biasing the controller to solve every sub-goal using System-2. However, note that System-2 with decomposition still applies System-2 planning to every step, making System-1.z more performant any given budget.
System-1 is cheaper but not accurate. On one extreme of the plot. System-1 Planner is cheaper since it only explores the states that are part of its plan, amounting to an average of 3 states. However, this comes at cost to performance: it also only generates 49% valid plans. Another limitation of System-1 1 Planner is that unlike System- 1.x: is not controllable and hence, there no easy affordance for improving performance by exploring more states.
System-2 is more accurate but expensive. On the other extreme, System-2 Planner achieves a maximum validity of 94% but explores a much higher 24.4 states in the process. While System-2 can be adapted to minimize #States- Expttee- such adaptivity is only possible during inference via trate-truncation. Test-time truncation of #States #States-Explored Explored is typically ad-hoe and not preferable because the loss in performance will only increase as the plan length grows. This is in contrast to the System-1.ar Planner, which allows for systematic control of compute both at training-time (via a user-defined hybridization factor r), and at test-time (by biasing the controller).
Results for Blocksworld. Fig. 3(b) (cf. Table 4) reports our results on Blocksworld (BW) that studies -iittiibbitiitto generalization to longer plan lengths. Our first observation is that System-1 only obtains a plan validity of 9% and although System-2's s validity is higher 28%, given the longer plan lengths, it also explores a significantly higher number of states (55). Thus, when we perform trunca- tion to get lower values of "SStates-xxplores without sub-goal decomposition also does not work for OOD test points because the controller predicts almost all samples as hard and solves them using System-2. Hence, System- without sub-goal de- composition practically becomes System-2 2 in such scenarios, suffering major losses to performance with truncation. However, in contrast to all baselines, System-1.zz with sub-gou sub-goal decomposition can solve larger fraction of problems at lower budgets (e.g.. 25% at 30 states compared to System-2 Plan- ner's 13%) because of its ability to leverage a System-1 Planner for simpler sub-goals that have shorter plan lengths and are more likely to be in-distribution (refer to example in Fig. 9). At maximum #States- Explored, System-1 1.r obtains comparable performance to System-2. 2. Overall, when testing on OOD data, the results in Fig. (b) highlight the key role that the System- . controller plays in decomposing longer-horizon planning problems. Crucially, what is generally a weak System-1 Planner for harder, OOD goals can be used effectively to solve easier, ID sub-goals, thereby saving a lot of compute.
System-2's valid plans drops to almost 0%. System-
4.2 NRURO-SYMBOLIC SYSTEM- OUTPERFORMS OR MATCHES A" AT ALL STATE BUDGETS
Study Setup. While primarily designed System-L.z as a fully neural planner with all three components (System-1, 1, 2, and the controller) developed on top of the same LLM, we note that System-1.r can also act neuro-symbolic planner. This can be achieved by using symbolic solver like A as the System-2 component. We follow this setup to report results on the Maze Navigation task, again setting 0.5.
Results. Fig. 4 (cf. Table 5) compares the validity of all planners at different #States- Explored. We note that LLM--base planner may not be directly comparable to symbolic
8

---

Published as a conference paper at ICLR 2025
planners like A with respect to #States-Exxe #States-Explored metric because the former generates to- kens for states which computationally can be more expensive than invoking a symbolic planner. However, for the purpose of this study, compare them only based on the number of explored states, without considering the derlying computation responsible for such ex- plorations. Our main result that the conclu- sions drawn with fully neural System- 1.2 also transfer to syuro-yymbolic System-1.aa Notably, at an average of 11.6 states, neuro- symbolic System-1.az outperforms A by large margin of 39% (31.0% .0% 70.5%). This can again be explained by the ineffectiveness of "States-Exxlorrs
sys1 Sys-1.5 sub-goais
Iruncation
Sys-2 +-
Sys-1.5 w/ sub-goals
-.... w/ Test-Time Control
100
Plan Valldity
48.7
45.7
31,0
20
truncation for System-2 Planners, symbolic or neural. The trend is also consistent all lower budgets. System- 1.r can be test-time controlled to obtain a near-perfect validity (99.2%), matching that of A"
10 11.6 15
20
#States- #States-Explored
Figure 4: Performance of System-1.r in conjunction with a symbolic (A") System-2 Planner on Maze.
4.3 ANALYSES AND ABLATIONS SYSTEM-1.2 PLANNER
We conduct all analyses and ablations on the maze navigation task.
Train-time Control: System-1.77 trades off efficiency for plan validity compared to System- 1.5. One of the core strengths of our planner is its training-time controllability. The hy- bridization factor x allows the user to specify the System-1 to System-2 balance they want in the final planner. To demonstrate this capability, we train System-1.75 Planner by setting = 0.75. Table 6 (cf. Table 6) shows our results on the maze task. By default, System- 1.75 uses 16.6 states on average and generates 75.7% valid plans, compared System-1.5 70.4% with default average of 13.6 states (Table 3) i.e., 5% gain at the cost of 3 more states. Thus, as we increase the value of æ, the System-1.r Planner will use more states and improve performance, converging to System-2 Planner that additionally per- forms sub-goal decomposition. Compared to System-2, System-1.75 is again significantly more performant at all budgets (e.g., by 28% 16.6 states) and eventually obtains the high- est fraction of valid plans at 96.7% when act- ing a System-2 with sub-goals. This echoes our 5ystem-1.5 55 Planner findings, which also demonstrated similar gains at high budgets.
5ys-1
Sys-1. w/o sub-goals
w/ Truncation
Sys-
Sys-1.75 sub-goals
Test:Time Control
100
Plan Validity
60
48.7
47.0
Broadly, these results showcase our con- troller's effectiveness, which in turn validates our hardness measures and sub-goal decom- positions. We further ablate features of the controller in Appendix B, comparing against a random controller, testing the effectiveness of the sliding window, and contrasting differ- ent hardness functions.
10
15
16.6
20
25
#States-Explores
Figure 5: Train-tine controllability of System-1.ar Planner by setting x 0.75. System-1.75 largely out- performs System-2. Compared Fig. 3(a), System- 1.75 yields 75.7% valid plans vs. 70.4% (with System-1.5) by utilizing 3 more states.
Generalizability: System-1.z beats System-2 with all search algorithms (A*, DFS, BFS). We conducted our main experiments by leveraging supervision from A" traces. In Appendix Fig. 6, we show that System-1.zri robust to different search algorithms, working with other options like Breadth First Search (BFS) and Depth First Search (DFS) and outperforming baselines by large margins (see Appendix B for details). In Appendix C, we show a representative example of System-1.x outperforming the baseline systems.

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Combining LLMs and Symbolic Planners. A large body of recent work has highlighted the shortcomings of LLMs on long-horizon planning problems (Valmeekam et al., 2023; 2024; Pallagani et al., 2023; Momennejad et al., 2024; Hirsch et al., 2024; Zheng et al., 2024; Aghzal al., 2023), persisting across popular prompting techniques like Chain-of-Thought (Wei et al., 2022), ReAct (Yao et al., 2022), and Reflexion (Shinn et al., 2024). These methods largely resemble System-1 inference- time methods in which the LLM does not explicitly search, backtrack, or learn from incorrect actions. Systematic search is either not performed all or delegated to symbolic solvers (e.g., Liu al.. 2023; Pan et al., 2023; Xie et al., 2023, discussed further below), thus deviating from the focus of this work on equipping L.MSs to solve harder planning problems via search. More related to our work are inference-time methods that do involve planning, tree search, and backtracking. These gystem-2-inspire methods have a search algorithm that operates on top of and guides an LLM with reward models, value functions, or verifiers (Hao et al., 2023; Yao et al., 2024; Besta et al., 2024; Zhou et al., 2023; Sel et al., 2023; Koh et al., 2024). In a similar spirit, neuro-symbolic planning systems like LLM-Modulo Frameworks (Kambhampati et al., 2024) have emerged that combine LLMs with symbolic planners (Nye et al., 2021; Liu et al., 2023; Pan et al., 2023; Xie et al., 2023; Zuo et al., 2024; Fabiano et al., 2024; Katz al., 2025; Zhuang et al., 2024). System-1.zr 1.z differs from this class of inference-ting that teaches the LLM to search, with the goal of enhancing the LLM's intrinsic planning capabilities and discovering more effective search strategies, making it self-contained (as opposed to complex inference-ting methods relying on external modules). Secondly, it is also hybrid system that can alternate between System-1 and System-2-style planning, whereas past work is capable of one or the other, but not both. Notably, this kind of hybridization distinguishes System- 1.2 from other work that either creates hybrids between weaker and stronger models in the form of model cascades (Lin et al., 2024; Yue et al., 2023) or between neural and symbolic methods (e.g.. LLM-Modulo frameworks).
methods in two major ways. Firstly, it is a training-time method
LLMs Generating Plans. Methods combining LLMs and symbolic planners need some com- munication channel between the LLM and the symbolic planning component. This is usually value function or reward model producing a single score for a state, making the method's s success contingent the quality of the reward model, the LLM'' ability act value function (as in Yao et al. (2024); Lightman et al. (2023)). Furthermore, off-loading planning or search adds additional modules test-time that may not always be available for certain domains. It also assumes that the model's is representation is already sufficient for the task and complicates finetuning the representation, since externalized search typically non-differentiable. In an attempt mitigate such concerns, a second line of work removes the dependency on separate reward models and external tools and teaches LLMs conduct search by themselves (Yang et al., 2022; Gandhi et al., 2024; Lehnert et al., (SoS), which teaches L.MMs
2024). For example, Gandhi et al. (2024) introduce to search by verbalizing trajectories and Lehnert et al. (2024) propose Searchformer, that learns from A search dynamics. This line of work predates LLMs: Anthony et al. (2017) generate data from tree search which they use to iteratively train a neural model to perform System-1 planning. However, Gandhi et al. (2024) and Lehnert et al. (2024) differ from our work in that their final models are gystem-2-only incurring a high computational cost for all examples and states by performing full planning all times, whereas allocating computation to more difficult parts of the plan. Moreover, to meet user budget constraints, System-2-ond methods also have rely on approaches like post-hoc truncation while System-1.ar offers more systematic controllability via its controller.
method is a hybrid between Systems and 2, dynamically
6 CONCLUSION
We introduced the System-1.z Planner, a novel LLM-based planner that adaptively interleaves quick and cost-efficient System-1 planning with more costly but performant System-2 planning. On two tasks, we showed that our method results in strong performance across a range of budgets: on Mazes, System-1.z obtains the best performance at every budget, and on Blocksworld it outperforms System- 2 at all budgets except the highest, where it is comparable. In our analysis, we find that System- is controllable, generalizes to various search algorithms, and performs better -ut-of-distributtin.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Large Language Models have been shown to reflect stereotypes, biases, and other negative traits present in their pre-training data (Weidinger et al., 2021). Hence, the outputs produced by our fine-tuned planner may exhibit undesirable behavior similar to the base model and have the same potential for misuse as other fine-tuned LLMs. Hence, more studies are needed evaluate and mitigate such biases in LLMs.
REPRODUCIBILITY STATEMENT
We are making our code and data available in the supplementary material enable replication of our findings. We also show the exact prompts in Appendix D.
ACKNOWLEDGMENTS
This work supported by NSF-CAREER Award 1846185, NSF-AI Engage Institute DRL-21 DARPA MCS Grant N66001-19-2-4031 and Google PhD Fellowships. The views contained in this article are those of the authors and not of the funding agency.
REFERENCES
Mohamed Aghzal, Erion Plaku, and Ziyu Yao. Can large language models be good path planners? benchmark and investigation on spatial-temporal reasoning. arXiv preprint arXiv:2310.03249, 2023. URL https: http://aavvvo.o///e wwwp:/iiioorroooooooo.oe abs 231/2.31....23
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. URL
https: arxiv. org/ abss/ 2204. 01691.
Thomas Anthony, Zheng Tian, and David Barber. deep learning and tree search. Advances in neural information processing sys- tems, 30, 2017. URL ntttps://proceeddigg ceneiieurpppp cce wttpc///pppp.. cctpppaee//01///aaattte wbwteedeeebbtrrt..... htmll.
Thinking fast and slow with
Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. arXiv preprint arXiv:2403.06963 2024. URL
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 17682-17690, 2024. URL https:///oosss.... aaai. org/ feg/inn ///iexdexx.///// php AAAI
Bernd Bohnet, Azade Nova, Aaron T Parisi, Kevin Swersky, Katayoon Goshvadi, Hanjun Dai, Dale Schuurmans, Noah Fiedel, and Hanie Sedghi. Exploring and benchmarking the planning capabilities of large language models. arXiv preprint arXiv:2406. /3094, 2024.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2xXiv:2303.12712 2023. URL https: ptattp:://ariii //axxiv.
org/abs/2303 12712.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. Faith and fate: Limits of transformers compositionality. Advances in Neural Information Processing Systems, neurips. ce/ cc/paper hacppper_iieeeeeeeeeeee https//eeeeepppeeeeeeeeee//eapee fites//appeeee22//// html.
36. 2023. URL org.://e0e.......caa......a.rarr htmi.. hast:/ceecc22819999999922222222

---

Published as a conference paper at ICLR 2025
Jonathan St BT Evans. In two minds: dual-process accounts of reasoning. Trends in cognitive sciences, 7(10):455 2003.
Francesco Fabiano, Vishal Pallagani, Marianna Bergamaschi Ganapini, Lior Horesh, Andrea Loreggia, Keerthiram Murugesan, Francesca Rossi, and Biplav Srivastava. Plan-sofai: A neuro-symbolic planning architecture. In Neuro-Symbolic Learning and Reasoning in the era of Large Language Models, 2024.
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D Goodman. Stream of search (sos): Learning to search in language. arXiv preprint arXiv:2404.03683, 2024. URL https: : www.i. org. abs / 2404. ..0044000 03683.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong. Zhen Wang, Daisy Wang. and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8154-8173, 2023.
Eran Hirsch, Guy Uziel, and Ateret Anaby-Tavor. What's s the plan? evaluating and developing planning-aware techniques for lims. arXiv preprint arXiv:2402. 1489, 2024. URL https:
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021. URL https: http//axvvvvoo .org abs/ / 2106. 09685.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.0682) 2023. URL https: //apiii////vvvv///// http//tww.v.rg....ea...oooo abs 22100. 06825.
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lueas Saldyt, and Anil Murthy. Llms can't plan, but can help planning in Ilm-modulo frameworks. arXiv preprint arXiv:2402.01817 2024. URL tittp:///arxint org. /abs/
2402. 01817.
Michael Katz, Harsha Kokel, Kavitha Srinivas, and Shirin Sohrabi Araghi. Thought of search: Planning with language models through the lens of efficiency. Ad- vances in Neural Information Processing Systems, 37:138491-138568 2025. URI
weureiinnnnnneuuuueeeeeeeeeeee http://perrf.lleepppe////// html.
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language
model agents. Preprint, 2024.
Lucas Lehnert, Sainbayar Sukhbaatar, Paul Mevay, Michael Rabbat, and Yuandong Tian. Be- yond a*: Better planning with transformers via search dynamics bootstrapping. arXiv preprint 240.. 2024. URL https: / //apss:////xiieee org/ abs/ 14083.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023, URL https:/ //wp///axiii..///i org/ abs/2305 20050.
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. Swiftsage: generative agent with fast and slow thinking for complex interactive tasks. Advances in Neural Information Processing Systems, 36, 2024. URL https:tpp /ttp://p///eeeeeeeeee ngs neurips hash/ 4b0eea 6 9deea 2c9e2c4691 8877787888 64 -
html.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Lim+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304. 1477, 2023. URL https: / wwwp:/wwww//////////2/
11477.
12

---

Published as a conference paper at ICLR 2025
Ida Momennejad, Hosein Hasanbeig. Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson. Evaluating cognitive maps and planning in large language models with cogeval. Advances in Neural Information Processing Systems,
36, 2024. URL or -otpra/eaaaaassssaattoooooerreee
neurips wwtpurii..ccpppappeeeeeeeepp cctpaper__eeeeeeeeee weserrriiiieeeeeeeeeeeeeeeeeeeee htalllesppper/222/2 html.
Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. Improv- ing coherence and consistency in neural sequence models with dual-system, neuro- symbolic reasoning. Advances in Neural Information Processing Systems, 34:25192- 25204, 2021. URL httpps:/prroceedini hew.riiiiinnnnnnnnnnooeeeoee.n neurips wwtparriss.cccpappee wttp//pppr//22///aaeee cb228ffi10 www.:e87777-Abbaattteeae html.
Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Biplav Srivastava,
Lior Horesh, Francesco Fabiano, and Andrea Loreggia. Understanding the capabilities of large language models for automated planning. arXiv preprint drXiv:2305.16151, 2023. URL https:
//arxiv. /tg/ abs 2305. 16151.
Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1306-3824, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1/2023/ 2e2.022...nnnnsseeee.e moddings-emnl..2440 findings-emn p..
URL
org 2023.
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language models. In Findings of the Association Computational Linguistics: NAACL 2024, pages 4226-4252, 2024.
Stuart Russell and Peter Norvig. Artificial intelligence: modern approach. Pearson, 2016.
Bilgehan Sel, Ahmad Al-Tawaha, Vanshaj Khattar, Lu Wang, Ruoxi Jia, and Ming Jin. Algo- rithm of thoughts: Enhancing exploration of ideas in large language models. arXiv preprint arXiv:2308. 10379, 2023. URL https: /arxivvvi..v//..v/aaaaaaaaaaavla org/ abs/
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Reflexion: Language agents with verbal reinforcement learn- Advances in Neural Information Processing Systems, 36, 2024.
Shunyu Yao.
ing.
URL
https: fttts::/proceet:: http://poeeednnnge/eeeee neur wssurrpp..ae
/2tp://lllsssappee/2/////aaa 1b44b8 (5507722285528888822000000000000
htmi.
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambham- pati. On the planning abilities of large language models-a critical investigation. Ad- vances Neural Information Processing Systems, 36.75993-76005. 2023. URL
httpa:://pooeedigg..neueee
html.
Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Advances in Neural Information Processing Sys- tems, 36. 2024. URL ht 2023/hase httph///aa9eccddeeeeeeeeaaa and_Benchmarte rks. html.
cc htcp/pppe_fflee/ppeeeeeeeeeeeeee /apper/ -aw.eeesse---aaaaat----ttteeeeee
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Peter C Wason and St BT Evans. Dual processes in reasoning? Cognition, 3(2):141-154, 1974.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems, 35:24824-24837, 2022. URL https: htttps://prooeeedingg neurips, wttpppppeeeiieeeeeeeeeeeeeeeeeee hhath//455055155544ffff11111 A html.
13

---

Published as a conference paper at ICLR 2025
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks harm from language models. arXiv preprint arXtv:2112.04339 2021. URL https:/ /ttp:///aavvvvv/aa org/ 2112.04355
Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to planning goals with lange-la models. arXiv preprint arXiv:2302.05/28, 2023. URL
https://arre http::///axxv.vvttt org/orgg//ss.. abtaaabs//2a.. 2.02.0220000 05128.
Mengjiao Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Chain of thought imitation with procedure cloning. Advances Neural Information Processing Systems, 35:36366- 36381, 2022. URL https: // //rxivivv.. org/ 2005/2205.00 10816.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022. URL https: www.wiiwwgggaaaa.eeaaao..aoooeee abs 2210.
03629.
Shunyu Yao, Dian Yu, Jeffreyy Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. URL https: / arxiv, httpa/aawii.aaaaaaaaa//oa wttp/r/////////eeeeeeee 23055 10601.
Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thoughts representations for cost-efficient reasoning. arXiv preprint 10X0022310.03094 2023.
Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng- Tze Cheng, Quoe V. Le, Ed H. Chi, and Denny Zhou. NATURAL PLAN: Bench- marking llms on natural language planning. arXiv preprint arXiv:2406. 04520, 2024. URL
https:/ aariiivv otggabs/////ooooooro/oooo/sossrr 2406. 04520.
Andy Zhou, Kai Yan, Michal Sanpenttok-Roothmae Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint DiXiv:2310.04400 2023. URL http: //ta://///xvvii org/ abs/ /2////311- 1333000000 04406.
Yuchen Zhuang, Xiang Chen, Tong Yu. Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang. Toolchain Efficient action space navigation in large language models with a* search. In The Twelfth International Conference on Learning Representations, 2024. URL
https: wpwnreviewweeeeeeeeeeeeee.eeewwe net wwwpeeeeeeeetttteeeeeeeeeeee fowum?id-B6ppxqqee
Max Zuo, Francisco Piedrahita Velez, Xiaochen Li, Michael L Littman, and Stephen H Bach. Planetarium: rigorous benchmark for translating text to structured planning languages. arXiv preprint arXiv:2407.033211 2024.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL DETAILS OF EYSTEM-1.r PLANNER
Algorithm Training Data Generation for System-1.zz Controller
Input:
System-1 Training Data Dsys1 {($b)) 8g
(0) P Ie-1 Hybridization Factor [0, 1].
Hardness Function h(+,-)
Output: Training Data Dr for System-1.zz Controller
Bort(Dsysl. key=h((8- 8g)) // Ranking data points in increasing order of hardness
D. O
for data point ((50) (a)), ).P(")) E DDsyst do
if (1 -x) N then
I/ For the first (1 - 23% of instances (easiest), solve them directly using System- planner
y - ((s) )g "Sys1"}
else:
// For the remaining x% hardest instances, find the best decomposition into three sub aub-goals such that the middle sub- sub-goal (8,, 8b) is solved using System-2 and has a length of x% of the total plan length
(j. k)) argmin (h((0
h(su: Su) + h(ss, Sq ( s.t. |0 u =
y() {(5)) (6) 85), "Sysl")
( 1(t 1-((85, sk), "Sys2")
0 + (( + [(kk, ) "Sys1"}
D. t D. + ((sp ,8
(1) ( ),00
return D.
A.1 BACKGROUND ON A
A is an informed search algorithm that decides the next best state to expand with function f(n) = g(n) + t(n) that considers (1) the cost g(n) from the start state to the current state n, and (2) heuristic function t(n) that gives estimate of the cost from state to the goal. Note that A' is optimal if the heuristic t(n) admissible (i.e., t(n) I t" (n) which means that the heuristic never overestimates the true cost to the goal h" (n)) and consistent (i.e., t(n) S c(n, n') + t(n') ) for all states n and its successor and e the cost to transition from n to n'). For the Maze Navigation task, Manhattan Distance an example of such admissible heuristic, and hence A with will always generate optimal plans.
A.2 IMPLEMENTATION DETAILS FOR MAZE NAVIGATION
System-2 Planner. We implement a System-2 Planner for the Maze domain by verbalizing all possible explorations at each state, including the ones that lead invalid states (i.e., outside the maze, have obstacles, or have already been visited), See example in Appendix D.
Hardness Function. Given two maze states 8; (a,b) and = (c. d) we define hardness of a sub-goal h(s;, sy) as the number of obstacles the rectangular sub-maze given the end-points. More obstacles will mean that the model might have to search more and would likely benefit from System-2 planning.
A.3 IMPLEMENTATION DETAILS FOR BL.OCKWWOLLD
System-2 Planner. In contrast to the Maze domain, the number of actions each state in Blocksworld problem is directly proportional to the total number of blocks. Consequently, the number of possible states that can be explored (both valid and invalid) is significantly larger this domain, often exceeding the maximum input token budgets of standard LLMs (in our case, 8096 tokens). Therefore, when verbalizing the search trajectories T in Dsysz using A search to train System-2 2 Planner, cap the number of valid and invalid states explored to and 2 respectively. The verbalized valid states are chosen by picking the top-3 scored states as per the A heuristic (that computes the number of mismatches between the start state and the goal state), while (up to) 2 invalid
15

---

Published as a conference paper at ICLR 2025
Sys-1 Sys-2
Syer 1.5 we sub goal- w/ Truncation
Sya-1
Byli-15 sut-goes Slye- suo-goats
Truncstion
Sys-1:5 sub-goals
Test-llmme Sye2 (DFS)
82.3
48.7
50.7
48.7
23.2
0
States-Explored
AStatos-Expiored
(a) System-1.z with BFS
(b) System-1.a with DFS
Figure 6: Comparison of System-1.a Planner on Maze Navigation using (a) Breadth-Fin Search and (b) Depth-First Search traces. The System-1.x Planner generally outperforms System-2 Planner in both cases, converging System-2, showing robustness to the underlying search algorithm.
states are chosen randomly to make the model learn from mistakes. Refer to Appendix D for an example.
Hardness Function. A state in Blocksworld is characterized by the configuration the blocks. Our hardness function computes distance metric for a sub-goal: For every block that is not in its correct position in the goal state (i.e., the blocks above and below it are different between the start and goal states), add cost of 1. Additionally, for block that is its correct position, we add more to the cost, indicating greater hardness for moving blocks that are in the middle of stack.
the table and also not in
B ADDITIONAL RESULTS
Generalizability: System-1.rr beats System-2 with all search algorithms (A", DFS, BFS). We conducted our main experiments by leveraging supervision from A traces. In this experiment, we show that System-1.r robust different search algorithms, working with other options like Breadth First Search (BFS) and Depth First Search (DFS). First, note that for mazes, BFS explores all four actions (up, down, left, right) each state and hence, on average, is the most expensive search strategy in terms of states explored. However, for an unweighted maze, BFS always generates optimal plans. DFS, on the other hand, explores fewer states because returns the first valid plan, though that plan may not necessarily be optimal. Appendix D contains examples of BFS and DFS trajectories. Based on Fig. 6 (cf. Tables 7 and 8), we summarize our key findings:
System-1.z continues to outperform System-2 and 'System-1.ar 1.. without sub-goal decomposition' by large margins (e.g., up to 39% at 17.4 states for BFS and at 10 states for DFS), showcasing our planner's ' flexibility learn from different algorithms.
System-1.z closely resembles the behavior of the search algorithm it is trained on. For example, System-1.zr (BFS) is more accurate and generates a significantly higher number of optimal plans than System- (DFS) because the former is trained on optimal plans (refer to Table for a study validity versus optimality). In the process, however, System-1.5 (BFS) explores more states than System-1.r (DFS), similar to their symbolic counterparts. Among the three algorithms studied in this paper, System-1.at (A*) is the most accurate and efficient.
System- 1.x Controller outperforms random controller. In Table 1, we compare our System 1.5 controller with a random controller that arbitrarily chooses 50% problems to be solved by System-2. Our trained controller outperforms the random controller by significant 10%.
Analyses of System-1.x Controller. this section, we analyze the different components of our System- controller.
16

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of trained controller in System-1.æ Planner with controller that randomly chooses x% of problems to solve with System-2.
Plan Validity
System 1.5 (w/ random controller) System 1.5 (w/ controller)
70.5
79.5
Sys 1.5 w/o Sliding Window
W/ Trunc
Sys-1.5 w/ Manhattan Distance
W/ Truncation
Sys w/ Sliding Window ...... w/ Test-Timee Control
Sys- 1.5 w/ botsacclss
W/ Test-Time Control
100
80
Plan Validity
Validity
40
40
10
20
25
35
Estates-Exploreo
#States-Explored
Figure 7: Comparison of System-l.z Planner with and without sliding window-based sub-goal decomposition on the Maze Navigation task. Sliding window-based decomposition outper- forms the variant without it at all budgets.
Figure 8: Comparison of two hardness functions (Manhattan Distance and #Obstacles) for Maze Navigation. Both yield comparable results, with #Obstacles being slightly better.
Effectiveness of Sliding Window Decomposition. Recall that generated training data for the controller by using sliding window approach that assigned a contiguous 2% of the plan to hardness function. Here, evaluate the effectiveness of this approach by
System-2 according comparing it to a System- 1. variant without the sliding window decomposition. In this variant, the -% System-2 allocation can either be at the beginning or at the end of the plan but not in the middle, resulting in a maximum of two decompositions instead of three. As shown in Fig. 9, our sliding window variant not only obtains higher plan validity but also far more efficient. This is because with three decompositions, weaker System-1 Planner can be used to solve two granular sub-goals (instead of one) and hence is more likely to succeed.
Comparison of Hardness Functions. Our experiments sub-maze as the hardness function. In Table 10, we compare it to Manhattan Distance, which rates goals that are further away (according to the distance function) more difficuul. We observe that both functions lead comparable performance, with obstacles being slightly more efficient. Taken together with the previous experiment, these results quantify the influence of two essential inputs to the controller's ' training: hardness function and the decomposition strategy.
maze used the number of obstacles in
Table Comparison of Plan Validity and Optimality for System- Planner in Maze Navigation using A* search. While not using sub-goal decomposition nearly guarantees optimality, using sub-goal decomposition may generate plans that are sub-optimal.
Plan Validity
Plan Optimality
#States-Explored
System- (w/o sub-goals)
33.5
33.2
4.8
System- 1.x (w/o sub- nob-goals)
38.2
38.0
9.7
System-1.xr (w/o sub-goals)
62.2
61.7
15.0
System-1.ar (w/o sub-goals)
79.5
78.7
17.4
System-1.ar (w/ sub-goals)
30.5
30.2
4.5
System- 1. (w/ sub-goals)
56.2
53.7
10.0
System-1as 1. (w/ sub-goals)
70.4
61.5
13.6
17

---

Published as a conference paper at ICLR 2025
System-1.a xx Controller's Decomposition
D
B
G
Use System-2 Planner
Use System-1
Planner
(Hard)
(Easy)
C
Init State Controller Generated Sub-goal 6,
Goal State Sy
Figure 9: An example decomposition generated by System 1.5 for Blocksworid. The controller generates intermediate state 51 where blocks 'C' and 'F' are block apart, with the sub-gou 80 + 91 solved using System-2 and Gi + 89 solved via System-1.
Optimality of System-1.z. In Table 2. compare the optimality and validity of the plans generated by our System-1.2 Planner. While System-1.2 without sub-goal decomposition generates plans that almost always optimal, observe that sub-goal decomposition may sometimes hurt optimality. This happens because the sub-goals generated by the controller may not always lie on the optimal path.
C QUALITATIVE ANALYSIS OF SYSTEM-1, SYSTEM-2, AND SYSTEM-1.2r
Fig. 10 shows example from our succeeds. We consider a maze configuration where the plan length is quite long (8) and there is solitary path the goal, making the problem challenging for LLMs. We find that the System-1 Planner generates an incorrect plan that ignores the placement of the obstacles. On the other hand, the System-2 Planner avoids obstacles successfully initially but then takes path that leads to dead end. It is unable to recover from that and the search does not terminate. Our System-1.r Planner is, however, able solve the problem successfully by first breaking up into sub-goals and solving the easy sub-goals close the start state and the goal state using System-1 and the hard sub-goal in middle (that includes a number obstacles) with deliberate System-2 planning.
where both System-1 and System-2 fail but System-1.ar
Input Maze
System-1 Plan
System-2 Plan
System-
Controller Sub-goal Decomposkion
1. Shart Stetes Th E1 Coat reater
solowd fear catiee syeeem
Shoestin fur satinne band
2. poates
scad scates
rlacoiig
dass saking ageses
Falled because of invelid plan
Falled because of deed-end plan
Succecded by combiiing System-1 and Syctem-2
A qualitative example showing (1) failure case for System-1 Planner because generates a plan through the obstacles, (2) failure case for System-2 Planner because the generated plan leads dead end and the model does recover from and (3) a successful case for our System-1.. I.z Planner that solves the easy sub-goals near the start state and the goal state using System-1 and the hard sub-goal in the middle using System-2.
Figure 10:
D PROMPTS AND EXAMPLES
See the subsequent pages for prompts and examples from both domains.
18

---

Published as a conference paper at ICLR 2025
System-1 System-2 Prompt cel-s walla.
You in maze of dimensions The walls placed in cells [[3, 41, [0, 4], [3, 31, 11, [2, 3), [0, 0], [2, [4, (1,
and
411.
Given start sequence of actions. The optimal plan is ene Lhat has the mininum number of stepa, The list of permisaible actions that you can take at any given cell are ('left', 'right', rup', 'down'l ].
goal state,
task is to generate the optimal plan
The optimel plan from [1, 4] Lo [3, 2] is
System-1 Output for Maze (#States-Exploredi-)
start (1, 4; | left (1, 3]
left (1, 2]
down [2, 2] down [3, 21
System-2 (A') Output Maze (#States-
Moved to state [1, 4] | Plan so far [] Exploring actior. 'left" to move to
state [1, 3]
State [1, 3] valid Exploring action 'right' Le sove te
state [1, 5]
State [1, 5] is invalid Exploring actior 'up" to mave to
state [0, 4]
State [0, 4] invalid Exploring action 'down" move to
state [2, 4]
State [2, 4] invalid Taking action 'left" from state [I, so ['left'] Exploring actien is valid Exploring action 'right' invalid Exploring
Moved state (1, 3]
State
move state
"up" move state 10, 31 State 10, 31 is valid Exploring action "down" to to state [2, 3] State [2, 3] is invalid Taking action 'left' from state [1, 3] Moved to state (1, 2] Plan so far ('left', 'left'] Exploring action
State
"ler=" to move to state [1, 11 State (1, 11 is invalid Exploring actien
"right" to mave to state 11, 31 State 11, 31 is invalid Exploring action
'up" to move to state [C, 2] I State [0, 2] is valid | Exploring action 'down' to move to state [2, 2] State [2, 2] is valid | Taking action 'down' from state [1, 21 Moved to state 12, 21 Plan ao far I'left', 'leit', "down"]
Exploring actior 'left" move to state [2, 1: State [2, 11 valid |
Exploring actior "right" to move to state [2, Exploring action 'up"
State [2, 3] is invalid
state [1, 2] State [1, 2] invalid
Exploring actior "down"
state [3, 2: State [3, 21 is valid
Taking action 'down" Irom stale [2, 21 Moved to state [3, 2] | Plan Iar ('left', 'left', 'down', 'down'] | Goal state [3, 2] reached!
19

---

Published as a conference paper at ICLR 2025
System-2 Output (BFS) Maze WiStates-Explored-50
Moved to state [1, 4] | Plan so faz [] Explozing action 'left' move to
state [1, 3]
State [l, 3] valid Exploring action 'right" to move to
state [1, 5] State (1, 5] invalid
Exploring action 'up" upp to move to State [0, 4] invalid Exploring action 'down'
state [0,
state [2,
State [2, 4] is invalid Taking action "left' from state [l, Plan faz ['le[t'] Exp-oring actien 'left" Lo
4] Moved Lo slete [1,
move to state [1, 2] State [1, 2] is valid Exploring action 'right" to
move to state [1, 4] State [1, 4] is invalid Exploring actior. 'up' to move
to state (0, 31 State 10, 3] is valid Exploring action 'down' to move to
state 12, 3] State (2, 3] is invalid Taking action 'left' from state [1, 3] Moved to state [1, 2] Plan so faz "laft', 'left"] Exploring action
"lett" to move to state [)
State [1, is invalid Exploring
"right" to move to state 12, 31 State 11, 31 is invalid Exploring action
"up" nove state [0,
SLate [0, is valid Exploring actios "dowa
state
State [2, 2] valid Taking action 'up' from
state [1, 3] Moved state [0, 3] Plan far ['laft', 'up'] Exploring
action "eett' to move to state 10, 2] State [o, 2] is valid Exploring
action 'right" to
state [0, 4]
State [0, 4] is invalid Exploring
action 'up" to move to state [-1, 3] State [-1, 3] is invalid Exploring
action 'down' to move to state 12, 3] State [1, 3] is invalid Taking
action 'up' from state [1, 21 Koved to state [0, 2] Plan so far ['left',
'left', 'up'l Exploring action 'left" to move to state [c,
State [0,
is valid Explaring actien 'right" to move to state [0, 3] State [0, 3] is
invalid Exploring action 'up' move to state [-1, 2 State [-1, 2]
invalid Exploring action
move
[1, 21
[1, [2,
Moved invalid Taking action 'down' from Exploring action "left" to move to
Plan far ['left', 'left', 'down']
state [2,
State [2, 1] is valid Exploring action 'right"
state [2, 3] State [2, 3] is invalid
Exploring action 'up" to move to
state [1, 2]
Exploring action 'down' to move to State [3, 2] is valid | Taking action 'left' from atate [0, 3]
State (1, 2] is invalid
state [3, 2]
Noved to state 10, 21 Flan so far ('left', 'up', 'leit'] Exploring
action 'left" to move to state [0, 1]
State [0, 1] is valid Exploring
action 'right" to move to state [0, 3]
state [0, 3] invalid Exploring
action 'up" move to state [-1, 2] State [-1,
invalid Exploring
action 'down! move
[1,
State
is invalid Taking
action "LefL" from state [0, 21 Moved Lo state [0, 1] Plan Ear ['left',
'left', 'up', 'left'] Exploring action 'left"
atate [0,
State 10, 0] is invalid Exploring action 'right" to move to state [9, 21
State [0, 2] is irvalid
Exploring action 'un' to move to state [-1, 1]
State [-i, 1] is invalid Expluring aution "down" to move to state [1, 11
State [1, 1] is irvalid | Taking ection "left" from state [2, 2] Maved to
atate 12, 1] Plan so far ['left', 'left', 'down', 'left'] Exploring action 'left! to move to state 12, 01 State 12, 01 is invalid Exploring action 'right" move to state 12, 2] State [2, 21 invalid Exploring action Exploring State [3, is valid Taking action 'down' ('lef"', 'left',
"up" nave to state [1,
State [2 is invalid
'down"
state
Xoved
'down', "down'] Goal state [3, 2] reached!
20

---

Published as a conference paper at ICLR 2025
System-2 Output (DFS) for Maze (#States-Explorke (SStates-xploordd=99
Moved to state [1, 4] Plan ao far [] Exploring action "left" to move 20
state [1, 3] Stata [1, 3] is valid Taking action 'left" from state [1, 4]
Moved to state [], 3] Plan so far ['left'] I ixploring action 'left' to
move to state (1, 2] State [1, 2] is valid Taking action 'left' from state
[1, 31
state [1, 2] Plan so far [*left', 'left'] Exploring
action 'left"
[1, 1] invalid Exploring [1, 11 State [1, 31 is invalid | Exploring
action "right" Lo Le state [1,
action "up'l to
state [0, 2] State [c, 2] valid | Taking action
'up' frem. state [1, 2] Noved to azate [0, 2] Plan so far ['left', 'left',
'up'] Exploring action "left" to move to state [0, 1] State [0, 1] is valid Taking action 'left' from atate [0, 2] Moved to state [0, 1] 2lar. so far ['left', 'left', 'up', 'left"] Exploring action "IefE" to mave to state [0, 0] State [C, 0] is invalid Exploring action "right" to move to state [0, 21 State IC, 21 is invalid Exploring action 'ap"l to move to [-1, is invalid Exploring
[-1, 1]
[1, State
to is valid Taking action 'right' from state [0, [0, Plan far ['left', 'left', 'up', 'right']
is invalid Exploring action 'zight"
state [0, Stata
Moved
Exploring action 'left" to move to state [0, 2] State [0, 2] is invalid
Exploring action 'right" to move to state [0, 4] | State [0, 4] is invalid
Exploring action 'up' to move to atate [-1, 3] State [-1, 3] is invalid
Exploring action 'down" to move to state [1, 3] State 11, 3] is invelid
exploring action 'up' to move to state [-1, 2] State [-1, 2] is invalid
Exploring action 'down" to move to state [1, Exploring action
State [1, 2] is invalid
to move to state [2,
State [2, 2] is Plan far
Taking action 'down' from
Noved state [2,
('left', 'left', 'dawn'] Exploring sctian 'left!
to state [2, 1]
State [2, is valid Taking action "left' from state state 12, 11 Plan far I'left', "left', 'down', 'ieft"] Exploring action "left" to move to state [2, 0] State [2, 01 is invalid Exploring sction
Moved
'right" to move to state [2, 2] szate [2, 2] is invalid Exploring action
'up" to move to state [1, 1] State [1, 1] is invalid Exploring action
'down! to move to state [3, 1] State [3, 1] is valid raking action 'down'
from state [2, 1] Moved to state [3, 11 Plan so fer ['lefL', 'left', 'down', 'left', 'down'] Exploring
'lefE" to move to state [3,
valid Taking action 'left' state [3, Plan far I'lefL', "left', 'dewn', 'lefL', 'dewn', "lefL"] Exploring action 'left"
[3,
Moved
State [3, -1] is invalid
Exploring action 'right" to move to state [3, 1] State [3, 1] is invalid
exploring action on 'up" to move to state [2, 0] Stace [2, 0] is inwalid
Exploring action "down" to move to state 14, 0] State 14, 0] is invelid
Exploring action "right! to move to state [3, 2] State [3, 2] is valid
Taking action "right" from state [3, 1] Moved to state [3, 2] Plan so far ['iest', 'left', 'down', "left', 'down', 'right"] Goal state [3, 21 reached!
21

---

Published as a conference paper at ICLR 2025
Controller Prompt Maze
You 2d dimensions and 5 and of the sells have walls. The walls are placed in cells 113, 4., 10, 4,, 13, 31, 11, 21, 12, 31, 10, 01, [2, G], [4, C), [I, 0), [2, 4]1.
viver start and goal state, your task is to first generate set of supgoals that can then be soived to generate the optimal plan between the stales. The optimal plan is one Lhal has the minimum number of sleps. The 1ist of permissible actions that you can take at any given cell are ['left', "right', 'ap', "d
Generale subgoals generaling plan [1, Lo 13, 2].
Controller Output (Sub-goal Decomposition Easy Hard Annotations) Maze
The walls between [1, 4J to 13, 21 are at [2, 31, 12, 41, 13, 31, 13, 41 which are a Lotal of walls. Based on this, the plan should be broken down into 2 sub-goals:
1. Start State: 12, 41 and Goad State: 12, 21. Planning for this subgoal is hard can solved using system
2. State: [2, 2] Goal State: [3, 2lanning for this subgoal is easy solved. fast using syatem
22

---

Published as a conference paper at ICLR 2025
System-1 System-2 Prompt for Blocksworid
Your tack is to generate a plan for a blocksworld problem given an initial state and a gual
initial state:
The hand is empty.
C ls on the Lable, on C, on B. clear. D. cleas.
D is the table.
goal is:
A is oz the table. C is x. is c, is clear. D. cleas.
on
System-1 Output for Blockswortd (#5tates-Exppored-gg
Move E From 3 to table Move from D to cable Move fron table to D Move fron
to tabla Nove from table to Nove from to
System-2 (A") Output for Blocksworid "'States-Exploredd
to stata Stack [C, B, 3], Stock [D,
for Exploring
Nove E from B to New atate => Stack 1: [C, B], Stack 2: [D, A, R] Exploring sct
Nove E from B to tabls New state +> Stock 1: (c, 2], Stack 2: (0, A], Stack 3: (E) Rxploring actioon
[C, B, SLack 3: [D] Explering acLion Hove 5 from C La New state => Invalid Taking scLion
table
1:
Stack
Stack [F]
I Plan far [Move from table] Exploring action Rove from to table New state -> Stack [8], Stack (C), Stack (0, A], Stack [R] Exploring acston table New
+> Stack 1: (A), Stack 2: [C, B], Stack 3: [D], Stack 4: [E] Exploring action Nove from table to E New state -> Invalid I Exploring from
Hove A from
Exploring action Nove C from table to New state -> Involid dd Toking action Move A action
Stack [c, Bl, Stack
Stack 3:
A]
Stack [D], Stack table Mova crom to table] | Exploring action New atate Stack [A], Stack (C), Stack [0], Stack [3, 8] Exploring action More from tabla to D New state +> Invalid Exploring action New state Stack 1: (A), Stack [C, B), Stack [D, R] >> Stack (A), Stack
[A],
2:
[3] Plan for [Mova from
from
Move from table
Stack 3: [CI, Stack 4: [D], Stock 5: [R] Exploring sction Move C from table to 8 New state Invalid
Exploring action
Nex
table
[A],
2: IC, 8], Stack 3: [D, R] Plan so for (Move E. from 8 to table Move A from D to table Move from
state [A, B], Stack 2: [C], Stock 3: (0, 3] Exploring action Move from C to table New
to Exploring action
to
state -> Stack [A], Stack (B), Stack 3: [C], Stack 4: [D, Ej Exploring accion table
New
Involid Exploring action
New state -> Stack 1: [C, B, Al, stack 2: [2, E] Exploring action Hove 2 from table to Invalid
Stack 1: [A], Stack 2: [8], Stack 3: [C], Stack 4: [D, B] Plan ao far [Nove E from B to table from to
Nove
from to table] Exploring Exploring sction Move New state +> Stack 1: [R, A], Stack 2: [C], Stack 3: 1D, C
to D
actton Hove a from table
8), Stack 3: (D, R] Exploring sction Move from table Stack [A], Stack [c, Exploxing action Hove è from table to Nex state -> Stack 1: (A, C), Stack 2: (B), Exploring
New atate +> Invalid
3:
New
Invalis Taking
action Hove C from table to A Moved to state >> stack 1: [A, Cl, Stack 2: [3], Stack 1: from
[D,
from table
to D Hove B from C to table Hove C from table to A] Exploring action Hove c from A table New state Stack 1: [A], Stock (R), Stack [C], Stack [D, E] Exploring action
-> Invalid Exploring action from tsble to d New state -> Stack 1: (A, C, Bl, stack 2: [D, El Exploring action from
A table
tabls Niew
Stack [A, cl, Szack (B), Stack
Stack
4: [E] Taking action Move from tabla to C Xeved to state +> Stack 1: [A, C, B), 2:
Move
Move
from table to D Nove B from C to table Move from table to A Nove B from table
C) Reached goal state!
23

---

Published as a conference paper at ICLR 2025
Controller Prompt for Blocksworld
You are given a blocksworld probles em and its initial state, and goal state, Your task is to first generate set of subgoals that can then be solved ganerate the plan between the states.
inilial state: The hand is empty.
is the table. 8 is on C. is
is clear.
is the table. is
is clear.
The goal is:
& is on the table. C is on A. 5 is on C. B is clear.
D is on the table, 8 is on D. 8 is clear.
Controller Output (Sub-goal Decomposition Easy / Hard Annotations) for Blocksworid
The plan should be broken down into 2 sub-goals:
1. Start State -> Stack 1: [C, B, B], Stack 2: [D, A] and Goal State -> Stack
1: [A], Stack 2: [C, 8], Stack 3: [0, R].
Plaaning for this subgoal is hard and can be solved slow using system 2
2. Start State -> Stack 1: [A], Stack 2: [C, B], Stack 3: [D, P] and Goal
State -> 1: [A, C, 8], Stack 2: [E,
Flanning for this subgoal easy can solved using system
24

---

Published as a conference paper at ICLR 2025
E TABLES CORRESPONDING TO RESULT FIGURES
In the tables below, we report the accuracies corresponding to all the plots in the paper.
Table 3:
Comparison of System-1.z Planner (r 0.5) with all baselines on Maze Navigation (corresponding to Fig. 3()),, The blue rows indicate plan validity obtained by methods when Sestates-Explores is matched that of System 1.5 with sub-goals, The green indicate plan validity obtained when #States #States-Explored is maximized. All tables, henceforth, will use the same color coding.
Plan Validity
#States Staates-Explored
System-1
48.7
3.1
System-2 (truncated)
12.2
4.8
System-2 (truncated)
19.7
9.9
System-2 (truncated)
37.2
13.6
System-2 (truncated)
37.2
14,8
System-2 (truncated)
63.7
19.8
System-2 (default)
93.7
24.4
System-1. w/o sub-goal (truncated)
33.5
4.8
System-
sub-goal (truncated)
38.2
9.7
System- 1.5 w/o sub-goal (truncated)
50.0
13.4
System-1.5 w/o sub-goal (truncated)
62.2
15.0
System-1.5 w/o sub-goal (default)
79.5
17.4
System-1.3 w/o sub-goal (test-time controlled)
83.7
20.0
System- 1.5 (w/o sub-goal) (test-time controlled)
24.4
System- 1.5 sub-goal (truncated)
30.5
4.5
System-1.3 1.5 sub-goal (truncated)
56.2
10.0
System-1.5 w/ sub-goal (default)
70.4
13.6
System-1.5 w/ sub-goal (lest-time controlled)
82.2
19.9
System-1.5 sub-goal (lest-lime controlled)
92.2
24.4
System- 1.5 sub-goal (lest-lime controlled)
27.3
Table 4: Comparison of System-1.x Planner (æ 0.5) with all baselines on Blocksworld (corre- sponding to Fig. 3(b)).
Plan Validity
#States- Sttates-Explored
System 1
9.0
4.5
System 2 (truncated)
0.0
9.8
System 2 (truncated)
0.0
19.9
System 2 (truncated)
13.0
29.3
System
(truncated)
38.2
System 2 (default)
28.0
55.5
System-1.5 1.5 w/o sub-goal (truncated)
0.0
9.8
System-1.5 w/o sub-goal (truncated)
0.0
19.9
1y5 w/o sub-goal (truncated)
13.0
29.3
System-1.5 sub-goal (truncated)
27.0
38.0
System-1.5 w/o sub-goal (default)
28.0
55.1
System- 5 w/ sub-goal (truncated)
1.5
9.6
System-1.5 1.5 w/ sub-goal (truncated)
17.5
20.0
System-1.5 1.5 w/ sub-goal (truncated)
25.0
30.0
System-1.5 w/ sub-goal (default)
25.0
38.3
Sy5tem-1.5 w/ sub-goal (test-time controlled)
26.0
53.5
25

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of Neuro-symbolic System-l.æ Planner (a - 0.5) with all baselines on Maze
Navigation (corresponding to Fig. 4).
Plan Validity
#States-Fxplored
A* (truncated)
12.5
A" (truncated)
22.5
9.8
A" (truncated)
31,0
11.3
A' (truncated)
42.0
14.5
A' (truncated)
67.7
19.8
A"
(default)
100.0
System- 1.5 w/o sub-gool (truncated)
33.0
System-13 1.5 w/o sub-goal (truncated)
40.2
9.8
System- 1.5 w/o sub-goal (truncated)
45.7
11.5
System- 1.5 w/o sub-goal (truncated)
65.7
14.9
1y5tem-1.5 w/o sub-goal (default)
84.2
16.5
System- w/o sub-goal (test-time controlled)
94.0
System- 1.5 w/o sub-goal (test-time controlled)
100.0
22.4
System- 1.5 w/ sub-goal (truncated)
30.5
System- 1.5 w/ sub-goal (truncated)
58.7
10.0
System- 1.5 (default)
70.5
11.6
System- w/ sub-goal (lest-time controlled)
76.0
14.8
System-1.5 w/ sub-goal (lest-time controlled)
89.7
19.9
System-1.5 w/ sub-goal (lest-time controlled)
99.2
23.9
Table 6: Comparison of System-1.ar Planner (x: = 0.75) with all baselines on Maze Navigation (corresponding Fig. 5).
Plan Validity
#States-Explored
System-2 (truncated)
12.2
4.8
System-2 (truncated)
19.7
9.9
System-2 (truncated)
37.2
14.8
System-2 (truncated)
47.0
16.6
System-2 (truncated)
63.7
19.8
System-2 (default)
93.7
24.4
System-1.77 w/o sub-goal (truncated)
24.2
4.6
System-1.75 w/o sub-goal (truncated)
29.2
9.8
System-1.75 w/o sub-goal (truncated)
44.5
14.6
System-1.75 w/o sub-goal (default)
59.7
16.6
System-1.75 w/o sub-goal (default)
86.5
20.3
System- sub-goal (test-time controlled)
93.7
24.4
System- 1.75 w/ sub-goal (truncated)
23.0
4,9
System-1.75 w/ sub-goal (truncated)
38.7
System-1.75 w/ sub-goal (truncated)
66.7
15.0
Sy5tem-1.75 w/ sub-goal (default)
75.7
16.6
System-1.75 w/ suh-goal (test-time controlled)
82.0
19.9
Dastem-1.75. sub-goal (test-time controlled)
91.2
24.4
Dystem-1.75 w/ sub-goal (test-time controlled)
96.7
26.8
26

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of System-1.r Planner (a 0.5 and BFS) with all baselines Maze Navigation (corresponding to Fig. 6(a)).
Plan Validity
#States- -States-Explored
System-2 (truncated)
11.7
System-2 (truncated)
28.0
17.4
System-2 (truncated)
28.0
19.9
System-2 (truncated)
56.2
29.7
System-2 (default)
91.2
System- 1.5 w/o sub-goal (truncated)
35.2
System- 1.5 w/o sub-goal (truncated)
45.0
17.2
System- 1.5 w/o sub-goal (truncated)
52.5
19.9
System-1.5 w/o sub-goal (default)
77.2
26.9
System-1.5 w/o sub-goal (test-time controlled)
81.2
30.0
System-1.5 w/o sub-goal (test-time controlled)
91.2
39.8
System- 1.5 5 w/ sub-goal (truncated)
43.0
System- 1.5 w/ sub-goal (default)
67.0
17.4
System- w/ sub-goal (test-time controlled)
78.5
29.7
System-1.5 w/ sub-goal (test-time controlled)
86.5
39,9
System-1.5 w/ sub-goal (lest-time controlled)
91.7
45.4
Table 8: Comparison of System- Planner (x= and DFS) with all baselines on Maze Navigation
(corresponding to Fig. 6(b)).
Plan Validity
Avg States
System-2 (truncated)
11.7
System-2 (truncated)
23.2
System-2 (truncated)
42.0
14.5
System-2 (truncated)
69.7
19.8
System-2 (truncated)
81.2
25.0
System-2 (default)
83.5
System-1.5 w/o sub-goal (truncated)
36.0
4.7
System-1.5 w/o sub-goal (truncated)
50.7
9.8
System-1.5 w/o sub-goal (truncated)
73.2
15.0
Sy5tem-1.5 w/o sub-goal (default)
74.0
18.7
System-1.5 w/o sub-goal (test-time controlled)
76.3
19.8
System-1.5 sub-goal (test-time controlled)
81.2
25.0
System- 1.5 w/o sub-goal (test-timn controlled)
83.5
30.4
System-1.5 (truncated)
40.2
System-1.5 (default)
62.3
10.0
1ystem-1.5 (test-time controlled)
76.3
14.9
System-1.5 (test-time controlled)
79.7
20.0
System- 1.5 (lest-time controlled)
82.7
33.6
27

---

Published as a conference paper at ICLR 2025
Table 9: Comparison of System-1.xr Planner with and without sliding window decomposition (corresponding to Fig. 7).
Plan Validity
#States-Fxplored
System-1.5 w/o Sliding Window (truncated)
27.5
4.6
System-1.5 w/o Sliding Window (truncated)
45.2
10.0
System-1.5 w/o Sliding Window (default)
67.2
17.0
System-1.5 w/o Sliding Window (test-time controlled)
72.1
20.0
System-1.5 w/o Sliding Window (test-time controllcd)
77.2
25.0
System-1.5 w/o Sliding Window (lest-time controlled)
83.2
30.0
System-1.5 w/o Sliding Window (lest-time controlled)
90.0
35.3
System-1.5 w/ Sliding Window (truncated)
30.5
4.5
System-1.5 Sliding Window (truncated)
56.2
10.0
System-1.5 w/ Sliding Window (default)
70.4
13.6
System-1.5 w/ Sliding Window (test-time controlled)
82.2
19.9
System-1.5 w/ Sliding Window (test-time controlled)
92.2
24.4
System-1.5 w/ Sliding Window (lest-time controlled)
96.7
27.3
Table 10: Comparison of hardness functions (Manhattan Distance and #Obstacles) for Maze
Navigation (corresponding to Fig. 8).
Plan Validity
#States-Explored
System-1.5 w/o sub-goal (#Obstacles)
33.5
4,8
System-1.5 w/o sub-goal (#Obstacles)
38.2
9.7
System-1.5 w/o sub-goal (#Ohstacles)
50.0
13.4
System-1.5 w/o sub-goal (#Obstacles)
62.2
15.0
System-1.5 w/o sub-goal (#Ohstacles)
79.5
17.4
Sy5 w/o sub-goal (#Obstacles)
83.7
20.0
System-1.5 w/o sub-goal (#Obstacles)
93.7
24.4
-y5 w/o sub-goal (Manhattan)
31.7
4.8
115 w/o sub-goal (Manhattan)
54.5
15.0
Sy5 w/o sub-goal (Manhattan)
35.2
10.0
System-1.5 w/o sub-goal (Manhattan)
80.0
18.8
System-1.5 w/o sub-goal (Manhattan)
81.5
19.9
System-1.5 w/o sub-goal (Manhattan)
93.7
24.4
28


Reviews:
[{"review_id": "HcyTaoq2iN", "reviewer": "Reviewer_eYzS", "summary": "This work proposes a hybrid planning system that combines a large language model (LLM) fine-tuned to act as either a System-1 or System-2 planner, with a Controller that decomposes tasks into sub-goals based on difficulty. The Controller generates a meta-plan by assigning \"easy\" sub-goals to System-1 for faster solutions and \"hard\" sub-goals to System-2 for more deliberate, search-based solutions. The final plan is constructed by concatenating sub-plans from both systems. However, it faces limitations due to compounding errors across systems, particularly as task complexity grows.", "questions": "1. The term plan validity should not be used interchangeably with plan accuracy. Validity is a binary concept that indicates whether a plan meets all required conditions or constraints (i.e., valid or invalid), whereas accuracy implies a measure of correctness or closeness to an ideal solution, which can vary in degree.\n2. The definition of a sub-goal as a \"pair of states\" adds unnecessary complexity and deviates from the usual understanding. Sub-goals are typically single intermediate states that help guide progress toward the final goal.\n3. The hardness function defined for Blocksworld, in Section A.3, seems wrong. It states that if a block is not on the table and not in its goal position, an additional 1 is added to the hardness cost. However, this approach doesn't account for scenarios where a block is in the air after a pick-up action—this intermediate position may not be the goal but could be one step away from it. Adding to the hardness cost in such cases could mistakenly label states as more challenging, even when they are actually closer to the goal. Similarly for the Maze Navigation task, counting all obstacles in the sub-maze as part of the hardness cost is problematic, as obstacles outside the plan trajectory can inflate the difficulty inaccurately.\n4. In Section 2.3, under Controller Training Data, Step 3, it’s unclear why only a contiguous chunk should be assigned to System-2. What is the rationale for restricting it to contiguous data?\n5. According to the authors' definition of sub-goals, are they explicitly training the Controller to decompose the task into exactly three sub-goals?\n\n**Minor Comments:**\n1. Examples in the figures, particularly Figure 2, would benefit from clearer annotations in the maze to improve the interpretation of the numbering. The current numbering follows a top-to-bottom, left-to-right order, but this is not immediately clear.\n2. In Figure 2, System-2 output, I do not understand why this is invalid “Exploring action left State [3, 1] is invalid”?\n3. The authors should maintain consistency in their notation. In Section 2.3, when discussing Controller training data, the notation for initial and goal states switches from $s_i$​ and $s_j$​ in Step 1 to $s_0$​ and $s_g$​ later in the paragraph.\n\n\n**References:**\n\n[1] Fabiano, F., Pallagani, V., Ganapini, M.B., Horesh, L., Loreggia, A., Murugesan, K., Rossi, F. and Srivastava, B., 2023, December. Plan-SOFAI: A Neuro-Symbolic Planning Architecture. In Neuro-Symbolic Learning and Reasoning in the era of Large Language Models.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "An innovative feature of this hybrid approach, which combines quick solutions from System-1 with thorough search-based solutions from System-2, is a user-controlled parameter x (between 0 and 1) that adjusts how much System-2 is used compared to System-1. This allows users to balance speed and accuracy based on the task’s needs.", "weaknesses": "1. A significant limitation of this approach is its dependency on multiple components performing flawlessly: System-1 must solve the easy sub-goals without hallucinating, System-2 must solve the harder sub-goals using search trajectories accurately, and the Controller must correctly decompose and assign tasks between the two systems. This compounded reliance on error-free performance from each component raises concerns about the approach’s reliability, especially as problem complexity increases. For even moderately difficult tasks, or as the number of objects scales, this method may struggle to maintain effectiveness, as errors are likely to accumulate across the system.\n2. A limitation of this hybrid approach is that selecting the x value too greedily can hurt performance and increase training computation costs. Meanwhile, setting x too high leads to more thorough searches by System-2, which increases compute requirements and makes the approach similar to existing methods like fine-tuning LLMs for planning or SOFAI architectures [1].\n3. For System-2, the hardness function should be explicitly defined for each specific domain in which the planner operates. Without domain-specific definitions, the hardness function risks being too generalized, leading to inaccurate difficulty assessments that don’t align with the unique challenges of each task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pgpSgjgiOb", "reviewer": "Reviewer_vcqi", "summary": "This paper introduces System-1.x which acts as a hybrid System 1 / System 2 planner that outperforms fully System 1 and 2 systems, an A* planner, and a System-1.x variant without subgoal decomposition. This paper contributes a method for collecting data to train a System-1.x system through the introduction of domain-specific hardness functions to determine which of the collected samples should be annotated for System-1 (easy samples) or System-2 (hard samples).", "questions": "- You make the case that System-1.x is advantageous over inference-time methods relying on symbolic planners since they may not exist for certain domains, yet System-1.x relies on defining a hardness function, which is essentially a heuristic function. How could System-1.x be leveraged in new domains where a user is not a domain expert?\n- In addition, it is non-trivial to construct hardness functions for multi-task domains like Alfworld [1], Robotouille [2] or Mini-BEHAVIOR [3]. How would System-1.x be applied to such domains?\n- There appears to be a tradeoff in Figure 3 with accuracy and states explored with Sys-1.5 with and without subgoals. Sys-1.5 with subgoals achieves higher performance with less states explored but as the state exploration budget increases, no subgoals is better. Why is that?\n\n[1] Alfworld: Aligning text and embodied environments for interactive learning (Shridhar et. al 2020)\n\n[2] Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought (Wang et. al 2023)\n\n[3] Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI (Jin et. al 2023)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "$Originality:$ This paper introduces a new method to collect data to finetune a hybrid System 1/2 system whose hybridization factor x can be customized during train and test time.\n\n$Quality:$ The figures and methodology are well-done and the experiments cover relevant baselines and ablations.\n\n$Clarity:$ The paper is well-written and clear.\n\n$Significance:$ Constructing LLM planners that can outperform classical planning approaches with minimal exploration is very important in domains where querying the world model can be expensive (e.g. real world systems like robotics).", "weaknesses": "The introduction makes the following claim\n\n\"System-1.x is a fully LLM-based neural planner ... not relying on any external solvers or verifiers... it has an advantage over inference-time methods that rely on symbolic planners which may not exist for certain domains.\"\n\nHowever, System-1.x strongly rely on the presence of a hardness function, which is essentially a heuristic function. This assumption is too strong and cannot simply be listed as a limitation of this approach; the domains that are tested on could be purely solved using classical planners like A* with access to the hardness (heuristic) function and a valid plan would be retrieved (regardless of consistency or admissibility of the hardness function) much faster than any LLM approach in terms of inference time.\n\nTo improve this paper, evaluating on environments where classical planners cannot be trivially applied would give more credibility to this LLM approach. A user-specified hardness function cannot be provided for the environment since a classical planner would be able to utilize this along with the other environment information provided to the System-1.x planner and quickly find a valid plan. \n\nOne potential idea is utilizing an LLM to provide a hardness value for a pair of states as other works have done, using an LLM to approximate a heuristic function [1,2,3]. This would require some new experiments to show the effectiveness of the hardness function and would necessitate a new approach other than the sliding window mechanism to be token-efficient.\n\nThis is a very important problem to solve in this space and I wish the authors the best of luck.\n\n[1] Large Language Models as Commonsense Knowledge for Large-Scale Task Planning (Zhao et. al 2023)\n\n[2] Reasoning with Language Model is Planning with World Model (Hao et. al 2023)\n\n[3] Toolchain*: Efficient Action Space Navigation in Large Language Models with A* Search (Zhuang et. al 2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "coTBO6tfLV", "reviewer": "Reviewer_z8RW", "summary": "The paper proposes System-1.x, a hybrid planning framework using language models to balance fast, intuitive System-1 planning with deliberate, accurate System-2 planning.", "questions": "1. In the experimental setup, the authors mention that the hybridization factor x allows System-1.x to control the degree of System-1 versus System-2 planning. How sensitive is this hybridization factor to changes in task complexity? Were there significant trade-offs observed with larger values of x in different domains?\n2. In discussing Blocksworld’s out-of-distribution generalization, the paper mentions that System-1.x effectively utilizes System-1 for ID sub-goals while reserving System-2 for OOD sub-goals. Could the authors explain how the controller identifies which sub-goals are in-distribution vs. out-of-distribution during inference?\n3. The authors claim that System-1.x can match or exceed a purely System-2 Planner’s accuracy with fewer states explored. What specific methods were used to prevent overfitting to the training traces, especially for tasks requiring extensive state exploration?\n4. Why is there no baseline comparison against other fast and slow architectures used for solving planning problems? It is not clear to me what is the advantage of using this approach against architectures like Plan-SOFAI that are both optimal and resource-efficient compared to System 2 solvers for planning tasks.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "System-1.x shows robust adaptability and performance across different planning tasks (e.g., Maze Navigation, Blocksworld), outperforming both purely System-1 and System-2 planners due to its effective sub-goal decomposition.", "weaknesses": "The related work section of the paper does not cover papers similar to this work. To cite a few papers that should have been used as baselines to compare System 1.x against - \n[1] Katz, M., Kokel, H., Srinivas, K., & Sohrabi, S. (2024). Thought of Search: Planning with Language Models Through The Lens of Efficiency. In The First Workshop on System-2 Reasoning at Scale, NeurIPS'24.\n[2] Fabiano, F., Pallagani, V., Ganapini, M. B., Horesh, L., Loreggia, A., Murugesan, K., ... & Srivastava, B. (2023, December). Plan-SOFAI: A Neuro-Symbolic Planning Architecture. In Neuro-Symbolic Learning and Reasoning in the era of Large Language Models.\n[3] Webb, T., Mondal, S. S., Wang, C., Krabach, B., & Momennejad, I. (2023). A Prefrontal Cortex-inspired Architecture for Planning in Large Language Models. arXiv preprint arXiv:2310.00194.\nRefer to the questions for more weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CJbX4AqeXo", "reviewer": "Reviewer_s6YJ", "summary": "This paper attempts to create a mixed-mode planning system that both immediately generates plans without deliberate state-space search, and does deliberate state space search.\n\nHowever, this paper seems deliberately misleading in terms of what it claims, namely the improvements over A*.", "questions": "In the results section for blocksworld, which numbers are from in-distribution testing and which are from out-of-distribution testing?\n\nWhat guarantees does System 1.75 have?\n\nDid you consider wall-clock time budgets? This would be a more fair comparison. Surely the authors know that querying LLMs repeatedly takes far longer than querying A*, so I'm curious why these times weren't reported in the paper?", "limitations": "", "rating": 1, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "In general, the goal of creating a neurosymbolic planning system is a very good one. However, the one presented in this paper is preliminary. \n\nI really like the idea of having a hybridization factor. It could be interesting to have a way to control this at inference time rather than at training time, i.e. like iterative-deeping based searches use more search budget when given more time.", "weaknesses": "While this work attempts to build a neurosymbolic planning system, which is a good goal, it seems like the work may not be ready for publication at ICLR. \n\nThe main results of the paper rest on a claim about the \"States explored\" metric, however, there are several issues with this evaluation. Primarily, computational costs of querying LLMs are not accounted for. A single pass of an LLM clearly can do a significant amount of computation. Furthermore, intermediate LLM-generated states are never checked, e.g. they could contain hallucinations, so it doesn't seem like they'd make a reasonable metric. More importantly there is simply no guaranteed correspondence between intermediate states inside of A* and reported intermediates produced by an LLM to the degree that the two can be compared meaningfully. Because of this, it is unreasonable to restrict A* to run in correspondence with the LLM output, when a full A* computation may several orders of magnitude less than a computational single pass of an LLM. Since the paper rests on the states explored metric, I highly encourage the authors to propose a better method for comparing computational costs between LLMs and A*. \n\nAnother ommission is the fact the the overall system is trained by calling A* thousands of times, which makes an efficiency argument extremely questionable. Regardless of the current state of such an argument, I do buy into the general idea that mixed-system planning algorithm could be more efficient than a system-2 one and more accurate than a system-1 one, but I think the authors would need to fairly evaluate this to make progress on building such an algorithm.\n\nThe other main metric used in the paper is \"valid plan accuracy\". Such a metric is not meaningful when using systems with guarantees, namely A*, which is guaranteed to find optimal plans, while the proposed system has no such guarantees. Because of the false equation between A* intermediate states and LLM-output, the budget restriction is not meaningful and skews the accuracy metric significantly. For instance, if wall time budget were used instead, A* would simply have an accuracy of 100%. Reporting otherwise is potentially misleading. \n\nThe paper claims to test out-of-distribution blocksworld, but it is unclear from figure 1 which results are out of distribution testing and which are in-distribution testing. Also, the left hand side of figure 1 has a correct maximum accuracy of 100%, but the right hand side has a maximum accuracy of only 30%, which seems extremely misleading. Can you clearly label which results in Figure 1 are from in-distribution versus out-of-distribution testing and present the results in Figure 1 with a uniform scale 0-100?\n\n\"model-based\" on line 144 seems misused, namely confusing parameterized models like neural networks with transition models used in planning and RL. In general, some terminology is unconventional or misused. \n\nline 253: \"we assume that they can be decomposed into sub-goals\". This is a very dangerous assumption that rules out true planning problems, as often sub-goals interact. This is the case for blocksworld (but not the maze task), so since the blocksworld dataset has been filtered in this way, it no longer represents the original task and is in fact far simpler to solve [1]. The method for decomposing plans is not obvious, and the sliding window approach, while interesting, is certainly not guaranteed to decompose an arbitrary problem successfully. In the worst case, the system must always be able to fall back on System 2. \n\nThe system is given tiny datasets and (for mazes) tested within distribution. 5x5 mazes are far too small and A* can scale to much bigger problems, while it's unclear if the proposed LLM-based approach does. The relatively high system 1 performance is a good indicator that parts of the problem are being memorized -- and in the case of mazes it is clear how this is possible since the task has decomposable subgoals that can be independently memorized. It's good that out-of-distribution was considered for blocksworld, but the results section doesn't always distinguish between numbers reported from in-distribution testing and out-of-distribution testing. \n\nIn general, planning is misunderstood. By default A* is sound, complete, and produces optimal plans. In contrast, an LLM generator is not sound, not complete, and not optimal. When A* is limited by search budget, it is not the same algorithm, but still will never produce an incorrect plan, and if given enough time will produce the optimal plan. In contrast, an LLM (or system that uses one) is capable of producing incorrect plans, and has no optimality guarantee even if given infinite computation. The proposed method does not have any guarantees, wheras systems like LLM-Modulo do have soundness guarantees (but not completeness!). Please consider adding a discussion section that explicitly compares the theoretical guarantees and limitations of A* versus the proposed LLM-based approach. This could help readers better understand the tradeoffs involved and the contexts in which each approach might be most appropriate.\n\n[1] Sussman, G. J. (1973). A computational model of skill acquisition.\n\nSee also: https://en.wikipedia.org/wiki/Sussman_anomaly", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)