Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programming

## Abstract

Abstract While large language models (LLMs) have recently demonstrated strong potential in solving planning problems, there is a trade-off between flexibility and complexity. LLMs, as zero-shot planners themselves, are still not capable of directly generating valid plans for complex planning problems such as multi-constraint or long-horizon tasks. On the other hand, many frameworks aiming to solve complex planning problems often rely on task-specific preparatory efforts, such as task-specific in-context examples and pre-defined critics/verifiers, which limits their cross-task generalization capability. In this paper, we tackle these challenges by observing that the core of many planning problems lies in optimization problems: searching for the optimal solution (best plan) with goals subject to constraints (preconditions and effects of decisions). With LLMs’ commonsense, reasoning, and programming capabilities, this opens up the possibilities of a universal LLM-based approach to planning problems. Inspired by this observation, we propose LLMFP, a general-purpose framework that leverages LLMs to capture key information from planning problems and formally formulate and solve them as optimization problems from scratch, with no task-specific examples needed. We apply LLMFP to 9 planning problems, ranging from multi-constraint decision making to multi-step planning problems, and demonstrate that LLMFP achieves on average 83.7% and 86.8% optimal rate across 9 tasks for GPT-4o and Claude 3.5 Sonnet, significantly outperforming the best baseline (direct planning with OpenAI o1-preview) with 37.6% and 40.7% improvements. We also validate components of LLMFP with ablation experiments and analyzed the underlying success and failure reasons.

## 1 Introduction

Figure: Figure 1: An overview of LLMFP and how it is applied to a coffee supply chain example. All sections in yellow are inputs, and all sections in blue are steps accomplished by LLMs. With task description, background information, and query as inputs, LLMFP defines the goal, decision variables, and constraints of this optimization problem, identifies all necessary variables and summarizes their key information into a JSON representation, generates codes to solve the optimization problem, executes the codes and formats the execution results, and performs self-assessment for each step.
Refer to caption: x1.png

Making complex plans subject to multiple constraints is a time- and labor-intensive process, but is critical in many aspects of our lives such as work arrangement, business management, logistics, and robotics. In the past, people used domain-specific tools and languages to make specific plans in their areas, which often required a steep learning curve and were hard to adapt to other domains. When large language models (LLMs) emerge with their versatile capabilities such as language understanding, reasoning, and tool-using, using LLMs for planning has gained significant traction.

For such planning systems to be deployed in complex, real-world applications, two desirable properties need to be satisfied:
1). *Zero-shot flexibility*:
Unlike many experimental settings where planning tasks usually come with labeled datasets, it is very challenging to request such datasets from users in many realistic settings. Ideally, a flexible planning system should be able to conduct planning with only a task description provided by users, and nothing else.
2). *High performance on complex tasks*:
Realistic planning problems usually require multi-step, long-horizon solutions, with many explicit and implicit constraints.

However, there is a trade-off between flexibility and task complexity, and thus existing LLM-based planning systems are typically unable to achieve both simultaneously.
On one hand, planning systems capable of performing zero-shot planning, utilizing the abundant knowledge and generalization capabilities in LLMs, in many successful applications can only solve single-objective tasks such as household chores, with step-by-step interactive planning and grounding (Huang et al., ; Ahn et al., ; Huang et al., ).
For complex, multi-constraint, and long-horizon tasks that involve iterative trials and errors even for humans, LLMs still do not have the capabilities to generate valid plans by themselves (Kambhampati et al., ). On the other hand, recent research efforts to empower LLM-based planners to solve complex tasks are often based on well-designed task-specific in-context examples and extensive task-specific pre-defined efforts (Liu et al., ; Xie et al., ; Li et al., ; Song et al., ; Gundawar et al., ), impacting their zero-shot flexibility. In short, few existing planning systems can flexibly resolve generic complex tasks with only task descriptions in natural language. Hence we ask: Can we build a universal LLM-based planning system that can solve complex planning problems without task-specific efforts?

In this paper, we observe that, although planning problems come with drastic variations, many of them can be recast as constrained optimization problems — The optimization problems aim to find the optimal solution, which is equivalent to locating the best plan that satisfies the goal for planning problems; the decision’s precondition and effect are equivalent to constraints of optimization problems. Furthermore, although solving complex planning tasks is generally beyond the capabilities of LLMs, converting any planning tasks into optimization problems is a much more tractable problem, and can be within the zero-shot capabilities of LLMs.

Motivated by this, we propose LLM-Based Formalized Programming (LLMFP, illustrated in Fig. 1), a general-purpose zero-shot planning framework that leverages LLM’s strong common reasoning, and programming capabilities to encode planning problems into optimization problems *without* any task-specific examples or designs, combined with a formal planner to solve the optimization problem.
LLMFP takes in natural language domain description, natural language query under this domain, and available background information or APIs as inputs, and solves the planning problem in five steps. First, LLMFP prompts LLMs to reason and propose the goal, decision variables, and key constraints necessary for the task. Second, based on the response, LLMFP asks LLMs to formulate a representation that includes all variables needed to construct and their detailed information and requirements. Third, with the representation, LLMs write codes to formally encode the problem into an optimization problem. Fourth, LLMFP executes the generated codes, converts the execution results into plans. Finally, LLMFP performs overall self-assessment and automatic modification to fix the broken parts of the previous steps. Currently, LLMFP uses the satisfiability modulo theory (SMT) to encode the optimization problems but can be adapted to any planners or solvers by updating the requirements and representation format in the prompts.

We evaluate our framework with 9 diverse planning problems, ranging from single-step supply chain problem to multi-step robot block stacking and moving. Experiment results demonstrate that LLMFP achieves strong performance across all tasks with an average of 83.7% and 86.8% optimal rates for GPT-4o and Claude 3.5 Sonnet, which greatly outperforms the baselines, including direct plan generation with OpenAI o1-preview. We conduct ablation experiments to validate the key components of our framework and investigate the underlying reasons why our framework is more effective than baselines.
In addition, although our framework does not require task-specific examples, we show the ease of adding task-specific examples to one stage of LLMFP, and how it could help to clarify unclear queries and therefore can further improve the performance within the same domain.

In summary, our key contributions are:

- •
We offer a novel perspective on using LLMs to solve planning problems by rigorously constructing optimization problems from scratch, alike how human experts use optimization tools for planning.
- •
We propose LLMFP, a general-purpose planning framework with zero-shot generalization capability. To our knowledge, LLMFP is the first to enable LLMs to build and solve diverse types of planning problems as optimization problems with no task-specific examples or external critics.
- •
LLMFP notably achieves 83.7% and 86.8% optimal rates for GPT-4o and Claude 3.5 Sonnet, outperforming the best baseline (direct planning with OpenAI o1-preview) by 37.6% and 40.7%. We examine the effectiveness of our framework and analyze the success and failure reasons.

## 2 Related Works

### 2.1 LLMs for Planning

The remarkable capabilities of LLMs in reasoning (Wei et al., ; Kojima et al., ; Yao et al., ; ) and tool-use (Qin et al., ; Schick et al., ) brings up interests of utilizing LLMs to solve planning problems. Based on LLMs commonsense and zero-shot generalization capability, many works capable of performing zero-shot planning are proposed (Huang et al., ; Ahn et al., ; Huang et al., ). However, their planning scenarios are limited to simple single-objective tasks such as household cleaning and they often require step-by-step interactive planning with grounding. To improve LLMs reasoning and planning capabilities for more complex problems, chain-of-thought (CoT) prompting appends reasoning steps before the answer to encourage LLMs to perform step-by-step reasoning (Wei et al., ). Recent works also propose to aid the LLM planning processes with external tools (Liu et al., ; Guan et al., ; Chen et al., ; Li et al., ; Gundawar et al., ; Hao et al., ; Chen et al., ). For example, (Liu et al., ; Xie et al., ; Gundawar et al., ) leverages LLMs as translators to convert problems into fixed formats and inputting them to external planners, (Li et al., ) prompts LLM to add short codes to existing optimization codes of certain domain to account for follow-up what-if questions, and (Gundawar et al., ; Chen et al., ) empowers LLMs to iteratively refine plans or prompts based on feedback from external task-specific critics/verifiers/humans. However, to achieve strong performance, these methods are often based on extensive task-specific pre-defined efforts. For example, CoT depends on task-specific examples to achieve strong performance, domain files written in the planning domain definition language (PDDL) (Aeronautiques et al., ; Haslum et al., ) are required for (Liu et al., ), codes that encode domains into a mixed-integer linear program (MILP) are necessary for (Li et al., ), and external constraint critics are needed for (Gundawar et al., ). These requirements limit the generalization capability of these works to new domains.

### 2.2 LLM + Solver

As existing LLMs still do not have the capability to perform long-horizon reasoning for complex tasks (Achiam et al., ; Valmeekam et al., ; ; ), many works propose to take advantages of both LLMs and external planners or solvers by combining them for reasoning or planning. (Wu et al., ; He-Yueya et al., ; Pan et al., ; Ye et al., ) combines LLM with symbolic solvers to solve logical reasoning problems. LLMFP differs from them in that we aim to solve complex planning problems, which could include implicit constraints not clearly described in the task description and could be sequentially long-horizon tasks with defined actions. Instead, most logical reasoning problems are single-step satisfiction problems and clearly specify all constraints in question. In addition, LLMFP proposes a general approach, which does not require task-specific examples or task-specific efforts. (Li et al., ) teaches LLMs to add constraints to existing MILP code snippets of planning problems. (Li et al., ) asks the developer to express planning problems into automaton and guide the LLMs for planning based on it. (Liu et al., ; Guan et al., ; Zhou et al., ; Xie et al., ) leverages PDDL planner to aid the planning processes. Except for the natural language task description, they require human efforts to design solver-related specifications and task-specific examples, which is not needed for LLMFP.

## 3 LLMFP

LLMFP aims to deliver a plan that can resolve generic planning problems. For example, consider a coffee supply chain problem, where a coffee company sources beans from three suppliers with fixed capacity, roasts them at two facilities into either dark or light coffee and ships the roasted coffee to three retail locations to fulfill their demands. Then a planning problem involves accomplishing the task at the cheapest cost.

To achieve this, LLMFP takes the following inputs from users, as shown in Figure 1 (top panels).

- •
Natural Language Task Description. A natural language description that details the problem settings and the planning objective, such as the above description of the coffee problem.
- •
Background Information & API. A list of background information about the tasks as well as information on APIs that the planner can use. An example of the background information for the coffee task is the variables containing specific numbers of supplier capacities, cafe demands, and costs for shipping and roasting.
- •
User Query. The question that either describes the detailed initial and/or goal states or adds/modifies existing requirements of the tasks. In the coffee planning task, one example query is ‘What would be the outcome if cafe1 experienced a 23% rise in demand’.

Example inputs for all 9 tasks can be found in Appendix A.5. Note that LLMFP does not require any task-specific examples from the users. Considering the diversity of user requests, LLMFP needs to accommodate a large variety of domains, planning problem setups, user queries, constraints, and complexity levels, which poses a great challenge.

### 3.1 Overview

Emulating how humans construct and write codes to solve optimization problems, LLMFP solves the planning problems via the following steps, as shown in Fig. 1. ❶ Definer: LLMFP first prompts an LLM to define the problem by analyzing and proposing the optimization goal, decision variables, and constraints of the optimization problem (Sec. 3.2). ❷ Formulator: LLMFP asks LLM to think about the necessary variables and steps to build when programming, and formulate a representation to summarize all key information of these variables (Sec. 3.3). ❸ Code Generator: Given this representation, LLMs generate codes that initialize all necessary variables, assert constraints, and add goals (Sec. 3.4). ❹ Result Formatter: After LLMFP executes the generated codes, it prompts LLMs to convert the execution result into a fixed format and provide a short assessment of the execution results (Sec. 3.5). ❺ Self Assessment and Modification: LLMFP assesses each step based on the execution result, and modifies the first incorrect step (Sec. 3.6). The generated plan is delivered when it passes self-assessments of all steps. Please refer to Appendix A.6 and A.7 for example outputs and prompts of all steps in LLMFP.

### 3.2 Definer

The first step of building an optimization problem is to identify the goal, decision variables, and constraints of the problem. This is accomplished by prompting the LLM to express in a natural language format (See Figure 1 for an example). The prompt includes ❶ all the user-supplied task information; ❷ a description of what goal, decision variables, and constraints mean; and ❸ an instruction to output the aforementioned information. The detailed prompt is listed in Appendix A.7.2.

While generating the goals and decision variables are straightforward, generating the constraints is challenging, because certain constraints are not explicitly stated and can only be inferred by commonsense reasoning. We refer to these as the *implicit constraints*.

For example, in the coffee supply chain task example, the implicit constraints include ‘the roasted coffee in each roastery does not exceed the beans it receives’, ‘the shipped coffee from each roastery does not exceed the coffee it roasts’, and importantly but easily overlooked, ‘all numbers of shipped and roasted beans and coffee need to be non-negative integer’.

To facilitate uncovering the implicit constraints, we include in the prompts (under the description of constraints) a three-step instruction to derive the constraints: ❶ Identify all decision variables in this task, ❷ for each pair of decision variables, consider relations (explicit, implicit, underlying assumption, unmentioned commonsense) between them to make sure all variables are connected, and ❸ provide a constraint reasoning first before answering. This effectively helps LLMs to better identify implicit constraints for multi-constraint planning problems. Since for multi-step planning problems the task description needs to explicitly define the preconditions and effects of each action, there will be no implicit constraint so this step is omitted.

### 3.3 Formulator

Figure: Figure 2: Examples of diverse variables and steps appeared in different tasks. Examples 1 and 2 are from single-step multi-constraint problems, and Examples 3 and 4 are from multi-step problems.
Refer to caption: x2.png

Although from the last step, we have defined the optimization goal, decision variables, and constraints, more efforts are needed to integrate these into executable and correct code snippets. For example, the decision variables may be continuous, binary, or integer; they may need to be arranged into data structures such as lists and dictionaries; they may have fixed values or belong to flexible ranges, etc. All this information and requirements need further declarations to better generate codes. Fig. 2 shows examples of diverse types of variables.

Single-Step Multi-Constraint Problem To foster LLM to better formulate the variables, we prompt LLM to utilize a fixed-format JSON representation to summarize all variables and their key information. As shown in Fig. 1 and Fig. 2, for each variable, the representation uses 6 fields to summarize the information related to this variable. The name field indicates the variable name. Since we are using SMT as the optimization solver. the SMT_variable field indicates whether the variable is an SMT variable or a normal variable. SMT variables are different from normal variables in that they don’t hold specific values upfront, rather, they are symbolic variables to represent unknown values. The number_of_variables field represents the length of the variable. For example, the total cost in the coffee supply chain task has a length of 1, but the number of coffee beans shipped from supplier to roasteries would be a list of length 6 since there are 3 suppliers and 2 roasteries. The data_source field denotes the dependencies of the variable. For example, since the total cost of the coffee task is the summation of shipping cost from suppliers to roasteries, roasting cost, and shipping cost from roasteries to cafes, they are the data source of the total cost. The value field further specifies the value of the variable. It could either be a real number or list, a string describing keys and values when the data structure of the variable is a dictionary, or it could be a description of operations to do with the data source. In the coffee example, the value is a string “sum of all costs” for the total cost variable. The specific_requirement field is where we point to the constraints or goals related to this variable. For example, the specific requirement for total cost is “minimize”, and for the number of coffee beans shipped from supplier to roasteries, it could be “non-negative integer, summation for each supplier greater than capacity”. With this intermediate step between Definer and Code Generator, LLMFP is capable of obtaining a more detailed, well-formulated, and overall coding plan. We teach LLM to formulate variables into this representation by including two simple examples in the prompt. Note these two examples are not task-specific examples of any of our testing tasks and we do not modify these two examples across tasks.

Multi-Step Planning Problem In addition to including information of variables in the JSON representation, more information is needed for multi-step planning problems. Since predicates, actions, and their preconditions and effects are involved in multi-step problems, we separate the JSON representation for multi-step problems into five stages: objects, predicates, actions, update, goal and each stage includes 6-field representations described above. The objects stage declares all objects in the scenario. The predicates stage defines the predicates, which represent the properties of objects and the relationships between them. The actions stage initializes variables to represent all actions. The update stage adds assertions to existing action variables to account for the preconditions and effects of actions. The goal stage adds constraint to existing predicate variables to encode the goal. As shown in Fig. 2, Example 3 shows a variable in predicates stage, which initializes a dictionary with keys (block, block, timestep) to represent whether a block is on another block at a certain timestep. Example 4 shows a step in update stage, which collects all defined actions in actions stage and asserts that only one action can be true for a timestep. For multi-step planning tasks, we replace the examples with a multi-step task example, and similarly, it is not a task-specific example and we do not modify it across tasks.

To generate the above information, the Formulator’s prompt includes ❶ A brief instruction for the Formulator, ❷ Example input-output pairs of Formulator as demonstrations, and ❸ The user-provided task information and Definer’s output. The detailed prompt is listed in Appendix A.7.2. Note that although example input-output pairs are used, they are task-agnostic examples fixed for all the planning tasks. No task-specific examples are needed.

### 3.4 Code Generator

With the representation generated from the Formulator, now we have all the information needed to build an optimization problem with codes. In the Code Generator’s prompt, we explain the meanings of different stages and fields in the JSON representation and ask LLMs to follow Python and Z3 SMT syntax (De Moura & Bjørner, ). By including user-provided task information and results from Definer and Formulator, with no examples, LLM could reliably generate reasonable, executable, and correct Python codes. Then, LLMFP executes the codes and returns to re-generation if there are runtime errors. We set our maximum re-generation times to be 5.

### 3.5 Result Formatter

Since the variable names are decided by LLMs and have chances to be very different across queries, after code generation, we use a Result Formatter to ask LLM to convert the execution result to a fixed output format. For example, the output for the coffee task would be a JSON with the following fields: ❶ number of coffee beans shipped from each supplier to each roastery, ❷ the number of light and dark coffee roasted in each roastery, and ❸ the number of light and dark coffee shipped from each roastery to each cafe. After filling in this result, we prompt the LLM to provide a brief evaluation of the result based on whether the result achieves the goal, satisfies constraints, and makes sense in common sense. Taking commonsense into consideration is important because sometimes if a necessary constraint is missing from the Definer step, it could result in unreasonable execution result thus could be unrealistic in commonsense. For example, for the coffee task, if the Definer does not include the non-negative constraint, to minimize the cost, the solver could propose negative units of shipped coffee. Detecting these unrealistic plans is helpful in the Self Assess & Modification step.

### 3.6 Self Assess & Modification

After we have the execution result and evaluation, LLMFP perform self-assessment to reason about the correctness and provide a rating for the Definer, Formulator, and Code Generator. If the assessment marks all three steps to be correct, this plan will be delivered as the final output of our framework. Otherwise, the assessment will reason about how to modify this step, and provide a modification by itself. This modification will replace the output of the incorrect step and LLMFP will loop back to continue the next steps from there again. That is, if the self Assessor thinks the output of the Formulator is incorrect, it will generate a JSON representation by itself, and the framework will use this modified representation to enter Code Generator again. We set the maximum number of loops to be 5.

### 3.7 Choice of Solver

As a framework that proposes to formulate and solve planning problems as optimization problems, LLMFP could be adapted to use any planner or solver by modifying the requirements in prompt to follow the syntax of new solvers. In this work, we compare the SMT solver with popular PDDL and MILP solvers and choose SMT solver with following reasons: SMT allows explicit goal and constraint assertion from scratch, which could be used to solve both single-step multi-constraint problems and multi-step problems, while PDDL solvers require a PDDL domain file and a PDDL problem file with strictly fixed formats, which limits its capability to solve non-PDDL problems. SMT is complete and sound, that is, it guarantees to find the optimal plan, while PDDL planners are not guaranteed to be complete. In addition, for all optimization solvers like SMT and MILP, the processes of building optimization problems for different optimization solvers or planners are the same: defining the goal, constraints, and decision variables, and writing codes to encode relationships between decision variables. Thus, utilizing any optimization planner has a similar process. We show how easily our framework could adapt to use MILP by including prompt differences and output examples in Appendix A.7.3. We selected SMT over MILP because the SMT Z3 solver is a publicly available package and is more accessible to all users than the Gurobi MILP solver, which requires licenses and limits the number of devices per license.

## 4 Experimental Results

### 4.1 Domains

We test on 9 planning problems, which includes 5 multi-constraint decision making tasks,Coffee, Workforce, Facility, Task Allocation, and Warehouse, and 4 multi-step tasks, Blocksworld, Mystery Blocksworld, Movie, and Gripper (Li et al., ; Valmeekam et al., ; Stein & Koller, ). Detailed task descriptions are included in Appendix A.1.
The queries are either what-if questions that change/add constraints to the existing scenarios or different task initial and goal conditions. Task inputs including example queries are given in Appendix A.5.

### 4.2 LLMFP Performance

We evaluate LLMFP on 9 tasks with two models: GPT-4o () and Claude 3.5 Sonnet () with temperature 0. Each task comes with a natural language task description, background information on the scenarios or info collection API, and natural language queries. For single-step decision-making problems, expected output formats are also provided. With no task-specific example, LLMFP takes these inputs and delivers plans as outputs. We use optimal rate as the evaluation metric, that is, whether the plans are optimal under the current task scenario and query.

Baselines We compare LLMFP against 1) Direct: LLM direct plan generation, 2) CoT: chain-of-thought prompting (Wei et al., ) by asking LLMs to reason before generating the final answer, and 3) Code: prompts LLM to generate Python codes to solve the problem, allowing the use of any package or solver. For all baselines, we use both GPT-4o and Claude 3.5 Sonnet and also include a direct plan generation baseline with OpenAI o1-preview (). All baselines are zero-shot with no task-specific examples. All baselines have the same input information as LLMFP, including task description, task background information or info collection API, and query. Please refer to Sec. A.7 for prompts of baselines.

**Table 1: Optimal rate (%) comparison of LLMFP with baselines on 5 multi-constraint problems.**
| Method | Coffee | Workforce | Facility | Task Allocation | Warehouse | Average |
| --- | --- | --- | --- | --- | --- | --- |
| Direct${}_{\textsc{GPT-4o}}$ | 0.8 | 2.6 | 0.0 | 0.0 | 0.0 | 0.7 |
| Direct${}_{\textsc{o1-preview}}$ | 25.9 | 47.6 | 4.8 | 4.0 | 66.0 | 29.7 |
| CoT${}_{\textsc{GPT-4o}}$ | 0.0 | 5.6 | 0.0 | 0.0 | 16.0 | 4.3 |
| Code${}_{\textsc{GPT-4o}}$ | 17.7 | 75.8 | 53.9 | 0.0 | 8.0 | 31.1 |
| LLMFP ${}_{\textsc{GPT-4o}}$ | 64.7 | 92.2 | 70.7 | 96.0 | 72.0 | 79.1 |
| Direct${}_{\textsc{Claude 3.5 Sonnet}}$ | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| CoT${}_{\textsc{Claude 3.5 Sonnet}}$ | 7.1 | 0.0 | 0.0 | 0.0 | 14.0 | 4.2 |
| Code${}_{\textsc{Claude 3.5 Sonnet}}$ | 59.8 | 71.9 | 47.3 | 0.0 | 42.0 | 44.2 |
| LLMFP ${}_{\textsc{Claude 3.5 Sonnet}}$ | 80.5 | 88.7 | 48.2 | 96.0 | 90.0 | 80.7 |

**Table 2: Optimal rate (%) comparison of LLMFP with baselines on 4 multi-step problems.**
| Method | Blocksworld | Mystery Blocksworld | Movie | Gripper | Average |
| --- | --- | --- | --- | --- | --- |
| Direct${}_{\textsc{GPT-4o}}$ | 41.5 | 0.8 | 85.7 | 0.0 | 32.0 |
| Direct${}_{\textsc{o1-preview}}$ | 88.4 | 31.9 | 100.0 | 52.0 | 68.1 |
| CoT${}_{\textsc{GPT-4o}}$ | 39.9 | 2.7 | 81.0 | 0.0 | 30.9 |
| Code${}_{\textsc{GPT-4o}}$ | 0.0 | 0.3 | 0.0 | 0.0 | 0.1 |
| LLMFP ${}_{\textsc{GPT-4o}}$ | 96.2 | 77.7 | 100.0 | 76.0 | 87.5 |
| Direct${}_{\textsc{Claude 3.5 Sonnet}}$ | 43.2 | 0.5 | 100.0 | 12.0 | 38.9 |
| CoT${}_{\textsc{Claude 3.5 Sonnet}}$ | 52.8 | 2.8 | 100.0 | 28.0 | 45.9 |
| Code${}_{\textsc{Claude 3.5 Sonnet}}$ | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| LLMFP ${}_{\textsc{Claude 3.5 Sonnet}}$ | 93.0 | 98.0 | 100.0 | 76.0 | 91.8 |

Results and Analysis We include the optimal rate comparison of LLMFP and baselines on 5 multi-constraint problems and 4 multi-step problems in Table 1 and 2. There are three key takeaways:

First, LLMFP achieves strong performance across all 9 tasks, significantly outperforming all baselines. For GPT-4o, LLMFP achieves an average of 83.7% optimal rate across 9 tasks (79.1% for 5 multi-constraint problems and 87.5% for 4 multi-step problems). For Claude 3.5 Sonnet, LLMFP achieve an 86.8% optimal rate across 9 tasks (80.7% for 5 multi-constraint problems and 91.8% for 4 multi-step problems). For 5 multi-constraint problems, LLMFP ${}_{\textsc{GPT-4o}}$ and LLMFP ${}_{\textsc{Claude 3.5 Sonnet}}$ outperform best baselines Code${}_{\textsc{GPT-4o}}$ and Code${}_{\textsc{Claude 3.5 Sonnet}}$ by a large margin of 48.0% and 36.5%. For 4 multi-step problems, LLMFP ${}_{\textsc{GPT-4o}}$ and LLMFP ${}_{\textsc{Claude 3.5 Sonnet}}$ outperform Direct${}_{\textsc{o1-preview}}$ and CoT${}_{\textsc{Claude 3.5 Sonnet}}$ by an average of 19.4% and 45.9%. This highlights both the effectiveness and the generalization capability of LLMFP.

Second, among baselines, Code works better for multi-constraint problems, while Direct and CoT work better for multi-step problems. This validates that the skills required for solving different tasks are different. For multi-constraint problems, as heavy calculations are required to test every possible solution, it is hard for LLMs to directly plan, even with the strongest o1-preview model. For multi-step problems, since Code tries to use a PDDL planner, which requires LLM to generate fixed-format PDDL domain and problem files, it almost always fails to generate and call them correctly. While it is easier for LLMs to directly devise plans as the preconditions and effects of each action are easier to reason about than calculations. This further proves that LLMFP can tackle problems that are fundamentally different because it uses a universal and formal approach for all tasks.

Third, for Direct and CoT, Mystery Blocksworld’s performance degrades largely compared to Blocksworld, though they are fundamentally same problems. Changing predicate and action names to illogical names makes LLMs hard to understand the problem and generate reasonable plans. However, LLMFP still can obtain an overall strong optimal rate of 77.7% and 98.0% on Mystery Blocksworld for GPT-4o and Claude 3.5 Sonnet. This shows LLMFP is robust to obfuscated problems, as it can encode the problem as long as the problem is clearly defined regardless of the names.

To summarize, the experiment results show that LLMFP is capable of solving all 9 tasks with strong performance and is robust to fundamentally different and obfuscated problems. We also show the performance of LLMFP across iterations, time and cost statistics, and failure analysis of LLMFP for different tasks in Appendix A.2,  A.3, and A.4.

### 4.3 Effectiveness of LLMFP Components

We then validate each component of LLMFP with ablation experiments on 9 tasks. We examine the effectiveness of Definer, Formulator, and Self Assess & Modification by removing these components from our framework one at a time and comparing with LLMFP. We do not remove Code Generator and Result Formatter because they are the necessary components of LLMFP to deliver outputs. We use GPT-4o as the LLMs and optimal rate as the evaluation metric.

Results and Analysis
We include the optimal rate performance comparison of LLMFP and baselines on 9 problems in Table 3.
From Table 3 there are two key takeaways:

First, removing any of the 3 components from LLMFP negatively affects the performance. For multi-constraint problems, removing Definer, Formulator, and Self Assess & Modification lowers the optimal rate by 15.4%, 22.2%, and 21.9%. For multi-step problems, removing Formulator and Self Assess & Modification reduces the optimal rate by 87.4% and 12.4%.

Second, for different problems, the most effective components are different. Coffee degrades the most for No Definer; Warehouse and all multi-step problems drop the most for No Formulator; and Workforce decreases the most for No Self Assess & Modification. This again validates the diversity of the 9 problems and how they require different efforts to be successfully solved. Thus, LLMFP is an overall framework that could aid the process of planning from all aspects.

Note that Warehouse with No Definer is the only entry that reaches a higher optimal rate than LLMFP. We notice that the main failure case for LLMFP in Warehouse is: LLM overwrites the provided API get_distance and provides 1 as the output during Code Generation. For Warehouse with No Definer, this happens less frequently. This kind of issue could be alleviated by adding non-task-specific modifications to our framework, such as emphasizing direct use of the provided API in prompt or implementing a checker that checks for overwriting of provided APIs.

To summarize, all three components in LLMFP are effective and could account for diverse problems by providing comprehensive aids to solve planning problems.

**Table 3: Optimal rate (%) comparison when removing some key components of LLMFP on all 9 tasks. LLMs used are GPT-4o.**
| Domain | No Definer | No Formulator | No Self Assess & Modification | LLMFP |
| --- | --- | --- | --- | --- |
| Coffee | 8.6 | 56.4 | 55.3 | 64.7 |
| Workforce | 84.4 | 80.5 | 27.3 | 92.2 |
| Facility | 61.6 | 53.7 | 53.7 | 70.7 |
| Task Allocation | 74.0 | 92.0 | 96.0 | 96.0 |
| Warehouse | 90 | 2.0 | 54.0 | 72.0 |
| Average | 63.7 | 56.9 | 57.2 | 79.1 |
| Blocksworld | N/A | 0.2 | 95.3 | 96.2 |
| Mystery Blocksworld | N/A | 0.0 | 74.4 | 77.7 |
| Movie | N/A | 0.0 | 66.7 | 100.0 |
| Gripper | N/A | 0.0 | 64.0 | 76.0 |
| Average | N/A | 0.1 | 75.1 | 87.5 |

### 4.4 LLMFP with Task-Specific Example

Although from Sec. 4.2 and Sec. 4.3 we prove that LLMFP is capable of achieving strong performance on a wide range of problems with no task-specific example, we test LLMFP by only replacing the two examples in Formulator to one task-specific example on Coffee task to see how much the task-specific example could further improve LLMFP. Queries of Coffee tasks are what-if questions and are categorized into 7 sets. Each set is a type of question. For example, the type of Set 1 is “demand-increase”, that is, every query in Set 1 asks about what if the demand in some cafes increases some amount. The number of queries in the 7 sets are 60, 55, 17, 33, 35, 12, and 54. For each set, we add one task-specific example to Formulator only, test the performance of LLMFP ${}_{\textsc{task-specific}}$ over this set, and compare with LLMFP performance over this set. We use GPT-4o for LLMs. We include the optimal rate performance comparison of LLMFP and LLMFP ${}_{\textsc{task-specific}}$ on 7 sets of queries of Coffee in Table 4. From the result, we observe that on average, LLMFP ${}_{\textsc{task-specific}}$ improves the performance of LLMFP by 24.2% for Coffee. Note that the performance of Set 3 increases the most. We investigate the reason and notice that the type of question for this set is “supply-roastery”, which asks queries like “What led to the decision to use supplier3 for the roasting facility at roastery1?”. To answer this question, it is both plausible to test “using supplier3” or to test “not using supplier3” to see the performance. However, the ground truth answer for this type of question is to “not using supplier 3”. As confusing queries even for humans, they are hard for LLMs to understand. Thus, for these queries, adding task-specific examples significantly improves the performance. To summarize, LLMFP is capable of achieving strong performance with no task-specific example, but easily adding task-specific examples only to Formulator could improve the performance, especially when the task description or query is not clearly presented.

**Table 4: Optimal rate (%) comparison of LLMFP and LLMFP with one task-specific example in Formulator on Coffee task. Sets represent different types of what-if questions. LLMs are GPT-4o.**
| Method | Set 1 | Set 2 | Set 3 | Set 4 | Set 5 | Set 6 | Set 7 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLMFP | 58.3 | 70.9 | 11.8 | 42.4 | 80.0 | 83.3 | 81.5 | 61.2 |
| LLMFP ${}_{\textsc{task-specific}}$ | 78.3 | 72.7 | 70.6 | 84.8 | 91.4 | 100.0 | 100.0 | 85.4 |

## 5 Conclusion

To account for the challenge of the trade-off between flexibility and task complexity for existing LLM planning works, we observe that the core of many planning problems lies in optimization problems and propose a universal approach for LLMs to solve planning problems. We propose LLMFP, a general-purpose LLM-based planning framework that captures key information from planning problems and formally formulates and solves them as optimization problems, with no task-specific examples needed. We test LLMFP on 9 diverse planning tasks with two LLMs to prove LLMFP’s capability of achieving strong performance over fundamentally very different tasks and show the effectiveness of components in our framework.

Limitations One limitation of LLMFP is that it needs clear and detailed task descriptions and queries. It is hard for LLMFP to define the problems’ goals and constraints if the task description is ambiguous or missing some important information. Another limitation is that since LLMFP encodes the planning problems into optimization problems and solves them with optimization solvers, the capability of LLMFP depends on the strength of the solver. For massive databases with numerous feasible plans, it may take a longer time for the solver to search for the optimal plan. Ways to mitigate this is to introduce heuristics to prioritize a portion of the choices or to switch from solving optimization problems to satisfaction problems for planning tasks that do not require optimality.

## Appendix A Appendix

### A.1 Domains

We test on 9 planning problems, including 5 multi-constraint decision making tasks and 4 multi-step tasks (Li et al., ; Valmeekam et al., ; Stein & Koller, ):

- •
Coffee Coffee company sources beans from three suppliers with fixed capacity, roasts them at two facilities into dark or light coffee, and ships the roasted coffee to three retail locations. The company aims to minimize the total shipping and roasting cost while fulfilling the demand at each retail location. There are 266 different queries of 7 types in the dataset.
- •
Workforce Assign workers to shifts; each worker may or may not be available on a particular day. The goal is to minimize the total payments to workers while fulfilling the shift requirements for two weeks. There are 231 different queries of 5 types in the dataset.
- •
Facility A company currently ships its product from 5 plants to 4 warehouses. It is considering closing some plants to reduce costs. The goal is to decide which plant(s) to close to minimize transportation and fixed costs. There are 165 different queries of 4 types in the dataset.
- •
Task Allocation Given tasks and three robots skilled in different tasks, the goal is to assign tasks to robots to minimize finish time. The finish time counts when the last robot stops working. There are 50 different queries describing the number of different tasks. This task and its data and queries are created by us.
- •
Warehouse The robots need to finish tasks by visiting stations that are capable of accomplishing corresponding tasks. The goal is to find the list of stations while minimizing the total distance traveled. There are 50 different queries that include the random-length list of tasks to finish. This task and its data and queries are created by us.
- •
Blocksworld The robot has four actions: pickup, putdown, stack, and unstack. The goal is to stack the blocks in the scene from their initial setup to a specific order with minimum steps. There are 602 different queries that describe blocks’ initial conditions and goal states.
- •
Mystery Blocksworld An obfuscated version of Blocksworld. The action and predicate names are replaced with names that logically make no sense. There are 602 different queries that describe objects’ initial conditions and goal states.
- •
Movie The goal is to get the required snacks, watch the movie, and recover the movie and counter to the original state with minimum steps. There are 21 different queries that describe objects’ initial conditions and goal states.
- •
Gripper There are robots and balls in different rooms. Each robot, with two grippers, can pick, drop, and move balls between rooms. The goal is to place balls in specific rooms with minimum steps. There are 25 different queries describing objects’ initial conditions and goal states.

The queries for Coffee, Workforce, and Facility are what-if questions that change or add constraints to the existing scenarios. The queries of the rest tasks are different task initial and goal conditions. Task inputs including example queries are given in Appendix A.5.

### A.2 LLMFP Performance over Iterations

Fig. 3 shows the performance of LLMFP over 5 iterations. The key observation is: number of iterations of Self Assess & Midification stage enables LLMFP to further improve the optimal rates, although we can observe that LLMFP does not need extensive iterations to achieve an overall satisfying performance.

Figure: Figure 3: Optimal rates of models across LLMFP Iterations
Refer to caption: x3.png

### A.3 LLMFP Time and Cost Statistics and Analysis

Table 5, 6, and 7 shows the time and cost statistics of LLMFP for GPT-4o on 9 tasks.

We could observe that for both LLM querying time and solver running time, all stages of LLMFP requires reasonable runtime. The longest runtime is prompting Formulator for Movie, which takes 31.9 seconds because it contains 9 actions thus requiring longer representation formulation.

From cost statistics we could observe that the average cost per query for all 9 tasks is around 0.1 dollar, indicating LLMFP is not costly.

**Table 5: Average time (s) spent per query for all components of LLMFP ${}_{\textsc{GPT-4o}}$ on all 9 tasks.**
| Domain | Definer | Formulator | Solver | Formatter | Code Gen. | Self Assess & Mod. |
| --- | --- | --- | --- | --- | --- | --- |
| Coffee | 5.6 | 10.8 | 17.1 | 0.1 | 14.2 | 11.3 |
| Workforce | 3.4 | 5.1 | 8.3 | 11.0 | 3.1 | 7.8 |
| Facility | 3.8 | 7.5 | 6.4 | 0.7 | 4.3 | 4.0 |
| Task Allocation | 8.6 | 23.8 | 5.2 | 0.2 | 6.5 | 5.9 |
| Warehouse | 3.9 | 3.1 | 6.2 | 0.2 | 4.1 | 3.3 |
| Blocksworld | N/A | 21.0 | 14.6 | 0.6 | 1.9 | 3.4 |
| Mys. Blocksworld | N/A | 24.3 | 14.6 | 0.6 | 2.3 | 4.1 |
| Movie | N/A | 31.9 | 15.6 | 0.5 | 11.8 | 6.5 |
| Gripper | N/A | 18.3 | 16.0 | 6.9 | 11.2 | 7.0 |

**Table 6: Average cost ($) per query of LLMFP ${}_{\textsc{GPT-4o}}$ on 5 multi-constraint problems.**
|  | Coffee | Workforce | Facility | Task Allocation | Warehouse |
| --- | --- | --- | --- | --- | --- |
| cost/query | 0.139 | 0.140 | 0.083 | 0.081 | 0.085 |

**Table 7: Average cost ($) per query of LLMFP ${}_{\textsc{GPT-4o}}$ on 4 multi-step problems.**
|  | Blocksworld | Mystery Blocksworld | Movie | Gripper |
| --- | --- | --- | --- | --- |
| cost/query | 0.122 | 0.105 | 0.131 | 0.128 |

### A.4 LLMFP Failure Case Analysis

Here we analyze the major failure cases for all 9 tasks.

#### A.4.1 Coffee

There are two major failure cases for Coffee tasks:

First, some queries are not clearly presented, indicating ambiguous information. Queries of Coffee tasks are what-if questions and are categorized into 7 sets. Each set is a type of question. We notice that the type “supply-roastery” asks queries like “What led to the decision to use supplier3 for the roasting facility at roastery1?”. To answer this question, it is both plausible to test “using supplier3” or to test “not using supplier3” to see the performance. However, the ground truth answer for this type of questions is to “not using supplier 3”. As confusing queries even for human, they are hard for LLMs to understand. Thus, for these queries, LLMFP sometimes generate codes with opposite meanings as what is expected.

Second, sometimes LLMFP Definer fails to consider all implicit constraints. The most easily neglectable implicit constraints are 1) the beans roasted in each roastery do not exceed the beans it receives, and 2) the beans ship from each roastery do not exceed the coffee it roasts. When any of the two constraints are missing, to minimize the cost, the model will automatically set the shipped beans or roasted coffee to be 0, assuming the company delivers coffee without sourcing beans or roasting coffee.

#### A.4.2 Workforce

There are two major failure cases for Coffee tasks:

First, sometimes LLMFP fails to understand the queries. Some of the queries asks questions like ‘Can Gu transition from Sun14 to Sun7 for work purposes?’. The meaning is to force Gu to work on Sun7 and take rest on Sun14. However, sometimes LLMFP builds variables to test both taking and not taking this transition, and returns solutions with less costs.

Second, sometimes when the solution space is large, it is hard to find the optimal solution within maximum runtime set for solver. We set the maximum solver runtime to be 15 minutes, which is exceeded when solving some hard queries.

#### A.4.3 Facility

Similarly as the first failure case of Coffee, some queries are not clearly presented. The queries are like “What justifies the opening of plant 0?”, which is confusing even for humans. Both opening plant 0 and closing plant 0 to report the costs make sense to answer this query. However, the ground truth meaning of this query is to close plant 0.

#### A.4.4 Task Allocation

LLMFP only fails one query in Task Allocation. The reason is the Formulator generates wrong values for robot finish time.

#### A.4.5 Warehouse

The major failure case for Warehouse is Code Generator overwrites the provided API get_distance and provide 1 as the output during Code Generation. Thus, the distance between each station is mistakenly set to be 1.

#### A.4.6 Blocksworld

One major failure case for Blocksworld is Code Generator fails to initialize the states of predicates correctly and thoroughly. Since the query will only meantion the predicates that are true, for example, block 1 is on block 2, but when initializing, LLMFP needs to initialize both mentioned states but also unmentioned states that are false. For example, block 2 is not on block 1. However, Code Generator sometimes fails to consider all unmentioned states.

#### A.4.7 Mystery Blocksworld

Similarly as Blocksworld, Mystery Blocksworld has same failure case. For Mystery Blocksworld, since the predicate and action names are not meaningful, more this kind of errors are made by GPT-4o. However, Claude seems to have better reasoning capability to support it from making more these errors.

#### A.4.8 Movie

There is no failure case for Movie.

#### A.4.9 Gripper

The major failure case for Gripper is when the solver fails to find the solution because there are some code generation errors, the Self Assess & Modification sometimes would think it is because the timestep is not enough, thus adding another loop within the original loop. However, this would result in the program to execute forever.

### A.5 Inputs on 9 tasks

We include the inputs, which includes task description, background information or API, and example queries, for all 9 tasks in Fig. 4 - Fig. 12:

Figure: Figure 4: Task description, background information or API, and example queries for Coffee
Refer to caption: x4.png

Figure: Figure 5: Task description, background information or API, and example queries for Workforce
Refer to caption: x5.png

Figure: Figure 6: Task description, background information or API, and example queries for Facility
Refer to caption: x6.png

Figure: Figure 7: Task description, background information or API, and example queries for Task Allocation
Refer to caption: x7.png

Figure: Figure 8: Task description, background information or API, and example queries for Warehouse
Refer to caption: x8.png

Figure: Figure 9: Task description, background information or API, and example queries for Blocksworld
Refer to caption: x9.png

Figure: Figure 10: Task description, background information or API, and example queries for Mystery Blocksworld
Refer to caption: x10.png

Figure: Figure 11: Task description, background information or API, and example queries for Movie
Refer to caption: x11.png

Figure: Figure 12: Task description, background information or API, and example queries for Gripper
Refer to caption: x12.png

### A.6 Example Outputs on Coffee tasks

In this section, we include the outputs of all models for the query “What is the potential impact of a 29% increase in demand at cafe cafe2?”. The optimal solution has total cost to be 2612.

#### A.6.1 Baselines

We include the outputs of four baselines for Coffee task in Fig. 13 - Fig. 15, and analyze the failure reason in the figure descriptions:

Figure: Figure 13: Output of Direct${}_{\textsc{GPT-4o}}$ and Direct${}_{\textsc{o1-preview}}$ for Coffee. LLMs fail to directly solve complex multi-constraint planning tasks that involve intensive calculations.
Refer to caption: x13.png

Figure: Figure 14: Output of CoT${}_{\textsc{GPT-4o}}$ for Coffee. Although the reasoning of constraints makes sense, LLMs still fail to locate optimal values to fulfill the task as it involves intensive calculations.
Refer to caption: x14.png

Figure: Figure 15: Output of Code${}_{\textsc{GPT-4o}}$ for Coffee. LLM also tries to use optimization solver to solve for the problem, however, without proper analysis of constraints, it fails to notice that the updated demand for cafes needs to be integers. Instead, to account for the 29% demand increase, it directly update with “30 * 1.29” and “20 * 1.29”
Refer to caption: x15.png

#### A.6.2 LLMFP

We include the outputs of all stages of LLMFP for Coffee task in Fig. 16 - Fig. 20:

Figure: Figure 16: Output of LLMFP ${}_{\textsc{GPT-4o}}$ Definer for Coffee. LLMFP successfully defines the goal, decision variables, and constraints.
Refer to caption: x16.png

Figure: Figure 17: Output of LLMFP ${}_{\textsc{GPT-4o}}$ Formulator for Coffee. LLMFP successfully formulates the decision variables with variable_1 to variable_5, account for the goal with variable_6, and consider additional constraints posed with query with variable_7 and variable_8.
Refer to caption: x17.png

Figure: Figure 18: Output of LLMFP ${}_{\textsc{GPT-4o}}$ Code Generator for Coffee. LLMFP successfully generates Python codes based on the representation formulation.
Refer to caption: x18.png

Figure: Figure 19: Output of LLMFP ${}_{\textsc{GPT-4o}}$ Formatter for Coffee. LLMFP format the execution result in fixed format, and analyze the correctness and reasonableness of the execution result.
Refer to caption: x19.png

Figure: Figure 20: Output of LLMFP ${}_{\textsc{GPT-4o}}$ Self Assess & Modification for Coffee. LLMFP checks for all 3 steps and provide ratings.
Refer to caption: x20.png

### A.7 Prompts

#### A.7.1 Baseline Prompt

Direct Prompt

CoT Prompt

Code Prompt

#### A.7.2 LLMFP Prompt

We use general templates for all tasks. The full prompts for all tasks will be released soon. Here we show the templates we have for GPT-4o. Since Claude naturally considers more constraints and is more strict in assessing, we edit the prompts a little to account for the different traits of Claude, and prompts are also included in the codes.

We include the prompt template we use for GPT-4o as below:
Definer Prompt

Formulator Prompt for single-step multi-constraint problems

Formulator Prompt for multi-step problems

Code Generator Prompt

Formatter Prompt

Self Assess & Modification Prompt

#### A.7.3 Prompts and output for LLMFP with MILP solver for Coffee Example

Definer and Formatter prompt remain exactly the same as for SMT solver.

We include the comparison of prompts for Formulator, Code Generator, and Self Assess & Modification in Fig. 21 to 23 and labelled all the differences with red. We then include the output of Formulator and Code Generator in Fig. 24 and Fig. 25.
The key takeaway is it is very easy to switch from one solver to another with LLMFP, as the inner logic is same: building an optimization problem.

Figure: Figure 21: The Formulator prompt difference when switching from using Z3 SMT solver to Gurobi MILP solver.
Refer to caption: x21.png

Figure: Figure 22: The Code Genetator prompt difference when switching from using Z3 SMT solver to Gurobi MILP solver.
Refer to caption: x22.png

Figure: Figure 23: The Self Assess & Modification prompt difference when switching from using Z3 SMT solver to Gurobi MILP solver.
Refer to caption: x23.png

Figure: Figure 24: Output of Formulator for Coffee after switching the solver to MILP, using GPT-4o.
Refer to caption: x24.png

Figure: Figure 25: Output of Code Generator for Coffee after switching the solver to MILP, using GPT-4o.
Refer to caption: x25.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.  As the authors noted, LLMs have been used to translate natural language to planning problems. Similarly, the mapping from planning to SMT is well known in the planning literature. So, is the novelty is limited to combining the two ideas together??
2. In page 3, just above the first paragraph, you seem to say that encoding to PDDL requires more human effort than encoding to SMT. Can you elaborate why?
3. How do you encode the length of the plan? When compiling planning to SAT or SMT, this is an issue because the solver (SAT/SMT) requires to set an upper bound while in PDDL it does not have too.
soundness: 3
strengths: Strength
- The paper is in general clear (even if it is sometimes hand-wavey)
- The problem is interesting and the related work seems to cover all bases
- The results are impressive, and much better than the baselines. 
- The proposed workflow makes sense and works well.
summary: This paper addresses the problem of solving planning problems that are given in natural language. The proposed algorithm they propose – LLMFP -  is a workflow of multiple LLMs, including an LLM to extract variables and constraints from the text, an LLM to formulate the extracted variables and constraints as an SMT problem in a specific format,  an LLM to convert this format to code that can be run by an SMT solver, and an LLM to verify and correct mistakes by the other LLMs. This LLM workflow is evaluated against the other LLM-based methods to solve planning problems, including one that is similar to the LLMFP but creates PDDL instead of an SMT problem. The authors also examine how results can be better by adding some task-specific expertise. The results over a set of benchmark problems show that LLMFP is, in general, much better than the baselines.
weaknesses: - I’m not sure if the novelty of the proposed work over the PDDL-based approach is sufficiently novel for a top conference. 
- The appendix is huge (~40 pages!). This seems to me not reasonable, as the main paper should be self contained. 
- The presentation is too much hand-wavy. It would be great to try to capture more of it in a more formal manner

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 2
questions: 1. What is the definition of a planning problem in this paper?
2. Why are the baselines only LLMs when the proposed approach is a framework/architecture? LLM-PFC [1] approaches planning problems similarly and there are other baselines to consider like Plan-SOFAI [2].
3. LLMs when used with API's are found to hallucinate new API functions or overuse a specific API call. Is such behavior observed here?
4. When it is a planning problem, why not directly use a symbolic planner and why is this architecture beneficial?
soundness: 2
strengths: LLMFP's ability to handle a wide variety of planning problems without task-specific examples is a significant strength.
summary: LLMFP is proposed which leverages LLMs to tackle complex planning problems by formulating them as optimization tasks.
weaknesses: 1. The baselines for comparison do not seem to be a fair comparison to LLMFP. See questions.
2. The related work does not cover relevant set of papers that should have been used a baseline to compare this work. Mentioning a few of them below - 
[1] Webb, T., Mondal, S. S., Wang, C., Krabach, B., & Momennejad, I. (2023). A Prefrontal Cortex-inspired Architecture for Planning in Large Language Models. arXiv preprint arXiv:2310.00194.
[2] Fabiano, F., Pallagani, V., Ganapini, M. B., Horesh, L., Loreggia, A., Murugesan, K., ... & Srivastava, B. (2023, December). Plan-SOFAI: A Neuro-Symbolic Planning Architecture. In Neuro-Symbolic Learning and Reasoning in the era of Large Language Models.
[3] Katz, M., Kokel, H., Srinivas, K., & Sohrabi, S. (2024). Thought of Search: Planning with Language Models Through The Lens of Efficiency. In The First Workshop on System-2 Reasoning at Scale, NeurIPS'24.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What are the most common failure modes of the baselines?

2. Are the baselines prompted vanilla models plus the components described in lines 410-416, or do they also include other components, e.g. formatter?

3. What are the success rates of the tested methods? Do they all achieve 100% and the question is only whether the solution is optimal, or do some methods fail to solve some instances at all?

4. What's the wall time of LLMFP compared to the baselines?

5. Are the methods explicitly instructed to provide optimal solutions?
soundness: 3
strengths: I like the general idea and the presented approach. One could argue that it is simply a combination of prompt engineering and the incorporation of external tools. However, showing an effective way of doing this can be a significant contribution.

The baselines and ablations are well-chosen for evaluating the performance of LLMFP.

The paper is written very clearly, making it easy to read. The figures are well-chosen (particularly Figure 1), they are helpful in understanding the pipeline. I like the section structure and the focus on key takeaways when discussing experimental results. Most of my questions that arose while reading the text were addressed in later sections.
summary: The paper presents a framework that pairs LLMs with optimization tools to solve planning tasks without using task-specific knowledge. The authors define consecutive stages of reasoning that, generally speaking, consist of understanding, coding, and refining. For each stage, they discuss the prompting, formatting, and other relevant decisions. Through experimental validation, they show that LLMFP outperforms baselines in 9 domains.
weaknesses: The goal stated in the introduction is "Can we build a universal LLM-based planning system that can solve complex planning problems without task-specific efforts?". However, my main concern is whether the tasks used for experiments are indeed complex planning problems. Specifically, the 5 multi-constraint problems resemble simply optimization problems rather than planning problems. Hence it's quite clear that adding an external optimizer to LLM would be much better than just using LLM. On the other hand, the multi-step problems seem to be rather simple and the main difficulty is to understand what we have to do rather than finding a good solution. Hence, I suggest adding at least one multi-step domain with high underlying complexity (e.g. Sokoban). If I missed something and some of your environments are actually NP-hard (or hard in any other reasonable sense), it should be remarked in the paper.

Since the method you propose is clearly subject to a tradeoff between performance and computation time, there should be a discussion of that. What's the wall time of LLMFP compared to the baselines? What's the cost of using SMT compared to querying LLM?

The description of baselines should be extended a bit. Are they prompted vanilla models plus the components described in lines 410-416, or do they also include other components, e.g. formatter? Also, the Code variant uses pure Python, but for a completely fair comparison you should also add variant which is forced to use SMT like LLMFP does. After reading the prompts used, it's also not clear to me whether they are explicitly instructed to provide optimal solutions, which is captured by the metrics. Also, I suggest discussing the failure modes of the baselines (in the Appendix).

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. How LLMFP handles generalization across different planning tasks. Please correct me, but it seems that we need a very Elaborate prompt with a high level of detail for each task. 
2. In section 3.4 (code generator), readers can Benefit from prior work such as "CAPE: Corrective Actions from Precondition Errors using Large Language Models" and "CoT-TL: Temporal Knowledge Representation of Natural Language Planning Task for Autonomous Agents using Chain-Of-Thought." Or is LLMFP doing differently compared to the above works; if yes, explain or not, and make sure to provide proper background.
soundness: 3
strengths: 1. LLMFP introduces a new perspective on using LLMs for formal optimization-based planning, a method that significantly expands the generalizability of planning tasks.
2. Experimental results are solid, with clear evidence of performance gains across diverse tasks and models. The ablation studies reinforce the utility of the framework components, which I really liked.
3. The ability to solve multi-step, multi-constraint problems without task-specific examples or extensive prior efforts is a major step forward in the area of LLM-based planning.
summary: This paper provides the LLM prompting-based framework for planning tasks. The main contribution lies in its use of prompt and pipeline templates, which can be used across various planning tasks. Here, planning problems from various domains were considered, and planning is treated as an optimization problem. In summary, LLMs are used as an optimizer. They used the formal planner to achieve the goal of planning as already shown in the previous works that LLM still lacks the coherent reasoning needed for planning. The main contribution is an end framework that deploys a zero-shot learning approach for both single and multi-stage planning tasks. Additionally, the author claims that their framework can handle self-critique to assess the problem in planning code to change and achieve the goal. Effectiveness of framework components is supported via the ablations.
weaknesses: 1. Complexity in FORMULATOR: Some parts, particularly the JSON representation and the code generation steps, could be simplified. While important, the handling of different variable types and constraints might be a bit dense for readers unfamiliar with optimization theory.
2. Regrading Multi-Step Planning Problem: The predicate, object, and update structure are not clear in multi-step planning. Also image shown for this is not utilized in conveying the idea. Overall, Figure. 2 examples are not clear and make things confusing. 
3. The author claims that their framework is a "general approach, which does not require task-specific examples or task-specific efforts"; however, in the paper, this statement is not supported in terms of explanations and prompt structure.
4. Some theoretical insights regarding performance would make this work more strong, right now its presented more like a experimental results.

Meta-review: The paper presents LLM-Based Formalized Programming (LLMFP), a framework for incorporating LLM's to solve natural language planning tasks. The framework uses an LLM iteratively with external planning tools to create a viable solution. The LLM is used to extract variables and constraints from text, construct and parse an instance of a SMT formula from the text, and catch errors in the process via another LLM. The experiments demonstrate improved performance compared to a direct application of an off-the-shelf LLM.

Decision: Accept (Poster)