Plansformer: Generating Multi-Domain Symbolic Plans using Transformers
Authors: Vishal Pallagani
Bharath Chandra Muppasani
Keerthiram Murugesan
Francesca Rossi
Lior Horesh
Biplav Srivastava
Francesco Fabiano
Andrea Loreggia
Venue: Submitted to ICLR 2023 (2023)
Keywords: automated planning
language models
transfer learning

Abstract: Large Language Models (LLMs) have been the subject of active research, significantly advancing the field of Natural Language Processing (NLP). From BERT to BLOOM, LLMs have surpassed state-of-the-art results in various natural language tasks such as question answering, summarization, and text generation. Many ongoing efforts are focused on understanding LLMs' capabilities, including their knowledge of the world, syntax, and semantics. However, extending the textual prowess of LLMs to symbolic reasoning has been slow and predominantly focused on tackling problems related to the mathematical field. In this paper, we explore the use of LLMs for automated planning - a branch of AI concerned with the realization of action sequences (plans) to achieve a goal, typically for execution by intelligent agents, autonomous robots, and unmanned vehicles. We introduce Plansformer; an LLM fine-tuned on planning problems and capable of generating plans with favorable behavior in terms of correctness and length with minimal knowledge-engineering efforts. We also demonstrate the adaptability of Plansformer in solving different planning domains with varying complexities, owing to the transfer learning abilities of LLMs. For one configuration of Plansformer, we achieve ~97\% valid plans, out of which ~95\% are optimal for Towers of Hanoi - a puzzle-solving domain.

Under review as a conference paper at ICLR 2023
PLANSFORMER: GENERATING SYMBOLIC PLANS USING TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
Large Language Models (LLMs) have been the subject of active research, significantly ad- vancing the field of Natural Language Processing (NLP). From BERT to BLOOM. LLMs have surpassed state-of the-at art results in various natural language tasks such as question swering, summarization, and text generation. Many ongoing efforts focus on understanding LLMs' capabilittes, including their knowledge of the world, syntax, and semantics. How- ever, extending the textual prowess of LLMS to symbolic reasoning has heen slow and predominantly focused on tackling problems related to the mathematical field. In this pa- per, we explore the use of L.LMs for automated planning branch of AT concerned with the realization of action sequences (plans) to achieve goal, typically executed by intel- ligent agents, autonomous robots, and unmanned vehicles. We introduce Plansformed an LLM fine-tuned on planning problems and capable of generating plans with favorable behavior in terms of correctness and length with reduced knowledg--cniineering efforts. We also demonstrate the adaptability of Plansformer solving different planning domains with varying complexities, owing to the transfer learning abilitics of LLMs. For one con- figuration of Plansformer, we achieve 97% valid plans, out of which 95% are optimal for Towers of Hanoi puzzle-solving domain.
INTRODUCTION
Large Language Models (LLMs), based transformer-based (neural) architecture (Vaswani al. 2017 Devlin et al. 2018 Brown et al., 2020 Chowdhery et al. 2022), have significantly advanced the field of Natural Language Processing (NLP). Their employment has grown dramatically in recent
times Li 2022), as researchers develop newer and bigger LLMs. From BERT to the most recent BLOOM, language models have surpassed state-of-the-ar results in various natural language tasks. For example, PaLM (Chowdhery et al. 2022), achieved breakthrough performance on a plethora of natural language tasks such as inference, question answering, and commonsense reasoning, and outperformed an average human performance on the BIG-bench benchmark.
Despite the textual prowess of LLMs, their significance has been limited in the domains that in- volve symbols. For example, domains with symbols such mathematics (Hendrycks et al. (02 Cobbe al. (02T), and coding problems (Hendrycks et al. 2021a Chen et 2021) deliberates the failures of LLMs when it comes to handling symbols. In Automated Planning. (Valmeekam et al.) 2022) suggests that even state-of-the-arr LLMs cannot reason with symbolic data and offer a new suite of benchmarks to test their reasoning capabilities. Recently, there has been a lot of interest in LLMs for code generation; for example, CodeT5 (Wang et al.) (021), CodeBERT (Feng et al.] 2020), Codex (Chen al. 202T), etc. In this paper, propose employ LLMS that trained to generate code and repurpose them to generate valid plans. To advance the research in L.-M-based automated planning,
create training and test dataset for several planning domains. We use CodeT5 (base), a transformer-base CodeXGlue, as the pre-trained LLM. We select CodeT5 due its ability to generate goal-directed, sequential instruction and semantically meaningful program codes with syntactic and structural con- straints. Then, we present, Plansformer, an LLM trained to generate symbolic plans of high quality in terms of correctness and length. Our experimental results indicate that the syntactic/symbolic knowledge learned from different programming languages in the CodeT5 model can be beneficial for the PDDL-based automated planning task. For example, in one configuration of Plansformer tested on a puzzle-solving domain, Towers of Hanoi hanoi, our model was able to generate 97%
code generation model that achieves state-of-the-at results in
'The dataset, code, and all fine-tuned model checkpoints will be released for public usage.

---

Under review as a conference paper at ICLR 2023
valid plans, out of which 95% are shortest length plans. The results reveal a promising direction to harness LLMs for symbolic tasks such as planning.
In the remainder of the paper. we present preliminaries on automated planning and language models and then propose an LLM repurposed planner called Plansformer. Next, we present the experimen- tal results comparing our approach with state-of-the-ar planners and other large language models. Furthermore, we demonstrate the ability of Plansformer to adapt to other domains and discuss the relevance instruction generation. We conclude with a discussion of the results and presentation of ongoing work.
2 BACKGROUND
2.1 AUTOMATED PLANNING
Given the initial and goal states, alongside set of legal actions, the objective of planning agent is to devise sequence of actions that advance the agent from the initial to the goal state. This paper adopts the Planning Domain Description Language (PDDL) (MeDermott et al. 1998 Fox & Long 2003) notations. In PDDL, a planning environment is described in terms of objects in the world, predicates that describe relations between these objects, and actions that modify the world by manipulating these relations. The output plan consists of series of time steps, each of which can have one or more instantiated actions with concurrency semantics (Ghallab et al. 2004). A planner devises plans by searching in the space of states, where a state is configuration of physical objects or partial plans. There is a single agent in the most basic formulation, called classical planning. The actions have unit cost, take constant time to execute, have deterministic effects, with the fully observable world, domain-specific conditions/constraints and all goals have to be achieved (Ghallab) et a 2004). In more sophisticated planning settings, many of these conditions relaxed. There may be multiple agents, and the cost and duration of actions can be non-uniform. At the same time, its effects can be non-deterministic, the world can be partially observable, and the agent may maximize as many goals as it can achieve in a given time and resource budget.
2.2 LARGE LANGUAGE MODELS AND SYMBOLIC TASKS
Large Language Models (LLM) such BERT (Woff et al. 2020), RoBERTa (Liu et al.) 2019), and GPT3 (Brown et al. 2020) are pre-trained on extensive unstructured knowledge from public data such as Wikipedia, Bookcorpus, and Commoncrawl, have shown impressive results in several NLP tasks. It demonstrated the ability to generalize to multiple tasks from question answering and machine translation to story generation and instruction following (Wang et al.. 2018:) 2019). LLMs
have shown the ability to generate output in natural language (Wolf et al. 2020 Raffel et al. 2020)
adapt to novel tasks zero or few-shot approach (Brown et al. 2020 Radford et al. (2019] and
decode with constraints on output space (Hokamp & Liu 2017: Welleck et al.] 2019 Kumar et al.)
2021]. Recent progress in LLMs has demonstrated the generation of structured output that requires precise syntactic/symbolic knowledge with structural constraints such as knowledge graphs (Petroni et al., 2019), protein structure (Unsal et al., 2022 Ferruz & Höcker 2022), and programming lan- guages (Ahmad et al. (02T). As the LLMs collect the related knowledge necessary to solve an NLP task, (Petroni et al. 2019) have shown that the LLMs are potential representations of the significant knowledge bases. In protein data (Unsal et al. 2022: Ferruz & Höcker 2022). LLMs generate the functional properties of the proteins by enforcing the structural constraints specific to protein science and determining the complex functional relationships in the protein binding. Code generation has recently become very popular in the LLM research community. Several models such as CodeBERT (Feng et al., 2020). Codex (Chen et al., 2021), ), and CodeT5 (Wang et al. 2021) have shown signifi- cant improvement in transfer from pre-trained models for natural language to structured codes. One of the key contributors to the success of these LLMs in code generation is fine-tuning the models on task-specific data. For instance, CodeXGlue (Lu et al.] 2021), benchmark dataset for code understanding and generation with sample codes from several programming languages, is used to fine-tune CodeBERT, CodeTT5, and others. In this paper, we harness C5 for further fine-tuning to the classical automated planning domain due to its ability to generate goal-directeo sequential instruction and semantically meaningful program codes with syntactic and structural constraints.
2

---

Under review as a conference paper at ICLR 2023
FastDowmarrd Planner
Plans
Domain Model
Training Data
Test
Medal (CodeTS)
Iinesunddd
Tagk/Problem
Input:
Plansformer
Instance
Generated
pian valicoioon
VAL (Plen Valdatiion Tool)
BLEU
Planner Testing
Model Testing
Figure Plansformer Model Architecture showing modeling and evaluation phases. Modeling phase involves finetuning CodeTT with data from planning domain. Evaluation phase shows both the planner and model testing.
The closest prior addressing the ability of LLMs to generate symbolic plans are the studies of
(Hernandez et al., 2021]; Valmeekam et al. 2022 Huang et al. (202). (Hernandez et al. (021) looks
different ways generate plans using GPT-3 versions, prominent generative LLM. (Valmeekam et al. 2022) discusses different scenarios for generating plans, like finding satisficing plan or
adapting a previous one, and discusses encodings generate plans and verify using a plan validator.
(Huang et al., 2022) generates step-by-step instructions for user-defined task using LLM prompt- ing. All these studies require the nuuman-guided, mapping of a natural language-bast sequence of instructions generated by the LLM to the admissible action in the planning domain an additional step.
3 PLANSFORMER FOR SYMBOLIC PLANS
Figure provides an illustrative overview of how we generate and test our planner, called Plans- former. The first phase, modeling, shows how fine-tune the CodeT5 to address planning syntax and semantics. The second phase, evaluation, deals with assessing the competency of Plansformer as model and planner. The key idea here is utilize an LLM (CodeT5) pretrained on code generation and further train it on planning problem instances with corresponding valid plans. We evaluate its competence generating valid plans (or almost valid plans for unseen planning prob- lem instances) using two types of testing: 1) Model testing measures if Plansformer could generate meaningful responses (as in the test dataset), 2) Planner testing measures if the generated plans are valid/optimal (independently of whether they were the same plans in the test dataset).
3.1
MODELING PHASE
In the modeling phase, we first create planning based dataset for finetuning the CodeT5 to generate plans. The modeling phase of Figure depicts the different modules employed.
3.1.1
PLANNING DATASET
We generate PDDL-based datase benchmark finetune pretrained CodeT5 and facilitate further research at the intersection of LLMS and automated planning. We use the domain model
Thhe link is omitted during the review process.

---

Under review as a conference paper at ICLR 2023
booksword
<GOAL> on b1 82, 58 b3, ontaple b3, on bi, dear be <NITs handempty. ontable deer on bit ortabie on bet clear bi
unskack b4 b2: pul-down b4 pilkop b1, slack to, pido- up 04, stack
<ACTION~ pok-u
<FRES> cler x, ontablle x. handerey
not intatte clear a, hardemptyy holding
<ACTION pul-cown
<PRES haiding
.EFFECT> not holding x, dlear handerphy, ortable
<ACTIONN> stack
<PRE>
y
<EFFECT> holding dearry
on
<ACTION-
<PRE> handemoty
<EFFECTs
y, handamply,
Figure 2: Snapshot of one instance of the plan dataset for blocksworld domain.
(in PDDL) to generate corresponding valid problem files with varying complexities automatically. In this paper, we focus on four different classical planning domains, i.e., Blocksworld, Towers of Hanoi, Grippers, Driverlog.
Blocksworld, or bw, is a well-studied domain GGupta & Nau 1991) with blocks placed on a table or arranged in vertical stacks. Here, one can alter the arrangement of the blocks with the available actions such as pick-up, put-down, stack, and unstack. We generate the problems with 2 to block configurations.
Towers of Hanoi, or hn, consists of pegs and multiple disks of varying diameters. Initially, all disks are placed in the first peg, and the end goal is move the disks to the last peg. The only limitation to consider when moving the disks is that only a smaller disk can be placed on top of bigger disk. Although the domain has only one action, the problem-solving Cull 1986). Here, we generate the problems with configurations of 2 to 5 disks.
recursive (Gerety &
Grippers, or gr domain involves moving balls across rooms using robotic grippers. has problems generated with configurations of 2 to balls, 3 to 5 robots, and 2 to 4 rooms.
Driverlog or dl domain involves moving packages on trucks between locations driven by drivers. has problems generated with configurations of to 3 drivers, 1 to 3 trucks, 2 to 4 packages, and 3 to locations.
Each planning domain explained above includes multiple problem instances. We generate the corre- sponding plans for each problem instance using FastDownward planner (Helmert 2006). FastDown- ward is a classical planning system based on heuristic search and offers different search algorithms such as causal graph heuristics and A search. FastDownward can generate optimal plans with A' LM-Cut heuristic (Helmert & Domshlak 201T). Hence, FastDownward can be regarded as potent planner for generating a dataset of optimal plans.
We show the snapshot of the generated dataset in Figure 2 with more examples in Section 1 of sup- plementary material. Unlike the traditional planner which requires two different files domain and problem (as pddl files, See Figure 2 in supplementary material), Plansformer reduces the knowledge- engineering efforts with a simplified input format that includes the problem instance coupled with a corresponding valid plan. The problem instance captures all the essential information in the domain and problem instance, such as the goal, the initial state, and the possible actions that can be taken in that domain. The generated dataset for each domain consists of 18,000 plans with different problem configurations. For training, we use 5-fold cross-vale validation with an 80%-20% split of the generated dataset for each domain. The average plan length (number of actions in the generated plan) for blocksworld is 9, gripper is 9, driverlog is 10, and hanoi is 12.
3.1.2 TOKENIZER
We use a Byte-level BPE tokenizer, following the standard practice in LLMs, with a vocabulary size of 32,005. We add PDDL-specific tokens, namely, [GOAL], (INIT), [ACTION], [PRE], [EFFECT] to represent the goal state, initial state, possible actions with their associated precondi- tions and effects these actions cause in the environment respectively. We do not re-train specific tokenizer for this task from scratch following the previous work (Chen et al. 202T). where GPT-3's tokenizer reused generate code.
4

---

Under review as a conference paper at ICLR 2023
3.1.3 FINE-TUNING CODETS
While there are many LLMs to select candidate for this work, we shortlist the models pre-trained on code generation to exploit the syntactic information in the programming languages implicitly captured in their weights. Although Codex (Chen et al. (021), built using GPT-3, has reported the best performance in solving code-related tasks, its lack of public access led us to choose equally competitive LLM: CodeT5 (Wang et al.) 2021). CodeTS is a masked language model consisting of encoder-decoder stack inspired by the transformer architecture (Vaswani et al. 2017). It is capable of performing a wide range of tasks including code generation and understanding tasks. The generation tasks include code summarization, code generation, translation, and refinement. The understanding tasks include code defect detection and clone detection. Code is pretrained with example codes from eight programming languages Python, Java, JavaSeript, PHP, Ruby, Go, C, and C#. Its pre- pre-training tasks include identifier awareness and bimodal generation, which optimizes code-to-code understanding. The CodeT5 model possesses several properties amenable to the plan- ning domain, such as its ability to generate goal-directed, sequential instruction and semantically meaningful program codes with syntactic and structural constraints. With this pre-trained knowl- edge already encoded within CodeTT5, we finetune it with 14400 samples (80% of the generated dataset) for each independent domain from the planning dataset. As a result of this finetuning, the weights of CodeT5 are updated to account for the task of plan generation. We give the planning problem instance as input to CodeT5's encoder and generate the intermediate features for the de- coder of C5deTT5 to output plan.
3.2 EVALUATION PHASE
Plansformer an LLM that ingests new problem instance as input and outputs a plan for that problem instance. Therefore, to evaluate its competency, we must test its quality as model and planner. The evaluation phase described in the lower part of Figure [ showing both testing phases.
3.2.1 PLANNER TESTING
Unlike in natural language, symbolic plans have richer information content, inherently captured in their structure. Thus, we have additional evaluation phase for plan validation to check how well Plansformer can mimic an automated planner. The sequence actions generated by Plansformer must help agent navigate from the initial state to the goal state for given problem instance. We call a generated plan cost-optimal if is the shortest possible among all other plans. Several metrics exist in the automated planning literature to evaluate plan generated by Plansformer. In this paper, we consider validity and optimality. We evaluate the plan generated by Plansformer using plan validation tool, called VAL (Howey al. 2004), to check for its optimality and validity. VAL is an automatic validation tool for PDDL. VAL takes as input the task posed to Plansformer and the corresponding generated plan. It applies PDDL-based relaxation conditions to check for validity and optimality.
|
3.2.2 MODEL TESTING.
typical evaluate natural language tasks such as summarization or generation using metrics such as BLEU and ROUGE. Both BLEU and ROUGE are widely used metrics in NLP. In general, BLEU measures precision and helps understand how closely a machine translation (here, plan gener- ated by Plansformer) is compared to a human translation (here, plan generated by automated plan- ner). On the other hand, ROUGE measures recall, i.c., how many of the words referenced in human summaries appeared in the summaries generated by the machine. In particular, we adopt ROUGE-L, which considers sentence-level structure similarity by identifying the longest co-occurring sequence n-grams. Although ROUGE and BLEU have no direct intuition in automated planning, use these metrics to look at the task of plan generation from the perspective of LLMs. The evaluation based on these metrics provides with an insight into the performance of Plansformer language model. In the next section, we evaluate Plansformer as planner to give conclusive evidence on how well Plansformer generates the plans.
We also refer to optimality interchangeably
5

---

Under review as a conference paper at ICLR 2023
Models
ROUGE-
ROUGE-L precision
ROUGE- Amessurre
BLEU
Codex
0.72
0.52
0.60
0.36
GPT-2
0.04
0.14
0.06
0.07
T5-base
0.16
0.70
0.26
0.02
CodeTT--be base
0.41
0.28
0.33
0.02
Plansformer
0.93
0.93
0.93
0.89
Plansformer-bw
0.97
0.99
0.98
0,90
Plansformer-hn
0.99
0.96
0,97
0.95
Plansformer-gi
0.94
0.94
0,94
0.92
Plansformer-dl
0.82
0.83
0.82
0.79
Table Results of model testing (best performance in bold).
4 EXPERIMENTAL RESULTS
In this section, present the quantitative and qualitative results obtained using Plansformer to gen- erate symbolic plans for multiple domains of varying complexities. We select a test-bed of 3, 600 unique and unseen problem instances (20% of the dataset) for each domain for evaluating Plans- former. All the results reported this paper are averaged over 5 randomly selected (80% 20%) train-test splits. We report the results for the Plansformer variants by evaluating the corresponding test-bed. For example, P Plans former bw''s results are reported based on the performance results obtained on bw test-bed. We evaluate Plansformer using both model and planner lesting find its efficiency as a language model and a planner.
4.1 Is PLANSFORMER GOOD MODEL?
Plansformer has an encoder-decoder pair, where the encoder attends to tokens on either side of the masked word, whereas the decoder auto-regressively generates plans. Table compares all the Plansformer models with other LLMs using the model evaluation metrics (ROUGE and BLEU). In this experiment, we consider the best-performing models from (bidirectional) masked language models (e.g.. T5 (Raffel et al.) 2020)) and (unidirectional) causal language models (e.g., GPT-2 (Radford et al. 2019)) for the experiments. We present the actual plan generations from a few of these models in Figure 5 of supplementary material.
We report the performance of the baseline models averaged over the four planning domains. We also show the performance of Plansformer on individual domains (Plansformer-bw, Plansformer- hn, Plansformer-g: gr and Plansformer-dl) We observe that Plansformer performs best on all metrics, followed by Codex, with significant ROUGE-Lucall from Codex compared to other baseline models is due to it's ability to relate the natural language that
We believe that the performance gain
understanding (a skill inherited from GPT-3) with code generation. It is interesting CodeTS performs poorly compared to Codex and Plansformer, demonstrating the advantages of the natural language understanding with code generation task this evalutation metries. We conclude that the models pre-trained with code-related tasks have an advantage over other models in plan generation task due to the similarities of PDDL with other programming languages. Despite with the best model testing metrics, We need to test Plansformer for plan validation to see its effectiveness as planner.
4.2 IS PLANSFORMER A GOOD PLANNER?
In this section, we report the results from the planner testing. We evaluate the generated plans for validity and optimality. We also report the average time taken solve the problem instances. Ta- ble 2 shows the plan validation scores obtained by different models. We consider FastDownward Helmert 2006) for plan validation to generate the ground truth plans. FastDownward planner gen- crates 100% valid and optimal plan for a given input (i.c., a combination domain description and problem instance) when the landmark-cut heuristic is used within a standard A search framework (Helmert & Domshlak 2011).
We that Blockworld bw domain achieved the highest performance gain via Plans former-Dr generated 90.04%, out which 88.44% are optimal. This better perfor- mance is analogous to the fact that bw is the easiest domain among the four domains. Although it

---

Under review as a conference paper at ICLR 2023
Models
Valid Plans (%)
Invalid Plans
Optimal Plans (%)
Avg. Time (sec)
Aacomplete/Wrongy (%)
Failed (%)
FastDownward
100%
100%
10.28s
(Ground Truth)
Codex
0.15%
0%
99.85%
0.15%
1s
GPT-2
0%
100%
0%
0%
0.05s
TS-hase
0.25%
82.7%
17.3%
0.25%
0l.47s
CodeT3-base
0.6%
99.4%
0%
0.6%
0.68s
Plansformer
83.64% %
16.18%
0.19%
73.27 %
0.06s
Plansformer-be
90.04%
9.94%
0.02%
88.44%
0.05s
Plansformer-hn
84.97%
14.72%
0.31%
82.58%
0.05s
Plansformer-g
82.97%
16.61% %
0.42%
69.47%
0.06s
Plansformer-dl 76.56%
23.44%
0%
52.61%
0.09s
Table 2: Results of plan validation.
hard to find optimal plans, we can find a valid plan linear in the number of blocks to any problem instances by putting them down and picking from the table (Gupta & Nau 1991).
On a relatively more complex domain, i.e., dl, Plans former-ddrrrr achieves 76.56% valid plans, of which 52.61% % optimal. We notice 20% difference between valid and optimal plans for dl, with an observation that the model can come up with completely new and valid action sequences, although may not optimal. We can see that the number of optimal plans generated reduces with the increasing complexity of the domains. We include both incomplete/wrong generations from the models and failed plans when reporting invalid plans. An incomplete/ wrong generation partially correct ordering of action sequences, whereas failed plan is an entire plan consists of impossible ordering of actions, not allowed by the domain definition. Figure an example of incomplete generation and failed plans. All Plansformer models generate close to 0% failed plans for respective domains. It is a testament to Plansformer models' ability to understand and generate valid action sequences for given domain.
Failed Plans
put-down b2, unetack b4 b1, put-down unstack put-down plck-up b2, stack b2 b1, up b4, stack b4 b2
Actuel
unstack b2 b4, put-down b2, unstack b4 b1, put-down b4, unstack b1 b3, put-down b4, unatack bé b1. put-down b4, unstack b1 b3, pul-down b4. unstack b4 b2, put-down b4, unstack b2 b4, stack pick
Generated Plan:
bz
Incomplete Generatione
Actual Plan: unstack b1 b3, pul-down b1. pick-up b4, stack b4 b1
Generated Plan: unslack b1 b3, pul-down picksup
Figure 3: Different types of invalid plans generated by Plansformer on Blocksworld domain (Plansformer-bw).
Codex, the second best performing model according to ROUGE and BLEU scores, only generates 0.15% valid plans, emphasizing the need for a two-stage evaluation phase where both model and generated plans are tested. We notice that the average time taken by Plansformer completely solve the test-bed of problems is 200x faster than the FastDownwwrr, an automated planner that generated ground truth plans. Plansformer may offer immense advantage in generating approx- imately correct plans in real-time applications. Interestingly, CodeT5, used to build Plansformer, takes considerable time to solve the same problem instances from the test bed. We believe that the Plansformer is faster since it generates valid and likely optimal plans shorter in length than usually long incoherent sequences generated by CodeT5, which tend to conecmnns.,ing. The difference between plans generated by Code'T5 and Plansformer for the same problem is shown in Figure
There have been very few relevant works that can be compared with Plansformer. Although some recent works such as (Huang et al.) 2022) use LLMs to generate "plans", ; it is different from auto- mated planning and not symbolic in nature. These "plans" are step-by-step actions to perform trivial everyday task such "Brush teeth". These methods use user-constructed prompt to enable an LLM to generate appropriate steps from its prior knowledge. We believe the work by Valmeekam et al. 2022) is similar in spirit to ours, using PDDL-based natural language prompt to obtain symbolic plans using GPT-3. The significant difference between their dataset and ours is the difference in the object names, for example, block is named b in our dataset opposed
7

---

Under review as a conference paper at ICLR 2023
to in VValmeeamm et al. 2022). This difference lets evaluate Plansforme on the dataset from Valmeekam et al. 2022) with randomized object names as opposed to our dataset introduced in Section 3.1.1 On this dataset, Plansformer generated 66% valid plans, whereas, GPT3 with PDDL- based natural language prompting generated only 0.6% valid plans. The significant difference in performance enables to validate the advantage of our approach in generating valid plans despite randomized object names.
Plansformer, trained and tested on the same domain, displays superior performance both as a model and a planner. However, LLMs are also well known for transfer learning, i.e., a model trained for solving one domain can be re-purposed solve other related domains. In the next section, we explore how Plansformer trained on one domain can be adapted to another.
CodeT5
handemply, ontable, clear, holding x. not x, handomply, ontable x, stack not x, not cloar unstack b2 b3, put-down b
b3, put-down b4, unstack unstack
Plansformer
unstack put-down b4
Figure 4: Long invalid generation from CodeT5 as opposed to valid optimal plan from Plansformer-bw for the same problem.
4.3 CAN PLANSFORMER ADAPT TO ANOTHER DOMAIN?
The base models, i.e., Plansformer-x, where can bw, hn, gr, and dl, cannot generate valid plans for other domains (i.e., Plansforme r-bw on hn, gr; and dl) since each domain differs from the others in terms of action space, complexity, and solving. However, LLMs allow us to utilize the model trained in one domain to adapt to another using either making use of prompt conditioning or the problem instances from the new domain. We have seen from the previous work on prompt conditioning (Valmeekam et al. (2022) that the performance prompt. A small perturbation to the prompt can significantly affect the model's performance, and creating a perfect prompt requires understanding the inner workings of LLM hand and trial and error. In recent years, researchers have started looking at automatic prompt generation (Shin et al. 2020). which we would like to explore in the future.
transfer learning with further fine-tuning
of the model on unseen domain is very sensitive to the mumually-identinn
Instead of the prompt conditioning, we follow the transfer learning approach by finetuning Plans- former base models with problem instances from other domains to check the ability of Plansformer to adapt to new domains. For brevity. we demonstrate variants of P Plansfo corfforrer-eee models on three other domains. Figure shows different Plans former-bee models and their plan validation scores on respective test-bed from target domains. We report that the results for transfer learning setup of Plansformer base models convey the same insights as P Plansfr rmer shown here and presented in Section 4.2 of supplementary material.
We consider different numbers of problem instances for finetuning P Plansformer- bow on a given domain to see how the performance of the model varies across the sample size. We use the model naming format to convey the details on the amount of problem instances used for finetuning the Plansformer base model, i.e., bw-hn [ [500] implies that further finetune P Plans f orerrbb using 500 problem instances from hn and report the results. In Figure 5 we can see an overall increase in the number of valid plans for every testing domain available for finetuning. We observe that the models fine-tuned with 2000, which is 14% of the training size of base models, achieves 50% of the valid plans recorded by Plansformer" hn, P Plansforme, z-gr. and P Plans Former--erer r-d Despite the complexity of these planning domains, we obtain > 90% valid plans for all testing domains by increasing the finetuning samples to that of the training size of base models. P Plansforme wwwssroommrrbwbeee
increase the problem instances
[144 4400] obtains the best performance among all models, by achieving 97.05 97.05% valid plans, out of which 95. 22.22% are optimal. Figure 5 d). we compare Plansformer-hn trained with different number of data points from hn domain
We would like to note that Plansformer is trained only on the dataset introduced in Section 3.1.1 and is not trained/fintuned the dataset in (Valmeekam et al. 2022 for this experiment
8

---

Under review as a conference paper at ICLR 2023
Testing Domain
hanoi
Testing domain gripper
Optimal Plans
Falled Plans
Valld Plans
Optimal Plans
Failed Plans
Valid Plana
100.0%%
75.0%
75.30%
90.00%
90.20%
20.00%
25
0.0%%
tepprag
belky1448g) beite
0.00%
becentsay
beaetson
Nocar
(a)
(b)
Testing domain- driverlog
Testing Domain
(Valid Plans)
Optimal
Failed
bw-hn
00%
00.0%%
20 20%
50.00%
25.00%
00%
1000
6000
10000
(c)
(d)
Figure 5: Plansformer-he bw as the base model fine-tuned with and tested on (a) hanoi (b) grippers (c) driverlog, and (d) shows the comparison valid plans generated by Plansformer-bw: derived models with Plansformer-hn trained using similar data points.
against the Plansformer-bw (base model) finetuned the same data points. We can clear advantage of the transfer learning capability in LLMs, both the bw and hn domains have similar plan semantics. Similar trends can be seen for the other domains and the results are reported in Section 4.2 of supplementary material.
We notice that the failed plans decrease with additional problem instances used for finetuning. Using the same amount of problem instances as training (14, 400), we observe that the number of failed plans is less than that of the base models built for the respective domains. The number of optimal plans consistently increases with the number of problem instances in hn domain for finetuning. It is 13% more than Plansformes We also report the fine-tuning results of all possible Plansformer models and their performance in Figure of the supplementary material. We have also trained a single Plansformer model on all the domains (multi-task setting) and found that the individual model has relatively comparable performance to that 5f supplementary material Section 4.2 for more details).
hn, whereas we can see some variations for the other two domains.
CONCLUSIONS AND ONGOING WORK
In this paper, we have explored using LLMs to generate symbolic plans for multiple domains. We have taken an LLM tailored to code and trained it further over a set of planning problem instances and corresponding valid plans. We then tested the model's capability to generate plans for unseen planning problem instances, evaluating the correctness and length of such plans. Our approach compared to an existing state-of-the-art planner, showing that our LLM-based planner, called Plansformer, can solve most instances with high quality both in terms of correctness and length while needing much less time to generate such plans. Although Plansformer shows promising results, this work is not done and can be extended in many directions. One seek to eliminate invalid plans comprising both failed or incomplete plans, drive efficiency further, explore more planning domains, e.g., from competitions (ICAPS 2022). and tackle other planning settings (Valmeekam al. 2022

---

Under review as a conference paper at ICLR 2023
REFERENCES
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for pro- gram understanding and generation. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp- 2655 2668, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 1rXii:2107.03374 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reicchrr Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /.1.v0080000000. 2018.
Zhangyin Feng. Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1536-1547, 2020.
Noelia Ferruz and Birte Höcker. Controllable protein design with language models. Nature Machine Intelligence, pp. 1-12, 2022.
M. Fox and D. Long. Pdd12.1: An extension to pddl for expressing temporal planning domains.
Journal of Artificial Intelligence Research, 20:61-124, Dec 2003. ISSN 1076-9757. doi: 10. 1613/jair. 129. URL httptttp//d....m dww.:oo/i..og/00 1et.
C Gerety and P Cull. Time complexity of the towers of hanoi problem. SIGACT 18(1):80-87 mar 1986. ISSN 0163-5700. doi: 10.1 15..111/331228300
831 2. 8320
Malik Ghallab, Dana Nau, and Paolo Traverso. Automated Planning: Theory and Practice. The Morgan Kaufmann Series in Artificial Intelligence. Morgan Kaufmann, Amsterdam, 2004. ISBN 978-1 118-1-55860-85644 URL lattp://www/ sciencedirect. www./ccinne//bo......o
9781553608568
Naresh Gupta and Dana S. Nau. Complexity results for blocks-world planning. In Proceedings of AAAI-91, pp. 629-633, 1991.
Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26: 191-246, 2006.
Malte Helmert and Carmel Domshlak. Lm-cut: Optimal planning with the landmark-cut heuristic. Seventh international planning competition (IPC 2011), deterministic part, pp- 103-105, 2011.
Dan Hendrycks, Steven Basart, Sauray Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song. et al. Measuring coding challenge competence with apps. arXiv preprint 10S. 099388, 2021a.
Dan Hendrycks, Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv: 103.038444 2021b.
10

---

Under review as a conference paper at ICLR 2023
Alberto Olmo Hermandez, Sarath Sreedharan, and Subbarao Kambhampati. Gpt3-to-plan: Extract- ing plans from text using GPT-3. CoRR, abs/2106.07131, 2021. URL WtLlhttps///ariiv.oo
abs 00ss/216.0077 07131
Chris Hokamp and Qun Liu. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1535-1546, 2017.
Richard Howey, Derek Long, and Maria Fox. Val: Automatic plan validation, continuous effects and mixed initiative planning using pddl. In 16th IEEE International Conference Tools with Artificial Intelligence, pp. 294-301. IEEE, 2004.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207, 2022.
ICAPS. International planning competitions international conference automated planning and scheduling (icaps).
2022.
Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yulia Tsvetkov. Controlled text generation as continuous optimization with multiple constraints. Advances in Neural Information Processing Systems, 34:14542- 4544--14554, 2021.
Hang Li. Language models: Past, present, and future. Commun. ACM, ACM, jun 2022. ISSN 0001-0782. doi: 10. 14533490444 URL
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco. Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: machine learning benchmark dataset for code understanding and generation. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Drew McDermott, Malik Ghallab, Craig Knoblock, David Wilkins, Anthony Barrett, Dave Chris- tianson, Mare Friedman, Chung Kwok, Keith Golden, Scott Penberthy, David Smith, Ying Sun, and Daniel Weld. PDDL the planning domain definition language. Technical report, Technical Report, 1998.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463 2663-2473, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21 (1410)) 2020.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
Serbulent Unsal, Heval Atas, Muammer Albayrak, Kemal Turhan, Aybar C Acar, and Tunca Dogan. Learning functional properties of proteins with language models. Nature Machine Intelligence, (3):227-245, 2022.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large lan- guage models still can't plan (a benchmark for Ilms on planning and reasoning about change). arXiv preprint 2206. 2ete::2020....9... 2022.
11

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre- trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696-8708, 2021.
Sean Welleck, Kianté Brantley, Hal Daumé lii, and Kyunghyun Cho. Non-monotonic sequential text generation. In International Conference on Machine Learning, pp. 6716-6726. PMLR, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pp. 38 45, 2020.
12


Reviews:
[{"review_id": "n8uNruMMpN", "reviewer": "Reviewer_X39q", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper study considers the question of whether zero-shot large pre-trained language models (LLMs) can generate plans by studying performance in symbolic planning problem domains. For each sample, the model receives the goal, the initial state and the available actions. The model used for fine-tuning is popular for code generation. The empirical results show that although some models are frequently able to generate sequences similar to the gold truth, they are not valid plans.\n\nIn the main body, I concluded –but it wasn't clear– that no model was trained using the data from all the domains. Fig 4 in the appendix confirms that.\n\n\n---\n\nincreasing scores after the updates", "summary_of_the_review": "The paper study how a LLM specialized in code generation cannot generate symbolic plans without specifying tunning. The contribution is two-fold: on one hand, the paper identifies symbolic planning as an interesting benchmark. Second, it proposes a method for achieving high-performing models, although they require a symbolic planner. This motivates researchers on LLMs for reasoning to attempt to get similar performance using other tools.\n\nI'm now more inclined to accept the paper, but it depends on the answers to the questions.\n", "strength_and_weaknesses": "Strengths\n- Symbolic planning seems to offer a challenging problem for code generation in new domains. Even if it can be argued that such code is different from the usual one, the sequence of actions could as well correspond to actions in a robot or an API call\n- Evaluating using ROUGE and BLEU allows seeing the behaviour of LLM with a popular metric.\n- The planning problems are reasonably simple\n\nWeaknesses\n- One can argue that zero-shot is not the recommended methodology for testing an LLM in a domain like this. Many reasoning tasks are evaluated including one demonstration. \n\t- I understand it's cumbersome to find the best prompt for them. However, I suspect the readers would prefer some prompt investigation. The fine-tuning time is short (at most 75mins), so the computing cost might not become too high.\n- In contrast, each model is specialized in a domain.\n\t- Moreover, the transfer to a new domain required a significant amount of examples to start observing significant performance. Providing one example seems only fair.\n- The size of the models is not said explicitly.\n\t- LLMs has shown additional generalization at larger sizes. Perhaps the conclusions would change with bigger models. I\n- Single set hyper-parameters (Appendix sect 3.2). \n\t- One can argue the hyper-parameters are to be fixed across experiments, but the models are being tuned.\n\t- See question below on hyper-parameters\n\nNon-Weaknesses (these are potential weak points that I think don't diminish the submission)\n- Code evaluation is usually complemented by constrained decoding.\n\t- When grammar is available, LLMs tend to be used with constrained decoding. I understand this is not the point of the paper as none of the methods used that, but it should be discussed in related work and make an explicit statement.\n\nAdditional comments\n- I miss an explicit token to indicate the beginning of the task. \n\t- For instance, in Fig 4, CodeT5 might be predicting the parameters of part of an effect in the prompt.\n\t- I wonder what would happen with an explicit token plus a demonstration.\n- Plan lengths are not reported. Instances with longer plans might be more prone to testing.\n- I suspect the 80-20% splits might be helping the proposed model.\n\t- The problems for testing are in the same distribution, although they are rather small.\n\t- The 20% might have share sub plans with the training data.\n\t- If that's the case, perhaps the proposed method would not actually help to plan in a new domain. I'm ok with that, but the conclusions of the paper should be toned down.\n- Please clarify in the main body\n\t- Plansformer (no suffix) in Table 1 is 4 different models. I suggest adding an explicit column on the data used for evaluation.\n\nQuestions:\n- What pre-trained models were used precisely? \n\t- How many parameters?\n\t- What hyper-parameters were used for testing them?\n- How similar are the sequences of the training vs the testing data?\n\t- Perhaps a pair-wise comparison using distance can reveal some patterns.\n\t- Is the generation repeating objects in the same order? There might exist artifacts that wouldn't affect a classical planner but would affect LLMs.\n- Is CodeT5 slower just because it is generating a longer sequence?\n\t- We are missing hyper-parameters for them.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FnWsY26XKlt", "reviewer": "Reviewer_CWqr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper attempts to use large language models for automated planning, by fine-tuning them on samples of plans (sequences of actions / a path determined by a planner). They essentially frame plan-formation as a supervised learning task, relying on the capability of a pre-trained language model to learn and adapt to the planning language’s syntax in the domains they test in. They create such a dataset to fine-tune the models on and they show that this fine-tuned language model (that they call “plansformer”) is capable of generating plans for 4 different domains or tasks they test on.\n", "summary_of_the_review": "This paper is well-motivated and executed, however the evaluation (mainly the test datasets) might not be as sound/correctly generated as needed to properly evaluate generalisation. If the authors can redo this and add a more concrete evaluation this would be a very nice paper with lots of interesting insights!\n", "strength_and_weaknesses": "Strengths\n\n1. The motivation of this paper is well-formed and the experimental setup is quite sound—the authors create a dataset to fine-tune a language model on, and then evaluate how well the model can reliably produce plans that are syntactically and semantically correct and executable.\n2. They evaluate using standard NLP metrics like ROUGE/BLEU—although it might be worth looking at more executable metrics like measure syntax (e.g., through the underlying grammar) or semantic aspects (e.g., through the propositions or predicates being referenced) rather than string-match metrics.\n3. The conclusion and ongoing work sections have interesting future work directions that draw off of this paper—however, some clarity on the future work could be helpful. Can the authors elaborate on the “learning planner” idea maybe in it’s own section, or otherwise remove from the conclusion section?\n\nWeaknesses / Questions\n\n1. Can a navigation-like domain also be handled by this model? It might be worth adding a domain that is different from the existing ones (in terms of its action space) to illustrate the generality of this training mechanism to show that the model can generate plans in that domain as well.\n2. What do the train/test splits look like in terms of distributional differences? From reading the paper, it sounds like the splits were made randomly? My main concern (especially from looking at the results which look very high) is that the test splits are almost identical, or at least not very structurally different, from the train splits, which means the model does not actually have to generalise very much to perform well on the test splits.\n3. Along those lines, can we re-create the test split to test the following. (a) if the test plans are of length >>> the length of plans in the train dataset (b) if the nested strcture/syntax of test plans is deeper/more complex than those in the train set (c ) if some set of objects/predicates are held out in samples in the test set \n4. I would expect performance to at least decrease a little on these splits—and this is what we want to test to truly measure test-time generalisation and whether the model really is learning the syntax/semantics of plans. But if the performance is significantly lower, this is an important thing to evaluate and put into the paper, so I would like to see this table of results.\n5. There are lots of things that warrant some more explanation/clarity: what is FastDownward and is it crucial to the process? If it is, there should be a small paragraph that briefly describes it as well as it’s relevance to the generated dataset.\n6. It is also unclear what the dataset looks like for the different domains, so Figure 2 could be updated to contain smaller snapshots for different domains to make it clearer.\n7. Can we have more details of the fine-tuning process even if only in the supplementary material, because there currently does not exist information about fine-tuning and what layers/components of the model are being updated.\n8. When talking about evaluation in terms of time taken to solve the test-bed, can you clarify why the models take longer? Is it an inference-time generation problem or are there other reasons that it might take longer?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EU-VjCiHdy", "reviewer": "Reviewer_T1dJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Continuing the successful pattern of fine-tuning large LMs on specialized domains, this paper presents Plansformer, a LLM (CodeT5) which is additionally fine-tuned to generate plans that solve problems expressed in PDDL. Plansformer is found to generate plans with far higher validity than LLMs (including CodeT5 itself) without fine-tuning on these domains. And Plansformer generates plans with far greater speed than exhaustive planners like FastDownward (though at a reduced validity rate). ", "summary_of_the_review": "The Plansformer is a promising application of large language models to planning problems. But this message may be lost on the community if the clarity of the paper’s writing is not significantly improved.\n\n***** POST-DISCUSSION UPDATE*****\n\nThe authors are to be commended for performing additional experiments on the impact of object name randomization, and providing a table of results from randomizing the object names for either train or test or both. In my view, this new table and the paper's Table 2 tell a complex story. When train and test use the **same** naming scheme (either original or fully randomized), then the impact of object names on valid plans is highly task-specific, ranging from a very small drop (0.21%) on Blocks World, to a significant drop (15.41%) on Hanoi. But when train and test **do not** employ the same object naming scheme, validation performance falls dramatically, from 76.56% to 2.69% in the worst case. \n\nMy conclusion is that the issue of object name randomization cannot be safely ignored, because it calls into serious question the proper interpretation of the previous results (without object name randomization). The only solution I see is to repeat all of the experiments using (2-character) object name randomization. My expectation is that the paper’s primary conclusions will still hold up, but I don’t think the experiments can be skipped.\n\nFor these reasons, I regret having to leave my score unchanged.", "strength_and_weaknesses": "**Strengths**\n\nThe work is well-executed, and the results are noteworthy. On four different planning domains, Plansformer significantly outperforms baseline LMs, and sometimes reaches 90% plan validity without the high computational cost of the baseline planner (FastDownward).\n\nThe adaptation experiments give interesting results, showing that a second round of fine-tuning (on a domain different than the first) produces a model which is at least as accurate as training on the second domain alone. This may indicate some degree of successful transfer between domains. But the missing experiment would be to fine-tune a single model for twice as long on the same domain, which would presumably produce the best results of all for each domain.\n\nThe “model-testing” evaluations (in Table 1) are useful as sanity checks, but they are not nearly as important as the validity and optimality tests, and could be moved to an appendix if space runs short.\n\n**Weaknesses**\n\nThe paper says “There isn’t much prior work to adopt as a baseline and perform a comparison with Plansformer.” But in fact, deep neural networks and even transformers have been applied to planning problems, including those described by PDDL:\n- DeConNet: Deep Neural Network Model to Solve the Multi-Job Assignment Problem in the Multi-Agent System, Lee et al., 2022, https://www.mdpi.com/2076-3417/12/11/5454\n- Learning to Delegate for Large-scale Vehicle Routing, Li et al., 2021, https://arxiv.org/abs/2107.04139\n- PDDLGym: Gym Environments from PDDL Problems, Silver and Chitnis, https://arxiv.org/abs/2002.06432\n\nIncluding non-LLM baseline models such as these in the experiments would provide a more accurate gauge of Plansformer’s abilities.\n\nThe paper says: “Plansformer uses masked language modeling, attending to tokens on either side of the masked word.” This gives the impression that Plansformer is just a transformer encoder, when in fact it is an encoder-decoder pair, as is CodeT5. And Plansformer’s decoder uses standard causal attention instead of bidirectional masked attention. Furthermore, the paper gives no details of bidirectional masked pretraining at all. Was it performed? Or was all of this work’s fine-tuning done on the decoder’s autoregressive generation of plans?\n\nIt is probably the case that each problem description is passed to Plansformer’s encoder, then the plan is produced by the decoder. But this basic flow of operation is never explicitly stated in the paper. \n\nBecause Plansformer’s plans always fall well short of 100% validity and optimality, the bolded phrase in the following sentence appears to be incorrect:  “We then compared its behavior to an existing state-of-the-art planner, showing that our LLM-based planner, called Plansformer, **can solve many more instances with high quality both in terms of correctness and length** while needing much less time to generate such plans.” \n\n“88.44% are optimal and have the highest performance compared to **other domains**.” But as explained in other parts of the paper, optimal means that the plan is the shortest possible (minimum cost). What does this have to do with “other domains”?\n\n“All Plansformer models generate 0% incomplete generations for respective domains.” This seems to contradict Fig. 3, which shows an incomplete plan generated by Plansformer on Blocksworld.\n\nFig. 5 and others like it would be less confusing if the first bar were removed, because it doesn’t represent testing on the same domain as all the other bars in the chart.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "02EAGW7zraR", "reviewer": "Reviewer_abxA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper showcases the use of LLM on solving classical planning problems. A dataset based on PDDL syntax which includes 4 tasks is generated for training and evaluation of the model. The proposed model, Plansformer, is fine-tuned on the generated dataset from pre-trained CodeT5 model. The model is evaluated on each planning domain using both language modeling metrics and planning validity and optimality metrics. Experimental results show that the model can predict valid plans for the problem and domains it has been trained on.", "summary_of_the_review": "I think this paper has a straightforward idea and is looking at an interesting problem, but the motivation and use case of the trained model seems unclear to me.  The experiment settings and baselines can be improved to showcase the benefits of learning such a model based off LLMs,  and illustrate the advantages over classical planners.\n\n-----------\nUpdate the score on the revised version.", "strength_and_weaknesses": "LLMs has shown some impressive results in several language modeling tasks so it would be interesting to see whether they can be useful in planning, and how. This paper is an attempt on the classical planning. The method is quite clear and straightforward, where the pretrained CodeT5 model is fine-tuned on the text sequence which is a compact representation of the domain and problem file. The evaluation on the plan validity and optimality is helpful. \nI also have some questions.\n- In terms of training and testing data, are the object names randomized in each problem? Is the testing problems having the same object names as the training set? If the initial and goal state of two problem instances has the same set of literals, but in a different order, are they treated as two different problems during training and testing? If swap the object name during testing, will there be a large performance drop?\n- In terms of evaluation, are all the baselines tested without fine-tuning? Does the choice of pre-trained model matter a lot to the result? For example, if fine-tuned based on T5, instead of CodeT5, will there be a big performance drop.\n- ‘On a relatively more complex domain, i.e., dl, Plansformer-dl attains 76.56% valid plans, out of which 52.61% are optimal. There is a 20% difference between valid and optimal plans for dl showcasing that the model was able to come up with completely new and valid action sequences…’ It’s not clear to me why the difference between valid and optimal plans are new plans. Are the plans in the training set all optimal?\n- It seems that, for experiments in 4.3, once fine-tuned on another task, the plan validity on the original problem will drop drastically, many of which drops to 0, according to Fig. 6 in supp materials. Can Plansformer adapt to other domains, while maintaining the ability to solve the problems in the original domain? Or can it generate valid plans if trained on multiple domains simultaneously? \n- For these domains, if using lm-cut or h-add heuristics with greedy search, what is the average planning time and percentage of optimal plan?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject