Venue: ICLR 2023 Conference

Reviews:

final_score: rating: null
review_content: Dear authors, 
as the discussion period is almost over, please don't bother answering this thread. 
I leave it here, just in case the authors find it useful.

----

Dear authors,

After discussing with other reviewers, I think the contribution of the paper is still confusing in the paper.

The current manuscript sometimes tries to show that transformers would be able to plan in many circumstances, including variations of the object names.

I think the original point is an empirical demonstration that transformers could plan, reporting what methods work better.
That doesn't mean that any object naming should work.

Perhaps it'd be simpler and more transparent to report how to make transformers plan in these non-trivial domains. 
Part of the how is a strategy for object naming is akin to prompt design. 
I'm not sure prompt design would make it into AI textbooks in 20 years but, for know, it's a valid contribution.
Showing how small changes affect performance is useful, but that's not the main point.

The significance of the work I think has been justified enough: pre-trained models are not performing well, and the paper presents a way to do it.
 
Revisiting the randomization, besides my point above, I suggest making it more clear that this does not imply a general strategy for any kind of text generation that involves planning-like problems. It's possible that the strategy used for randomization here might not be feasible in other domains. Furthermore, getting plans of high quality might be very expensive or even impossible in some domains –unless a symbolic model is available, that would in turn enable the object naming strategy here.

We cannot get another version of the paper, so perhaps the authors can answer this two questions:
* What's the contribution, and what's not a contribution?
* What would you tune down or even remove from the introduction and the discussion of results to make the paper more clear around the contribution?
* How would you use that space?


review_title: [post discussion reflection] What's the contribution?

final_score: rating: null
review_content: Dear authors, I think randomization is not an issue.
The main reason is that the diversity of the problems is even harder.

Would you summarize the issue, the concerns, and your answer?

Please do your best on summarizing the outstanding issues.

I think this would help us settle the differences. 

Feel free to summarize any outstanding issue or angle. 

review_title: Randomization

final_score: rating: null
review_content: I wonder if the authors could comment on how difficult is the tasks. For instance, what’s the performance we should expect from

- solving them by blind search
- how hard it would be to train RL models for them? In this case, the proper comparison depends on the domain. One thing is a model for the domain blocks, something else is a generalist model.

Another question is about the consequences of this work.
I don’t think commenting on specific work on reasoning using LLM would not be illuminating because any synthesis would be disputed.
Instead, could you summarize your points or elaborrate on how could this be used in other domains beyond the planning benchmarks.

review_title: Significance of the problem 

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper attempts to use large language models for automated planning, by fine-tuning them on samples of plans (sequences of actions / a path determined by a planner). They essentially frame plan-formation as a supervised learning task, relying on the capability of a pre-trained language model to learn and adapt to the planning language’s syntax in the domains they test in. They create such a dataset to fine-tune the models on and they show that this fine-tuned language model (that they call “plansformer”) is capable of generating plans for 4 different domains or tasks they test on.

strength_and_weaknesses: Strengths

1. The motivation of this paper is well-formed and the experimental setup is quite sound—the authors create a dataset to fine-tune a language model on, and then evaluate how well the model can reliably produce plans that are syntactically and semantically correct and executable.
2. They evaluate using standard NLP metrics like ROUGE/BLEU—although it might be worth looking at more executable metrics like measure syntax (e.g., through the underlying grammar) or semantic aspects (e.g., through the propositions or predicates being referenced) rather than string-match metrics.
3. The conclusion and ongoing work sections have interesting future work directions that draw off of this paper—however, some clarity on the future work could be helpful. Can the authors elaborate on the “learning planner” idea maybe in it’s own section, or otherwise remove from the conclusion section?

Weaknesses / Questions

1. Can a navigation-like domain also be handled by this model? It might be worth adding a domain that is different from the existing ones (in terms of its action space) to illustrate the generality of this training mechanism to show that the model can generate plans in that domain as well.
2. What do the train/test splits look like in terms of distributional differences? From reading the paper, it sounds like the splits were made randomly? My main concern (especially from looking at the results which look very high) is that the test splits are almost identical, or at least not very structurally different, from the train splits, which means the model does not actually have to generalise very much to perform well on the test splits.
3. Along those lines, can we re-create the test split to test the following. (a) if the test plans are of length >>> the length of plans in the train dataset (b) if the nested strcture/syntax of test plans is deeper/more complex than those in the train set (c ) if some set of objects/predicates are held out in samples in the test set 
4. I would expect performance to at least decrease a little on these splits—and this is what we want to test to truly measure test-time generalisation and whether the model really is learning the syntax/semantics of plans. But if the performance is significantly lower, this is an important thing to evaluate and put into the paper, so I would like to see this table of results.
5. There are lots of things that warrant some more explanation/clarity: what is FastDownward and is it crucial to the process? If it is, there should be a small paragraph that briefly describes it as well as it’s relevance to the generated dataset.
6. It is also unclear what the dataset looks like for the different domains, so Figure 2 could be updated to contain smaller snapshots for different domains to make it clearer.
7. Can we have more details of the fine-tuning process even if only in the supplementary material, because there currently does not exist information about fine-tuning and what layers/components of the model are being updated.
8. When talking about evaluation in terms of time taken to solve the test-bed, can you clarify why the models take longer? Is it an inference-time generation problem or are there other reasons that it might take longer?


clarity,_quality,_novelty_and_reproducibility: 1. There are many typos and incorrect citations in the paper (see list below) that should be corrected–especially in the case of citations to alleviate any confusion
2. Citations:
“Transformer-based (neural) architecture) Rogers et. al., → I’m not sure why this paper is being cited for transformers, instead maybe cite Vaswani et. al., Devlin et. al., Brown et. al., even for LM-specific transformers 
Helmert 2006 citations are incorrectly typeset
Gerety & Cull citation is incorrectly spaced with other text

3. Typos:
Quotations are incorrectly typeset

summary_of_the_review: This paper is well-motivated and executed, however the evaluation (mainly the test datasets) might not be as sound/correctly generated as needed to properly evaluate generalisation. If the authors can redo this and add a more concrete evaluation this would be a very nice paper with lots of interesting insights!
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Continuing the successful pattern of fine-tuning large LMs on specialized domains, this paper presents Plansformer, a LLM (CodeT5) which is additionally fine-tuned to generate plans that solve problems expressed in PDDL. Plansformer is found to generate plans with far higher validity than LLMs (including CodeT5 itself) without fine-tuning on these domains. And Plansformer generates plans with far greater speed than exhaustive planners like FastDownward (though at a reduced validity rate). 
strength_and_weaknesses: **Strengths**

The work is well-executed, and the results are noteworthy. On four different planning domains, Plansformer significantly outperforms baseline LMs, and sometimes reaches 90% plan validity without the high computational cost of the baseline planner (FastDownward).

The adaptation experiments give interesting results, showing that a second round of fine-tuning (on a domain different than the first) produces a model which is at least as accurate as training on the second domain alone. This may indicate some degree of successful transfer between domains. But the missing experiment would be to fine-tune a single model for twice as long on the same domain, which would presumably produce the best results of all for each domain.

The “model-testing” evaluations (in Table 1) are useful as sanity checks, but they are not nearly as important as the validity and optimality tests, and could be moved to an appendix if space runs short.

**Weaknesses**

The paper says “There isn’t much prior work to adopt as a baseline and perform a comparison with Plansformer.” But in fact, deep neural networks and even transformers have been applied to planning problems, including those described by PDDL:
- DeConNet: Deep Neural Network Model to Solve the Multi-Job Assignment Problem in the Multi-Agent System, Lee et al., 2022, https://www.mdpi.com/2076-3417/12/11/5454
- Learning to Delegate for Large-scale Vehicle Routing, Li et al., 2021, https://arxiv.org/abs/2107.04139
- PDDLGym: Gym Environments from PDDL Problems, Silver and Chitnis, https://arxiv.org/abs/2002.06432

Including non-LLM baseline models such as these in the experiments would provide a more accurate gauge of Plansformer’s abilities.

The paper says: “Plansformer uses masked language modeling, attending to tokens on either side of the masked word.” This gives the impression that Plansformer is just a transformer encoder, when in fact it is an encoder-decoder pair, as is CodeT5. And Plansformer’s decoder uses standard causal attention instead of bidirectional masked attention. Furthermore, the paper gives no details of bidirectional masked pretraining at all. Was it performed? Or was all of this work’s fine-tuning done on the decoder’s autoregressive generation of plans?

It is probably the case that each problem description is passed to Plansformer’s encoder, then the plan is produced by the decoder. But this basic flow of operation is never explicitly stated in the paper. 

Because Plansformer’s plans always fall well short of 100% validity and optimality, the bolded phrase in the following sentence appears to be incorrect:  “We then compared its behavior to an existing state-of-the-art planner, showing that our LLM-based planner, called Plansformer, **can solve many more instances with high quality both in terms of correctness and length** while needing much less time to generate such plans.” 

“88.44% are optimal and have the highest performance compared to **other domains**.” But as explained in other parts of the paper, optimal means that the plan is the shortest possible (minimum cost). What does this have to do with “other domains”?

“All Plansformer models generate 0% incomplete generations for respective domains.” This seems to contradict Fig. 3, which shows an incomplete plan generated by Plansformer on Blocksworld.

Fig. 5 and others like it would be less confusing if the first bar were removed, because it doesn’t represent testing on the same domain as all the other bars in the chart.

clarity,_quality,_novelty_and_reproducibility: **Clarity**

Some parts of the paper are clear enough, but most of the paper is very poorly written. I hope these problems are corrected in the rebuttal version.

Why is Rogers et al. referenced on the first line of the intro?

Most references are improperly parenthesized.

Poorly worded sentences like the following make the reader’s job much harder than it needs to be:
- “promisingly tell us the failures”
- “with a few samples than learning from scratch”
- “due to the fine-tuneup of the models”
- “We call the model with new weights that can generate symbolic plans as Plansformer.”
- “The two stages mentioned in the evaluation phase checks for finding out the efficiency of Plansformer as a language model and a planner.”
- “Plansformer… requires to be tested for plan validation”
- “A drop in the number of optimal plans reduces with the increasing complexity of the domains.”
- “200x faster than the ground truth”  Instead, this should say “200x faster than the FastDownward planner, which generated the ground truth plans”

**Quality**

The overall quality of the work is quite good.

**Novelty**

Fine-tuning of LLMs for planning appears to be novel and timely.

**Reproducibility**

The code and fine-tuned models would be very useful, but I didn’t see any mention that they would be provided.

summary_of_the_review: The Plansformer is a promising application of large language models to planning problems. But this message may be lost on the community if the clarity of the paper’s writing is not significantly improved.

***** POST-DISCUSSION UPDATE*****

The authors are to be commended for performing additional experiments on the impact of object name randomization, and providing a table of results from randomizing the object names for either train or test or both. In my view, this new table and the paper's Table 2 tell a complex story. When train and test use the **same** naming scheme (either original or fully randomized), then the impact of object names on valid plans is highly task-specific, ranging from a very small drop (0.21%) on Blocks World, to a significant drop (15.41%) on Hanoi. But when train and test **do not** employ the same object naming scheme, validation performance falls dramatically, from 76.56% to 2.69% in the worst case. 

My conclusion is that the issue of object name randomization cannot be safely ignored, because it calls into serious question the proper interpretation of the previous results (without object name randomization). The only solution I see is to repeat all of the experiments using (2-character) object name randomization. My expectation is that the paper’s primary conclusions will still hold up, but I don’t think the experiments can be skipped.

For these reasons, I regret having to leave my score unchanged.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper study considers the question of whether zero-shot large pre-trained language models (LLMs) can generate plans by studying performance in symbolic planning problem domains. For each sample, the model receives the goal, the initial state and the available actions. The model used for fine-tuning is popular for code generation. The empirical results show that although some models are frequently able to generate sequences similar to the gold truth, they are not valid plans.

In the main body, I concluded –but it wasn't clear– that no model was trained using the data from all the domains. Fig 4 in the appendix confirms that.


---

increasing scores after the updates
strength_and_weaknesses: Strengths
- Symbolic planning seems to offer a challenging problem for code generation in new domains. Even if it can be argued that such code is different from the usual one, the sequence of actions could as well correspond to actions in a robot or an API call
- Evaluating using ROUGE and BLEU allows seeing the behaviour of LLM with a popular metric.
- The planning problems are reasonably simple

Weaknesses
- One can argue that zero-shot is not the recommended methodology for testing an LLM in a domain like this. Many reasoning tasks are evaluated including one demonstration. 
	- I understand it's cumbersome to find the best prompt for them. However, I suspect the readers would prefer some prompt investigation. The fine-tuning time is short (at most 75mins), so the computing cost might not become too high.
- In contrast, each model is specialized in a domain.
	- Moreover, the transfer to a new domain required a significant amount of examples to start observing significant performance. Providing one example seems only fair.
- The size of the models is not said explicitly.
	- LLMs has shown additional generalization at larger sizes. Perhaps the conclusions would change with bigger models. I
- Single set hyper-parameters (Appendix sect 3.2). 
	- One can argue the hyper-parameters are to be fixed across experiments, but the models are being tuned.
	- See question below on hyper-parameters

Non-Weaknesses (these are potential weak points that I think don't diminish the submission)
- Code evaluation is usually complemented by constrained decoding.
	- When grammar is available, LLMs tend to be used with constrained decoding. I understand this is not the point of the paper as none of the methods used that, but it should be discussed in related work and make an explicit statement.

Additional comments
- I miss an explicit token to indicate the beginning of the task. 
	- For instance, in Fig 4, CodeT5 might be predicting the parameters of part of an effect in the prompt.
	- I wonder what would happen with an explicit token plus a demonstration.
- Plan lengths are not reported. Instances with longer plans might be more prone to testing.
- I suspect the 80-20% splits might be helping the proposed model.
	- The problems for testing are in the same distribution, although they are rather small.
	- The 20% might have share sub plans with the training data.
	- If that's the case, perhaps the proposed method would not actually help to plan in a new domain. I'm ok with that, but the conclusions of the paper should be toned down.
- Please clarify in the main body
	- Plansformer (no suffix) in Table 1 is 4 different models. I suggest adding an explicit column on the data used for evaluation.

Questions:
- What pre-trained models were used precisely? 
	- How many parameters?
	- What hyper-parameters were used for testing them?
- How similar are the sequences of the training vs the testing data?
	- Perhaps a pair-wise comparison using distance can reveal some patterns.
	- Is the generation repeating objects in the same order? There might exist artifacts that wouldn't affect a classical planner but would affect LLMs.
- Is CodeT5 slower just because it is generating a longer sequence?
	- We are missing hyper-parameters for them.

clarity,_quality,_novelty_and_reproducibility: The line of work is a popular line of investigation. Focusing on symbolic planning is novel and almost a necessary question.
The manuscript is clear enough for writing a review but it should be revised. For instance, 
- there is some confusion on phases and stages. 
- "All Plansformer models generate 0% incomplete generations for respective domains". Is that rounding down 0.x%?

The core of the evaluation is clear, and the observations are not trivial.
summary_of_the_review: The paper study how a LLM specialized in code generation cannot generate symbolic plans without specifying tunning. The contribution is two-fold: on one hand, the paper identifies symbolic planning as an interesting benchmark. Second, it proposes a method for achieving high-performing models, although they require a symbolic planner. This motivates researchers on LLMs for reasoning to attempt to get similar performance using other tools.

I'm now more inclined to accept the paper, but it depends on the answers to the questions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper showcases the use of LLM on solving classical planning problems. A dataset based on PDDL syntax which includes 4 tasks is generated for training and evaluation of the model. The proposed model, Plansformer, is fine-tuned on the generated dataset from pre-trained CodeT5 model. The model is evaluated on each planning domain using both language modeling metrics and planning validity and optimality metrics. Experimental results show that the model can predict valid plans for the problem and domains it has been trained on.
strength_and_weaknesses: LLMs has shown some impressive results in several language modeling tasks so it would be interesting to see whether they can be useful in planning, and how. This paper is an attempt on the classical planning. The method is quite clear and straightforward, where the pretrained CodeT5 model is fine-tuned on the text sequence which is a compact representation of the domain and problem file. The evaluation on the plan validity and optimality is helpful. 
I also have some questions.
- In terms of training and testing data, are the object names randomized in each problem? Is the testing problems having the same object names as the training set? If the initial and goal state of two problem instances has the same set of literals, but in a different order, are they treated as two different problems during training and testing? If swap the object name during testing, will there be a large performance drop?
- In terms of evaluation, are all the baselines tested without fine-tuning? Does the choice of pre-trained model matter a lot to the result? For example, if fine-tuned based on T5, instead of CodeT5, will there be a big performance drop.
- ‘On a relatively more complex domain, i.e., dl, Plansformer-dl attains 76.56% valid plans, out of which 52.61% are optimal. There is a 20% difference between valid and optimal plans for dl showcasing that the model was able to come up with completely new and valid action sequences…’ It’s not clear to me why the difference between valid and optimal plans are new plans. Are the plans in the training set all optimal?
- It seems that, for experiments in 4.3, once fine-tuned on another task, the plan validity on the original problem will drop drastically, many of which drops to 0, according to Fig. 6 in supp materials. Can Plansformer adapt to other domains, while maintaining the ability to solve the problems in the original domain? Or can it generate valid plans if trained on multiple domains simultaneously? 
- For these domains, if using lm-cut or h-add heuristics with greedy search, what is the average planning time and percentage of optimal plan?
clarity,_quality,_novelty_and_reproducibility: This paper is overall clear and straightforward on the approach used.  The reproducibility looks good with the dataset provided.  
The experiment section has provided quantitative and qualitative results on both modeling metrics and plan validity metrics. It would be better if the data generation and representation can be illustrated clearer. The title and some claims in the paper may be revised as the models are mainly tested on one single domain at a time. Specifically, there is no experiment showing one model generating plans on multiple domains. Section 4.3 (and the supp material) suggests that the performance drops a lot if finetuned on another domain.
In terms of motivation, it would be helpful to discuss the use case of the trained model. If the model can only be used on the same domain, with object names and numbers identical to the training distribution, the use case seems a bit limited. More experiments on, for example, planning/inference speed compared with different search and heuristics, can be added if the time consumption is a plus. Or any other properties that might come with LLM pre-training, e.g., zero-shot generality.
summary_of_the_review: I think this paper has a straightforward idea and is looking at an interesting problem, but the motivation and use case of the trained model seems unclear to me.  The experiment settings and baselines can be improved to showcase the benefits of learning such a model based off LLMs,  and illustrate the advantages over classical planners.

-----------
Update the score on the revised version.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper provides a strategy for solving classical AI planning problems by fine-tuning a code-oriented large language model.    The model is fine-tuned using data generated automatically by calling the planner on the problem domain specification.  It is then applied to different problem instances, including larger ones, drawn from the same formal domain.   Additional results on transfer are included, as well as experimentation (done at the behest of the reviewers) on the effect of randomizing object names.

Viewed in terms of the effectiveness of the resulting artifact as a planning algorithm, the results are mixed:
- It solves instances of relatively simple domains with high reliability
- It is less reliable (75%) on more difficult domains
- It generates solutions much more quickly than a planner
- It's not clear what the amortized cost of the fine-tuning is

Despite the lack of reliability as a planner, the contribution raises interesting points:
- It teaches us about the strengths and weaknesses of LLMs for planning
- It explores fine-tuning (rather than prompt conditioning, which had been the strategy of previous papers) and finds it to be helpful

The paper was a bit muddled in terms of
- exactly what points it was trying to make
- presentation of results (it is critical to make clear, for every single number, what data it is being trained and tested on)
But the authors did clarify substantially in a constructive dialog with the reviewers

A smaller point is that I don't think anyone found the Bleu-score results motivating---the big question is whether the plans are valid.  Those results could be moved to the appendix in favor of more results about actual planning performance.
justification_for_why_not_higher_score: The paper is at best an interesting exploration of how one might fine-tune LLMs for use in classical AI planning.  It does not constitute a substantial technical advance.
Decision: Reject