Kati8/ICLR_2023_2025 huggingface dataset:
authors: Bernd Bohnet
authors: Azade Nova
authors: Aaron T Parisi
authors: Katayoon Goshvadi
authors: Kevin Swersky
authors: Hanjun Dai
authors: Dale Schuurmans
authors: Noah Fiedel
authors: Hanie Sedghi
reviews: 
Comments: This paper presents datasets for large language model (LLM) reasoning tasks from three classical planning domains. The paper explores how LLM performs in those classical planning domains for in-context learning and fine-tuning settings. The paper also provides a comparison with a similar benchmark set, PlanBench, and the paper investigates failure cases.
Strengths: In terms of originality, there are similar benchmarks such as PlanBench available in the literature.  The authors argue that the paper also considers many-shot in-context learning settings, as shown in Figure 2.  The number of few-shot exemplars ranges from 1 to 400.
Figure 4 shows results from three inference-time ICL methods: ToT, MCTS, and Debate-as-reasoning in non-PDDL-based domains.
Lastly, this paper also shows the experiment results from the fine-tuning setting.
Weaknesses: On the coverage of classical planning domains.
This paper argues that the proposed work lays the foundation for improving the planning capabilities of LLMs by presenting three classical planning domains. From a classical planning viewpoint, those domains were more than 20 years old and they don't have many features in PDDL and only have almost the minimal subset of the language features. Namely, it is almost STRIPS with a negative precondition. If we view the direction of translating PDDL to NL, the work should consider the coverage of the translation of the language.

On the quality of translations.
The paper mentioned that the translation is done by regular expressions. LLMs are supposed to be trained in human-written texts and natural languages. However, the translations are done by hand-crafted regular expressions that translate formal PDDL descriptions. It is doubtful how such a dataset would enhance the reasoning capability of LLMs.
If we consider the LLM as a symbolic reasoning engine like a planner such as fast-downward, the scale of problems is so tiny that it is doubtful how the proposed methods will be useful.
Recommendation: 3
Reviewer confidence: 5
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The paper provides a PDDL and NL planning benchmark, evaluates ICL and SFT with LLMs on this benchmark dataset, and probes for failure models and plan generalization abilities.
Strengths: 1. A new NL planning benchmark dataset.
Weaknesses: 1. Did not find the paper to have novel results (please refer to questions)
2. No details on how the plans are being generated using FD (which heuristic is being used to generate optimal plans?)
3. The NL translated prompt does not include PDDL domain, making me question how any LLM is supposed to know all actions, preconditions, and effects for a given domain?
4. Not a complete coverage of literature on plan generation. Especially, authors in [1] introduce thought of search which steers LLMs to solve 100% planning problems. [2] already provided similar conclusions on SFT using even smaller parameter models and also talked about multiple plan generalization issues.

[1] Katz, M., Kokel, H., Srinivas, K., & Sohrabi, S. (2024). Thought of Search: Planning with Language Models Through The Lens of Efficiency. In The First Workshop on System-2 Reasoning at Scale, NeurIPS'24.

[2] Pallagani, V., Muppasani, B., Murugesan, K., Rossi, F., Srivastava, B., Horesh, L., ... & Loreggia, A. (2023). Understanding the capabilities of large language models for automated planning. arXiv preprint arXiv:2305.16151.
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper benchmarks the planning capabilities of Large Language Models on the PDDL and travel planning domains, and also discusses the impact of fine-tuning LLMs on optimal planning solutions. The presented experiments include results with Gemma 2 27b, Gemini 1.5 Pro, Gemini 1.5 Flash, and GPT-4-turbo, and the authors primarily analyze the planning performance of these models when varied with the number of few-shot examples in the prompt.
Strengths: 1) The paper presents a consolidated set of results on two recently popular natural language planning benchmarks, along with additionally generated problems in PDDL planning domains.
2) This is the first work that systematically analyzes LLM planning capabilities with fine-tuning on optimal solutions, and if fine-tuning is effective with respect to generalizing across different problem categories under a domain (for example, training on PDDL blocksworld problems with 3-7 blocks and testing with PDDL problems of 8-9 blocks, and so on).
Weaknesses: 1) Novelty - One of the biggest contributions that the work claims is of the planning dataset. However, the paper only combines existing PDDL planning dataset (PlanBench) and travel planning domains, with the addition of some newly generated test problems. This seems to be more like packaging the two domains together than presenting a new dataset in a paper. Can the authors provide more details to counter this argument - such as - what % of problems are new in this dataset that are not present in the two existing datasets?
2) Unjustified evaluations for improved performance - It seems that the paper aims to establish that LLMs perform exceptionally well in planning domains, but at the cost of prompt engineering (line 73 - authors say that ICL methods show boost in performance with 'carefully instructing the model') and cherry-picking evaluation data (line 89 - authors say that 'data curation during training' can be helpful to avoid biases thereby helping LLMs). Why do we have to give all the benefit of the doubt to the LLMs at the expense of human effort during prompting and data processing? It is worthwhile noting the challenges that LLMs face (using failure mode analysis) without such human efforts, as they can be constructively useful for further improving their planning capabilities.
3) Fine-tuning results - The authors show near-perfect accuracy for SFT results in Tables 3 and 4, but only in the cases where the train and the eval data consisted of the similar problem cases. This is a classic over-fitting example, and thus, does not lead to any conclusions regarding the so-called 'generalization' abilities of LLMs.
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 1
Clarity: 2