reviews: 
Comments: This paper runs various experiments on 3 PDDL datasets (Blocksworld, Logistics, Mini-Grid) and 2 natural language datasets (Trip Planning, Calendar Scheduling) to investigate (1) the effect of many-shot prompting on all 5 datasets (2) the effect of inference-time prompting techniques on 2 datasets (Trip Planning, Calendar Scheduling) (3) the effect of SFT using ID and OOD on 2 datasets (Blocksworld, Logistics) (4) the performance differences on PlanBench's Blocksworld/Logistics dataset versus this paper's and (5) failure case analysis.
Strengths: $Originality:$ This paper runs large-scale experiments on environments explored by the LLM planning community using new models.

$Quality:$ This paper has plenty of results that substantiate past findings.

$Clarity:$ The paper is written very clearly.

$Significance:$ Some of the experiments may suggest that newer models deviate from past findings (see questions).
Weaknesses: The main issue with this paper is that it has very little novelty to contribute to the LLM planning community since it reproduces many results and echoes insights that have been communicated in other papers. There is a contrast in narrative to existing work that claims LLMs cannot plan using base models with no inference time techniques [1, 2]; however, plenty of work exists that individually investigate and demonstrate that prompting with examples boosts performance [3, 4], inference-time prompting techniques boosts performance [5, 6, 7, 8], finetuning boosts performance [9, 10, 11], and do similar failure analyses [2, Figure 5].

The conclusion ends with a quick note on 3 key findings regarding plan generalization: (1) the superiority of SFT (2) curriculum learning effectiveness and (3) limitations of hard example training. These points have already been brought up in [12] that uses SFT to tackle plan generalization but uses a better approach of process supervision [13].

In addition, while there are 5 datasets total experimented on in this paper, the experiments focus mainly on Blocksworld and Logistics which has already been focused on in [2], with TravelPlan and CalendarPlan only being used for many-shot and inference-time experiments and MiniGrid only being used for the many-shot experiment. It is unclear why these 5 datasets are introduced as part of a benchmark suite but only subsets are experimented on.

Overall, this paper reads primarily as a summary of well-investigated trends in the LLM planning community. To strengthen this paper, it would be useful to propose an approach atop the experiment findings to add novelty to the paper and further support the insights (for example, the balanced approach incorporating easy and hard examples in the conclusion). Since it would require non-trivial rewriting to improve this paper, I have suggested a rating of reject.

[1] On the planning abilities of large language models-a critical investigation (Valmeekam et. al 2023)

[2] Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change (Valmeekam et. al 2023)

[3] Language Models are Few-Shot Learners (Brown et. al 2020)

[4] Many-shot in-context learning (Agarwal et. al 2024)

[5] ReAct: Synergizing Reasoning and Acting in Language Models (Yao et. al 2022)

[6] Tree of Thoughts: Deliberate Problem Solving with Large Language Models (Yao et. al 2023)

[7] Graph of Thoughts: Solving Elaborate Problems with Large Language Models (Besta et. al 2023)

[8] Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (Zhou et. al 2023)

[9] FireAct: Toward Language Agent Fine-tuning (Chen et. al 2023)

[10] Plansformer: Generating Symbolic Plans using Transformers (Pallagani et. al 2022)

[11] LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks (Kambhampati et. al 2024)

[12] Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping (Lehnert et. al 2024)

[13] Let's Verify Step by Step (Lightman et. al 2023)
Recommendation: 3
Reviewer confidence: 5
Soundness/correctness: 2
Clarity: 3