Planetarium: A Rigorous Benchmark for Translating Text to Structured
  Planning Languages
Authors: name: Max Zuo, name: Francisco Piedrahita Velez, name: Xiaochen Li, name: Michael L. Littman, name: Stephen H. Bach
Organization: Bats Research

Many recent works have explored using language models for planning problems.
One line of research focuses on translating natural language descriptions of
planning tasks into structured planning languages, such as the planning domain
definition language (PDDL). While this approach is promising, accurately
measuring the quality of generated PDDL code continues to pose significant
challenges. First, generated PDDL code is typically evaluated using planning
validators that check whether the problem can be solved with a planner. This
method is insufficient because a language model might generate valid PDDL code
that does not align with the natural language description of the task. Second,
existing evaluation sets often have natural language descriptions of the
planning task that closely resemble the ground truth PDDL, reducing the
challenge of the task. To bridge this gap, we introduce \benchmarkName, a
benchmark designed to evaluate language models' ability to generate PDDL code
from natural language descriptions of planning tasks. We begin by creating a
PDDL equivalence algorithm that rigorously evaluates the correctness of PDDL
code generated by language models by flexibly comparing it against a ground
truth PDDL. Then, we present a dataset of 132,037 text-to-PDDL pairs across
13 different tasks, with varying levels of difficulty. Finally, we evaluate
several API-access and open-weight language models that reveal this task's
complexity. For example, 87.6% of the PDDL problem descriptions generated by
GPT-4o are syntactically parseable, 82.2% are valid, solve-able problems,
but only 35.1% are semantically correct, highlighting the need for a more
rigorous benchmark for this problem.