is there anything useful in this paper? can you distill key findings from it and any useful techniques or approaches that could aid me in my own prompting or problem solving with llms or models?:

```
LLMS STILL CAN’T PLAN; CAN LRMS?
A PRELIMINARY EVALUATION OF OPENAI’S O1 ON PLANBENCH
Karthik Valmeekam∗
SCAI, Arizona State University
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Kaya Stechly∗
SCAI, Arizona State University
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Subbarao Kambhampati
SCAI, Arizona State University
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
ABSTRACT
The ability to plan a course of action that achieves a desired state of affairs has long been considered a
core competence of intelligent agents and has been an integral part of AI research since its inception.
With the advent of large language models (LLMs), there has been considerable interest in the question
of whether or not they possess such planning abilities. PlanBench [1], an extensible benchmark we
developed in 2022, soon after the release of GPT3, has remained an important tool for evaluating
the planning abilities of LLMs. Despite the slew of new private and open source LLMs since
GPT3, progress on this benchmark has been surprisingly slow. OpenAI claims that their recent o1
(Strawberry) model has been specifically constructed and trained to escape the normal limitations of
autoregressive LLMs–making it a new kind of model: a Large Reasoning Model (LRM). Using this
development as a catalyst, this paper takes a comprehensive look at how well current LLMs and new
LRMs do on PlanBench. As we shall see, while o1’s performance is a quantum improvement on the
benchmark, outpacing the competition, it is still far from saturating it. This improvement also brings
to the fore questions about accuracy, efficiency, and guarantees which must be considered before
deploying such systems.
1 Introduction
The recent release of OpenAI’s o1 (Strawberry) [2] brings with it the opportunity to both freshly evaluate progress on
PlanBench and to consider directions for extending the benchmark. In particular, unlike the LLMs which came before it,
which can roughly be viewed as approximate retrievers, o1 seems to have been trained to be an approximate reasoner.1
Following OpenAI2
, we draw a distinction between previous Large Language Models and o1, a Large Reasoning Model
(or LRM), as its new (unknown) architecture, operation, and capabilities all seem to be fundamentally different from
those of vanilla LLMs, both at pretraining phase and at inference time. To properly evaluate this new kind of model and
understand its abilities and limitations will require new tools and evaluation methods, especially if details of the overall
model structure are kept secret and internal traces remain inaccessible to outside researchers.3
Since PlanBench [1] first debuted on arXiv in 2022, we have continually retested new models on one particular
subset: a static dataset of 600 three to five block problems. Even though LLMs have gotten ever larger and required
substantially more investment per model, their performance on even the simplest planning problems has never come
close to saturating this test set, and the improvements we have seen have not been robust or generalizable [6]. Our
benchmark has thus served as a useful marker of LLM progress (or lack thereof) on planning and reasoning tasks,
though it is important to note that this analysis–especially when confined to a static test set–can only serve as an upper
bound on performance. When we first designed PlanBench, it was intended to be more than just this set, but rather to be
∗Equal Contribution
1We speculate that the complete system learns to improve its ability to make appropriate Chain-of-Thought (CoT) moves useful
for reasoning in a pretraining RL step with synthetic data, and does inference time prompt-specific rollouts [3]. In other words, it may
be an RL-trained system in the same vein as AlphaGo, but where the ‘moves’ being generated and evaluated are Chains of Thought.
2
Per the blogpost announcing the model: "A new series of reasoning models [...] for complex reasoning tasks this is a significant
advancement and represents a new level of AI capability. Given this, we are resetting the counter back to 1 and naming this series
OpenAI o1."[4]
3There are reports that OpenAI is threatening to revoke access to o1 from anyone who tries to extract internal reasoning traces [5].
arXiv:2409.13373v1 [cs.AI] 20 Sep 2024
A PREPRINT - SEPTEMBER 23, 2024
Domain Shots Claude Models OpenAI GPT-4 Models LLaMA Models Gemini Models
Claude 3.5
(Sonnet)
Claude 3
(Opus)
GPT-4o GPT-4o
-mini
GPT-4 GPT-4
Turbo
LLaMA
3.1 405B
LLaMA
3 70B
Gemini
1.5 Pro
Gemini
1 Pro
Blocks
world
One
Shot
346/600
(57.6%)
289/600
(48.1%)
170/600
(28.3%)
49/600
(8.1%)
206/600
(34.3%)
138/600
(23%)
284/600
(47.3%)
76/600
(12.6%)
101/600
(16.8%)
68/600
(11.3%)
Zero
Shot
329/600
(54.8%)
356/600
(59.3%)
213/600
(35.5%)
53/600
(8.8%)
210/600
(34.6%)
241/600
(40.1%)
376/600
(62.6%)
205/600
(34.16%)
143/600
(23.8%)
3/600
(0.5%)
Mystery
Blocks
world
One
Shot
19/600
(3.1%)
8/600
(1.3%)
5/600
(0.83%)
0/600
(0%)
26/600
(4.3%)
5/600
(0.83%)
21/600
(3.5%)
15/600
(2.5%)
-
2/500
(0.4%)
Zero
Shot
0/600
(0%)
0/600
(0%)
0/600
(0%)
0/600
(0%)
1/600
(0.16%)
1/600
(0.16%)
5/600
(0.8%)
0/600
(0%)
-
0/500
(0%)
Table 1: Performance on 600 instances from the Blocksworld and Mystery Blocksworld domains across large language
models from different families, using both zero-shot and one-shot prompts. Best-in-class accuracies are bolded.
an extensible suite of tools for evaluating LLM planning capabilities. Now that LRMs score so highly on at least parts
of the original test set, those tools will become ever more important for future evaluations.
In this preliminary evaluation (at time of writing, o1-preview and o1-mini have only been out a week, and the full
o1 model is yet to be released), we examine the performance jump that these new Large Reasoning Models promise.
We record the slow progress we’ve observed in vanilla LLM performance since the release of the benchmark, discuss
o1’s performance, and then tackle the question of how PlanBench’s domains and tests can be elaborated on in order
to remain relevant metrics for LRMs. We argue that, to be complete, new approaches to measuring LRM reasoning
capabilities must take into account efficiency, cost, and guarantees.
2 State-of-the-Art LLMs Still Can’t Plan
PlanBench remains a challenging benchmark for vanilla LLMs (massive transformer models which have been fine-tuned
via RLHF), and their lackluster performance on even our easiest test set leads us to continue to believe that planning
cannot be generally and robustly solved by approximate retrieval alone. In Table 1, we present the results of running
current and previous generation LLMs on a static test set of 600 three to five block Blocksworld problems, as well as on
a set of 600 semantically identical but syntactically obfuscated instances which we call Mystery Blocksworld. Across
these models, the best performance on regular Blocksworld is achieved by LLaMA 3.1 405B with 62.6% accuracy.
Despite the underlying problems being identical, Mystery Blocksworld performance lags far behind–no LLM achieves
even 5% on our test set–and performance on one version of the domain does not clearly predict performance on the
other.4
In the original paper, we tested both natural language prompts and PDDL, and found that vanilla language models
perform better when tested on the former, even though natural language prompts can introduce uncertainty due to
polysemanticity and syntactic ambiguity. To make our comparisons "fair" for the models being tested, the results we
have been reporting are the higher accuracy natural language prompting numbers.
LLMs are highly capable of providing translations between equivalent representations [7]. This fact, combined
with their significantly higher performance on the unobfuscated version of the Blocksworld domain, predicts that–if
they are capable of composing reasoning operations–the performance gap between Mystery Blocksworld and classic
Blocksworld should shrink substantially if the translation from Mystery Blocksworld back into Blocksworld is explicitly
provided. However, when we provide this in the prompt (see Appendix C), performance only improves a very small
amount: GPT-4 achieves 10%.
We also find that, contrary to previous claims, one-shot prompting is not a strict improvement over zero-shot. In fact,
for many models it seems to do significantly worse!5 This is most notable in our tests of LLaMA family models.
4We do not provide Mystery Blocksworld data for Gemini 1.5 Pro only because we haven’t been able to generate it. The model
refuses to produce any output, instead claiming that responding to these queries would be harmful. We include this output in
Appendix A.
5While the reverse is generally true for Mystery Blocksworld problems, it’s important to note that the performance of vanilla
LLMs on Mystery Blocksworld has consistently and uniformly been poor (the same as it was when this benchmark was first released),
2
A PREPRINT - SEPTEMBER 23, 2024
Figure 1: These examples are on Mystery Blocksworld. Fast Downward, a domain-independent planner [8] solves all
given instances near-instantly with guaranteed perfect accuracy. LLMs struggle on even the smallest instances. The
two LRMs we tested, o1-preview and o1-mini, are surprisingly effective, but this performance is still not robust, and
degrades quickly with length.
Domain Shots Instances correct Average Time Taken (in secs)
o1-preview o1-mini Fast Downward o1-preview o1-mini Fast Downward
Blocksworld Zero Shot 587/600
(97.8%)
340/600
(56.6%)
600/600
(100%) 40.43 10.84 0.265
Mystery
Blocksworld
One Shot 247/600
(41.6%) -
600/600
(100%) 82.03 - 0.265
Zero Shot 317/600
(52.8%)
115/600
(19.1%)
600/600
(100%) 83.37 35.54 0.265
Randomized Mystery
Blocksworld Zero Shot 224/600
(37.3%) -
600/600
(100%) 111.11 - 0.265
Table 2: Performance and average time taken on 600 instances from the Blocksworld, Mystery Blocksworld and
Randomized Mystery Blocksworld domains by OpenAI’s o1 family of large reasoning models and Fast Downward
The original iteration of our benchmark did not take efficiency into consideration, as the time taken by a vanilla LLM to
produce some output is only dependent on the length of that output, but otherwise independent of the semantic content
or difficulty of the instance. However, as LRMs adaptively vary their time taken and dollar cost per instance in response
to the input, measuring efficiency has become much more important. As a comparison point between LRMs and LLMs,
we compute prompting costs across models and present them in Table 4.
3 From Approximate Retrieval to Approximate Reasoning: Evaluating o1
Many researchers, including us, have argued that "standard" autoregressive LLMs generate outputs via approximate
retrieval, and that, while they show impressive performance on a range of System 1 tasks, they are unlikely to achieve
more System 2-like approximate reasoning capabilities critical for planning tasks (c.f. [9]). Until now, the best way to
coax sound planning capabilities out of LLMs has been to pair them with external verifiers in a generate-test framework,
so those results do not provide too clear a picture. Most models do not solve even a single instance in zero-shot mode, and only one
(LLaMA 3.1 405B) manages more than one.
3
A PREPRINT - SEPTEMBER 23, 2024
(a) Average number of reasoning tokens used by o1-preview
when solving Mystery blocksworld instances correlates with
the length of the optimal plan required to solve the instance.
(b) Average number of reasoning tokens used by o1-preview
does not increase as expected with the number of steps
required to solve the instance.
Figure 2
in what are called LLM-Modulo systems [10, 11]. o1 attempts to supplement an underlying LLM with System 2-like
abilities in a different way.
As far as we can tell, o1 combines an underlying LLM, most likely a modified GPT-4o, into an RL-trained system that
steers the creation, curation, and final selection of private Chain-of-Thought reasoning traces. Exact details are currently
sparse, and so we can only speculate about its exact mechanisms. Our best guess is that there are two major differences
between o1 and LLMs: an additional reinforcement learning pre-training phase (perhaps to learn the q-values of
different CoTs from massive amounts of synthetic data) and a new adaptively scaling inference procedure (maybe it
further refines learned q-values by something like rollout before selecting a particular CoT; see [3]). Regardless, what
looks clear from the detail available is that this model is fundamentally different in nature from previous LLMs.
Evaluating LRMs on the Original Test Set: We test o1-preview and o1-mini on the static PlanBench test set.6
The full results can be seen in Table 2. Our 600 Blocksworld instances range from three to five blocks, and require
plans of between 2 to 16 steps to solve. Far surpassing any LLM, o1 correctly answers 97.8% of these instances.
On Mystery Blocksworld, the model does not maintain this level of performance, but it does far surpass all previous
models, answering 52.8% correctly. To test whether the exact obfuscation might be compromised because of data
contamination, we also generated a new obfuscation using completely random strings, and presented these problems in
a new, semantically equivalent prompt format with fully specified and unambiguous PDDL descriptions of both the
domain and problem. This is presented in the table as Randomized Mystery Blocksworld. While performance did dip
further, 37.3% of instances are answered correctly, sharply contrasting the flat zeroes we saw when testing previous
models.
Accuracy with Increased Problem Size: Standard LLM chain-of-thought prompting approaches are brittle, do not
robustly scale with problem size, and fail to induce general algorithmic procedure-following [6]. We test the models on
a set of 110 larger Blocksworld problems (see Figure 3). Problems in this set range from 6 to 20 blocks in length and
require 20 to 40 step optimal plans. Without any obfuscation, we see performance quickly degrade from the 97.8%
reported earlier. In fact, over these 110 instances, o1-preview only manages 23.63%, and most of this accuracy comes
6While for previous models, we had the model itself enforce the plan format we wanted, some modifications had to be made
to accurately test o1’s abilities. In its current form, o1-preview does not always conform to explicit formatting restrictions. This
is right in line with OpenAI’s injunction to keep o1 prompts "simple and direct" [12]. In order to extract the generated plans, we
used GPT-4o-mini to translate them into PDDL, and wrote a small Python parser to strip any remaining extraneous symbols before
evaluating each proposed plan.
4
A PREPRINT - SEPTEMBER 23, 2024
Figure 3: Extending even the (regular, not obfuscated) Blocksworld dataset to problems requiring greater numbers of
steps worsens the performance of o1-preview. When tested on 110 instances which each require at least 20 steps to
solve, it only manages 23.63%.
Domain Shots o1-preview
True Negatives False Negatives
Blocksworld 0-Shot 27% 0%
Randomized Mystery
Blocksworld 1-Shot 16% 11.5%
Table 3: Rate of claiming that a problem is impossible by OpenAI’s o1-preview on 100 unsolvable and 600 solvable
instances in the Blocksworld and Randomized Mystery Blocksworld domains. The True Negative rate is the percent
of unsolvable instances that were correctly marked as unsolvable. The False Negative rate is the percent of solvable
instances that were incorrectly marked as unsolvable. Previous models are not shown in this table as their true negative
and false negative rates were generally 0% across the board.
from correctly solving problems which require fewer than 28 steps. While these models are overall impressive, this
shows that their performance is still far from robust.
Performance on Unsolvable Instances: While planning problems normally require the agent to formulate a course
of action to achieve a goal, an equally valid use of planning abilities is to recognize that a given goal cannot be
accomplished by any plan. A real-world example of this is network vulnerability analysis, where an agent may wish
to certify that no plan of attack exists for a specified system [13]. So far, LLMs have struggled to recognize that
some problems cannot be solved, instead confidently confabulating nonsensical answers. o1 was launched with the
claim that it has started to overcome this issue, and can now accurately identify unsolvable problems [14]. To test this
systematically, we modified 100 instances from our original three to five block test set by adding one on(x,y)-type
conjunct to each instance’s goal state, making the goal unsatisfiable.7 The results are in Table 3. On Blocksworld, only
27% of all instances were correctly and explicitly identified by o1 as unsolvable. In 19% of all cases, the model returned
a dot or some kind of "[empty plan]" marker, without any explanation or indication of unsolvability. We consider these
incorrect, as "empty plan" is only the correct answer if the goal is already satisfied. In the remaining 54% of cases, the
model generated a full (and therefore impossible and incorrect!) plan.
On Randomized Mystery Blocksworld, these numbers are worse: 16% of cases were correctly identified as unsolvable,
5% returned an empty plan, and the remaining 79% were answered with a full (impossible or goal-unsatisfying) plan.
Therefore, unsolvable instances continue to be a problem for LRMs. Furthermore, this ability to sometimes note
impossible plans correctly comes at a cost: now the model sometimes falsely claims that solvable problems are actually
unsolvable. On Randomized Mystery Blocksworld, 11.5% of instances are incorrectly claimed to be impossible. These
results can be seen in Table 3.
7We ensured our instances were unambiguous by giving the full PDDL representation of both the domain and the instance, to
avoid quibbles such as "A is on B because A is on C and C is on B" where the model redefines the meanings of an ambiguous natural
language statement.
5
A PREPRINT - SEPTEMBER 23, 2024
Costs per 100 instances (in USD)
Large Language Models Large Reasoning Models
Claude 3.5
(Sonnet)
Claude 3
(Opus)
GPT-4o GPT-4o
-mini
GPT-4 GPT-4
Turbo
Gemini
1.5 Pro
Gemini
1 Pro
o1-preview o1-mini
$0.44 $1.70 $0.65 $0.02 $1.80 $1.20 $0.33 $0.03 $42.12 $3.69
Table 4: Cost per 100 instances (in USD). LRMs are significantly more expensive than LLMs.
Accuracy/Cost Tradeoffs and Guarantees: With LRMs showing better performance on planning problems, our
evaluations must explicitly take into account the trade-offs that come from choosing general models over established
deep and narrow systems. While o1-preview may provide higher accuracy than LLMs, it still fails to provide any
correctness guarantees, and it is unclear that it is at all cost-effective. Unlike previous models, whose APIs only charge
based on the number of input tokens and the number of output tokens (usually at a rate that is five times higher for the
latter), o1’s price-per-call includes a surcharge based on the number of "reasoning tokens" it used–tokens generated as
part of inference and not revealed to the user–which are charged at the significantly higher output token rate. Currently,
end users have no control over the number of these tokens generated, a number which is expanded or limited by the
model in its own opaque way. Since the launch of these models less than a week ago, we have already run up a bill of
$1897.55 for just the o1 model experiments on this benchmark!8
)
The early version of o1-preview that we have access to seems to be limited in the number of reasoning tokens it uses per
problem, as can be seen in the leveling off in Figure 2 and more clearly from the scatterplot in Appendix 4. This may be
artificially deflating both the total cost and maximum performance. If the full version of o1 removes this restriction, this
might improve overall accuracy, but it could also lead to even less predictable (and ridicuously high!) inference costs.
o1-mini is cheaper, but generally less performant.
Without exposing the ability to scale inference time to particular specifications, influence the internal ‘thinking’ process
in task-specific ways, or ensure that intermediate steps are evaluated by trusted or sound verifiers, the o1 models are a
coarse-grained choice in the space of cost, inference time, guarantees, and performance trade-offs. They aren’t, however,
the only choices in that space, and reasonable LRM evaluations must take this into account (see similar arguments in
[15, 16]).
Classical planners like Fast Downward [8] achieve 100% on our dataset in a fraction of the time, compute, and cost,
while providing guarantees that their answers are correct. Running Fast Downward on a personal computer was
essentially free in dollar terms and averaged 0.265 seconds per instance, which is many orders of magnitude faster than
the average o1 clock times listed in table 2. It is also generally predictable, and can be scaled to harder instances very
directly. Vanilla LLMs are typically very good at translating problems between formats, and could be used to do so in
concert with a classical planner at a fraction of the cost of LRMs (e.g. [7, 17]). For problems which don’t have simple
PDDL domain and instance specifications, LLM-Modulo systems may be a safer and cheaper approach: run a smaller,
faster LLM in a loop with a sound verifier, so that the combined system will only output guaranteed correct solutions
(e.g. [10, 18, 11]). Previous testing showed that this LLM-Modulo approach can already achieve 82% accuracy on a
subset of our original Blocksworld test set, as well as 70% in the Logistics domain (see Table 4 in the Results section of
[19]).
The correctness guarantees provided by these latter two methods are sorely lacking in LRMs like o1. A general
reasoning system cannot be deployed in safety critical and non-ergodic domains if it continues to confidently make
incorrect plans. o1 is a fully black box system, even more so than previous models, and OpenAI’s decision to not only
keep the architecture under wraps and hide the reasoning traces, but to warn away and even ban anyone who attempts to
understand what is going on inside them [5], makes interpretability nearly impossible, and reduces trust in the system
overall.9
o1’s Creative Justifications: While our main focus has been on providing a quantitative evaluation of o1’s performance on PlanBench, we have also noticed an o1 idiosyncracy that is worth commenting on. When the model gives an
8The rich irony of researchers using tax payer provided research funds to pay private companies like OpenAI to evaluate their
private commercial models is certainly not lost on us.
9The current model is also set to a default temperature of 1.0, which further reduces replicability and interpretability–for any
given problem, it is never clear whether the result is merely the result of stochasticity. This compounds a problem with OpenAI
models that has existed since at least GPT3. Temperature 0 never gave deterministic outputs, and worse, the logprobs provided by
the OpenAI API for any given prompt have long been known to fluctuate wildly [20].
6
A PREPRINT - SEPTEMBER 23, 2024
incorrect answer, it also sometimes provides a creative, but nonsensical, justification for its decision. It is almost as if
o1 has gone from hallucinating to gaslighting! In one case, it decided that an unsolvable problem is solvable because
a goal condition, while not present in the final state, had been true for at some point during the execution, and thus
should continue to count. In another, it declared that on(a,c) was true because, as it explained in a brief parenthetical,
a was on b which was on c, and was thus a was somewhere above c, which should count as being "on top" of it. As we
mentioned earlier, we changed our unsolvable instance prompts from natural language to PDDL in order to make it
extremely clear that divergences from our exact definitions were disallowed.
4 Conclusion
We took a fresh look at the planning capabilities of both SOTA LLMs, and examined the performance of OpenAI’s new
o1 models on PlanBench. Over time, LLMs have improved their performance on vanilla Blocksworld–with the best
performing model, LlaMA 3.1 405B, reaching 62.5% accuracy. However, their dismal performance on the obfuscated
("Mystery") versions of the same domain betrays their essentially approximate retrieval nature. In contrast, the new o1
models, which we call LRMs (Large Reasoning Models)–in keeping with OpenAI’s own characterizations–not only
nearly saturates the original small instance Blockworld test set, but shows the first bit of progress on obfuscated versions.
Encouraged by this, we have also evaluated o1’s performance on longer problems and unsolvable instances, and found
that these accuracy gains are not general or robust. We also discussed the critical accuracy/efficiency tradeoffs that are
brought up by the fact that o1 that uses (and charges for) significant inference-time compute, as well as how it compares
to other LLM-based approaches (such as LLM-Modulo [10]) and dedicated solvers. We hope this research note gives a
good snapshot of the planning capabilities of LLMs and LRMs as well as useful suggestions for realistically evaluating
them.
```