ActionReasoningBench: Reasoning about Actions with and without Ramification Constraints
Authors: Divij Handa
Pavel Dolin
Shrinidhi Kumbhar
Tran Cao Son
Chitta Baral
Venue: ICLR 2025 Poster (2025)
Keywords: Reasoning about Actions and Change (RAC)
Benchmark
Large Language Models (LLMs)
Primary area: datasets and benchmarks
TL;DR: The paper introduces ActionReasoningBench, a diagnostic benchmark for Reasoning about Actions and Change (RAC) covering novel complex questions and indirect effects of actions, known as ramifications.

Abstract: Reasoning about Actions and Change (RAC) has historically played a pivotal role in solving foundational AI problems, such as the frame problem. It has driven advancements in AI fields, such as non-monotonic and commonsense reasoning. RAC remains crucial for AI systems that operate in dynamic environments, engage in interactive scenarios, or rely on commonsense reasoning. Despite substantial advances made by Large Language Models (LLMs) in various AI domains, their performance in RAC remains underexplored. To address this gap, we introduce a new diagnostic benchmark, $\textbf{ActionReasoningBench}$, which encompasses 8 domains and includes questions for up to 19 action sequences. This benchmark rigorously evaluates LLMs across six key RAC dimensions: $\textit{Fluent Tracking}$, $\textit{State Tracking}$, $\textit{Action Executability}$, $\textit{Effects of Actions}$, $\textit{Numerical RAC}$, and $\textit{Composite Questions}$. LLMs demonstrate average accuracy rates of 73.55%, 65.63%, 58.73%, and 62.38% on the former four dimensions, which are frequently discussed in RAC literature. However, the performance on the latter two dimensions, which introduce complex and novel reasoning questions, the average performance of LLMs is lowered to 33.16% and 51.19%, respectively, reflecting a 17.9% performance decline. We also introduce new ramification constraints to capture the indirect effects of actions, providing deeper insights into RAC challenges. Our evaluation of state-of-the-art LLMs, including both open-source and commercial models, reveals challenges across all RAC dimensions, particularly in handling ramifications, with GPT-4o failing to solve any question and o1-preview achieving a score of only 18.4%.

Published as a conference paper at ICLR 2025
ACTIONREASONINGBENCH: REASONING ABOUT AC- TIONS WITH AND WITHOUT RAMIFICATION CON- STRAINTS
Divij H Pavel Pavel Do Shrinidhi Kumbhar" Tran Cao Son", Chitta Baral Arizona State University, New Mexico State University
{ [chanda, pdolin, skumbha4, chitta)@asu edu, st strannnnsse edu
ABSTRACT
Reasoning about Actions and Change (RAC) has historically played a pivotal role in solving foundational problems, such as the frame problem. has driven advancements in AI fields, such non-monotonic and commonsense reasoning. RAC remains crucial for AI systems that operate in dynamic environments, en- gage in interactive scenarios, rely commonsense reasoning. Despite sub- stantial advances made by Large Language Models (LLMs) in various Al do- mains, their performance we introduce a new diagnostic benchmark, ACTION BENOONRRRSONNGBEENCCEC encompasses 8 domains and includes questions for up to 19 action sequences. This benchmark rigorously evaluates LLMs across six key RAC dimensions: Flu- ent Tracking. State Tracking, Action Executability, Effects of Actions, Numerical RAC, and Composite Questions. L.MS demonstrate average accuracy rates of 73.55% 65.63%, 58.73%, and 62.38% on the former four dimensions, which are frequently discussed in RAC literature. However, the performance the lat- ter two dimensions, which introduce complex and novel reasoning questions, the average performance of LLMs is lowered 33.16% and 51.19%, respectively, reflecting a 17.9% performance decline. We also introduce new ramification con- straints to capture the indirect effects of actions, providing deeper insights into RAC challenges. Our evaluation of state-of-the-art LLMs, including both open- source and commercial models, reveals challenges across all RAC dimensions, particularly in handling ramifications, with GPT-4o failing to solve any question and ol ooerreveww achieving a score of only 18.4%.
RAC remains underexplored. To address this gap, which
INTRODUCTION
Reasoning about actions and change (RAC) is fundamental problem in artificial intelligence, with its roots tracing back to early work from the 1960s (McCarthy et al. 1963). Initially, research focused on developing logical systems capable of effectively modeling and reasoning about actions and their effects in dynamic environment. One of the significant challenges in this domain has been succinctly expressing how actions influence changeable properties of the world, known as fluents. For example, consider the statement: "Moving object from location X to location Y results in describe the direct effects
the object being at location Y." While it is relatively straightforward on the affected fluents, such as the object's location, it is much more complex to account for the unaffected fluents, a challenge known as the frame problem. This challenge becomes exacerbated when the descriptions involve relationships between fluents in a state, such as "an object can not be at two different places at the same time". While such constraints simplify action descriptions by decoupling them from fluents, they introduce indirect effects, ramifications. For example, the statement "A block is said to be clear if there isn't any block on top of it" describes a ramification fluent, "elear" dependent on another fluent "on top of."
took multiple decades research to create a comprehensive logical formalization that adequately addressed these issues. It involved the laborious creation of numerous handcrafted rules and logic
"Equal contribution.

---

Published as a conference paper at ICLR 2025
detailing the effects and preconditions of actions (Reiter 2001). However, these tools are limited since they rely on manual effort to translate natural language descriptions of actions and their effects into formal logic representations. To address this challenge, recent research in natural language processing (NLP) has begun exploring the capabilities of LLMs in RAC tasks, as demonstrated by works of He et al. (2023), Spilliopoulou al. (2022), and [aanereee et al. (2020). However, these studies have not systematically decomposed the complex RAC problem into multiple categories and overlook the critical ramifications of actions seen in real-world scenarios. To address this gap, introduce ACTIONREASE modern state-of- the-ehr art LLMs struggle.
a diagnostic RAC benchmark that aims to pinpoint where
We decompose the RAC task into six distinct categories- FFuent Tracking, State Tracking, Action Executability, Effects of Actions, Numerical RAC, and Composite Questions. The first four cate- gories focus on assessing fundamental aspects of RAC, while the latter two introduce more complex and novel question types. The questions every category span action sequences ranging from to 19 steps, allowing to test the RAC capabilities at series of action sequence ranges. Additionally, we introduce ramification constraints to represent the indirect effect of actions. These constraints simplify action descriptions and align more closely with real-world conditions but introduce ad- ditional complexity, as highlighted by Mcllraith (2000). Specifically, we expand the domains by adding ramification fluents with dependencies up to four levels deep, where actions propagate their effects through multiple layers.
Highlights of benchmark, benchmarks on RAC, are presented in Table [ We evaluate four LLMs-two open-source models, -lama-3.J-BB-Inttruee and Llama-3.1 1-0..------mm Instruct (Dubey al. 2024). as well as two leading proprietary models, GPT-4o (Achiam et al. 2023) and ol op-revieww (OpenAl (2024). These LLMs across various RAC categories under different prompt were tested on Zero-shot-Cor CoT (Kojima et al. (202)) and 3 (Brown. 2020), to assess settings, including how performance varies based on these contigurations.
along with the comparison previous
Our findings indicate that LLMs face substantial challenges, particularly when addressing complex RAC ques- tions. The average performance of all LLMs on the complex categories de- creases by 17.88% compared to the first four basic categories. The best performing I.M,, GPT-4o, achieves an average accuracy of 59.91% % on these categories. Notably, GPT-4o failed to produce any correct an- swers for questions involving ram- ifications constraints, while the ol- preview model achieved accuracy of only 18.42%. Performance was es- pecially poor in categories like Ac- tion Executability. Numerical RAC, and Composite Questions, with furr ther declines observed as the length of action sequences increased. Ad- ditionally, LLMs struggled with rea- soning in scenarios that combined both true and false fluents, experiencing an average performance drop of 12.16% compared to tasks involving exclusively true or false fluents.
PlanBench
TRAC
(Ours)
Number of domains Number of queries Max Action Sequence length Max number of objects Binary Questions (TVF)
2
1
26k
15k
152k
48
3
19
24
5
28
V
Free Answers
X
State Tracking Action Executability
Y
Fluent Tracking Effects of Actions Numerical Reasoning Composite Questions Ramifications Constraints Subcategories Fluents
X
X
X
X
Table 1: Differences between ACTIONREASONINGGEENG (Ours) and previous benchmarks on RAC. PlanBench (Valmeekam et al. 2024): TRAC (He al. 2023)
2 RELATED WORKS
Benchmarking reasoning capabilities of LLMS Evaluating the reasoning capabilities of LLMs
using synthetic datasets has become key focus in NLP, with increasing efforts to create challenging
benchmarks. Notable areas of interest include the legal reasoning (Fei et al. 2023 Guha et al. 2023),
logical reasoning (Luo et al. 2024 Han et al. 2024 Patel al. 2024
Parmar et al. 2024) arithmetic
2

---

Published as a conference paper at ICLR 2025
Fatemi it al. 2024), and commonsense reasoning (Onoe et al. 2021 Lin et al. 2021 Geva et al. 2021
reasoning (Cobbe et al. 2021 Miao et al. 2021 temporal reasoning (Uddin et al. 2024
Lourie et al. 202I). Despite this progress, RAC remains significanly under-explored, even though it plays a crucial role in several of these reasoning tasks, such as commonsense and legal reasoning.
To fill this gap, create ACTIONREASONINGEENCT using synthetically generated data.
Evaluating RAC and Planning Benchmarking planning capabilities of LLMs is well-studied area, with recent works (Zheng et al., 2024) Xie et al., 2024) demonstrating the challenges LLMs face. While planning is son-polynomial problem that remains inherently difficult to solve, believe that RAC, which is a polynomial problem, is a prerequisite for effective planning. Without comprehending the effects of actions, LLMs are unlikely construct coherent plans. In this work, we address this gap by proposing ACTIONREASONIGGEENCH a diagnostic benchmark for RAC.
Previous research, such [Banerjee et al. (2020). has investigated RAC capabilities in models like RoBERTa, focusing primarily on binary questions or single-word answers. Extending this, He al. (2023) assessed LLMSs on broader range of question categories. However, only two directly to RAC, with the remainder addressing the broader domain of planning. Similarly. (aalmekam et al. (024) introduced a bench- mark evaluating both RAC and plamning, with detailed emphasis on planning tasks but maintaining a limited focus on RAC categories, specifically State Tracking and Action Executability. A compar- ative analysis of the benchmarks from He et al. (2023), Valmeekam et al. (2024), and our proposed benchmark summarized in Table [
of these- Action Executability and State Tracking-pertan
Parallel to our work, Kokel et al. (2024) introduced ACPBench to evaluate both RAC and plan- ning. While ACPBench overlaps with our work, it primarily covers fundamental aspects of RAC. In contrast, our benchmark introduces greater complexity by incorporating categories such as State Tracking, Numerical RAC, Composite Questions, and constraints related to ramifications.
In this work, we concentrate on RAC and develop a benchmark that encompasses wider range of categories, enabling more precise identification of areas where LLMs underperform. Furthermore, our benchmark introduces novel constraints involving ramifications, incorporating the indirect ef- fects of actions. To the best of our knowledge, ramification constraints have not been addressed in any existing benchmark, marking a significant advancement in the evaluation of RAC capabilities.
ACTIONREASONINGEEER
This section provides a detailed overview of our benchmark, including its categorization, creation methodology, and validation process. A sample instance is presented in Appendix B where we also describe the objects, actions, and fluents within the domain.
3.1 QUESTION CATEGORIES
In our benchmark, the questions are organized into six distinct categories, each designed to assess specific dimension of RAC. Below, we provide detailed description of each category.
1. Fluent Tracking Given the initial state and the sequence of actions performed, this cate- gory contains questions about the fluents, i.e. properties of the domain, of an object from the changed state. For instance, in the Grippers domain, fluent--racking question might be "List all valid properties associated with ball2."
2. State Tracking This category extends the concept of Fluent Tracking. It involves query- ing about the complete set of fluents in the final state. For instance, in the Blocksworld domain, a state- tracking question might be "What are all the valid properties in this state?"
3. Action Executability This category encompasses two types of questions related to exe- cutability of actions. The types of questions within this category are as follows:
(a) Given an initial state, and sequence of actions, the question focuses on identifying the first action in the sequence that is not executable.
(b) Given an initial state and a sequence of actions leading to a final state, the task is to identify all actions that can be executed in the final state. For instance, in the Visitall
3

---

Published as a conference paper at ICLR 2025
domain, an action-exccutability question might be "List all executable actions present in the current state."
4. Effects of Actions-
This category contains questions that explore the outcomes of perform- ing specific action. For instance, in the Mystery domain, an Effects- -oo------atoooo question can be "From the current state, the vehicle vO moves from location to 10, and has fuel-level f6 and J5, which properties of the state will be true now?"
5. Numerical RAC - Questions requiring a numerical response fall under this category. These questions may derive from any of the four previously mentioned categories. For example, in the Spanner domain, a Numerical-RAC question can be "What are the number of exe- cutable actions in the current state?"
6. Composite Question This category contains questions that integrate multiple above- mentioned categories, combining up to three distinct categories. These questions require multiple steps of reasoning to arrive the correct solution. For example, in the Satellite domain, a composite question may combine aspects of Fluent Tracking and Action Exe- cutability. An example of such question could be "List all the properties of the state for satelliteO before the first infeasible action in the sequence?"
3.2 FLUENT CATEGORIES
We further divide the fluents of all 8 domains into three distinct categories, each representing different aspect of ramifications within RAC.
1. Static Properties- - These properties remain unchanged regardless of any action performed. For instance, the property "connected" in the Visitall domain represents whether two loca- tions are connected, a relationship that remains constant irrespective of the robot's 's action, which may involve moving, picking up, or placing down objects. In this domain, the con- nectivity between locations remains unchanged, irrespective of any action.
2. Base Fluents These fluents can change as direct result of an action and do not depend on other fluents. For example, in the Grippers domain, the fluent "carry" indicates whether an object is being carried by a robot's 's gripper. This fluent can change if the action pick or drop is performed.
3. Ramification Fluents These fluents are influenced indirectly by other fluents, and action descriptions do not explicitly dictate their changes. Instead, they are determined by the dependencies and relationships within the system. Ramification fluents are further divided into two sub-categories:
(a) Derived Fluents These fluents rely the state of other fluents, reflecting level of dependency. Changes to them occur as a consequence changes in the fluents they depend on rather than through direct action. For instance, the fluent "stable" in the Blocksworld domain is considered a derived fluent as its state depends on the fluents "clear" and "on table.. This relationship can be described as: "Blocks are stable when clear and on the table". Furthermore, fluent "clear" " is itself a derived fluent, dependent on the fluent "on" ", which makes "stable" second-level indirect effect.
(b) Self-Derived Fluents These fluents rely on constraints related themselves rather than other fluents. For example, in the Depot domain, the fluent "at" represents the location of a truck, which can only be at one location at any given time. If the truck is at location 10. it cannot simultaneously be at location 11. Such constraints plicitly included in the domain description, for example, "A truck can only be in one location al a time".
Classification of every fluent across all 8 domains can be found in Appendix Furthermore, for each fluent type, we generate questions involving negative fluents, i.e. fluents that are false, which allows us to evaluate LLM's comprehension of negation within RAC contexts.
3.3 DATASET STRUCTURE AND VARIATIONS
Selected Domains ACTIONREASONING requires domains that facilitate the evaluation of L.LMs on both short and long sequences of meaningful actions, where the effects and preconditions
4

---

Published as a conference paper at ICLR 2025
of these actions succinetly described. Additionally, these domains should reflect real-world scenarios. To meet these criteria, we selected 8 domains Blocksworld, Depots, Driverlog, Crippers, Mystery, Satellite, Spanner, and Visitall-soureea from the International Planning Competition (IPC), covering the years 1998 to 2014. These domains are commonly used as benchmarks for evaluating advanced planning systems and provide robust foundation for research in automated planning. Appendix G provides a detailed description of each domain. Notably, even state-of-the-ar LLMs like GPT-4o are not capable of generating diverse domains action sequences that conform the precise constraints outlined in these domain descriptions, justifying the reliance on IPC domains.
Domain Descriptions and Ramifications The domains provided by the IPC are described using the Planning Domain Definition Language (PDDL), a formal language designed to model deter- ministic actions and state transitions for planning problems. Further information on PDDL can be found in Appendix H. In this study, initially translated the PDDL-based domains into nat- ural language. Subsequently, ramification constraints were introduced into these natural language descriptions. The process underwent validation by two experts in the RAC domain
correct- ness. Given that the category Action Executability focuses on determining whether the action can be performed rather than analyzing its effect, we concentrated on the categories Fluent Tracking, State Tracking. and Effects of Actions when introducing ramifications.
Action-Sequence Lengths In order to fine-tune tune LLMs, we generate a comprehensive set of ques- tions that span various action-sequee -eeton--equene lengths, specifically 1, 5, 10, 15, and 19. This range is chosen to capture increasing complexities in RAC. We curate a distinct subset of questions with action-sequence lengths of 1, 10, and 19 for evaluation. This subset is selected to assess the model's performance across the action sequences range.
Answer Types We formulate two distinct types of questions based on the expected answer format. The first type consists of binary questions, where the response is either True or False. The second type involves subjective answers, which encompass a range of multiple objects, actions, or fluents.
3.4
DATA CREATION & VALIDATION
Stage Generating Instances
Stage 2 Converting to ASP
Stage 3 Generating State Space
Stage 4 Generating Questions
IPC Domain (PDDL)
Templated Script
IPC Domain (ASP)
Templated Question
ASP Code
Templated Questions
Templated Questions 19
Initial State
Final State
PDDL Instances Plans
Instance Solver
0
Llama-3.1- 70B-Instruct
Templated Script
Action-State Space every Action (from 1 to 19)
Plan ASP Instances Plans
Validator
NL Questions Len
NL Questions Len 19
Figure 1: Overview of the question generation pipeline for ACTIONREASONINGEENCT Blue blocks represent "Generated Data", and green blocks represent "Code used in the pipeline". Stage involves generating states and plans using Helmert (2006) and validating them with Howey et al.) (2004). Stage 2, PDDL instances and plans are converted to ASP. Stage computes the action- state space through ASP. Stage generates questions using templates, which are then rephrased to natural language via Llama-3.1 1-----O--Inttuue
The question generation process follows four-stage pipeline, illustrated in Figure 1 The lected domains from the IPC are represented in PDDL (see Appendix G for examples). First, these PDDL representations are used to generate 10 pairs of initial and goal conditions. A PDDL solver
5

---

Published as a conference paper at ICLR 2025
(Helmert 2006) and validator (Howey al. 2004) then employed to obtain and validate the action sequences necessary to transition from the initial to the goal state. In the second stage, the PDDL domains, instances, and action sequences are converted into Answer Set Programming (ASP) descriptions using Python-based templates.
In the third stage, ASP solvers used to generate the action-state space and extract fluents for each state, along with identifying all executable and inexecutable actions. Further details on these formal languages are provided Appendix H Finally, the fourth stage involves converting the action-state data into questions using a Python template. Up to three natural language variations are created for every object, action, and fluent. These templated sequences are then paraphrased to -.--ttt--nstt to ensure they sound natural and avoid repetition in long action sequences. Three independent annotators review both the templated and the paraphrased versions to assess their naturalness, as detailed in Appendix D Additionally, all eight domain descriptions were manually translated from PDDL to natural language.
Test Set
Train Set
3.5
DATA SPL.TS
Fluent Tracking
438
57,906
The benchmark was divided into two parts: one for training and the other for testing the LLMs, ensuring balanced representation of question categories across the domains. The Com- posite Questions category is slightly larger in the test set, as it combines multiple categories, leading increased questions. Table 2 pro- vides an overview of the distribution of ques- tions and their corresponding categories across both the training and testing sets. The test set contains 3,498 questions, including 2,195 bi- nary and 1,303 free-answer questions. Finally, we designed both zero-shot-CoT and few-shot- 3 prompts for all the questions in the test set.
State Tracking
382
12,636
Action Exccutability
450
9,562
Effects of Actions
417
8,939
Numerical Reasoning
414
31,506
Composite Questions
1,397
28,688
Static Properties Base Fluents
237
12,458
231
10,461
Derived Fluents Self-Derived Fluents
366
15,946
390
23,436
Mixed Fluents
2,274
86,936
Total Unique Questions 3,498
149,237
Table Overview of the test and train sets across Question and Fluent Categories. The "Mixed Flu- ents" category represents questions that involve more than one type of fluent.
4 EXPERIMENTS AND EVALUATION
Models To evaluate our benchmark, we tested four LLMs and employed two prompting tech- niques. The LLMs include two proprietary models-GPT-40 (Achiam et al. 2023) and ol -preview and Llama-3.1-7088
(OpenA] 2024)-alongside two open-source models, LJama-3.1-8BB-11
Instruct (Dubey et al. 2024). Each LLM was evaluated using both few-shot prompting with three
examples (few-shot-3) Brown (2200) and zero-shot-CoT Kojima al. (2022) prompting.
While the entire dataset requires reasoning abilitics, the ramification subset involves the most com- plex and challenging reasoning tasks. Given that ol-preview is specifically optimized for reasoning tasks and incurs significantly higher costs compared to GPT-4g") we restricted its use to the ramifi- cation subset, where its superior reasoning capabilities are expected provide the greatest benefit. Utilizing ol oprrevee across the entire dataset would not be cost-effective, as its advantages would be less pronounced for simpler reasoning tasks.
To ensure a fair evaluation of the standalone reasoning capabilities of LLMs, we deliberately avoided incorporating external tools or systems. Although integrating formal solvers (e.g., PDDL-based planners) with LLMs could potentially improve performance, our primary focus is on assessing the intrinsic reasoning abilities of these models.
Evaluation & Metrics ACTIONREASONINGEENT
includes two types of answer formats, as outlined in Section 3.3) binary (true/false) and free-form responses. The evaluation process was per- formed separately for each answer type. For binary questions, we extracted "true" and "false" key- words from the model's response and compared them to the ground truth. Since free-form answers can't be evaluated using exact string matching, we employed human evaluation for the ramification
As of Oct 2024, ol-preview is six times more expensive than GPT-4o https: / com/api/ pricing/

---

Published as a conference paper at ICLR 2025
questions. While human evaluation is highly accurate, it is not scalable, used Llama-3.1-70 Instruct to assess all free-form responses. The specific prompt used to evaluate the LLMs and the correlation between Llama-3.1-70B 1...-700--Insiiie Inntruuttt and human evaluations are detailed in Appendix(F] For all experiments, we report the accuracy along with the standard error the mean (SEM), calculated SEM yn where represents the standard deviation, and n. is the sample size.
Fine-tuning We fine-tuned the Llama-3. 3.1-8B model using the training data split outlined Sec- tion 3.5 Due to the limited computing power, excluded any data samples that exceeded context length of 4096 tokens. The fine-tuning process was performed separately for free-response and bi- nary questions. Detailed information the fine-tuning procedure is provided in Appendix All experiments were executed using 8xH100 GPUs.
5 RESULTS AND DISCUSSION
This section presents the results and analysis using ACTIONREASONINGEENCH The Zero-shot- CoT results for each LLM on both the binary and free-response subsets of the test set are provided in Tables 5 and 3 respectively. Similarly, the Few-shot-3 results for each LLM evaluated on these same subsets are displayed in Tables and 7 The detailed analysis of the effects when ramification constraints are incorporated into the descriptions is discussed in Section 5.1
Action Seq.
Ques Categories
GPT-4o
Llama-8B-Inst
Llama-70B-Ins
Finetuned Llama-8B
Fluent Tracking
88.46 88.464.43
30.776.40
71.156.25
76.92;
State Tracking
73.33
28. 89..7
61.447.14
75.56..41
Action Executability
27 27.086.41
08. 33.333..0
33 33.33g.sr
31.25
Effects of Actions
82.50s.01
20.006.32
67.507.41
60.537.90
Numerical RAC
11.114.68
06.673.72
04.449.00
08.894,24
Composite Questions
64.533.36
24.63g.02
43.353.48
72.413.14
Average
60.282.35
21.711.98
45.962.30
61.022.35
Fluent Tracking
82.00s 43
36.006.79
62.00, 86
80.00s.68
State Tracking
74.422..5
18.605.93
60.177.46
66.67,.27
Action Executability
34.097.15
11.364.77
40.917.41
47. 737.53
10
Effects of Actions
76.096.20
19.57
65.227.02
62,227
Numerical RAC
10.20
02,042.02
06.
10.20
Composite Questions
59.11
16.262.59
45.32,.49
58.623.46
Average
57.012.37
17.01,.80
45.982.39
55.662.39
Fluent Tracking
67.447.15
27.916.81
67.447.15
67.447.18
State Tracking
75.516.14
16.33g.28
51.027.14
65.316.90
Action Executability
41.677.12
08.333.99
29.79..67
37.50..99
19
Effects of Actions
76.606.18
14.89
46.817.28
61.707.00
Numerical RAC
10.204.32
06. 12.422.42
08. 08.166.91
06. 12..2... 42
Composite Questions
60.30,.47
08. 18.54..95
38.19;.44
49. 49.255..44
Average
56.782.98
11.721.51
39.172.34
48.052.40
Table 3: Performance comparison of GPT-4o, 1amma-3.1-BB 1..1--BIInsttcc..
1.---...--7-------- and
fine-tuned Llama-3.1 --B on the free-answer subset of the benchmark, evaluated without the ramifi- cations constraints using the zero-shot-CoT. The results categorized up by the action-sequenee lengths and question categories.
Performance across Domains In our evaluation, GPT-4o demonstrated the highest performance on the Grippers domain and the lowest on the Satellite domain, with a performance gap of 15.53%. For both 1--ma-3.1-B--mm
the best performance also on the Grippers domain, but their worst performance occurs on the Mystery domain, with differences of 19.24% and 24.04%, respectively. Interestingly, the second lowest performing domain for GPT-4o is Mystery, while for the Llama models, it is Satellite. A detailed breakdown by domains is presented in Appendix C3 This suggests that despite potential differences in pre- pre-trainnn training data, these models exhibit similar relative understanding of the domains.
and 1--ma-

---

Published as a conference paper at ICLR 2025
+ Base Fluents +
Derived Fluents
Self-Derived Fluents + Static Properties
1.0
Accuracy
0.8
0.6
0.4
10 Action-Sequence Length
19
10
10 Action-Sequence Length (c) Llama-3.1-BB-Instruuc
Action-Sequence Length
GPT-4o
(b) Liama-3.1-700-Inst
Figure Performance for every Fluent Category for both binary and free-answer questions for every Action-Sequue Action-Sequence length for GPT-4o, Llama-3.1-70B-Inss and Llama-3.1 1.1-..--IInstrreee for Zero-shot-CoT prompt. Note: Bars represent SEM.
Furthermore, when correlating these results with the State Space Complexity metric (described in App. G), traditionally used to gauge complexity for classical AI systems, we observe an inconsistent trend. This discrepancy implies that LLMs may employ different heuristics from those of traditional Al systems when tackling RAC problems, an observation that opens avenues for future exploration.
Performance across Question Categories LLMs perform well in Fluent Tracking, State Track- ing, and Effects of Actions, demonstrating their strength in keeping track of changes. However, they struggle with Action Executability, Composite Questions, and Numerical RAC. The performance of all LLMs free-response complex questions highlights significant challenges, especially in the Nu- merical RAC category. This category reformulates existing question types into numerical formats, domain where all tested LLMs exhibit notable difficulty. Specifically, performance on numerical questions related to Action Executability is 8.65% lower than on questions in the Fluent Tracking category. Previous research, such as (Ahn et al., 2024 McCoy al.) (023), indicates that LLMs struggle with arithmetic reasoning and counting, which, when mixed with the RAC questions, likely contributes to the poor performance the Numerical RAC category.
For Composite Questions, the combination of Fluent Tracking and Action Executability proves easier to answer than the combination of State Tracking and Action Executability, with a 16.32% perfor- mance difference. This can be attributed to the fact that the State Tracking category is superset of the Fluent Tracking category, thereby explaining the observed difference in difficulty.
Performance across Fluent Categories As evident from Figure 2 and Table 9 across all LLMs examined in the study, a consistent trend emerges in which performance on Static Properties de- creases as the length of action-sequene increases. While these static properties remain unchanged throughout the actions, they might get overlooked in longer sequences, likely due to their absence in the effect of any action. This phenomenon resembles the "needle in haystack" challenge in long-context scenarios, where LLMs struggle to recall specific information embedded within a long context (Zhang et al. (2024). Conversely, Base Fluents maintain stable performance across all action sequences. indicating that the LLMs consistently capture the direct effects of actions. Ramification fluents exhibit steady decline in performance the sequence lengthens, particularly affecting the subcategory of Derived Fluents, which suggests that LLMs have more difficulty handling indirect effects. Finally, Mixed Fluents, which involve more than one fluent type, show consistent decline in performance as the length of action sequences increases.
Performance across Action-Sequence Lengths Figure 3 illustrates the performance of the three models-GPT-4o, 1lama-3.1-70B- ---ttt--nnttt and -lama-3.1-8B-Insteee
varying action se- quence lengths in a zero-shot-CoT setting. The results combine both binary and free-response formats, with detailed data available Tables 5 and 3 Generally, model accuracy declines as the action-sequence length increases, a pattern that holds for most categories. However, GPT-4o deviates from this trend in the State Tracking category, where performance first decreases and then improves. Since this trend is not observed with the other models, and the results lie within the
8

---

Published as a conference paper at ICLR 2025
Fluent Trk. +
State Trk.
Action Exec.
+ Effects of Ac.
Num. Reas.
Composite
1.0
0.8
Accuracy
0.6
0.4
0.2
10
19
10 Action-Sequence Length
1
Action-Sequence Length
Action-Sequence Length (c) -lama-3.1-BB-Inttrucc
GPT-4o
(b) Llama-3.1-70B-Inst
Figure 3: Performance for every QUestion Category for both binary and free-answer questions for every Action-Sequence length for GPT-4o, 1aae-3.1-70B-Isttrree and Llama-3.1-4 3......-----.. --8B-Instruct for Zero-shot-CoT prompt. Note: Bars represent SEM.
margin of error, believe this an outlier. In contrast, the Effects of Actions category consistently deviates from this trend, likely due to the nature of the task, which focuses on changes resulting from the last action, making less dependent the sequence of actions.
Model Parameters and Fine-tuning As demonstrated in Tables and the 1amma-3.11-7B- Instruct model consistently outperforms the smaller performance improvement of 20.84%. This improvement likely due the larger model's supe- rior reasoning capabilities stemming from its increased number of parameters. A similar trend is observed when comparing GPT-4o Llama-3. 1-0.....--7BBIIstrucet where GPT-4o exhibits an average performance increase of 8.64%. Although the specific size of GPT-4o remains undisclosed, it is widely speculated be the trillions of parameters Notably. tine-tuning the Llama-3. 3.1--BBBBBBBBBB model on the training set resulted in substantial gains in both binary and free-ans wee-arswer tasks, with an average performance increase of 33.68% across the test set, even outperforming GPT-4o by 4.2%.
model, with an average
Impact of Few-Shot Examples Model Performance As shown in Tand the inclusion of few-shot examples for binary answer categories does not significantly enhance model accuracy. This limitation is especially pronounced in models such GPT-4o and which exhibit relative performance decline of approximately 3.5% compared to zero-shot-Col conditions. We hypothesize that this decrease may be attributed the few-shot examples inadver- tently leading the model toward detecting spurious correlations. In contrast, the model relies more heavily on its internal reasoning capabilities zero-shot-CoT setting, potentially mitigating bi- ases introduced by example-driven patterns. However, seen from Tables 3 and 7 the few-shot approach shows effectiveness in free-answer questions only open-source LLMs.
LLMs Struggle with Negative Fluents Our study reveals consistent pattern across all the LLMs examined as demonstrated in Table IO wherein their performance declines when tasked with ques- tions involving negative fluents compared those focused on fluents that are true. Specifically, we observed 12.16% decrease in accuracy on these negative fluent tasks. Furthermore, when questions required reasoning about both true and false fluents simultaneously. LLMSs exhibited competence in identifying the true fluents but demonstrated difficulty in correctly recalling the false ones.
5.1 RAMIFICATIONS RESULTS
As discussed in Section the performance on ramification fluents evaluated for two LLMs: GPT-4o, the highest-performing LLM, and ol op-rreve the most recent state-of-the-art LLM. Table presents the performance of both models when ramification constraints are introduced. Further examples of the model responses to ranification-related questions can be found in the Appendix J
4
GPT-4o and Gemini 1.5 Pro: How the New Al Models Compare CNET

---

Published as a conference paper at ICLR 2025
Action Sequence
Question Categories
Free Answer
Binary Questions
GPT-4o
ol-preview
GPT-4o
ol-preview
Fluent Tracking
00.00
00.00
71.4317.10
100.000o.00
State Tracking
00.00po.00
20.0017.50 7.88
100.00po.00
100.00
Effects of Actions
00.0000.00
40.0021.91
71.4317.10
57.1418.70
Average
00.00g0.00
25.0012.49
80.95ps.57
85.7107.60
Fluent Tracking
00.00 00.00go.00
00.00go.or
57.14 97.1418.70
57.1418.70
10
State Tracking
00.00co.00
33.3319.22 19.24
42.8618.70
Effects of Actions
00.00go.00
14.28
71.4317.10
100.00g0.00
Average
00.00g0.00
23.0711.68
57. 57.1410.80
78.5710.90
Fluent Tracking
00.00oo.00
33.3327.21
42.8618.70
57. 57.1418.70
19
State Tracking
00.00po.o0
0000000.00
57. 57.1418.70
Effects of Actions
00.00go.00
00.00po.00
71.4317.10
85.71
Average
00.00g0.00
07.6907.38
57.14 57.1410.80
71.4312.10
Table 4: Performance comparison of GPT-4o and ol-preview on both the free-answer and binary question subset the benchmark, evaluated with the ramifications constraints using the zero-shot- COT. The results are categorized up by the action-sequence lengths and question categories with s_" indicating no response (due to longer prompts). We have provided some responses in App. !
GPT-4o Performance GPT-4o did not answer any ramification-related questions correctly, de- picted in Table Upon manual inspection of its outputs, it became evident that GPT-4o frequently failed to mention ramification fluents, even when these were explicitly detailed in the domain de- scription. In instances where it did address ramification fluents, the responses were incorrect or incomplete, with some fluents being omitted. We hypothesize that GPT--4o may have encountered the domain data during pre-training and relied on memorized effects of actions, as the experimen- tal domains were derived from publicly available IPC datasets. Since the ramification fluents were manually created and integrated for this study, this evaluation assesses the model's reasoning abili- ties without leveraging pre-existing knowledge. This likely explains why GPT-4o failed to generate ramification fluents, its pre- pre-training included only the original fluents from the domains.
ol-preview Performance ol -lerreee a recently developed LLM optimized for reasoning tasks and incorporating novel run-time inference mechanism OpenAl (2024), demonstrated significantly better performance on ramification-related questions compared to GPT-4o, as presented in Table A detailed review of its outputs showed that ol-preview fluents. However, the model struggles with fluents involving negation, which consistently poses challenge. often omitted certain fluents its final answers and, in some cases, incorrectly evaluated the ramification fluents.
(2024)
correctly identify most ramification
6 CONCLUSION
In this work, we introduced a new diagnostic benchmark, ACTIONREASONINGEENCE designed comprehensively evaluate the performance of large language models (LLMs) reasoning about actions and change (RAC). By assessing various LLMs across eight domains and six key dimensions of RAC, our findings indicate that while LLMs demonstrate moderate proficiency on traditional RAC tasks, such as Fluent Trucking, State Trucking, Action Executability, and Effects of Actions, they exhibit significant challenges when addressing more complex and novel questions, particularly in areas like Numerical RAC and Composite Questions, with average performance drop of 17. 7.9%. This highlights a substantial gap in the current ability of LLMs to handle complex reasoning tasks.
Additionally, we explored the indirect effects of actions, known as ramifications, where even state- of-the-ar art models show considerable limitations. For example, GPT-4o could not solve any questions involving ramifications, and the ol-preveew model achieved a low score of 18.4%. These results underscore the pressing need for further research and advancements in RAC reasoning, especially in addressing indirect effects and more advanced reasoning tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
We thank the anonymous reviewers for their constructive suggestions.
We extend our gratitude to the Research Computing (RC), and Enterprise Technology at ASU for providing computing resources, and access to the ChatGPT enterprise version for experiments.
We acknowledge support by a 2023 Spring Amazon Research Award (ARA). This material is also based upon work supported by the Engineering Research and Development Center Information Technology Laboratory (ERDC-ITL) under Contract No. W912H224C0022.
Son Tran acknowledges the partial support from NSF grant #1914635 during the time working on this paper.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.00 08774, 2023.
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157 2024.
Pratyay Banerjee, Chitta Baral, Man Luo, Arindam Mitra, Kuntal Pal, Tran C Son, and Neeraj Varshney. Can transformers reason about effects of actions? arXiv preprint arXiv:20/2.09933, 2020.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems, 2021. URL https://urxiv. 1ng/abs/2110.441666 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 4rX1v:2407.21783, 2024.
Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. Test of time: A benchmark for evaluating Ilms on temporal reasoning. arXiv preprint arXiv:2406.09172 2024.
Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. LawBench: Benchmarking Legal Knowledge Large Language Models, September 2023.
Richard E Fikes and Nils J Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, 2(3-4):189-22 208, 1971.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? Question Answering Benchmark with Implicit Reasoning Strategies, January 2021.
Neel Guha, Julian Nyarko, Daniel Ho. Christopher Ré, Adam Chilton, Aditya K. Alex Chohlas- Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan Choi, Kevin Tobia, Margaret Hagan, Megan Ma. Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. LegalBench: A Collaboratively Built Benchmark for Measur- ing Legal Reasoning in Large Language Models. Advances in Neural Information Processing Systems, 36:44123 44279, December 2023.
11

---

Published as a conference paper at ICLR 2025
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi. Martin Riddell, Wenfei Zhou, James Coady, David Peng. Yujie Qiao, Luke Benson, Lucy Sun, Alex Wardle-Solano, Hannah Szabo, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, An- song Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander R. Fabbri, Wojciech Kryscin- ski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingho Zhou, Caiming Xiong. Rex Ying, Arman Cohan, and Dragomir Radev. FOLIO: Natural Language Reasoning with First-Order Logic, May 2024.
Patrik Haslum, Nir Lipovetzky. Daniele Magazzeni, Christian Muise, Ronald Brachman, Francesca Rossi, and Peter Stone. An introduction the planning domain definition language, volume 13. Springer, 2019.
Weinan He, Canming Huang, Zhanhao Xiao, and Yongmei Liu. Exploring the capacity of pretrained language models for reasoning about actions and change. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4629- 4643, 2023.
Malte Helmert. The fast downward planning system. Journal of Artificial Intelligence Research, 26: 191-246, 2006.
Richard Howey, Derek Long. and Maria Fox. Val: Automatic plan validation, continuous effects and mixed initiative planning using pddl. In 16th IEEE International Conference Tools with Artificial Intelligence, pp. 294-301. IEEE, 2004.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances neural information processing systems, 35:22199-2223 2022.
Harsha Kokel, Michael Katz, Kavitha Srinivas, and Shirin Sohrabi. Acpbench: Reasoning about action, change, and planning. arXiv preprint arXiv:2410.05669 2024.
Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, and William W. Cohen. Differentiable Open-Ender Ended Commonsense Reasoning, June 2021.
Do Xuan Long, Hai Nguyen Ngoc, Tiviatis Sim, Hieu Dao, Shafiq Joty, Kenji Kawaguchi, Nancy Chen, and Min- Yen Kan. Llms biased towards output formats! systematically evaluating and mitigating output format bias 1lms. arXiv preprint arXiv:2408.
2024.
Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark, March 2021.
Man Luo, Shrinidhi Kumbhar, Ming shen, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya, and Chitta Baral. Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models, March 2024.
John McCarthy et al. Situations, actions, and causal laws. Comtex Scientific, 1963.
R Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309. 13638, 2023.
Sheila A Mcllraith. Integrating actions and state constraints: A closed-fom form solution to the ramifi- cation problem (sometimes). Artificial Intelligence, 116(1-2)87--21, 2000.
Shen- Yun Miao, Chao-Chun Liang, and Keh- Yih Su. A Diverse Corpus for Evaluating and Devel- oping English Math Word Problem Solvers, June 2021.
Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. Creak: A dataset for common- sense reasoning over entity knowledge. arXiv preprint arXiv:2109.01653 2021.
OpenAl. Openai ol system card, 2024.
URL
opw.aa....ppp.......... com/
o1 system- card-20240917... .df Accessed: 2024-09-30.
12

---

Published as a conference paper at ICLR 2025
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models, April 2024.
Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja. Mutsumi Nakamura, Neeraj Varsh- ney, and Chitta Baral. Multi-logieval: Towards evaluating multi-step logical reasoning ability of large language models. arXiv preprint arXiv:2406.17169 2024.
Raymond Reiter. Knowledge action: logical foundations for specifying and implementing dy- namical systems. 2001.
Evangelia Spiliopoulou, Artidoro Pagnoni, Yonatan Bisk, and Eduard Hovy. Events realm: Event reasoning of entity states via language models. In Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing, pp.
2022.
Zhi Rui Tam, Cheng- Wu, Yi-Lin Tsai, Chich-Yo Yen Lin, Hung-yi Lee, and Yun-Nung Chen. Let me speak freely? study on the impact of format restrictions on performance of large language models. arXiv preprint arXiv:2408. 202.22408.02442,, 2024.
Md Nayem Uddin, Amir Saeidi, Divij Handa, Agastya Seth, Tran Cao Son, Eduardo Blanco, Steven R Corman, and Chitta Baral. Unseentimeqa: Time-sensitive question-answering beyond Ilms' memorization. arXiv preprint arXiv:2407.03525, 2024.
Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kamb- hampati. Planbench: An extensible benchmark for evaluating large language models planning and reasoning about change. Advances in Neural Information Processing Systems, 36, 2024.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for world planning with language agents. arXiv preprint arXiv:2402.01622, 2024.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv: 09675, 2019.
Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. Found in the middle: How language models use long contexts better via plug-and-pe aad--ad-plaa play positional encoding. arXiv preprint arXiv:2403.04799 2024.
Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Hou, Heng-Tze Cheng, Quoc V Le, Ed Chi, et al. Natural plan: Benchmarking Ilms on natural language planning. arXiv preprint MxXiv:2406.045200 2024.
13

---

Published as a conference paper at ICLR 2025
A LIMITATIONS & FUTURE WORK
While ACTIONREASONINGBENG provides a diagnostic assessment of LLMs on RAC, it repre- sents an early step in this area and has several limitations, including but not limited to the following:
1. Although RAC is not inherently dependent on the English language, the current version of ACTIONREASONINGBENCE
limited to questions formulated in English.
2. Although there are more complex types of RAC, including incorporating more reasoning types, exploring those remains beyond the scope of this work and is left as direction for future research.
3. While the IPC domains in our work cover many scenarios, they may introduce a bias to- wards caaniig--specifie domains. Expanding the dataset to include more domains could help mitigate this bias.
4. Despite our efforts evaluate variety of LLMs, including open and proprietary LLMs, our assessment did not cover models with different architectures or training approaches due to resource limitations.
5. Recent studies by Long et al. (202)) and Tam et al. (024) indicate that LLM performance fluctuate depending the prompt format. This variation may lead to marginal im- provement in the performance of LLMs on ACTIONREASONNGGEENC..
6. Our free-answer evaluation, which relies prompting Llama-3. 1-1.3.1----m- 1-ste--nsttttt,, isn't isn't per- fect and reflects an ongoing challenge of evaluating the free-answers within the NLP com- munity.
B DESCRIBING AN INSTANCE FROM ACTIONREASONINGEENCH
Blocksworld domain with Ramifications for single sequence of action for Fluent Tracking
[DOMAIN DESCRIPTION]
A block can only be picked if is clear, on the table, and the hand is empty, resulting in the block being held. held block can be put down, placing it back on the table. Blocks can be stacked if the first block is held and the second block is clear, causing the first block to rest on top of the second. Unstacking occurs when the hand is empty, the first block clear, and on top of the second, resulting in the first block being held again. A block can't 'n't be at two locations at the same time and considered clear if nothing is on top of it and it's not held, and the hand is empty if it's not holding anything. Blocks are stable when clear and on the table, and they can be painted if stable and the hand is empty. A block is considered on display if it can be painted and has other block top of it.
[INITIAL CONDITIONS]
Block bl is situated on the table, block b2 is not stacked with any other block, block b2 is also on the table, block b3 is not stacked with any other block, block b3 is positioned on top of block b7, block b4 is stacked on top of block bl, block b5 is not stacked with any other block, block b5 is placed on top of block b4, block b6 is on the table, block b7 is stacked on top of block b6, and the hand is empty.
[QUESTION]
Starting from the initial condition, the following actions are taken: block b3 is unstacked from the top of block b7 to achieve the current state. In this state, what the valid proper- ties (including both affirmative and negated properties) for b7? If there are no valid proper- ties, write None.
In the domain description, actions and their corresponding effects on the state are outlined, including the necessary conditions for performing these actions. The initial conditions describe the starting configuration of objects within the domain. A typical scenario involves executing a sequence of actions that alter the configuration of the state, followed by query. In the example provided, the
14

---

Published as a conference paper at ICLR 2025
question falls under the category of fluent tracking, which asks about specific set of properties associated with the object "block b7" after one action has been performed.
The objects involved in this example follows:
block bl
block b2
block b3
block b4
block
block b6
block b7
hand
The properties of the "block" object include:
Block X on top of block Y
Block X is on the table
Block X is clear
Block X is stable
Block X can be painted
Block can displayed
Block X is held
The properties of the "hand" object are:
The hand is empty
The hand is holding block
Several ramification constraints (i.e., properties that depend on other properties) are present in "Blocksworld";
Clear: depends on the properties "on top of" and "held"
Stable: depends on the properties "clear" and "on the table"
Paint: depends the properties "stable" and "hand is empty"
Display: depends on the properties "painted" and "on top"
On top of: depends on itself, since block can't 't be at two locations at the same time
Hand holding block: depends on itself, since the hand cannot hold two blocks the same time
The valid actions that can be performed within this domain include:
Picking up block
Putting down a block
Stacking a block on top of another block
Unstacking block from top of another block
C ADDITIONAL RESULTS
C.I ZERO-SHOT-COT" BINARY RESULT
Table 5 shows the results on GPT-4o, Llama-3.1-8B. IIstrrcct and Llama-3.1 1.1ma-3.1170B.. on zero- Shot-CoT prompting.
15

---

Published as a conference paper at ICLR 2025
Action Seq.
Ques Categories
GPTT-4o
Llama-8B-Ins
Llama-708-n
Fine-tuned Llama-8B
Fluent Tracking
94.442.70
80.004.78
90. 12.31
89.063.90
State Tracking
85.9-4.35
80.00s.00
79.116.93
98. 98.551.44
Action Executability
93.142.50
79.124.26
14.122.33
94.743.61
Effects of Actions
78.494.26
70.89s.11
77.174.35
96.552.44
Numerical RAC
62.82..47
42.11s.01
57.386.33
49.375.62
Composite Questions
74.564.08
50.434662
67.39g.99
94.421
Average
81.261.71
66. 66.330
77.761.85
87.761.43
Fluent Tracking
91.433.35
68. 68.426.14
84.934.19
89.86s,
State Tracking
75.80;.62
76.19
86.216.40
96.432.48
Action Executability
70.874.45
65.67s.80
74.494.40
81.82s.22
10
Effects of Actions
61.224.92
63.8 23.80
60.824.96
96.672.32
Numerical RAC
55.21 55.21s.08
65. 65.528.83
54.396.60
51. 51.145.35
Composite Questions
82.963.24
71.004.54
68.92g.80
88.79, 08.799.07
Average
72.50
67.792.47
70. 70.122.04
84.061 .c0
Fluent Tracking
91.533.63
67 67.35
77.275.16
85.714.69
State Tracking
85.944.35
83.338.78
81.825.81
96.362.52
Action Executability
66.67
53.667,79
66.994.63
77.27p.32
19
Effects of Actions
70.004.83
54.696.22
69.324.92
97.10g
Numerical RAC
50.00s.21
42. 12..22.60
47.544..9
66.29s.u
Composite Questions
48.423.49
53.096.64
68.973.84
86. 18,22
Average
72.31
56.642.93
68.242.07
84.62
Table 5: Performance comparison of GPT-4o, 1lama-3.1-8BB 1-S--tttttt 1aiia-...-70--Inttttttt and fine-tuned Llama-3. 1-BBB on the binary subset (Truc/False) of the benchmark, evaluated without the ramifications constraints using the zero-shot-CoT. The results are categorized up by the action- sequence lengths and question categories.
C.2 FEW-SHOT-3 RESULTS
Table and 7 presents the results using the Few-shot-3 setting. These tables support the results showed in Section
C.3 RESULTS BY DOMAINS
Table shows the results of the binary questions across every domain in our benchmark.
C.4 RESULTS BY FLUENTS
Tables 10 show the results of the binary questions across fluent types in benchmark.
D DATA VERIFICATION
To ensure the soundness of our benchmark. we employed three independent annotators who had no prior involvement with the project. Their task was to evaluate the naturalness the sentences by assigning score from to 5, where indicates the least natural and 5 most natural. To make sure that rephrasing the templated questions using Llama-3.1-700 70B-Instruct helps, we sample from each domain in the dataset was represented by 5 randomly sampled instances, resulting in total of 65 samples across all domains for both templated questions and rephrased questions, resulting in a total of 130 samples. The annotators were provided with both the sampled instances and the following instructions:
Instruction the Annotators
Rate the Prompts from to based on how natural they appear English.
Table II summarizes the naturalness scores assigned by annotators across all domains in ACTION- REASONINGBENCH. The templated dataset received an average naturalness score of 4.2 of
16

---

Published as a conference paper at ICLR 2025
Action Seq.
Ques Categories
GPT-4o
Llama-8B-Inst
Llama-70B-Ime
Fluent Tracking
85.003.57
48.474.77
82.653.82
State Tracking
91.573.00
71.954.96
89.293.37
Action Executability
95.102.14
84.313.60
93.142,50
Effects of Actions
74.194.54
73.914.58
69.894.76
Numerical RAC
58.545.44
56. 76..66
56.105.48
Composite Questions
82.662.40
58.373.23
74.302.77
Average
81.921.45
67.261.80
77.261.58
Fluent Tracking
84.543.0r
69.661.87
82.473.86
State Tracking
91.553.30
64.185.86 86
86. $6.111.08
Action Executability
73.794.33
63.004.83
74.764.28
10
Effects of Actions
59.184.96
53.68r.19
65.314.81
Numerical RAC
41.67s.us
42.255.86
46.883.09
Composite Questions
82.963.24
71.004.34
68.923.80
Average
68.961.71
59.511.92
67.351.74
Fluent Tracking
73.26
63.38s.12
71.084.95
State Tracking
91 91.013.03
63.645.13
84.273.86
Action Executability
62.864.72
54.55s.00
62.864.72
19
Effects of Actions
69.574.80
56.636.44
66.674.89
Numerical RAC
56.995.18
49.156.51
48.39s.18
Composite Questions
65.572.88
57.313.11
62.042.93
Average
68.561.71
57.581.93
64.721.76
Table Performance comparison of GPT-4o, on the binary subset (True/False) of the benchmark, evaluated without the ramifications constraints using the Few-shot-3 setting. The results are categorized up by the action-sequence lengths and question categories.
and 100ma---.-70--Instrtue
5, while the paraphrased version scored 4.5, indicating the high effectiveness of Llama-3. 1amma-3.1-70: Instruct in enhancing the fluency of the data. We would like to point that the annotators partici- pated on a voluntary basis and were informed beforehand that no financial compensation would be provided for their contribution.
FINE-TUNING DETAILS
In this section, we describe the fine-tuning performed the training split of ACTIONREASONING, BENCH described in section 3.5 We fine-tuned Llama-3.1-8 1-8B separately for binary (true/false) and free answer questions, using 6 epochs for the former and 18 epochs for the latter. The AdamW optimizer was used, with batch size of 4 and gradient accumulation steps set to 8 for both of the training setups. Due to the available compute resources, we were limited to a maximum context length of 4096 tokens. This leaves us roughly with 27k samples for binary answers and 14.4k sam- ples for free answer. Tables and show the statistics of the training set that we used train the models.
17

---

Published as a conference paper at ICLR 2025
Action Seq.
Ques Categories
GPT-4o
Llama-&B-Inst
-lama-70B-Bll
Fluent Tracking
7..925.81
50.006.93
75.006.00
State Tracking
73.336.59
44.447.41
73.687.14
Action Executability
56.257.16
14.58p.09
38.307.09
Effects of Actions
80.00
35.007.54
63.648.37
Numerical RAC
08.894.24
04.443.07
13.33s.07
Composite Questions
49.263.51
41.383.46
55.613.63
Average
54.502.39
35.332.30
53.732.49
Fluent Tracking
74.006.20
24.006.0z
64.006.79
State Tracking
81.405.93
34.887.27
65.718.02
Action Executability
50.007.64
09.094.33
34.097.15
10
Effects of Actions
65.227.u2
41.307.26
62.86g.17
Numerical RAC
14.29s.ue
10.204.32
12.244.68
Composite Questions
43.843.48
37.933.41
18.133.65
Average
50.572.40
30.342.20
47.002.50
Fluent Tracking
74.42s.65
44.197.57
65.127.27
State Tracking
61 61.226.96
28.576.45
54.557.51
Action Executability
47.927.21
10.424.41
25.006.25
19
Effects of Actions
73.916.47
27.666.32
60.987.62
Numerical RAC
06.123.42
06.123.42
00.00g.00
Composite Questions
38.193.44
26.133.11
46.23.68
Average
45.622.39
24.372.06
42.542.44
Table 7: Performance comparison of GPT-4o, Llama-3. 1.maa-..1-BB.... I-Stuuntt and Llama-3.1 1--11----Innteeee on the free answer subset of the benchmark, evaluated without the ramifications constraints using the Few-shot-3 setting. The results categorized up by the action-sequence lengths and question categories.
Act. Seq.
Ques Categories
Blocksworld
Depots
Driverlog
Grippens
Mystery
Satellite
Spanner
Visitall
Tracking
100.0..
92.31 as
29.22ha.a
76
100.0c.
oz
88. 88.80+ se
100.0c.
State Tracking
83.33;
93.333.07
66.6716.07
93.756
85.716.1
61.511444
62.512.5
100.0c.
Action Executability
71.43.o.
76.47
76.47
78.95
68.67
10.0
47.06
Ellects of Act.
82.61.
66.67
85.7lam
77.27
100.0c.
60.0ha.s
80.0ho.o
87.5x
Numerical RAC
61.711100
53 80
16.00 66.67a
50. 50.0g1a7
35.7111.9
31.55 31.581c.s 4..44/
31.8210.1
Composite
78.57e.
67.5r.
68.427.61
ia
60.677 as
63.85s on
60.987 m
78.3% an
Avernge
78.955.
73.45
08.87
77.44,
73.214.
64.0%
65.80 1o
70.00
Tracking
75.0g5.
91.67
100.0..
100.0..
80.0
73.68
19..2
94.74,
State Tracking
86.67s
63.6415.a)
83.30
73.33
63.6415..1
61.54
81.2510.
87.5
Action Executability
12.3811.17
64.291a.29
73.68jo.se
37.141..77
46.6713.ss
61.111.95
65 65.0ho.94
55.011.4
Effectt of Act.
68. Os
86.67 10.64
78.57
60.57
78.57
30.77
61.54
68.67
Numerical RAC
57. 141
41.0ho.3
43.7519.81
37.5
33.33
42.86
29. 29.41
27.27e
Composite
71.74..
78 78.05..s
71.42a.s rs
77.55g
55.56,
54.35, as
64,29, as
71.437
Average
67. 12440
67. 67.74
74.364
73.91s
57.43,
35.044
63.57
61.8,2
Flucut Tracking
75e
66.67 uu.
62.5uw.
100.0..
71443
93.33
85.71o
61.5hum
State Tracking
91 91.67
81. 81.2510.05
86.67s.00
70.0
87.5g.44
58.33
78.571..8
83.39
Executability
m2.17
64.29h1.29
57.8th1.s
70.0hu.s1
45.4510.77
56.25mum
58.82
68p18
19
Effects of Act.
72.733.
80.96g.s
85.71am1
70. 120.0
76.92
41.07,aa5
08.6711.45
76.47.0.
Numerical RAC
50.0
57.89
38.4614.4
30.0jo.c4
12.5g.64
37.512.
26.67
31.82
Composite
72.92 as
60.477 54
72.09k
79.55
53.85g on
55.1
67.44r as
86.21,
Average
71.95.05
67.19.
68.75,
70.31,
51.877
58.67
64.46
66.97,
Table 8: Performance across domains GPT-4o both the binary and free-answer subsets of the benchmark, evaluated without the ramifications constraints using the zero-shot setting. The results categorized by the action-sequence lengths and question categories.
18

---

Published as a conference paper at ICLR 2025
Action Seq.
Fluent Types
GPT-4o
Llama 1aeaa-8B-mmn Inst
Llama-70B
Finetuned Llama-
Base Fluents
81.255.63
65.22,
74.55s,
96.32.57
Derived Fluents
82.814.72
76.36s.73
80,284.72
36.753.72
Self-Derived Fluents
85.873.63
68.24s.05
81 81.443.95
98.041.37
Static Properties
85.196.81
72.737.75
80.0g.32
89.474.06
Base Fluents
88.894.08
78.797.12
75.93s.
90.57
Derived Fluents
75.314.79
55.38
72.73s.o
79.384.11
10
Self-Derived Fluents
77.664.3
74.29
69.394.66
95.152.12
Static Properties
79.416.93
63.167,83
69.777.0
94.522.66
Base Fluents
90.244.63
63.64
76.196.57
83.61
Derived Fluents
74.424.
44.076.40
67.47s.14
76.47
19
Self-Derived Fluents
80.04.34
70.916.12
71.744.69
93.466..9
Static Properties
82.767.01
60.719.23
67.57.41
96.882.17
Table 9: Performance comparison of GPT-4o, Alama-3.118B-Instruut. fine-tuned Llama-3.1-88 1-8 on the binary subset (True/False) of the benchmark evaluated without the ramifications constraints using the zero-shot-CoT. The results categorized up by the fluent types
and
Action Seq.
Fluent Types
GPT-4o
Llama-8B-Ina
Mlama-70B-Inss
Finetuned Llama-8B
Positive Fluents
83. 33.163.84
80.0s.39
79.314.34
72.844.o .99
Negative Fluents
77.784.82
60.344.42
70.77..64
77.53
Pos. and Neg. Fluents
81.562.08
65.152.72
78.652.17
93.771.29
Positive Fluents
82.473.96
80.775.47
81.164.71
80.04.34
10
Negative Fluents
66 19
53.667.79
69.23s.12
72.294.91
Pos. and Neg. Fluents
71.322.92
67.422.88
68.212.43
87.741.73
Positive Fluents
79.314.34
67 17.447.15
72.86;.39
84.933.19
19
Negative Fluents
71.084.98
58.1 68.147.52
67 67.535.34
75.644.66
Pos. and Neg. Fluents
70.982.33
54.0g.52
67. 67.52.47
86.271.7
Table 10: Performance comparison GPT-4o, 1--ma-3.--B-IInstteee fine-tuned Llama-3.1 1-BB on the binary subset (True/False) of the benchmark evaluated without the ramifications constraints using the zero-shot-CoT. The results are categorized up by the fluent types
-lama-3.1-700-Instreee and
Domain
Annotator 1 Rephrased
Annotator
Anmotator 3
Average Templated
Templated
Templated
Rephrased
Templated
Rephrased
Rephrased
Blocksworld
3.8
4.8
5.0
4.6
3.8
4,2
4,2
4.5
Depots
4.6
4.8
3.6
4.8
4.0
4.0
4.1
4.5
Driverlog
4.8
4.8
3.6
4,2
4.4
4.6
4.3
4.5
Grippers
4.6
4.8
4.6
4.8
4.9
Mystery
4.0
4.8
4.4
3.6
4.0
3.9
4.4
Satellite
4.4
4.8
4.6
4.4
4.4
4.2
4.5
4.5
Spanner
5.0
5.0
5,0
4.6
3.8
3.8
4.6
4,5
Visitall
3.6
4.2
3.8
4.6
4.0
4.2
3.8
4.3
Average
4.3
4.8
4.2
4.6
4.1
4.2
4.2
4.5
Table 11: Naturalness scores assigned by three annotators on a scale of to 5, where 1 indicates completely incoherent text and indicates natural-sounding questions. The table presents scores for both the templated questions and paraphrased questions.
Answer category
No of Samples
False
13,793
True
13,319
Free-Response
14,476
Total
41,588
Table 12: Data distribution used for fine-tuning, categorized by response type. The binary question responses are split into "True" and "False". Srree-Response" indicates the count of open-ended questions in the training set.
19

---

Published as a conference paper at ICLR 2025
Question category
No of Samples (Binary)
No of Samples(Free-Responsee
Fluent Tracking
11,674
4,946
State Tracking
2,264
1,133
Action Executability
1,534
1,094
Effects of Actions
1,196
1,040
Numerical RAC
5,757
3,293
Composite Questions
4,687
2,970
Total
27,112
14,476
Table 13: Data distribution used for fine-tuning, categorized by Question Categories.
Domain
No of Samples (Binary)
Samples (eree-Resspose)
Blocksworld
3,540
1,978
Depots
2,799
1,394
Driverlog
3,404
1,785
Grippers
3,494
1,910
Mystery
2,825
1,636
Satellite
4,037
2,073
Spanner
3,866
1,992
Visitall
3,147
1,708
Total
27,112
14,476
Table 14: Data distribution used for fine-tuning, categorized by Domains.
20

---

Published as a conference paper at ICLR 2025
F FREE ANSWERS EVALUATION DETAILS
We evaluate the free answers using 1.ama-3.1-27 1-70B-Insttrct. The following is the few-shot-7 prompt that we used for evaluating the responses:
Prompt for Free Answer Evaluation with Llama-3. 3.1--00b-...--.---- ---ttt--nnsttctt Instruct
Evaluate whether the LLM response and the ground truth response semantically the same. Examine the responses, provide reasoning for your evaluation, and then Write "True" if the responses the same "False" if they different. LLM Response Ground Truth could be "None".
Example 1:
[LLM Response]
Location fl 4f is connected to 3f, fo 4f, and f2 4f.
[Ground Truth]
location fl-4f and location f0.4f are connected, location fl and location f2.4f connected, location f0 4f and location fl 4f are connected, location fl 3f and location fl .fff connected, there is connection between location f2. 4f and location fl 4f, there is connection between location f1_4f and location fl _f
[Reasoning for the evaluation] all of the connections the same
[Answer] True
Example 7:
[LLM Response] balll is at room2, ball2 at room2, ball3 room4, ball4 is at rooml, ball5 is at rooml, ball6 is at room4, ball7 is being carried by robotl's 's Igripperl, ball8 is at room4, robot is at room2, robotl's Igripper1 not free, robotl's 's rgripperl free, robotl is carrying ball7 with Igripperl, robot is carrying bal16 with rgripper!
[Ground Truth]
balll is present at room2, ball2 is at room2, ball3 present at room4, ball4 at rooml, ball5 is located at rooml, ball7 is being carried by robotl's 's Igripperl, ball8 is present at room4, rgripper1 robotl is carrying ball6 and robotl is present in room2
[Reasoning for the evaluation] According the ground truth, robotl is carrying ball6, but LLM gives a contradictory response: that ball6 is at room4
[Answer] False
Given the examples and instructions above, evaluate the following responses:
[LLM Response] (Ilmrrepposse
[Ground Truth] {ground )
21

---

Published as a conference paper at ICLR 2025
Table 15 shows the results of the free-answers using automatic metries based on semantic similarity, F1-BERTScoRE (Zhang et al. 2019).
Action Seq.
Ques Categories
GPT-4o
Llama-8B-Img
Llama-70B-Imm
Finetuned Llama-8B
Fluent Tracking
71.762.11
48.522.84
55.212
90.22u.90
State Tracking
68.241.0 189
54.853.18
61.072.59
82.061.88
Action Executability
56.933.3
49.492.80
57.253.17
75.242.70
Effects of Actions
66.431.u 94
49.823.53
59.482.67
82.12 92.122.07
Numerical RAC
87.07
39.40g.05
61.94,.30
89.521.36
Composite Questions
55.031.5 65
40.591.52
40.621.17
82.581.17
Average
67.552.10
47.113.15
55.92
83.62
Fluent Tracking
66.4 66.462.29
51.242.95
56.89
85.401.74
State Tracking
70.09
60.432.32
64.10, 23
83. 83.781.49
Action Executability
61.31
54.142.79
68.28 68.282.78
79.042.40
10
Effects of Actions
66.191.a 1.44
52.993.09
62.052.1e
82.031.78
Numerical RAC
82.38
19.144.13
50.39
85.202.62
Composite Questions
54.57
45.671.59
54.411.39
79.831.56
Average
66.831.m
48.932.85
59.352.87
82.471.93
Fluent Tracking
60.783.19
47.69 17.692.71
56.35..12
87.79 17.791.3
State Tracking
70.331.00
58.062.77
62.062.4r
83.511.41
Action Executability
61.723.41
49. 49.402.93
56.90;.48
78.842.40
19
Effects of Actions
65.711.99
52.533.51
62.78..22
84.901.20
Numerical RAC
81.30, 29
32.554.31
53.795.u
80.75a.62
Composite Questions
55.481.65
50.871.40
52.751.63
77.101.76
Average
65.882.23
48.51
57.433.21
82.141.96
Table 15: FI-BERTScore GPT-4o, Llama-3. 1-8B-Instruct, Llama-3. 1--0--..1-7-B-11 --BB-IIstttt and fine- tuned Llama-3.1-8 the free-answer subset of the benchmark, evaluated without the ramifications constraints using the pero-shot-CoT. The results are categorized up by the secion-sequence lengths and question categories.
Below we share some examples where Fl-BertScore fails to give accurate score.
Low FI-BERTScore but the response correct
[GROUND TRUTH]
robotl's 's rgripperl is available
[RESPONSE]
rgripperl free.
[BERT SCORE]
0.71
[Llama-3.1 1-70B--Intruu Evaluation] The L.MM response and the ground truth response are semantically the same. The term "free" in the LLM response is equivalent "available" in the ground truth response, both indicating that the rgripperl is not being used. True.
[ANALYSIS]
The response is a simple paraphrase of the ground truth which is easily explained by the LLM evaluation.
22

---

Published as a conference paper at ICLR 2025
High F1-BERTScore but the response is incorrect
[GROUND TRUTH]
loc_x0_y1 is not connected to loc_x2_y4
loc_x0_y3 is not connected to loc_x2_y4
loc_xl_y0 and loc_x2.y4 are not connected
loc_xl.yl is not connected to loc.x2.y4
there is no connection between loc_x3_y0 and loc_x2.y4 there is no connection between loc_x3.yl and loc loc_x2.y4
there is no connection between loc_x3.y2 and loc loc_x2.y'
there is no connection between loc loc_x3.y3 and loc x2.y4
[RESPONSE]
llo_x2_y4 is not visited.
-loc_2__4 is connected to loc_xl_ -y4.
-oc__2_y4 is connected to loc_x3-y4.
-oc_x2 x2 y4 is not connected to loc x2 y3. loc_x2 x2 y4 is not connected to loc x2.
llc__2_yy y4 is not connected to loc_x 1._.x..x Ly2. loo_x2_y4 is not connected to loc_xl_y3 loc x2 y4 is not connected to loc xl yy.
[BERT SCORE]
0.89
[Llama-3.1- 3.1-70B-----mmm Intruct Evaluation]
The LLM response and the ground truth response not semantically the same. The LLM response states that loc x2 y4 is not connected to loc x2 y3, but the ground truth response states that loc x2 y4 and loc x2 y3 connected. Additionally, the LLM response states that loc 1oc_x2_y4 is not connected to loc_x2.y5. but the ground truth response states that 1oc_x2.y4 and loc 1oc_x2.y5 are connected. False.
[ANALYSIS]
The response contains a lot of incorrect responses. Moreover, the response fails to mention the location of the robot. The Llama-3. 1-70BB-Instruud evaluation also fails mention the lack of the robot but correctly identifies some incorrect connections.
G DOMAINS IN ACTIONREASONIGGENNTT
In the following sub-sections, we provide details regarding all the domains used in our study. We first present the PDDL description that is given in the IPC and then present how state-space calculation performed for that domain. State space represents the possible number of interactions that can be performed particular state. higher state-space represents more difficult problem for traditional Al solvers.
G.1 BI.OCKSWORLD
In this domain, have of blocks that be manipulated using four basic actions: picking up a block from the table, putting down a block onto the table, stacking one block onto another, and unstacking block from atop another block. The goal is to move and stack these blocks using robotic hand, following specific rules and conditions.
This domain is formally represented in the IPC using the PDDL outlined below:
23

---

Published as a conference paper at ICLR 2025
Domain
# Fluents
Actions Defined
# Object Defined
State Complexity
IPC Year
Name
Defined
Blocksworld
2000
0(2N4
Depots
2002
0(21N*+25 +2N
0(25Nt-N)
DriverLog
2002
Grippers
3
4
1998
0(28
Mystery
1998
O(27N--3N
Satellite
5
4
2002
O(20N
Spanner
3 4
2011
D(28N++20)
Visitall
2014
1(2
in- cluding the number of fluents, actions, and objects defined within each domain. Domains are cate- gorized by year of introduction in the IPC and state space complexity. which reflects the difficulty level for planners to solve each domain. larger state space typically indicates greater complex- ity and presents more significant challenges for traditional Al planners. "N" represents the number of objects in each instance. For example, in the Spanner domain, refers to number of locations, spanners and nuts.
Table 16: Summarizing key characteristics of various domains in
PDDL description for Blocksworld domain
(define (domain BLOCKS)
(rrquuirements :strips :typing) (ttypes block)
(:predicates (on ?x block ?y block)
(ontable ?x block)
(clear ?x block)
(handempty) (holding ?x block))
(:action pick-up
:parameters (?x block)
:precondition (and (clear ?x) (ontable ?x) (handempty))
:effect
(and (not (ontable ?x))
(not (clear ?x))
(not (handempty))
(holding ?x)))
24

---

Published as a conference paper at ICLR 2025
(:action put-down
sparameters (?x block)
:precondition (holding ?x)
:effect
(and (not (holding ?x))
(clear ?x)
(handempty)
(ontable ?x)))
(caction stack
:parameters (?x block ?y block)
:precondition (and (holding 2x) (clear ?y))
:effect
(and (not (holding ?x))
(not (clear ?y))
(clear ?x)
(handempty)
(on ?x ?y)))
(:action unstack
:parameters (?x block ?y block)
:precondition (and (on ?x ?y) (clear ?x) (handempty))
:effect
(and (holding ?x)
(clear ?y)
(not (clear ?x))
(not (handempty))
(not (on ?x ?y))))
Predicates define the number of fluents state:
on(b1,b2)
ontable(b)
clear(b)
holding(b)
handempty
The complexity of state is, where N is the number of objects
V24N2
(1)
G.2 DEPOTS
The Depots domain models logistics environment where crates transported between different locations using trucks and manipulated using hoists. The goal is to efficiently move crates from location another, utilizing the available resources (hoists and trucks) while adhering to the constraints defined by the predicates and actions.
This domain formally represented in the IPC using the PDDL as outlined below:
PDDL description for Depot.s domain
(define (domain depots)
(rreuuirements :strips :typing) (:types place locatable object
depot distributor place
25

---

(action Lift Published as a conference paper at ICLR 2025
truck hoist surface locatable pallet crate surface)
(:predicates (at ?x locatable place)
(on ?x crate ?y - surface)
(in ?x crate ?y truck)
(lifting ?x hoist ?y crate)
(available ?x - hoist)
(clear ?x surface))
(:action Drive
:parameters (?x truck ?y place ?z place)
:precondition (and (at ?x ?y))
:effect (and (not (at ?x ?y)) (at ?x ?2)))
:parameters (?x hoist ?y crate ?z surface ?p -llac) :precondition (and (at ?x ?p) (available ?x) (at ?y ?p) (on ?y ?2) (clear ?y)) :effect (and (not (at ?y ?p)) (lifting ?x ?y) (not (clear ?y)) (not (available ?x)) (clear ?z. (not (on ?y ?z))))
(:action Drop
:parameters (?x hoist ?y crate ?z surface 3p- place)
:precondition (and (at ?x ?p) (at ?z ?p) (clear ?z) (lifting ?x ?y))
:effect (and (available ?x) (not (lifting ?x ?y)) (at ?y ?p) (not (clear ?2)) (clear ?y) ?y 22)))
(:action Load
"parameters (?x hoist ?y crate truck ?p -llac))
:precondition (and (at ?x ?p) (at ?z ?p) (lifting ?x ?y))
:effect (and (not (lifting ?x ?y)) (in ?y ?z) (available ?x)))
(:action Unload
:parameters (?x hoist ?y crate ?z truck place)
:precondition (and (at ?x ?p) (at ?z ?p) (available ?x) (in ly ?z))
:effect (and (not (in ?y ?z)) (not (available ?x)) (lifting ?x ?y)))
Predicates define the number of fluents state:
(at ?x locatable ?y -place)
(on ?x crate ?y surface)
(in ?x crate ?y truck)
(lifting ?x - hoist ły crate)
(available ?x hoist)
(clear surface))
O(22N4+2N)
G.3 DRIVERLOG
This domain is modeled to simulate logistics operations where drivers, trucks, and objects must be moved between different locations. The primary focus is on transporting objects via trucks, either driven by drivers moved manually by walking. The key actions in this domain include loading and unloading trucks, drivers boarding and disembarking trucks, driving trucks between connected locations, and walking when truck involved.
26

---

Published as a conference paper at ICLR 2025
This domain is formally represented in the IPC using the PDDL as outlined below:
PDDL description for Dr viiellloo domain
(define (domain driverlog)
(rrequirmmenss :typing) (:types location locatable object
driver truck obj locatable)
(predicates
(at ?obj locatable ?loc - location)
(in ?obj1 obj ?obj truck)
(driving ?d driver ?v truck)
(link ?x ?y location) (path ?x ?y location)
(empty ?v truck))
(aation
:parameters
(?obj obj
?truck truck
?loc location)
:precondition (and (at ?tuck ?loc) (at ?obj ?loc))
:effect (and (not (at ?obj ?loc)) (in ?obj ?truck)))
(:action UNLOAD-TRUCK
:parameters
(?obj obj
?truck truck
?loc - location)
:precondition (and (at ?truck ?loc) (in ?obj ?truck))
:effect (and (not (in ?obj ?truck)) (at ?obj ?loc)))
(:action BOARD-TRUCK
:parameters
(?driver driver
?truck truck
?loc location)
:precondition (and (at ?truck ?loc) (at ?driver ?loc) (empty ?truck))
:effect (and (not (at ?driver ?loc)) (driving ?driver ?truck) (not (empty ?truck)))
(sactoon DISEMBARK-TRUCK
:parameters
(?driver driver
?truck truck
?loc location)
:precondition (and (at ?truck ?loc) (driving ?driver ?truck)
:effect (and (not (driving ?driver ?tuck)) (at ?driver ?loc) (empty ?truck)))
(aaction DRIVE-TRUCK
:parameters
(?truck truck
?loc-from location
Ploc-to - location
?driver driver)
:precondition
(and (at Ptruck floc-from)
(driving ?driver ?truck) (link ?loc-from ?loc-to))
:effect (and (not (at ?truck Ploc-from)) (at ?truck ?loc-to)))
27

---

Published as a conference paper at ICLR 2025
(saction WALK
:parameters
(?driver driver
?loc-from location
?loc-to location)
:precondition (and (at 7driver ?loc-from) (path ?loc-from ?loc-to))
:effect (and (not (at ?driver ?loc-from)) (at ?driver ?loc-to)))
Predicates define the number of fluents state:
(at ?obj locatable ?loc location)
(in ?obj1 obj ?obj truck)
(driving ?d driver ?v truck)
(link ?y location)
(path ?x 2y location)
(empty ?v -ruck)
O(25N2-N)
G.4 GRIPPER
This domain represents a transportation domain where a robot with two grippers can move between rooms, pick up objects, and drop them off. The robot can only hold one object in each gripper at time. This domain could solve tasks where the robot needs to transport multiple objects from one room to another by strategically moving, picking up, and dropping items.
This domain is formally represented in the IPC using the PDDL as outlined below:
PDDL description for Gr i pper domain
(define (domain gripper-strips)
(rrequirements :strips :tpiigg)
(ttypes room object robot gripper)
(preddictees (at-robby ?r robot ?x room) (at ?o object ?x room)
(free ?r robot ?g gripper)
(carry 2r robot ?o object ?g gripper))
(:action move
:parameters (?r - robot ?from ?to room)
:precondition (and (at-robby !r ?from))
:effect (and rat-robby 2r ?to) (not (at-robby ?r ?from)))
(:action pick
:parameters (?r robot ?obj - object ?room room ?g gripper) :precondition (and (at ?obj ?room) (at-robby ?r ?room) (free ?r ?g)) :effect (and (carry ?r ?obj ?g)
(not (at ?obj ?room)) (not (free ?g)))
28

---

Published as a conference paper at ICLR 2025
(saction drop
sparameters (?r - robot ?obj object 'room room ?g gripper) :precondition (and (carry ?r ?obj ?g) (at-robby ?r ?room)) :effect (and (at ?obj ?room) (free ?g) (not (carry 3r ?obj ?g)))))
Predicates define the number of fluents state:
(carry ?r robot ?0 object ?g gripper)
(at-robby ?r robot ?x room)
(at ?o object ?x room)
(free robot gripper))
O(2N'-3N3
(4)
G.5 MYSTERY
The Mystery domain represents transportation system where vehicles move between locations, constrained by fuel levels, and can load or unload cargo, constrained by available space. The key aspects of this domain are managing fuel for vehicle movement and managing space for loading and unloading cargo. Locations are connected, and the system also handles fuel transitions, space transitions, and the movement of objects across a grid of locations.
This domain formally represented in the IPC using the PDDL as outlined below:
PDDL description for Mystery domain
(define (domain mystery-strips)
(requirements :typing)
(ttypes space fuel location movable object vehicle cargo- movable) (:predicates
(at ?v movable ?1 location)
(conn ?11 ?12 location)
(has-fuel ?1 location ?f fuel)
(fuel-neighbor ?f1 ?f2 fuel)
(in ?c cargo ?v vehicle) (has-space ?v vehicle ?s space)
(space-neighbor ?s ?s2 space))
(aaction move
:parameters (?v vehicle ?11 ?12 location ?fl ?f2 fuel) :precondition (and (at 2v ?11)
(conn ?11 ?12)
(has-fuel ?11 ?f1)
(fuel-neighbor ?12 ?fl))
:effect (and (not (at ?v 211))
(at ?y ?12)
(not (has-fuce 211 ?f1))
(has-fuel 211 ?f2)))
29

---

Published as a conference paper at ICLR 2025
(saction load
sparameters (?c cargo ?v - vehicle ?l location ?s ?s2 space)
iprecondition (and (at ?c
(at ?v ?1)
(has-space ?sl)
(space-neighbor ?s2 1))
:effect (and (not (at ?
(in ?c ?v)
(not (has-space ?v ?s{))
(has-space ?v ?s2)))
(saction unload
sparameters (?c cargo ?v - vehicle ?l location ?s1 ?s2 122--50 sspace) :precondition (and (in ?c ?v)
(at ?v ?I)
(has-space ?sl)
(space-neighbor ?s1 ?s2))
:effect (and (not (in 2e ?v))
(at ?c ?!)
(not (has-space ?v ?s))
(has-space ?v ?s2)))
Predicates define the number of fluents state:
(at ?v movable ?1 location)
(has-fuel location ?f fuel)
(in ?c cargo ?v vehicle)
(has-space ?y vehicle spacee)
(conn ?11 ?12 location)
neighoo ?f1 ?f2 fuel)
(space-neighbor ?s1 ?s2 spacee)
0(27N9-3N)
(5)
G.6 SATELLITE
The Satellite domain represents simplified model for managing and controlling satellites and their onboard instruments. The goal in this domain to coordinate the behavior of satellites, including turning them toward desired directions, powering instruments and off, calibrating instruments, and capturing images.
This domain is formally represented in the IPC using the PDDL outlined below:
PDDL description for Sa t lite domain
(define (domain satellite)
(rreuurements :strips :typing)
(ttyps satellite direction instrument mode)
30

---

Published as a conference paper at ICLR 2025
(:predicates
(on_board ?i instrument ?s satellite)
(supports instrument ?m mode)
(pointing ?s satellite ?d direction)
(power_avall ?s satellite)
(power_on on ?i instrument)
(calibrated ?i instrument)
(have_image ?d direction ?m mode)
(calibration.targe ?i instrument direction))
(:action turn to :parameters (?s satellite ?d_new direction ?d_prev direction)
:precondition (and (pointing ?s ?d_prev))
:effect (and (pointing ?s ?d_new) (not (pointing ?s ?d_prev))))
(action switch.on
:parameters (?i instrument ?s satellite)
:precondition (and (on board ?s) (power avail ?s))
:effect (and (power_on ?i) (not (calibrated ?i)) (not (power_avail ?s))))
(:action switch.ov ooff
:parameters (?i instrument satellite)
:precondition (and board ?i ?s) (power_on ?i)) :effect (and (not (power on ?i)) (power avail ?s)))
(:action calibrate
:parameters (?s satellite ?i instrument ?d direction) :precondition (and (on board ?i ?s) (calibration target ?i 2d)
(pointing ?s ?d)
(power_on ?i))
:effect (calibrated ?i))
(caction take_image
:parameters (?s satellite ?d direction ?i instrument ?m mode)
:precondition (and (calibrated ?i)
honrd ?i ?s)
(supports ?m)
(power_on ?i)
(pointing ?d))
:effect thave-image 2d ?m)))
Predicates define the number of fluents state:
(on-board ?i instrument satellite)
(supports ?i instrument 7m mode)
(pointing ?s satellite direction)
(have-image ?d direction ?m mode)
(calibration-targett ?i instrument ?d direction)
(power-avail ?s satellite)
(power-on ?i instrument)
(calibrated li instrument)
0(28N4+3N)
31

---

Published as a conference paper at ICLR 2025
G.7 SPANNER
This domain models simple world where man moves between locations, picks up spanners, and uses them to tighten loose nuts. The actions available to the man involve walking between locations, picking up the spanner, and tightening nuts using the spanner if all conditions are met.
This domain formally represented in the IPC using the PDDL as outlined below:
PDDL description for Spanner domain
(define (domain spanner)
(requirements :typing :strips)
(:types
location locatable object
man nut spanner locatable
(:predicates
(at ?m locatable ?l location) (carrying 7m man ?s spanner)
(useable ?s spanner)
(link ?11 location ?12 location)
(tightened ?n nut)
(loose ?n nut))
(:actinn walk :parameters (?start location ?end location ?m man) :precondition (and (at ?m ?start) (link ?start ?end))
:effect (and (not (at ?m ?start)) (at ?m lend)))
(:action pickup.spanner :parameters (?1 location ?s spanner ?m man)
:precondition (and (at ?m ?I) (at ?s ?1)) :effect (and (not (at ?s ?I)) (carrying ?m ?s)))
(saction tighten_nut :parameters (?1 - location ?s spanner ?m man ?n nut)
:precondition (and (at ?m ?I)
(at ?n ?1)
(carrying ?m ?s)
(useable ?s)
(loose ?n))
:effect (and (not (loose ?no) (useable ?s)) (tightened ?n)))
Predicates define the number of fluents state:
(at ?m locatable location)
(carrying ?m man ?s spanner)
(link ?11 location ?12 location)
(useable ?s spanner)
(tightened ?n - nut)
(loose ?n nut))
0(23N4+2N)
32

---

Published as a conference paper at ICLR 2025
G.8 VISITALL
The VisitAll domain is focused on controlling a robot that needs to visit all places on connected grid. The robot's movement is governed by the connectivity of places, and each move changes the robot's location and marks the visited place. The task is essentially to traverse the entire grid, visiting every place, while ensuring the robot follows connectivity constraints.
This domain is formally represented in the IPC using the PDDL as outlined below:
PDDL description for vi visia ttAll domain
(define (domain grid-visit-all)
(requirements :typing)
(types place obbject
(:predicates (connected ?x place)
(at-robot ?x place)
(visited ?x place))
(caction move
:parameters (?curpos ?nextpos place)
:precondition (and (at-robot ?curpos) (connected ?curpos ?nextpos))
:effect (and (at-robot ?nextpos) (not (at-robot ?curpos)) (visited (nextpos)))
Predicates define the number of fluents state:
(connected ?x ?y place)
(at-robot ?x place)
(visited ?x place)
O(2
H PLANNING DESCRIPTION AND TOOLS
Planning, at its core, involves determining sequence of actions that transforms the world from an initial state to a goal state. A world state specifies which fluents are true or false at any given time. The planning domain, denoted D, specifies the fluents, actions, and their effects within the system. Typically, planning domains are represented using formal languages such as PDDL or ASP. In these languages, transition function & states x actions states defines how actions transform an initial state into a resulting state.
H.1 PLANNING DOMAIN DEFINITION LANGUAGE (PDDL)
PDDL a formal language developed for expressing planning problems and domain models. Since its inception, PDDL has been extended to address increasingly complex planning scenarios, par- ticularly those involving deterministic problems (Haslum et al. 2019 PDDL facilitates the spec- ification of both the planning domains and problem instances, including objects, initial, and goal states. In this study, we employ the "STRIPS" (Stanford Research Institute Problem Solver) subset of PDDL (Fikes Nilsson 197T). Additionally, the domains "typed", meaning that objects in the planning problem are assigned specific types and subtypes, ensuring a structured representation of the problem space.
H.2 ANSWER SET PROGRAMMING (ASP)
ASP is declarative approach to problem-solving based on logic programming and non-monotonic reasoning. Unlike traditional planning languages like PDDL, ASP focuses on defining constraints and rules that describe potential solutions, rather than directly encoding state transitions. In ASP,
33

---

Published as a conference paper at ICLR 2025
problem is encoded as logic program consisting of rules, facts, and constraints, and the solution is an "answer set" that satisfies all the constraints of the problem. this study, we use ASP to generate the complete state-space by applying the sequence of actions starting from the initial state.
CLASSIFICATION OF FLUENTS
The subsequent sections provide detailed classification of fluents across all 13 domains included in ACTIONREASONINBENNC, described in section 3.2
1.1 BLOCKSWORLD
In the BLOCKSWORLD domain, the fluents are categorized as follows:
Static Properties No static properties are present
2. Base Fleunts onTable (block)
3. Derived Fluents clear (block), handEmpty
4. Self-Derived Fluents holding (block), on (block,
I,2 DEPOTS
In the DEPOTS domain, the fluents categorized follows:
1. Static Properties No static properties are present
2. Base Fleunts No base fluents are present
3. Derived Fluents clear (surface), avai lable (hoist)
4. Self-Derived Fluents at paaatttalee,,lcceee, on (nrate, surface), in (crate, truck), lifting (hoist, crate)
1.3 DRIVERLOG
In the DRIVERLOG domain, the fluents are categorized as follows:
1. Static Properties 1ink (location, location), path (location, location)
2. Base Fleunts No base fluents are present
3. Derived Fluents empty (truck)
4. Self-Derived Fluents at (locatable, location), in (object, driving (driver, truck)
1.4 GRIPPERS
In the GRIPPERS domain, the fluents are categorized as follows:
1. Static Properties No static properties are present
2. Base Fleunts carry (robot, object, gripper)
3. Derived Fluents free (robot, gripper)
4. Self-Derived Fluents at robby (robot, room), at (object, room)
I.5 MYSTERY
In the MYSTERY domain, the fluents are categorized as follows:
Static
Properties
space neighbor (space, space),
fuel net thigr (fuel, fuel) ), conn (location. location)
34

---

Published as a conference paper at ICLR 2025
2. Base Fleunts No base fluents present
3. Derived Fluents No derived fluents are present
4. Self-Derived Fluents
at (movable, location), in (cargo, vehicle),
has.space (vehicle, space), has_fuel (location, fuel)
1.6 SATELLITE
In the SATELLITE domain, the fluents are categorized as follows:
1. Static supports (instrume rument, mode), calibration.target (instrument direction)
Properties
on_board (instrument, satellite),
2. Base Fleunts power_on (instrument), have.image (direction, mode)
calibrated (instrument),
3. Derived Fluents power.avail (satellite)
4. Self-Derived Fluents - pointing (satellite, direction)
1.7 SPANNER
In the SPANNER domain, the fluents categorized follows:
1. Static Properties 1ink (location, location)
2. Base Fleunts carrying (man, spanner), tightened (nut)
3. Derived Fluents loose (nut), useable (spanner)
4. Self-Derived Fluents at (locatable, location)
1.8 VISITALL
In the VISITALL domain, the fluents are categorized follows:
1. Static Properties- connected (place, place)
2. Base Fleunts No base fluents are present
3. Derived Fluents visited (place)
4. Self-Derived Fluents at.robot (place)
J RESPOSNES ON RAMIFICATION DATASET
In the following sections, we illustrate some instances and responses that we see from GPT-4o and ol ol-preveew.
35

---

Published as a conference paper at ICLR 2025
J.1 GPT-40
Ramificaiton question for State Tracking
[DOMAIN DESCRIPTION]
A block can only be picked up if it is clear, on the table, and the hand is empty, resulting in the block being held. held block can be put down, placing back on the table. Blocks can be stacked if the first block is held and the second block is clear, causing the first block to rest on top of the second. Unstacking occurs when the hand is empty, the first block is clear, and top of the second, resulting in the first block being held again. A block is clear nothing is on top of it and it's not held, and the hand empty if it's not holding anything. Blocks are stable when clear and on the table, and they can be painted if stable and the hand empty. A block is considered on display if can be painted and has no other block on top of it.
[INITIAL CONDITIONS]
Block b1 is stacked on block b4, block b2 has no blocks on top of it, block b2 is stacked on block b6, block b3 has no blocks on top of it, block b3 is stacked on block b5, block b4 is positioned on the table, block b5 is stacked on block b7, block b6 is positioned on the table, block 67 stacked on block bl and the hand is empty.
[QUESTION]
Given the initial condition, the following actions are taken: block b3 is unstacked from block b5 to achieve the current state. In this state, if block b3 is placed down, what are all the valid properties of the state (including both affirmative and negated properties)? If there are no valid properties, state None.
Response to the above question
[REASON]
To determine the valid properties of the state after block b3 placed down, we need to follow the sequence of actions and check the conditions for each property.
1. ** Initial Conditions:
-b1 on b4.
-b2 on b6.
-b3 is on b5.
b4 on the table.
b5 is on b7.
b6 on the table.
bb7 is on bl.
The hand is empty.
Actioon: Unstack b3 from b5:**
-The hand must be empty.
-b3 must be clear and on top of b5.
-Result: b3 held, b5 is now clear.
** **Current State after unstacking b3:**
-b1 is on b4.
-b2 on b6.
.b3 is held.
36

---

Published as a conference paper at ICLR 2025
b4 is the table.
b5 is on b7 and is clear.
b6 on the table.
b7 is on bl.
-The hand is not empty (holding b3).
** **Action: Place b3 down:**
- -The hand must be holding b3.
Result: b3 placed on the table, the hand is empty.
5. **Final State after placing b3 down:**
-b1 is b4.
-b2 is on b6.
-b3 on the table and clear.
b4 is on the table.
b5 on b7 and is clear.
b6 is on the table.
b7 on bl.
The hand is empty.
6. Valid Properties:
.1 on b4.
b2 is b6.
b3 on the table.
b3 clear.
b4 on the table.
b5 on b7.
b5 is clear.
b6 on the table.
b7 is on bl.
The hand is empty.
bl is not clear (b7 is on it).
b2 not clear (nothing on it but is on b6).
b4 is clear (nothing it and the table).
b6 clear (nothing on it and on the table).
b7 not clear (b5 is it).
b3 stable (clear and on the table).
b4 stable (clear and the table).
b6 is stable (clear and on the table).
b3 can be painted (stable and hand is empty).
b4 can be painted (stable and hand is empty).
b6 can be painted (stable and hand is empty).
b3 is on display (can be painted and no block on top).
b4 on display (can be painted and no block on top).
b6 is on display (can be painted and no block on top).
[ANSWER]
-b1 on b4.
-b2 is on b6.
-b3 is on the table.
-33 clear.
b4 is on the table.
b5 on b7.
b5 is clear.
b6 on the table.
67 is on bl.
37

---

Published as a conference paper at ICLR 2025
The hand is empty.
bl is not clear.
b2 clear.
b4 is clear.
b6 clear.
b7 is not clear.
b3 stable.
b4 is stable.
b6 is stable.
b3 can be painted.
b4 can be painted.
b6 can be painted.
b3 is on display.
b4 on display.
b6 is on display.
From the above response, we can see that block b4 is not stable, but the model says it is stable.
J.2 O1-PREVIEW
A Ramificaiton question for State Tracking
[DOMAIN DESCRIPTION]
block can only be picked up if it is clear, on the table, and the hand is empty, resulting in the block being held. A held block can be put down, placing it back on the table. Blocks can be stacked if the first block is held and the second block is clear, causing the first block to rest top of the second. Unstacking occurs when the hand is empty, the first block is clear, and on top of the second, resulting in the first block being held again. A block is clear if nothing is top of it and it's not held, and the hand empty if it's not holding anything. Blocks are stable when clear and on the table, and they can be painted if stable and the hand empty. A block is considered on display if can be painted and has other block on top of it.
INITIAL CONDITIONS]
Block is positioned above block b7, block b2 has no blocks on block b2 is stacked on block b6, block b3 is stacked on block b4, block b4 is situated on the table, block b5 has no blocks on it, block b5 is situated on the table, block is stacked on block b3, block on the table, block b8 is positioned above block b1, block b9 has blocks on it, block b9 stacked on block b8, and the hand is empty.
[QUESTION]
Based on the initial condition, the following actions taken: block b9 is removed from block b8 to achieve the current state. In this state, identify all valid properties that do not include negations and list them: if there are such properties, indicate None.
38

---

Published as a conference paper at ICLR 2025
Response to the above question
[REASON]
After removing block b9 from block b8, block b9 is placed on the table, and the hand remains empty. Block b9 now has no blocks on top and is not held, so it is *$clear*s, Since is the table, it is also **taabl**, With the hand empty and block b9 being stable, c*n be painted**. As there are no blocks on top of it, block b9 **on display**.
Similarly, block b5 is on the table with no blocks on top and is not held, making **clear** and **stable** It **aan be painted** and is **on display**.
Block b7 is on the table and, with blocks explicitly on top of it after the action, is **clear**, **stable**, **can be painted**, and is ** display**,
Blocks bl and b8 have no blocks on top of them and are not held, so they are **clea**
Block b2 has no blocks top and is not held, so it is **clear**, though it is not the table and thus not stable.
The hand is explicitly stated to be **mppy*** in the current state.
[ANSWER]
The hand is empty
Block bl is clear
Block b2 is clear
Block b5 is clear
Block b5 is on the table
Block b5 is stable
Block b5 can be painted
Block b5 is display
Block b7 is clear
Block b7 is the table
Block 67 is stable
Block b7 can be painted
Block b7 is on display
Block b8 is clear
Block b9 is clear
Block b9 is the table
Block b9 is stable
Block b9 can be painted
Block b9 is display
From the above response,
that block b1 is not clear, but the model says it is clear.
K IMPACT OF INPUT FORMATS ON LLM PERFORMANCE
In this section, we explore the effects of different input formats on LLMs. Specifically, we compare the performance of LLMs when inputs are presented in formal language (PDDL) in templated formats generated during the data creation pipeline (described in Section 3.4), opposed to natural language. For our analysis, we sampled 10% of the test set from ACTIONRAASNNINGBENCC covering all question categories and action sequence lengths.
K.
FORMAL LANGUAGE (PDDL)
presents the performance of LLMs when provided inputs in the formal language PDDL. Our results reveal significant drop in performance for most models. GPT-4o shows notable per- formance degradation of 16.03% compared to the natural language baseline. Similarly, Llama-3.1 7 experiences 6.7% decrease in accuracy. Interestingly, Llama-3. 1..1--8-Instuue exhibits
Table 17
39

---

Published as a conference paper at ICLR 2025
Action Seq.
GPT-4o
Llama-8B-Inst
Llama-70B-Inst
13.907.75
26.83
53. 63.667.77
10
48.787.81
17.07
31.71
19
33.337.55
17.95..15
25.646.99
Table 17: Performance comparison of GPT-4o, Llama-3.1-88 1-8--Intttutt and Llama-3. 1lama-3.17705 Inrtttsrtttt on the free-answer subset of the benchmark evaluated without the ramifications constraints using the zero-shot-Coo The input is given in the formal language, i.c. PDDL. The results are categorized up by the action sequence length.
a 3.78% increase in performance, likely attributable to its initially low baseline performance. These findings suggest that the pretraining phase of LLMs, predominantly focused on natural language, plays crucial role in shaping their reasoning capabilities. Consequently, formal language inputs, that deviate from this training paradigm, may hinder model performance.
K.2 TEMPLATED LANGUAGE
Action Seq.
GPT-4o
Llama-8B-Inst
Lllam--7O-Inn
76.92..75
32.507.
63.417.52
10
66.677.55
19.51
47.507.89
19
64.107.68
17.956.15
25.646.99
Table 18: Performance comparison of GPT-4o, 1.-.aa....----Immtmmm and Llama-3.1-700 3.1--00BIInsrruue Instruct the free-answer subset of the benchmark evaluated without the ramifications constraints using the zero-shot-Con The input is given the templatized language. The results are categorized up by the action sequence length.
Table 18 summarizes the performance of LLMs when inputs are presented in the templated formats deseribed in Section 3.4 rather than in fully paraphrased natural language. The results indicate a consistent improvement across all models. Notably, GPT-4o achieves the highest gain, with an average performance improvement of 11.2%. 1amma-3.-8B-Inntruu exhibits the second largest improvement, with 6.5% increase, while Llama-3,1 - demonstrates a modest gain of 1.81%. These results suggest that templated inputs reduce the verbal reasoning burden on LLMs, leading to more accurate outputs.
40


Reviews:
[{"review_id": "hr9oe9iNTz", "reviewer": "Reviewer_byEu", "summary": "This paper introduces ActionReasoningBench, which evaluates LLMs' ability to reason about actions and change in PDDL-based domains. The benchmark introduces novel ramification fluents which require deeper reasoning to infer correctly, such as fluents which are dependent on combinations of other properties, and other properties like \"if object is at location a, it cannot also be at location b\". The results show that LLMs predict the standard fluents well, but are almost completely failing at such ramification fluents.", "questions": "- In Table 3, are only the base fluents inclulded in the \"Fluent tracking\" category?\n- Is my understanding correct that when you refer to \"introducing ramification constraints\", it indicates introducing the ramification fluents in the benchmark?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- The paper studies an important but overlooked problem of evaluating RAC capabilities of LLMs in a systematic way, beyond measuring surface RAC. The paper demonstrates through the difference in performance on basic fluents and the ramification fluents, that this is indeed a weakness of current LLMs.\n\n- The paper's conceptualization and discussion of ramification fluents is useful, and the benchmark has potential to lead to interesting new research such as entirely new models for RAC.", "weaknesses": "I have two main concerns with the paper in its current form:\n\n- Organization: The organization of the paper could be improved, as the main results of the paper appears to be Table 5 (the ramification results), but these experiments are conducted on a smaller set of models and limited to free-answer. The experiments seem a bit incomplete. Other sections like \"Performance across Fluent Categories\" (l415) should be shown as table/figure. \n- Human baseline: It's difficult to gauge the human upper bound on the benchmark with ramifications, and I would suggest that a human baseline would offer a better way to contextualize the low numbers shown by LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HRTkPsdMjR", "reviewer": "Reviewer_sFGq", "summary": "This paper proposes a reasoning benchmark for reasoning about actions and changes. The focus of this benchmark is specifically on tasks that require state tracking and commonsense, which often requires a full understanding of the current state. The paper aggregates this benchmark using the IPC advance planning competition challenge, by using a pipeline to convert them to natural language and testing it on the latest LLMs. The benchmark reveals headroom for LLMs in the context of state tracking, specifically when ramification constraints are incorporated in the challenge.", "questions": "- Why should an LLM alone be used to solve this and not in combination with other systems ? What aspects of the challenges in these individual domains are LLMs most suitable ? Could you please elaborate a bit on this ? \n\n- What are the current state-of-art (including non-LLM) methods for this ? How do LLMs perform in comparison to them ? \n\n- Are there any Insights into why numerical RAC performance doesn't seem to improve even after fine-tuning? I was wondering if it was because “number of possible states” is a well-posed problem ? Love to know the authors’ opinion on this \n\n- (Line 193) What are negative fluents ?\n\n- What happens if we don’t convert the examples into natural language ?  Does the LLM perform any well at all in this task without the natural language conversion step ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is a step forward towards producing a more challenging benchmark in the domain of reasoning about actions and changes, compared to the previous PlanBench and TRAC work. \n\n- It also incorporates additional metrics in state tracking that is not part of previous benchmarks. \n\n- The latest models are also thoroughly evaluated with a good sense of headroom, and will provide a meaningful resource for making progress in this domain.", "weaknesses": "- Related work is poorly written. It doesn’t cover the nature of the dataset in the context of existing benchmarks (I had to read the other papers to get a good context). It was hard to identify why this benchmark was the right direction to continue as a benchmark. I highly recommend the authors to expand the related work section placing the contribution in context (particularly for readers like me, who don’t have the full context). \n\n- Regarding the benchmark itself, the paper doesn’t give enough details on how challenging the individual domains are. It is hard to judge the contribution of the paper well, without those details. Specifically, how were these tasks solved before LLMs were used for these domains? What progress have they made ? While LLMs can be a general model for these tasks, how would they different from building custom models for these tasks ? \n\nNote: I still very much like this paper, and happy to increase the score if these points above are addressed well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVoRlOzyLc", "reviewer": "Reviewer_QWAg", "summary": "This paper introduces ActionReasoningBench, a benchmark for measuring LM reasoning on action sequences across multiple categories. The benchmark is created by adapting 8 planning domains from the International Planning Competition, and translating them into natural language using a combination of templates and language models. The natural language descriptions and questions are reviewed by annotators. The authors evaluate 4 models on these datasets: GPT-4o, Llama-8b-instruct, LLama-70B-instruct, and fine-tuned Llama-8b. They find that performance generally decreases as a function of action sequence length, and state-of-the-art LMs cannot solve the benchmark, performing particularly poorly on tasks that require numerical reasoning or a combination of other types of action-sequence reasoning (e.g. fluent tracking and action executability). Even GPT4 o1-preview struggles to perform well on inferring ramifications of actions.", "questions": "1. The use of Llama to generate question translations (into natural language) may affect the reasoning capabilities of LLMs, in particular requiring LLMs to use verbal reasoning skills to comprehend the questions. Are LLMs better at handling questions in natural language compared to templated?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "A strength of this paper is in its comprehensiveness and quantity of analysis. The benchmark  explores many different aspects and ways of querying for action reasoning capabilities, and discovers many interesting trends. The dataset is very large and is able to identify reasoning capabilities which even the most state-of-the-art LLMs do not possess. The method for creating the dataset seems sound as there is a programmatic solution to these types of questions, and humans are used to validate paraphrases. The paper is overall clearly written.", "weaknesses": "1. The choices of tasks in the paper could be better motivated -- why are the 8 tasks present a representative survey of the important types of action reasoning? Why do we care about these 8 specifically? What are the consequences of LMs being able (or not able) to perform some of these types of reasoning? Why is numerical reasoning included (which is not fully related to action sequences), but not other types of reasoning (e.g. logical)?\n2. Without a larger takeaway, this paper risks reading like simply a collection of empirical facts. This paper presents 7-8 separate takeaways in section 5, without too deeply investigating any single one. For example, what kind of errors do LLMs make on ramification fluents? What kind of intuition can we gain about LLMs and their limitations in reasoning from these errors. Having a more coherent high-level claim about all the takeaways from these results can be enlightening. Also having a table summarizing the limitations of LLMs in these 8 aspects could be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zsxmdR4Vzx", "reviewer": "Reviewer_24Bn", "summary": "This paper introduces a new benchmark, ActionReasoningBench, to evaluate LLMs ability to reason about actions and change. This benchmark covers various RAC dimensions, including fluent tracking, state tracking, action executability, effects of actions, numerical RAC, and composite questions. Additionally, the authors introduce a new ramifications constrains to the reasoning task. The paper presents the evaluation of several sota LLMs across the proposed RAC dimensions proving that there is a significant margin of improvement for advanced LLMs such as GPT-4o and o1-preview.", "questions": "Have you considered incorporating a combination of automatic evaluation metrics (e.g., BertScore) alongside the current evaluation?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow. \n- It provides a comprehensive evaluation framework for LLMs' RAC capabilities, covering a wide range of tasks and difficulties. This benchmark allows for a thorough assessment of LLM performance and identifies areas for improvement such as new key challenges and LLMs limitations.", "weaknesses": "- I am not sure about the exclusive use of LLM to evaluate the free-form answers. It may introduce inaccuracies due to the limitations of language models. While the correlation between Llama-3.1-70B-Instruct and human evaluations is reported, it would be of interest to report evaluation from automatic metrics based on semantic similarity such as Bertscore (among others).\n- The fine-grained methodology for evaluating RAC could be further enriched considering aspects such as causal reasoning (this is partially covered by “effects of actions”) and temporal reasoning (for example reasoning about concurrent actions). \n- A short discussion beyond the examples provided in the appendix on how/why LLMs fail (with error analysis) the task would help better motivate the study beyond the performance in term of accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)