Title: Exploring the Capacity of Pretrained Language Models for Reasoning about Actions and Change
Benchmark: Textual Reasoning about Actions and Change (TRAC)
Summary: The paper introduces four core Reasoning about Actions and Change (RAC) tasks as a unified textual benchmark, carefully designed to minimize confounding linguistic factors (e.g., grounding) and maintain a sharp focus on RAC. The resulting benchmark, TRAC (Textual Reasoning about Actions and Change), includes problems of varying complexity and enables more fine-grained evaluation of language models, with an emphasis on assessing their structural generalization capabilitie — crucial for effective RAC.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Two fundamental tasks, Projection and Executability, which directly target the essential
knowledge of RAC; two composite tasks, Plan-Verification and Goal-Recognition for
more comprehensive problem settings. 
Phenomenon defined by authors: Yes
Phenomenon definition: While transformers are able to induce and utilize knowledge from a large number
of training examples, it remains a great challenge to efficiently generalize to structurally more complex problems on TRAC.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: There are four tasks introduced in this benchmark. There definitions are listed below. 

Project: This task assesses the outcome of executing actions. Given an initial state s and a sequence of N applicable actions ⃗a, the goal is to determine whether a proposition q would hold after executing ⃗a. The context includes s and ⃗a, and the query is q.

Executability: This task focuses on action preconditions. Given an initial state s and a sequence of N actions ⃗a, the goal is to determine whether ⃗a can be executed sequentially starting from s. Here, the context is s, and the query is ⃗a.

Plan Verification (PV): In planning, the goal is to generate a sequence of actions to achieve a desired outcome. TRAC adopts a verification variant of this task: Given an initial state s, a goal g (expressed as a proposition), and a sequence of N actions ⃗a, the task is to determine whether executing ⃗a from s will successfully achieve g. The context includes s and g, while the query is ⃗a.

Goal Recognition (GR): This task involves identifying the goal based on a partial observation of actions. In the simplified version used in TRAC, given an initial state s, a candidate goal g, and a partial action sequence ⃗a, the system must decide whether g is the true goal—that is, whether ⃗a is a prefix of an optimal plan for achieving g. The context consists of s and ⃗a, and the query is g.
Task item definition: Context, Query and Answer. 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 3k
Dataset size notes: 10k, 2k
Dataset metadata available: Yes
Dataset metadata detail: length of action sequences. 
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: action sequences of length 4 and 5. 
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: No
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Planning
New taxonomy: General Capability / Planning