Benchmark: CaT-Bench: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: (causal/ temporal) reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: While LLMs appear to generate good plans, it’s unclear how well they understand important aspects of the steps themselves. We thus use CAT-BENCH to test whether LLMs can identify step dependencies that reflect the causal and temporal structure of the plan. We find that current LLMs struggle to identify step dependencies, often performing close to random chance, raising more questions about their understanding of instructional text. (Section1)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Purpose notes: Faithfulness in explanation qualities
Task definition: A step order prediction with plan text and question of how many step happens before/after the certain step. The output is expected to be yes/no binary answer with optional explanation generation.
Task item definition: One binary question about a specific order pair of steps and ground-truth label
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: End‑users would not normally answer yes/no (binary) dependency questions so it is constructed task.
Plus, the benchmark captures a key capability (causal/temporal reasoning over plans) that underlies practical applications such as recipe‑guidance, robotics and agent planning, so it is representative of real‑world needs even though the evaluation setting is synthetic.
Dataset size: 4260
Dataset size detail: 2840 (balanced subset for evaluation) + 1420 (held‑out non‑dependent questions used for analysis)
Dataset metadata available: Yes
Dataset metadata detail: Recipe, temporal step, causal dependency
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric aggregation: Weighted Mean
Metric statistics: Means, standard deviations, weighted Fleiss‑k
Metric subscores: Yes
Metric subscores detail: Analysis by class, relation direction (before/after), step distance (close/distant)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: High inter‑annotator agreement and consistency metrics 
Taxonomy: Reasoning / Planning
New taxonomy: General Capability / Planning