Title: PAGED: A Benchmark for Procedural Graphs Extraction from Documents
Benchmark: PAGED
Summary: Propose a dataset. Find that baseline models cannot extract optimal procedural graphs well, and that LLMs have advantages in building relevant structures.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: procedural graph extraction
Phenomenon defined by authors: Yes
Phenomenon definition: "Procedural graphs... intuitively represent the execution of actions for goal achievement", and this paper focuses on the "automatic extraction of procedural graphs from procedural documents"-p10829
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: given a procedural text, the model has to extract the procedural graph out of it
Task item definition: a procedural text
Task source: Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1131 (total is 3394, train:val:test=3:1:2)
Dataset size notes: train: 1697, val: 566 (total is 3394, train:val:test=3:1:2)
Dataset metadata available: Yes
Dataset metadata detail: actor, action, constraint, gateway, flow
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric subscores: Yes
Metric subscores detail: Different constraints, different gateways, different flows
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: No
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: they conduct human evaluation
Taxonomy: NLP / Extraction
New taxonomy: General Application / Extraction