Title: STREET: A MULTI-TASK STRUCTURED REASONING AND EXPLANATION BENCHMARK
Benchmark: STREET (Structured REasoning and Explanation Multi-Task benchmark)
Summary: This paper introduces STREET, a unified multi-task benchmark designed to evaluate natural language reasoning and explanation capabilities. Unlike typical QA datasets, STREET requires models not only to answer questions but also to generate structured, step-by-step explanations (reasoning graphs) detailing the derivation process. Evaluations using T5 and GPT-3 indicate that current models struggle to produce accurate reasoning graphs, lagging behind human performance.
Contribution: The paper proposes the STREET benchmark, unifying diverse reasoning tasks (math, logic, science QA) under a common framework. It introduces "reasoning graphs" as a novel format for structured explanations. The benchmark provides reasoning graph annotations (human-annotated or programmatically generated) for over 35k questions from existing datasets. It evaluates T5 and GPT-3 on generating these structured explanations, revealing limitations of current models. The dataset and code are publicly released.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multi-step structured reasoning and explanation generation in natural language.
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to perform multi-step reasoning to answer a question and concurrently generate a structured explanation ("reasoning graph"). This graph explicitly links premises (represented as Textual Logical Units or TLUs) to intermediate conclusions and the final answer, showing the derivation path.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: To provide a benchmark for evaluating the generation of structured explanations, going beyond free-form rationales, and focusing on reasoning primarily from the provided input context rather than external knowledge retrieval.
Task definition:  Given a question (potentially with context and answer options), generate the correct answer and a corresponding reasoning graph. The reasoning graph is represented as a sequence of textual reasoning steps, where each step explicitly references the premise Textual Logical Units (TLUs).
Task item definition: A QA instance (from ARC, SCONE, GSM8K, AQUA-RAT, or AR-LSAT) comprising context, question, answer options (if any), and the gold answer. This is augmented with segmented Textual Logical Units (TLUs) for all components and a reasoning graph represented by links between premise TLUs and conclusion/reasoning step TLUs.
Task definition detail: Explanations are structured as Directed Acyclic Graphs (DAGs). The benchmark focuses on reasoning where premises are mostly contained within the input text. It adapts multiple existing datasets into this unified structured explanation format.
Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: Reasoning graphs were added to existing datasets (ARC, SCONE, GSM8K, AQUA-RAT, AR-LSAT). This involved programmatic generation (SCONE), expert annotation based on existing rationales (GSM8K, AQUA-RAT), expert annotation from scratch (AR-LSAT), or adapting existing structured explanations (ARC from ENTAILMENTBANK). Annotators were experts with relevant educational backgrounds.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The benchmark uses established QA problems and adds a requirement for structured explanations, aligning with the need for explainable AI in complex reasoning scenarios.
Dataset size: Total questions included: 19,096. Test set sizes vary per task (not explicitly summed). Total reasoning steps: 151,093.
Dataset size notes: Train and Dev splits exist, derived from source datasets.
Dataset metadata available: Yes
Dataset metadata detail: Source Task/Domain, Answer Type, Textual Logical Units (TLUs) with IDs for all components (context, question, options, answer, rationale steps), Reasoning Graph Edges (dependency links between TLU IDs).
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph)
Response format detail: The model generates a single text sequence containing the reasoning steps and their dependencies encoded using the specific syntax (e.g., premise IDs -> conclusion ID: conclusion text;) followed by the final answer.
Metric definition: Exact Match (accuracy, F1, precision, recall), Reasoning Graph Accuracy and Reasoning Graph Similarity based on graph edit distance and textual similarity e.g. BLEURT.
Metric definition detail: 1. Answer Accuracy (Exact Match for numerical/MCQ, custom state match for SCONE). 2. Reasoning Graph Accuracy (strict structural and textual match). 3. Reasoning Graph Similarity (normalized graph edit distance using task-specific node text similarity - exact, numeric, or BLEURT)
Metric aggregation: Simple Mean
Metric statistics: Answer Accuracy (Exact Match %), Reasoning Graph Accuracy (%), Reasoning Graph Similarity (%).
Metric subscores: Yes
Metric subscores detail: Results are reported separately for each of the 5 source tasks (ARC, SCONE, GSM8K, AQUA-RAT, AR-LSAT).
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Built on reputable QA datasets. Used expert annotators (undergrad/grad level) with guidelines and multiple passes for quality control. Achieved substantial inter-annotator agreement (Fleiss Kappa κ=0.79) for graph structure annotation. Dataset analysis shows complex reasoning structures (avg 7.8 steps, multi-premise steps)
Taxonomy: Reasoning / 
New taxonomy: General Capability / Logical Reasoning
Procedural notes: Details the reasoning graph linearization format. Specifies models used (T5-large, GPT-3 text-davinci-002) and training/prompting setup. 
Defines custom evaluation metrics (Graph Accuracy, Graph Similarity) and node similarity functions. 
An annotation tool is described, and a screenshot is provided.   

Other notes: The core novelty lies in the structured "reasoning graph" representation for explanations, contrasting with free-form rationales. The benchmark explicitly tests the generation of these structures, revealing it's harder for models than just getting the final answer right.