Title: LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Benchmark: LEGALBENCH
Summary: LEGALBENCH is a comprehensive benchmark for evaluating language models' legal reasoning capabilities, consisting of 162 tasks spanning six distinct categories of legal reasoning that were designed and hand-crafted by legal professionals. The benchmark provides evaluation code, prompts, and a common vocabulary bridging legal frameworks and AI development, enabling rigorous assessment of both open-source and commercial language models on practically useful legal reasoning skills.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: (legal) reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: Six reasoning types: issue‑spotting, rule‑recall, rule‑application, rule‑conclusion, interpretation, rhetorical‑understanding. 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: input text (from single sentences to two‑page documents) + prompt + label (classification/ generation). 
Task item definition: one legal question, text snippet, gold answer/label.
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: LEGALBENCH was constructed from a mix of existing legal datasets (restructured for the few-shot LLM paradigm), and hand-crafted datasets created and contributed by legal professionals (included as authors on this work).
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Clause classification, supply-chain disclosure compliance, privacy-policy entailment
Dataset size: 91000
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Response format detail: LEGALBENCH tasks also span different formats: multiple-choice questions (35 tasks),
open-generation (7 tasks), binary classification (112 tasks), and multi-class/multi-label classification (8 tasks). Tasks range from 50–2 000 samples (avg. approximately 563 samples per task)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Weighted Mean
Metric statistics: Mean and Standard deviation
Metric subscores: Yes
Metric subscores detail: 162 individual task scores (task structure/reasoning types/legal domains/language variation)
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Partially; addressed their own limitations.
Author validity detail: We note that the purpose of this work isn’t to evaluate whether computational systems should replace lawyers and legal officers, or to understand the positive and negative
impacts of that replacement. Our goal is to construct artifacts that enable the relevant stakeholders and affected communities to better understand, empirically, the capacity for LLMs to perform different types of legal tasks. Given the proliferation of computational legal tools, we believe that answering this question is vital for ensuring their safe and ethical usage. (section 1) 
Taxonomy: Law / 
New taxonomy: Specific Application / Law