SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Incorrect	Correct/no analysis	Correct/yes analysis
The contract is for Alice to sell her bike to Bob. The contract is governed by the common law, because all goods are governed by the common law.	The contract is for Alice to sell her bike to Bob. The contract is governed by the UCC, because the UCC governs all goods.	The contract is for Alice to sell her bike to Bob. The contract is governed by the UCC, because a bike is a good and all goods are governed by the UCC.
Table 1: An example of how different generations are evaluated for correctness and analysis.
Interpretation LEGALBENCH evaluates interpretation through tasks which require the LLM to parse and understand a legal text. Interpretive tasks provide the LLM with a text, and ask the LLM to either extract a relevant piece of information, answer a question, or categorize the text by some property. Interpretive tasks are among the most studied and practically relevant tasks in LEGALBENCH, and many have been taken from actual use-cases. An example of an interpretive task is cuad_audit_right, which asks the LLM to determine if a contractual clause contains an "audit right." An example is shown below:
Interpretation example: cuad_audit_right
Does the clause give a party the right to audit the books, records, or physical locations of the counterparty to ensure compliance with the contract?
Clause: "We shall have the right at all times to access the information system and to retrieve, analyze, download and use all software, data and files stored or used on the information system."
Rhetorical-understanding LEGALBENCH evaluates rhetorical-understanding through tasks which require an LLM to reason about legal argumentation and analysis. In these tasks, an LLM is provided with a legal argument (usually excerpted from a judicial opinion), and asked to determine whether it performs a certain
function or has a certain property. An example is the definition_classification task, in which an LLM must determine if a sentence from a judicial opinion provides a definition of a term.
Rhetorical-understanding example: definition_classification
Does the sentence define a term?
Sentence: "To animadvert carried the broader implication of "turn[ing] the attention officially or judicially, tak[ing] legal cognizance of anything deserving of chastisement or censure; hence, to proceed by way of punishment or censure." 1 Oxford English Dictionary 474 (2d ed.1989).
We emphasize one aspect of LEGALBENCH: IRAC in this work is used as an organizing principle for grouping tasks. On a law exam, a student would be expected to generate an answer which structurally resembles IRAC, where each step builds on the inferences of the previous step [69, 29]. LEGALBENCH tasks, in contrast, each evaluate a single type of legal reasoning. Hence, a task like learnedHands_benefits can only be used to evaluate issue-spotting, and not rule-recall. In future work we hope to add tasks which evaluate multiple steps jointly.
4 LEGALBENCH tasks
Appendix G discusses each task in detail, providing a description of the reasoning that each task evaluates, how task data was constructed, task examples, and evaluation protocols. This section provides an overview of LEGALBENCH.
4.1 Construction process
Task sources LEGALBENCH tasks are drawn from three sources. The first source of tasks are existing available datasets and corpora. Most of these were originally released for non-LLM evaluation settings. In creating tasks for LEGALBENCH from these sources, we often significantly reformatted data and restructured the prediction objective. For instance, the original CUAD dataset [60] contains annotations on long-documents and is intended for evaluating extraction with span-prediction models. We restructure this corpora to generate a binary classification task for each type of contractual clause. While the original corpus emphasized the long-document aspects of contracts, our restructured tasks emphasize whether LLMs can identify the distinguishing features of different types of clauses. The second source of tasks are datasets that were previously constructed by legal professionals but never released. This primarily includes datasets hand-coded by legal scholars as part of prior empirical legal projects (e.g., [27]). The last category of tasks are those that were developed specifically for LEGALBENCH, by the authors of this paper. Overall, tasks are drawn from 36 distinct corpora.
Collaborative component In August 2022, we published a call for tasks, describing the goals of the project and its structure [57]. We publicized the project through mailing lists and legal computational conferences. Submitted tasks were vetted for legal correctness and task validity. Task contributors are drawn from diverse professional backgrounds within the law (e.g., academics, practitioners, computational legal researchers) and constitute the authors of this paper.
Infrastructure LEGALBENCH comes with support designed to enable non-law AI researchers to use and study LEGALBENCH tasks. First, each LEGALBENCH task is accompanied by extensive documentation describing how the task is performed, its legal significance, and the construction procedure. The objective of this documentation is to provide AI researchers with a working understanding of the mechanical processes behind each task, for the purposes of better understanding LLM performance. Second, each task is accompanied by a "base" prompt, which contains task instructions and demonstrations. The base prompt is provided to promote replicability and standardization. We anticipate that future research efforts building off of LEGALBENCH will identify higher performing prompts/prompt formats. We intended to update the LEGALBENCH GitHub repository with these prompts as they are discovered.
**Limitations** We note several limitations of the current LEGALBENCH tasks (additional limitations are noted in Appendix C). First, when this project began, most LLM context-windows were constrained to a few pages of text. As a result, the initial round of LEGALBENCH tasks does not involve longer documents. We hope to include such tasks in future work, particularly as recent technical developments have resulted in significantly longer context windows [41, 53, 42, 107]. Second, LEGALBENCH's tasks focus on legal reasoning questions with objectively correct answers. LEGALBENCH is thus not helpful for evaluating legal reasoning involving degrees of correctness or tasks where "reasonable minds may differ." Third, LEGALBENCH only considers English language tasks, is skewed towards certain jurisdictions (American law), and certain areas of the law (contracts). Thus, the current iteration of the benchmark limits inferences regarding how LLMs may generalize to legal tasks involving other jurisdictions. As we continue to solicit and incorporate contributions to LEGALBENCH, we hope
to add tasks addressing these limitations. Finally, LEGALBENCH evaluates IRAC abilities independently, while law exams and other legal work requires lawyers to generate outputs which follow IRAC in a multi-hop matter (i.e., each aspect is applied to the same fact pattern).
4.2 Dimensions of variation
Task structure All LEGALBENCH tasks contain at least 50 samples, with an average task size of 563 samples (Appendix E.4). These tasks are comparable in size to those used in benchmarking efforts like BigBench [128], HELM [78] or RAFT [1]. LEGALBENCH tasks also span different formats: multiple-choice questions (35 tasks), open-generation (7 tasks), binary classification (112 tasks), and multi-class/multi-label classification (8 tasks).
Reasoning types and legal domains LEGALBENCH provides tasks for each of the reasoning categories discussed above: rule-recall (5 tasks), issue-spotting (16 tasks), rule-application (16 tasks), rule-conclusion (16 tasks), interpretation (119 tasks), and rhetorical-understanding (10 tasks). Tasks are predominantly drawn from areas of law implicating civil matters, including contracts (58 tasks), civil procedure (8 tasks), evidence law (1 task), and corporate law (58 tasks). The skew towards interpretation tasks and tasks from contract law can be explained by the ubiquity of legal documents from these areas (e.g., contracts, terms-of-service agreements, disclosures, and etc.) and their immediate commercial implications [60, 74].
Language variation Legal language is highly heterogeneous, varying in sentence structure, vocabulary, and rhetorical style across different legal areas and document types [58]. This poses a distinct challenge for LLMs, which are extremely sensitive to structure of input text and the vocabulary used [78]. LEGALBENCH tasks are drawn from a diverse set of legal language types, thus enabling researchers to study performance variation across different categories of legal text. Specifically, LEGALBENCH encompasses tasks with language drawn from plain English (32 tasks), legal opinions (11 tasks), merger agreements (34 tasks), contracts (55 tasks), statutory text (3), and other sources.
We use LEGALBENCH to conduct a three-part study. In the first part (Section 5.2), we evaluate 20 LLMs from 11 different families, at four different size points. In the second part (Section 5.3), we show how LEGALBENCH can be used to conduct in-depth evaluations of models. To illustrate, we use LEGALBENCH to highlight similarities and differences in the performance of three popular commercial models: GPT-4, GPT-3.5, and Claude-1. In the final part (Section 5.4), we show how LEGALBENCH can support the development of law-specific LLM methods. We focus on prompting, and conduct experiments illustrating tradeoffs and challenges with regards to guiding LLMs towards certain tasks. Ultimately, our study here serves to illustrate the types of analyses that LEGALBENCH enables, and highlight potential directions for future work. We summarize the findings of our study here, and provide more details in the Appendix.
Models We study 20 LLMs from 11 different families. These are: GPT-3.5 [14] (text-davinci-003) and GPT-4 from OpenAI [96]; Claude-1 (v1.3) from Anthropic [3]; Incite-Instruct-7B, Incite-Base-7B, and InciteInstruct-3B from Together [34, 133]; OPT-2.7B, OPT-6.7B, and OPT-13B from Meta [154]; Falcon-7B-Instruct from TII [2, 103]; MPT-7B-8k-Instruct from MosaicML [129]; Vicuna-7B-16k and Vicuna-13B-16k from LMSYS [26]; Flan-T5-XL (3B parameters) and Flan-T5-XXL (11B parameters) from Google [31]; LLaMA2-7B, and LLaMA-2-13B from Meta [134]; WizardLM-13B [150]; and BLOOM-3b and BLOOM-7B from BigScience [116]. All inference was performed on two-GPU GCP 40GB A100s, using the Manifest library [97]. HuggingFace links for each model are provided in the Appendix.
**Prompts** We designed a prompt for each task by manually writing instructions for the task, and selecting between zero and eight samples from the available train split to use as in-context demonstration. The number of samples selected depended on the availability of data and the sequence length of samples. For application evaluation, we augmented the prompt with an instruction for the LLM to explain its reasoning. We used the same prompt for all models with one exception (Claude-1). LLM outputs were generated using next-token generation at a temperature of 0.0. For classification/extraction tasks, we terminated at a new-line 353 token. For rule_qa and all application tasks except diversity_jurisdiction_6 we generated 150 tokens. For diversity_jurisdiction_6 we generated 300 tokens.
Evaluation Classification tasks are evaluated using "exact-match" (following HELM [78]). Because some tasks contain significant label imbalances, we use balanced-accuracy as a metric. For extraction tasks, we perform normalization on generated outputs to account for differences in tense/casing/punctuation. A few tasks (e.g., successor_liability and ssla_inividual_defendants) requires the LLM to produce multiple classes or extracted terms per instance. For these, we evaluate using F1. Rule-application tasks were evaluated manually by a law-trained individual, who analyzed LLM responses for both correctness and analysis. The
Appendix provides more details on our approach to manual grading. All manual evaluation was performed with reference to a grading guide, which we additionally make available.
5.2 Performance trends
LLM	Issue	Rule	Conclusion	Interpretation	Rhetorical
GPT-4	82.9	59.2	89.9	75.2	79.4
GPT-3.5	60.9	46.3	78.0	72.6	66.7
Claude-1	58.1	57.7	79.5	67.4	68.9
Flan-T5-XXL	66.0	36.0	63.3	64.4	70.7
LLaMA-2-13B	50.2	37.7	59.3	50.9	54.9
OPT-13B	52.9	28.4	45.0	45.1	43.2
Vicuna-13B-16k	34.3	29.4	34.9	40.0	30.1
WizardLM-13B	24.1	38.0	62.6	50.9	59.8
BLOOM-7B	50.6	24.1	47.2	42.8	40.7
Falcon-7B-Instruct	51.3	25.0	52.9	46.3	44.2
Incite-7B-Base	50.1	36.2	47.0	46.6	40.9
Incite-7B-Instruct	54.9	35.6	52.9	54.5	45.1
LLaMA-2-7B	50.2	33.7	55.9	47.7	47.7
MPT-7B-8k-Instruct	54.3	25.9	48.9	42.1	44.3
OPT-6.7B	52.4	23.1	46.3	48.9	42.2
Vicuna-7B-16k	3.9	14.0	35.6	28.1	14.0
BLOOM-3B	47.4	20.6	45.0	45.0	36.4
Flan-T5-XL	56.8	31.7	52.1	51.4	67.4
Incite-3B-Instruct	51.1	26.9	47.4	49.6	40.2
OPT-2.7B	53.7	22.2	46.0	44.4	39.8
Table 2 provides the average task performance for all 20 models in five reasoning categories. The first block of rows corresponds to large commercial models, the second block corresponds to models in the 11B-13B range, the third block corresponds to models in the 6B-7B range, and the final block corresponds to models in the 2B-3B range. Table 3 provides the average task performance for the three large models on rule-application. The Appendix provides full results for each model on each task.
We highlight several observations. First, within LLM families, larger models usually outperform smaller models. For instance, Flan-T5-XXL (11B parameters) outperforms 372 Flan-T5-XL (3B parameters) on average across all five reasoning categories, and LLaMA-2-13B outperforms 373 LLaMA-2-7B on average across four reasoning categories. Second, even for LLMs of the same size, we find considerable differences in performance. For instance, we observe significant gaps in performance between Flan-T5-XXL (11B parameters) and Vicuna-13B-16k (13B parameters), across all reasoning categories. This suggests, unsurprisingly, that the choice of pretraining data, regime of instruction-tuning, and architecture play an important role in determining performance, and that certain configurations may be better aligned for LEGALBENCH tasks. Interestingly, such choices may affect which types of reasoning categories LLMs appear to perform well at. For instance, WizardLM-13B performs worse than all peers on issue-spotting tasks, best on rule-recall tasks, and nearly matches the performance of the best-performing peer on rule-conclusion tasks. Third, we find evidence that open-source models are
Table 2: Average performance for each LLM over the different LEGALBENCH categories. The first block of rows corresponds to large commercial models, the second block corresponds to models in the 11B-13B range, the third block corresponds to models in the 6B-7B range, and the final block corresponds to models in the 2B-3B range. The columns correspond to (in order): issue-spotting, rule-recall, rule-conclusion, interpretation, and rhetorical-understanding. For each class of models (large, 13B, 7B, and 3B), the best performing model in each category of reasoning is underlined.
LLM	Correctness	Analysis
GPT-4	82.2	79.7
GPT-3.5	58.5	44.2
Claude-v1	61.4	59.0
Table 3: Average performance for the large LLMs on rule-application tasks.
capable of performance that matches or exceeds certain commercial models. For instance, Flan-T5-XXL outperforms GPT-3.5 and Claude-1 383 on two categories (issue-spotting and rhetorical-understanding), despite the relative gap in parameter count. Notably, the gap between closed and open-source models is largest for the rule-conclusion category. Amongst LEGALBENCH tasks, rule-conclusion tasks most like the other types of multi-step/common-sense reasoning tasks where commercial LLMs have been found to perform well.
5.3 Comparing commercial models
Next, we conduct a more in-depth study of performance, focusing on the three commercial models (GPT-4, GPT-3.5, and Claude-1). In particular we highlight how LEGALBENCH can provide more rigorous empirical support for anecdotal observations arising out of the legal community's use of these models, and explain performance differences between models. The Appendix provides a more in-depth analysis.
We highlight two major findings. First, evaluation on LEGALBENCH reveals that the largest performance difference between GPT-4 and GPT-3.5/Claude-1 occurs (on average) for rule-application tasks. On rule application tasks, we observe that GPT-4 outperforms both GPT-3.5 ( $p < 0.01$ ) and Claude-1 ( $p < 0.01$ ) on both correctness and analysis. Across LLMs, we find that variation in performance across tasks is consistent with subjective impressions of task difficulty. For instance, performance on diversity Jurisdiction_1 (an easy task requiring a model to determine if an amount is greater than  $\$75\mathrm{k}$  and if the plaintiff and defendant are from different states) is much higher than performance on successor_liability (a harder task requiring a model to identify multiple successor liability exceptions in a fact pattern describing a complex transaction). Second, on the interpretation tasks, we find that on average GPT-4 outperforms GPT-3.5 ( $p < 0.01$ ), and GPT-3.5 outperforms Claude-1 ( $p < 0.01$ ). Here, the larger API-models are highly performant on tasks which involve binary classification over short clauses. Averaged across the 38 CUAD tasks (contract clauses), for instance, GPT-4, GPT-3.5, and Claude-1 all have a balanced-accuracy  $\geq 88\%$ . And on proa (statutory clauses), both GPT-4 and GPT-3.5 have a balanced-accuracy  $\geq 90\%$ . Notably, performance degrades on tasks which contain longer text sequences or involve multi-class classification. On the Supply Chain Disclosure tasks for instance—in which LLMs must classify disclosures which are 1-2 pages in length—the average balanced-accuracy of the large commercial models ranges between  $74 - 75\%$ . And on the MAUD tasks—which require answering multiple choice questions about merger deals—the average balanced-accuracy of GPT-4 drops to  $47.8\%$  accuracy.
5.4 Prompt engineering strategies
Finally, we illustrate—through a series of micro-studies—how LEGALBENCH can be used to explore different aspects of prompt-engineering for LLMs in legal settings. We focus on three questions: (1) Can LLMs rely on their latent knowledge of a rule for rule-conclusion tasks? (2) Does simplifying task descriptions to plain language affect performance? (3) Are LLMs sensitive to the choice of in-context demonstrations?
We highlight our findings (full results can be found in the Appendix). With regards to (1), we find considerable variation over studied tasks—some rules appear to be sufficiently well known that explaining the rule to the LLM in the prompt has a negligible impact on performance. With regards to (2), we find that a plain-language prompt significantly outperforms the technical language prompt, by up to 21 points (balanced-accuracy) on the subset of studied tasks. And with regards to (3), we find that LLMs are highly sensitive to the choice of in-context samples, with performance differences of up to 20pts (balanced-accuracy). These results illustrate that further work on understanding how to effectively and efficiently prompt LLMs for legal tasks is needed.
Our work here describes LEGAL BENCH: a collaboratively constructed benchmark of 162 tasks for measuring the legal reasoning capabilities of LLMs. In future work, we hope to expand this project, by continuing to solicit and collect interesting and useful tasks from the legal community
We are grateful to the following individuals and groups for feedback on this project: Alex Chao, Amit Haim, Arjun Desai, Armin Thomas, Avanika Narayan, Ben Spector, Brandon Yang, Eric Nguyen, Gautam Machiraju, Javed Qadrud-Din, Jian Zhang, Jonathan Zittrain, Karan Goel, Khaled Saab, Joshua Arp, Krista Opsahl-Ong, Laurel Orr, Lisa Ouellette, Lucia Zheng, Martin Gajek, Mayee Chen, Michael Zhang, Mike Wornow, Pablo Arredondo, Percy Liang, Rishi Bommasani, Roland Vogl, Sabri Eyuboglu, Sarah Hooper, Sergio Servantez, Simran Arora, Tengyu Ma, Tony Kim, Tri Dao and Vishnu Sarukkai. We presented and recieved feedback on earlier versions of this project at various forums, including: the Center for Research on Foundation Models, the Stanford Regulation and Governance Lab, the New York LLM x Law Hackathon (June 2023), the 2023 Stanford Data Science Conference, the 2023 Stanford CodeX Conference, and the Stanford Generative AI and Foundation Models Workshop. We are grateful to the organizers and attendees of these events for engaging with our work.
We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), 2204926 (Computational Statutory Reasoning), and 1937301 (RTML); US DEVCOM ARL under No. W911NF-21-2-0251 (Interactive Human-AI Teaming); ONR under No. N000141712266 (Unifying Weak Supervision); ONR N00014-20-1-2480: Understanding and Applying Non-Euclidean Geometry in Machine Learning; N000142012275 (NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the Center for Research on Foundation Models, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Facebook, Google, and VMWare. We thank Casetext for assistance with evaluating GPT-4. PH is supported by an Open Philanthropy AI Fellowship. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
For the sake of clarity, we provide a brief outline of the Appendix.
- Appendix B provides more information on the availability of LEGAL BENCH.
- Appendix C discusses limitations of the project and potential negative social impacts.
- Appendix D provides a top-level datasheet.
- Appendix E provides an overview of the LEGALBENCH tasks, and contains information on licenses, which datasets have been published before, evaluation protocols, and dataset statistics.
- Appendix G provides a description of each task/task-family. For each set of tasks, we provide background on the relevant legal concepts, describe the tasks, discuss how the task was constructed, provide examples of inputs/outputs, and explain their legal significance.
- Appendix H provides the full-results for evaluated LLMs on all LEGALBENCH tasks, along with additional information about prompting.
B Availability
- The LEGALBENCH website is available at https://hazyresearch.stanford.edu/legalbench/.
- Data can be downloaded from Huggingface here: https://huggingface.co/datasets/nguha/legalbench.
- The Github repository with evaluation code and prompts is available here: https://github.com/HazyResearch/legalbench/
Gradebooks to guide manual evaluation can be found on the website.
C Limitations and social impact
**Limitations** We note several limitations of our work. Legal applications—and what constitutes "legal reasoning"—is broad. Thus, LEGALBENCH will necessarily be an incomplete effort, and important tasks/document types/reasoning types are not included. To enumerate a few examples: