SlowGuess/NeurIPS_2023_OCR huggingface dataset:
LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Neel Guha\*, Julian Nyarko\*, Daniel E. Ho\*, Christopher Re\*, Adam Chilton\*, Aditya Narayana\*, Alex Chohlas-Wood\*, Austin Peters\*, Brandon Waldon\*, Daniel N. Rockmore\*, Diego Zambrano\*, Dmitry Talisman\*, Enam Hoque\*, Faiz Surani\*, Frank Fagan\*, Galit Sarfaty\*, Gregory M. Dickinson\*, Haggai Porat\*, Jason Hegland\*, Jessica Wu\*, Joe Nudell\*, Joel Niklaus\*, John Nay\*0, Jonathan H. Choi\*1, Kevin Tobia\*2, Margaret Hagan\*3, Megan Ma\*0, Michael Livermore\*4, Nikon Rasumov-Rahe\*3, Nils Holzenberger\*5, Noam Kolt\*7, Peter Henderson\*1, Sean Rehaag\*6, Sharad Goel\*7, Shang Gao\*20, Spencer Williams\*8, Sunny Gandhi\*9, Tom Zur\*9, Varun Iyer, and Zehua Li
$^{1}$ Stanford University,  $^{2}$ University of Chicago,  $^{3}$ Maxime Tools,  $^{4}$ Dartmouth College,  $^{5}$ LawBeta,  $^{6}$ South Texas College of Law Houston,  $^{7}$ University of Toronto,  $^{8}$ St. Thomas University Benjamin L. Crump College of Law,  $^{9}$ Harvard Law School,  $^{10}$ Stanford Center for Legal Informatics - CodeX,  $^{11}$ University of Southern California,  $^{12}$ Georgetown University Law Center,  $^{13}$ Stanford Law School,  $^{14}$ University of Virginia,  $^{15}$ Télécom Paris, Institut Polytechnique de Paris,  $^{16}$ Osgoode Hall Law School, York University,  $^{17}$ Harvard Kennedy School,  $^{18}$ Golden Gate University School of Law,  $^{19}$ Luddy School of Informatics - Indiana University Bloomington,  $^{20}$ Casetext
The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LEGALBENCH: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LEGALBENCH was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning—which distinguish between its many forms—correspond to LEGALBENCH tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LEGALBENCH, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LEGALBENCH enables.
Advances in large language models (LLMs) are leading American legal professionals to reexamine the practice of law [52, 61, 158, 55, 13]. Proponents have argued that LLMs could alter how lawyers approach tasks ranging from brief writing to corporate compliance [158]. By making legal services more accessible, they could eventually help alleviate the United States' long standing access-to-justice crisis [35, 132]. This perspective is informed by the observation that LLMs possess special properties
$^{2}$ In using "LLMs", we are referring to language models which evince in-context learning capabilities (also referred to as "foundation models" [13]). This behavior has traditionally been observed in models with at least a billion parameters.
which, it is argued, make them more suited for legal tasks. The models' capacity to learn new tasks from limited labeled data would reduce the manual data annotation costs that ordinarily burden the development of legal language models [13]. Their apparent proficiency at sophisticated reasoning tasks would also make them ideal for the rigor of law, which requires parsing obtuse texts with heavy jargon, and inferential processes which combine different modalities of reasoning [155].
This excitement, however, is tempered by the fact that legal applications often involve significant risk [47]. Existing work has shown that LLMs are capable of generating content that is offensive, misleading, and factually incorrect [10, 78]. Such behaviors—if replicated in legal applications [112]—could result in substantial harms [144], with much of the potential burden imposed on traditionally marginalized and under-resourced populations [125, 136]. The safety implications thus create a pressing need to develop infrastructure and processes for benchmarking LLMs in legal contexts.
However, significant challenges face practitioners seeking to assess whether LLMs can perform legal reasoning. The first challenge is the limited ecosystem of legal benchmarks [155]. The majority of existing benchmarks, for example, focus on tasks which models learn by finetuning or training on task-specific data [21]. These benchmarks do not measure the aspects of LLMs which generate excitement for law—namely, their ability to perform many different tasks using only few-shot prompts. Relatedly, benchmarking efforts have focused on professional certification exams like the Uniform Bar Exam [69], but these are not always representative of the actual use-cases for LLMs. The second challenge is the incongruity between the ways in which existing benchmarks and lawyers frame "legal reasoning." Existing benchmarks coarsely generalize all tasks involving legal data or laws as measuring "legal reasoning." In contrast, lawyers recognize that legal reasoning is a broad umbrella term encompassing many distinct types of reasoning [46]. Different legal tasks require different skills and bodies of knowledge. Because existing legal benchmarks fail to draw these distinctions, it is difficult for legal professionals to contextualize the performance of modern LLMs within their own understanding of legal competency. In short: legal benchmarks do not use the same vocabulary or conceptual frameworks as the legal profession.
In light of these limitations, we believe that rigorously evaluating the legal reasoning capabilities of LLMs will require the legal community to take a more proactive role in the process of benchmarking. To that end, we present LEGALBENCH: the first steps towards constructing an interdisciplinary collaborative legal reasoning benchmark for the English language.<sup>3</sup> Over the past year, the authors of this paper—drawing from their diverse legal and computer science backgrounds—came together to assemble 162 tasks (from 36 different data sources), each of which measures a specific type of legal reasoning. LEGALBENCH is thus, to the best of our knowledge, the first open-source legal benchmarking effort. We believe that this style of benchmark construction—where domain experts take an active and participatory role in the crafting of evaluation tasks—illustrates one approach to interdisciplinary collaboration in LLM research. Importantly, we believe it also shows that legal professionals have an essential role to play in the assessment and development of LLMs for law.
As a research project, we highlight three components of LEGALBENCH:
1. LEGALBENCH was constructed from a mix of existing legal datasets (restructured for the few-shot LLM paradigm), and hand-crafted datasets created and contributed by legal professionals (included as authors on this work). The legal professionals involved in this collaboration were asked to contribute datasets that they believed to either measure an interesting legal reasoning skill, or to capture a practically useful application for LLMs in the law. High performance on LEGALBENCH tasks thus provides useful information, allowing lawyers to validate their assessment of an LLM's legal competency, or identify an LLM that could be used in their workflow.
2. LEGAL BENCH tasks are organized into an extensive typology which describes the types of legal reasoning required to perform the task. Because this typology is drawn from frameworks familiar to the legal community, it enables legal professionals to meaningfully engage in discussions of LLM performance, using a terminology and conceptual framework familiar to them [46, 122].
3. Finally, LEGALBENCH is intended as a platform to support further research. For AI researchers who lack legal expertise, LEGALBENCH comes with significant support for understanding how to prompt and evaluate different tasks. And as more of the legal community begins to engage with
<sup>3</sup>https://github.com/HazyResearch/legalbench/
the potential impact and role of LLMs, we hope to grow LEGALBENCH by continuing to solicit and incorporate tasks from legal professionals.4
In this paper, we make several contributions. First, we present a typology for organizing and describing legal tasks in terms of the types of reasoning they require. This typology is drawn from frameworks lawyers use to describe legal reasoning [122]. Second, we provide an overview of the tasks in LEGALBENCH, describing the process by which they were constructed, important dimensions of heterogeneity, and limitations. A full description of each task is provided in the Appendix. Finally, we use LEGALBENCH to evaluate 20 LLMs from 11 different families, across a range of size points. We make observations regarding the performance of different models and present an initial study into different prompt-engineering strategies. Ultimately, these results are intended to highlight different directions of future work that LEGALBENCH may enable.
We hope that this benchmark will be interesting to a diverse set of communities. Practitioners may use these tasks to determine whether and where LLMs can be integrated into existing workflows to improve outcomes for clients. Legal academics may benefit from observing the types of annotation that LLMs are capable of [157], and different forms of empirical scholarly work they may enable. Computer scientists may benefit from studying the performance of these models in a domain like law, where distinct lexical properties and unique tasks may surface new insights.
Before we progress further, we note that the purpose of this work isn't to evaluate whether computational systems should replace lawyers and legal officers, or to understand the positive and negative impacts of that replacement [47, 126, 4]. Rather, our goal is to construct artifacts that enable the relevant stakeholders and affected communities to better understand, empirically, the capacity for LLMs to perform different types of legal tasks. Given the proliferation of computational legal tools, we believe that answering this question is vital for ensuring their safe and ethical usage.
**Benchmarking legal reasoning** Understanding the extent to which NLP models can perform tasks or skills traditionally associated with lawyers—or be useful in legal analysis—has been the focus of significant work [70, 6, 108, 124, 82, 84, 102, 86, 22]. Prior work has identified manually arduous tasks currently performed by lawyers (e.g., document review, case summarization) and developed corresponding benchmarks [60, 140, 118, 119, 88, 67]. Researchers have focused on the technically challenging asepts of legal tasks, like document length, jargon, or inferential reasoning [21, 109, 85, 40, 77, 21, 78, 155, 63]. Other work has focused on creating datasets for pretraining models [58, 127], non-English/multilingual tasks [93, 94, 64, 149, 51, 92, 68, 20, 23, 101, 18], legal judgement prediction [87, 37, 17, 156], legal role labeling [83], and different forms of retrieval [66].
Importantly, the majority of previous benchmarking efforts have focused on language models which learn by supervised training or finetuning (e.g., BERT variants [44]), and researchers have consequently studied questions related to the role of domain specific datasets [155, 19, 20]. More recently, researchers have begun to ask whether large language models (LLMs) like GPT-3/4 can perform legal reasoning [71, 151, 65, 12, 28, 30, 152], citing to evidence of these models' capacity to perform sophisticated reasoning tasks in domains like math or programming [143, 24]. Unlike BERT-based models, LLMs are evaluated on their ability to learn tasks in-context, primarily through prompting. While a few works have experimented with LLMs on existing benchmarks [16, 12], most evaluations focus on standardized tests or other exam equivalents [69, 91, 29]. Studies have explored the role of prompt-engineering [152, 151, 73], potential applications [91, 28, 145, 115, 114], questions regarding human-LLM interaction [30, 61], and comparisons to older finetuned-models [89].
Connections to other LLM benchmarking efforts We highlight connections to two broader research efforts. First, we draw inspiration from existing efforts within NLP and machine learning to define fine-grained measures of performance, which allow researchers to discuss model capabilities with precision and specificity. Examples include the diagnostic set of the GLUE Benchmark [139], the "reasoning patterns" studied in [98], the task organization used in HELM [78], and the BigBench effort [121]. We believe this paradigm of expert-driven evaluation is essential for specialized domains like law.
Cognizant of LEGALBENCH's current skew towards American law, we hope that additional contributions incorporate tasks from other jurisdictions.
3 The LEGALBENCH typology
LEGALBENCH identifies six types of legal reasoning that LLMs can be evaluated for: (1) issue-spotting, (2) rule-recall, (3) rule-application, (4) rule-conclusion, (5) interpretation, and (6) rhetorical-understanding. We first justify the selection of these types by providing background on how the legal profession frames "legal reasoning," and the connections to our typology. We then illustrate how task datasets may be used to evaluate LLMs for each type, using examples from LEGALBENCH.
Though this framework draws heavily on American legal thought, we believe it can be extended to characterize LEGALBENCH tasks that implicate non-American bodies of law. We also note that our types are non-exhaustive, and in future work hope to consider extensions.
3.1 Frameworks for legal reasoning
IRAC American legal scholars often describe "legal reasoning" as the process of determining the legal conditions that arise from a set of events or occurrences, with reference to both prior cases and codified laws [46]. A common framework for executing this type of legal reasoning is the Issue, Rule, Application and Conclusion (IRAC) framework [146, 122]. In this framework, legal reasoning decomposes into four sequential steps.
First, lawyers identify the legal issue in a given set of facts (issue-spotting). An issue is either (1) a specific unanswered legal question posed by the facts, or (2) an area of law implicated in the facts. Depending on the setting, a lawyer may be told the issue, or be required to infer a possible issue.
Second, lawyers identify the relevant legal rules for this issue (rule-recall). A rule is a statement of law which dictates the conditions that are necessary (or sufficient) for some legal outcome to be achieved. In the United States, rules come from different sources: the Constitution, federal and state statutes, regulations, and court opinions. Importantly, rules often differ between jurisdictions. Hence, the relevant rule in California might be different than the relevant rule in New York.
Third, lawyers apply these rules to the facts at hand (rule-application). Application, or the analysis of rule applicability, consists of identifying those facts which are most relevant to the rule, and determining how those facts influence the outcome under the rule. Application can also involve referencing prior cases involving similar rules (i.e. precedent), and using the similarities or differences to those cases to determine the outcome of the current dispute.
Finally, lawyers reach a conclusion with regards to their application of law to facts, and determine what the legal outcome of those facts are (rule-conclusion).
Example Suppose that BusinessMart—a large manufacturing corporation—is being sued by Amy in federal court on diversity jurisdiction.5 BusinessMart sells the majority of its goods in Texas, has its headquarters (where its CEO and board members sit and work) in California, and maintains a factory in Florida. A court is trying to determine—for the purposes of diversity jurisdiction—where BusinessMart's "principal place of business is."
- Issue-spotting: Here, a narrow issue is offered—where is BusinessMart's principal place of business?
- Rule-recall: A lawyer would recognize that the most relevant rule here comes from the case Hertz Corp. v. Friend, in which the Supreme Court determined "that the phrase 'principal place of business' refers to the place where the corporation's high level officers direct, control, and coordinate the corporation's activities."
- Rule-application: Applying this rule to the facts above yields two observations. First, a corporation's CEO and board members are examples of high level officers referred to in  $Hertz$  that control and conduct a company. Second, the place where BusinessMart's high level officers control the company is California, as that is where the CEO and board sit and work.
- Rule-conclusion: Based on the chain of inference spelled out in the application stage, a lawyer would thus conclude that California is BusinessMart's principal place of business.
<sup>5</sup>Diversity jurisdiction gives federal courts the ability to hear cases between parties that are "citizens" of different states.
6Hertz Corp. v. Friend, 559 U.S. 77 (2010).
The extent to which the outcome of the application and conclusion steps follow each other is dictated by the level of ambiguity in the fact patterns. When the law on a particular question is clear and there is little ambiguity in the facts (as the case in the above example), then the application and conclusion steps point towards the same outcome. Sometimes however, the facts may be unclear or contested, and reasonable minds may differ as the conclusion step. For now, LEGALBENCH focuses entirely on the former setting (unambiguous answers), and all tasks are considered to have objectively "correct" answers.
Other types of reasoning Though IRAC is the most formal framework for legal reasoning, lawyers recognize a variety of skills which are useful to practice of law [46, 75]. For instance, lawyers are often required to exercise interpretive skills, in order to identify the rights, obligations, or limitations of certain legal language (e.g., what a contractual clause may or may not enable). They must also exhibit rhetorical skills, and understand the types of arguments that are made. Though these tasks require the knowledge base and skill set of lawyers, they, arguably, do not always fit neatly within the IRAC framework. Hence, we consider these to be distinct from the examples offered in the previous section.
3.2 Evaluating legal reasoning in large language models
LEGALBENCH identifies six categories of legal reasoning. For each category, we describe how a LLM task may evaluate the typified legal reasoning, using examples from LEGALBENCH.
Issue-spotting LEGALBENCH evaluates issue-spotting through tasks in which an LLM must determine if a set of facts raise a particular set of legal questions, implicate an area of the law, or are relevant to a specific party. Issue tasks evaluate a LLM's ability to reason over the legal implications of different activities, events, and occurrences. An example of an issue-spotting task is the learned_hands_benefits task, which requires an LLM to determine (Yes/No) whether a post on a public legal aid forum raises issues related to welfare law (i.e., public benefits or social services). The box below shows how a LLM might be prompted for this task.
Issue-spotting example: learnedHands_benefits
Does the post discuss public benefits and social services that people can get from the government, like for food, disability, old age, housing, medical help, unemployment, child care, or other social needs?
Post: "I am currently receiving support from social services, idk why, this is just how my life turned out. They have asked for all of my bank information for the past 12 months. I don't know what this means. Why would they want that?"
Answer: Yes
Rule-recall LEGALBENCH evaluates rule-recall through tasks which require the LLM to generate the correct legal rule on an issue in a jurisdiction (e.g., the rule for hearsay in US federal court). A rule task can be an open-ended generation task—in which the LLM must generate the text of the rule for a jurisdiction—or a classification task—in which the LLM must determine whether the rule exists in that jurisdiction. Anchoring to jurisdiction is important, as legal rules differ across different jurisdictions. Rule tasks are particularly useful for measuring hallucinations [79]. An example of a rule-recall task is rule_qa, a question-answer task where questions include asking the model to state the formulations for different legal rules, identify where laws are codified, and general questions about doctrine.
Rule-recall example: rule_qa
Question: What are the four requirements for class certification under the Federal Rules of Civil Procedure?"
Answer: Numerosity, commonality, typicality, adequacy
Rule-conclusion LEGALBENCH evaluates rule-conclusion through tasks which require an LLM to determine the legal outcome of a set of facts under a specified rule. LLMs are evaluated purely on whether their predicted outcome is correct. For example, the ucc_v_common_law task asks a LLM to determine whether a contract is governed by the Uniform Commercial Code (UCC) or the common law of contracts. The LLM is always provided with the relevant rule, via the prompt (see below).
Conclusion example: ucc_v_common_law
The UCC (through Article 2) governs the sale of goods, which are defined as moveable tangible things (cars, apples, books, etc.), whereas the common law governs contracts for real estate and services. For the following contracts, determine if they are governed by the UCC or by common law.
Contract: Alice and Bob enter into a contract for Alice to sell her bike to Bob for $50. Is this contract governed by the UCC or the common law?
Governed by: UCC
Rule-application LEGALBENCH evaluates rule-application through the same tasks used to measure rule-conclusion. When evaluating rule-application however, we prompt the LLM to provide an explanation of how the rule applies to a set of facts, and evaluate the quality of the generated explanation along two dimensions: (1) whether the explanation is correct, and (2) whether it contains analysis. Each metric captures a different dimension upon which a particular rule-application may be good.
Correctness corresponds to the criteria that explanations should not contain errors. We focus on five types of errors: misstatements of the legal rule, misstatements of the fact pattern, incorrectly asserting the legal outcome, logic errors, and arithmetic errors. Analysis corresponds to the criteria that explanations should contain inferences from the facts that are relevant under the rule, and illustrate how a conclusion is reached. Consider, for example, an explanation which restates the rule, the fact pattern, and the predicted legal outcome. If the predicted legal outcome is correct, than the explanation in its entirety would be correct, because it contains no error. However, as prior works have noted [69, 29], examples like this are conclusory, and often unsatisfactory in the context of legal work.
To standardize evaluation and enable future work, we have released an "answer guide" for each task used for rule-application, which contains the inferences required for each sample, and describes common modes of errors. All evaluations in LEGALBENCH for rule-application have been performed with respect to this answer-guide.
Table 1 presents an examples of how three generations (corresponding to the Alice/Bob example above) would be evaluated under the above metrics. The first generation is incorrect, because it misstates the rule. The second generation is correct because it contains no falsehoods, but performs no analysis because it does not articulate inferences. The third generation is both correct and contains analysis, because it has no errors, and explicitly mentions an essential inference (e.g., that a bike is a "good").