LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Authors: Neel Guha
Julian Nyarko
Daniel E. Ho
Christopher Re
Adam Chilton
Aditya Narayana
Alex Chohlas-Wood
Austin Peters
Brandon Waldon
Daniel Rockmore
Diego Zambrano
Dmitry Talisman
Enam Hoque
Faiz Surani
Frank Fagan
Galit Sarfaty
Gregory M. Dickinson
Haggai Porat
Jason Hegland
Jessica Wu
Joe Nudell
Joel Niklaus
John J Nay
Jonathan H. Choi
Kevin Tobia
Margaret Hagan
Megan Ma
Michael Livermore
Nikon Rasumov-Rahe
Nils Holzenberger
Noam Kolt
Peter Henderson
Sean Rehaag
Sharad Goel
Shang Gao
Spencer Williams
Sunny Gandhi
Tom Zur
Varun Iyer
Zehua Li
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: large language models
law
benchmarks
legal reasoning
foundation models
TL;DR: We work with the legal community to develop a benchmark of 160+ tasks to evaluate legal reasoning in large language models

Abstract: The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning—which distinguish between its many forms—correspond to LegalBench tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LegalBench, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LegalBench enables.

LEGALBENCH: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Neel Guha*, Julian Nyarko"!, Daniel E. Ho Christopher Re*1, Adam Chilton?, Aditya Narayana* Alex Chohlas-Woodd Austin Peters', Brandon Waldon', Daniel N. Rockmore", Diego Zambrano', Dmitry Talisman", Enam Hoque Faiz Surani', Frank Fagan Galit Sarfaty', Gregory M. Dickinson Haggai Porat Jason Hegland', Jessica Wu', Joe Nudell', Joel Niklaus' John Nay"", Jonathan H. Choi" Kevin Tobia 12 Margaret Hagan" 13 Megan Ma Michael Livermore Nikon Eesumovv-Rahe' Nils Holzenberger 15 Noam Kolt?, Peter Henderson' Sean Rehaag'6, 16 Shhaadd "oel" Shang 2oo Spencer Williams Sunny Gandhi Tom Zur", Varun lyer, and Zehua Li"
Sttanord University, University of Chicago, Maxime Tools, *Datmmouth College, LLawBeta, "South Texas College of Law Houston, UUniversity of Toronto, *St. Thomas University Benjamin Crump College of Law, Harvard Law School, Stanford Center for Legal Informatics CodeX, Universitty of Southern California, 'eGorgeoown University Law Center, Stanfoodd Law School, Univvrrity of Virginia, 15Télécom Paris, Institut Polytechnique de Paris, "Osgoode Hall Law School, York University, 17 Harrard Kennedy School, 18 Golden Gate University School of Law, Luddy School of Informatics Indiana University Bloomington, CCCsstte
Abstract
The advent of large language models (LLMs) and their adoption by the legal com- munity has given rise to the question: what types of legal reasoning can LLMS perform? To enable greater study of this question, we present LEGAI.BENCT a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LEGALBENCH was built through interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals, Because these subject matter experts took leading role construction, tasks either measure legal reasoning capabilities that are practi- cally useful, or measure reasoning skills that lawyers find interesting. To enable doossddiscppli conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning-whic distinguish between its many comms-correspond to L.GAIL BENCH tasks, thus giving lawyers and L.LM developers common vocabulary. This paper describes LEGALL BENCH, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LEGALBENCI enables.
Introduction
Advances in large language models (LLMs) are leading American legal professionals to reexamine the practice of law 52 61
Proponents have argued that LLMs could alter how lawyers approach tasks ranging from brief writing to corporate compliance [18]. By making legal services more accessible, they could eventually help alleviate the United States' long standing sccesss-o-jusste This perspective is informed by the observation that LLMS possess special properties
158
55
13
crisis 33 [32].
In using "LLMs", we are referring to language models which evince in-context learning capabilities (also referred to as "foundation models" [3] ). This behavior has traditionally been observed in models with at least a billion parameters.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

which, it is argued, make them more suited for legal tasks. The models' capacity to learn new tasks from limited labeled data would reduce the manual data annotation costs that ordinarily burden the development of legal language models [3] Their apparent proficiency at sophisticated reasoning tasks would also make them ideal for the rigor of law, which requires parsing obtuse texts with heavy jargon, and inferential processes which combine different modalities of reasoning (155).
This excitement, however, is tempered by the fact that legal applications often involve significant risk [47). Existing work has shown that LLMs are capable of generating content that is offensive,
misleading, and factually incorrect [00, 78 Such behaviors Sitpaviioss-rreppeatttee. replicatee in legal applications 112
could result in substantial harms [144] with much of the potential burden imposed on traditionally
marginalized and under-resoureed populations [25] [36]. The safety implications thus create a
pressing need to develop infrastructure and processes for benchmarking LLMs in legal contexts.
However, significant challenges face practitioners seeking to assess whether L.MS can perform legal reasoning. The first challenge is the limited ecosystem of legal benchmarks [15]. The majority of existing benchmarks. for example, focus on tasks which models learn by finetuning or training on task-specific data (21). These benchmarks do not measure the aspects of LLMS which generate excitement for law-namely, their ability to perform many different tasks using only few-shot prompts. Relatedly, benchmarking efforts have focused on professional certification exams like the Uniform Bar Exam 69], but these are not always representative of the actual use-cases for LLMs. The second challenge is the incongruity between the ways in which existing benchmarks and lawyers frame "legal reasoning." " Existing benchmarks coarsely generalize all tasks involving legal data or laws as measuring "legal reasoning." In contrast, lawyers recognize that legal reasoning is broad umbrella term encompassing many distinct types of reasoning [46). Different legal tasks require different skills and bodies of knowledge. Because existing legal benchmarks fail to draw these distinctions, it is difficult for legal professionals to contextualize the performance modern LLMs within their own understanding of legal competency. In short: legal benchmarks do not use the same vocabulary or conceptual frameworks as the legal profession.
In light of these limitations, we believe that rigorously evaluating the legal reasoning capabilities of LLMs will require the legal community to take more proactive role in the process of benchmarking. To that end, we present LEGALBENCH: the first steps towards constructing an interdisciplinary collaborative legal reasoning benchmark for the English language Fl Over the past year, the authors of this paper paper-drawing from their diverse legal and computer science backgrounds came together to assemble 162 tasks (from 36 different data sources), each of which measures a specific type of legal reasoning. LEGALBENCI is thus, to the best of our knowledge, the first open-source legal domain experts
benchmarking effort. We believe that this style of benchmark construction take an active and participatory role in the crafting of evaluation tasks iillustatsss one approach to interdisciplinary collaboration LLM research. Importantly, we believe it also shows that legal professionals have essential role to play in the assessment and development of LLMs for law.
As research project, we highlight three components of LEGALBENCH:
1. LEGALBENCII was constructed from a mix of existing legal datasets (restructured for the few-shot LLM paradigm), and hand-crafted datasets created and contributed by legal professionals (included as authors on this work). The legal professionals involved in this collaboration were asked to contribute datasets that they believed to either measure an interesting legal reasoning skill, or to capture practically useful application for LLMs in the law. High performance on LEGALBENCH tasks thus provides useful information, allowing lawyers to validate their assessment of an L.M'ss legal competency, or identify an L.M that could be used in their workflow.
LEGAI.BENCH tasks are organized into an extensive typology which describes the types of legal reasoning required to perform the task. Because this typology is drawn from frameworks familiar to the legal community, it enables legal professionals to meaningfully engage in discussions of LLM performance, using terminology and conceptual framework familiar to them [6] 122).
3. Finally, LEGALBENCH is intended as a platform to support further research. For AI researchers who lack legal expertise, LEGALBENCH comes with significant support for understanding how to prompt and evaluate different tasks. And as more of the legal community begins to engage with

---

the potential impact and role of LLMs, we hope to grow LEGALBENCH by continuing to solict and incorporate tasks from legal professionals
In this paper, we make several contributions. First, we present typology for organizing and describing legal tasks in terms of the types of reasoning they require. This typology is drawn from frameworks lawyers use to describe legal reasoning [122]. Second, we provide overview of the tasks in LEGALBENCH, describing the process by which they were constructed, important dimensions of heterogeneity, and limitations. A full description of each task is provided in the Appendix. Finally, we use LEGALBENC to evaluate 20 LLMs from 11 different families, across a range of size points. We make observations regarding the performance of different models and present an initial study into different Prompt-engineeet directions of future work that LEGALBENCH may enable.
strategies. Ultimately, these results are intended to highlight different
We hope that this benchmark will be interesting to a diverse set of communities. Practitioners may use these tasks determine whether and where L.LMs can be integrated into existing workflows to improve outcomes for clients. Legal academics may benefit from observing the types of annotation that LLMs capable of [17], and different forms of empirical scholarly work they may enable. Computer scientists may benefit from studying the performance of these models in domain like law, where distinct lexical properties and unique tasks may surface new insights.
Before we progress further, we note that the purpose of this work isn't 't to evaluate whether computa- tional systems should replace lawyers and legal officers, or to understand the positive and negative impacts of that replacement 447 126 4]. Rather, our goal is to construct artifacts that enable the relevant stakeholders and affected communities to better understand, empirically, the capacity for LLMs to perform different types of legal tasks. Given the proliferation of computational legal tools, we believe that answering this question is vital for ensuring their safe and ethical usage.
Related work
Benchmarking legal reasoning Understanding the extent to which NLP models can perform tasks
or skills traditionally associated with lawyers oo be useful in legal analysis- been the focus of
significant work (70 [ [08 124
[8]
84
102
86 22]. Prior work has identified manually arduous
tasks currently performed by lawyers (e.g., document review, case summarization) and developed
67 Researchers have focused on the technically challenging asepets of legal tasks, like document length, jargon, or inferential reasoning [21 [09] 78
corresponding benchmarks (60, 140 [18
19
88
85 40
77
21
155
63), Other work has focused on creating datasets for pretraining models (58]
127.. Eonn-Engiis/mmuullnnggg
tasks 993 94
64
149
51 92 68
20
23 101 18] legal judgement
prediction 87 37 17 156), legal role labeling 83 and different forms of retrieval [66].
Importantly, the majority of previous benchmarking efforts have focused on language models which learn by supervised training or finetuning (e.g., BERT variants (44)), and researchers have conse- quently studied questions related to the role of domain specific datasets [55 19 20). More recently,
researchers have begun to ask whether large language models (LLMs) like GPT-3/4 can perform
legal reasoning [71 151 65] [2] 28 30 [52), citing to evidence of these models' capacity to perform sophisticated reasoning tasks in domains like math or programming [143] 24]. Unlike basTd models, LLMs are evaluated on their ability to Iearn tasks in-context, primarily through prompting.
While a few works have experimented with LLMSs on existing benchmarks [6] (1),, most evaluations
focus on standardized tests or other exam equivalents [69 91 |9]. Studies have explored the role of
prompt-engineering [52 [31] 73), potential applications 11 145 [13 114), questions regarding
- interaction 30 61. and comparisons to older
[89
Connections to other LLM benchmarking efforts We highlight connections to two broader research efforts. First, we draw inspiration from existing efforts within NLP and machine learning to define ane-grained measures of performance, which allow researchers to discuss model capabilities with precision and specificity. Examples include the diagnostic set of the GLUE Benchmark [39), the "reasoning patterns" studied in [98], the task organization used in HELM (8), and the BigBench effort [11].. We believe this paradigm of expert-driven evaluation is essential for specialized domains like law.
'Cognizant of BENCH's current skew towards American law, we hope that additional contributions incorporate tasks from other jurisdictions.
3

---

3 The LEGALBENCH typology
LEGALBENCH identifies six types of legal reasoning that LLMs can be evaluated for: (l) issue- spotting, (2) rule-recall, (3) rule-application, (4) rule-conclusion, (5) interpretation, and (6) rhetorical- understanding. We first justify the selection of these types by providing background on how the legal profession frames "legal reasoning." and the connections datasets may be used to evaluate LLMs for each type, using examples from LEGALBENCH.
typology. We then illustrate how task
Though this framework draws heavily on American legal thought, we believe it can be extended to characterize BFNCH tasks that implicate non-American bodies of law. We also note that our types are non-exhaustive, and in future work hope to consider extensions.
3.1
Frameworks for legal reasoning
IRAC American legal scholars often describe "legal reasoning" as the process of determining the legal conditions that arise from set of events occurrences, with reference to both prior cases and codified laws [4]. A common framework for executing this type of legal reasoning is the Issue, Rule, Application and Conclusion (IRAC) framework [166] [22]. In this framework, legal reasoning decomposes into four sequential steps.
First, lawyers identify the legal issue in given set of facts (issue-spotting). An issue is either (1) specific unanswered legal question posed by the facts, or (2) an area of law implicated in the facts. Depending on the setting, a lawyer may be told the issue, or be required to infer possible issue.
Second, lawyers identify the relevant legal rules for this issue (rule-recall). rule a statement of law which dictates the conditions that are necessary (or sufficient) for some legal outcome to be achieved. In the United States, rules come from different sources: the Constitution, federal and state statutes, regulations, and court opinions. Importantly, rules often differ between jurisdictions. Hence, the relevant rule in California might be different than the relevant rule in New York.
Third, lawyers apply these rules to the facts at hand (rule-application). Application, or the analysis of rule applicability, consists of identifying those facts which are most relevant to the rule, and determining how those facts influence the outcome under the rule. Application can also involve referencing prior cases involving similar rules (i.e. precedent), and using the similarities differences to those cases to determine the outcome of the current dispute.
Finally, lawyers reach conclusion with regards their application of law to facts, and determine what the legal outcome of those facts are (rule-conclusion).
Example Suppose that Business large manufacturing corporation-is being sued by Amy in federal court on diversity jurisdiction 5 BusinessMart sells the majority of its goods in Texas, has its headquarters (where its CEO and board members sit and work) in California, and maintains factory in Florida. A court is trying to determine for the purposes of diversity jurisdiction where BusinessMart's "principal place of business is."
Issue-spotting: Here, narrow issue is offered where is BusinessMart's principal place of business?
Rule-recall: A lawyer would recognize that the most relevant rule here comes from the case Hertz Corp. Friend in which the Supreme Court determined "that the phrase 'principal place of business' refers to the place where the corporation 's high level officers direct, control, and coordinate the corporation's activities."
Applying this rule to the facts above yields two observations. First, corporation's CEO and board members are examples of high level officers referred to in Hertz that control and conduct company. Second, the place where Business.Mart's high level officers control the company is Califomia, as that is where the CEO and board sit and work.
Rule-application:
Rule-conclusion: Based the chain of inference spelled out in the application stage. a lawyer would thus conclude that California is BusinessMart's 's principal place of business.
$ 'Dieersity jurisdiction gives federal courts ability hear cases between parties that are "citizens" of different states.
Hertz Corp. Friend, 559 U.S. 77 (2010).

---

The extent to which the outcome of the application and conclusion steps follow each other is dictated by the level of ambiguity in the fact patterns. When the law on a particular question is clear and there is little ambiguity in the facts (as the case in the above example), then the application and conclusion steps point towards the same outcome. Sometimes however, the facts may be unclear or contested, and reasonable minds may differ as the conclusion step. For now, LEEGALBENCI focuses entirely on the former setting (unambiguous answers), and all tasks are considered to have objectively "correct" answers.
Other types reasoning Though IRAC the most formal framework for legal reasoning, lawyers recognize a variety of skills which are useful to practice of law [46 [5]. For instance, lawyers are often required exercise interpretive skills, in order to identify the rights, obligations, or limitations of certain legal language (e.g., what a contractual clause may or may not enable). They must also exhibit rhetorical skills, and understand the types of arguments that are made. Though these tasks require the knowledge base and skill set of lawyers, they, arguably, do not always fit neatly within the IRAC framework. Hence, we consider these to distinct from the examples offered in the previous section.
3.2 Evaluating legal reasoning in large language models
LEGALBENCI identifies six categories of legal reasoning. For each category, we describe how LLM task may evaluate the typified legal reasoning, using examples from LEGALBENCH.
Issue-spotting LEGALBENCH evaluates issue-spotting through tasks in which an LLM must determine if set of facts raise particular set of legal questions, implicate area of the law, are relevant specific party. Issue tasks evaluate a LLM's ability reason over the legal implications of different activities, events, and occurrences. An example of an issue-spotting task is the learned_ hands_ satss__eeeffft its task, which requires an LLM to determine (Yes/No) whether a post on public legal aid forum raises issues related to welfare law (i.e., public benefits or social services). The box helow shows how LLM might he prompted for this task.
Issue-spotting example: learned hands, benef
Does the post discuss public benefits and social services that people can get from the government, like for food, disability, old age, housing, medical help, unemployment, child care, or other social needs?
Post: "I am currently receiving support from social services, idk why, this is just how my life turned out. They have asked for all of my bank information for the past 12 months. don't know what this means. Why would they want that?"
Answer: Yes
Rule-recall LEGAI evaluates rule-recall through tasks which require the LLM to generate the correct legal rule on issue in jurisdiction (e.g., the rule for hearsay in US federal court). A rule task can he an open-ended generation task-in which the I.I.M must generate the text of the rule for jurisdiction- a classification task-in in which the LLM must determine whether the rule exists in that jurisdiction. Anchoring to jurisdiction is important, as legal rules differ across different jurisdictions. Rule tasks are particularly useful for measuring hallucinations [79] An example of rule-recall task is rule_qa, question-answer task where questions include asking the model to the formulations for different legal rules, identify where laws are codified, and general questions about doctrine.
Rule-recall example: rule_g
Question: What are the four requirements for class certification under the Federal Rules of Civil Procedure?"
Answer: Numerosity, commonality, typicality, adequacy
Rule-conclusion LEGAL BENCH evaluates rule-conclusion through tasks which require LLM to determine the legal outcome of set of facts under a specified rule. LLMs are evaluated purely on whether their predicted outcome is correct. For example, the ccc_v_coomant law law task asks a LLM to determine whether contract is governed by the Uniform Commercial Code (UCC) or the common law of contracts. The LLM is always provided with the relevant rule, via the prompt (see below).
5

---

Conclusion example: ucc_v_ Common_las
The UCC (through Article 2) governs the sale of goods, which defined moveable tangible things (cars, apples, books, etc.), whereas the common law governs contracts for real estate and services. For the following contracts, determine they are governed by the UCC by common law.
Contract: Alice and Bob enter into contract for Alice to sell her bike to Bob for S50. Is this contract governed by the UCC the common law? Governed by: UCC
Rule-application LEGAL BEnCH evaluates rul-aapllication through the tasks used to measure rule- conclusion. When evaluating rule-application however, we prompt the LLM to provide an explanation how the rule applies to a set of facts, and evaluate the quality of the generated explanation along two dimensions: (1) whether the explanation is correct, and (2) whether it contains analyxis. Each metric captures a different dimension upon which particular rule-application may be good.
Correctness corresponds to the criteria that explanations should not contain errors. We focus five types of errors: misstatements of the legal rulc, misstatements of the fact pattern, incorrectly asserting the legal outcome, logic errors, and arithmetic errors. Analysis corresponds to the criteria that explanations should contain inferences from the facts that are relevant under the rule, and illustrate how conclusion is reached. Consider, for example, explanation which restates the rule, the fact pattern, and the predicted legal outcome. If the predicted legal outcome is correct, than the explanation in its entirety would be correct, because contains no error. However, as prior works have noted (69 29). examples like this are conclusory, and often unsatisfactory the context of legal work.
To standardize evaluation and enable future work, we have released an "answer guide" for each task used for rule-application, which contains the inferences required for each sample, and descrihes common modes of errors. All evaluations in L.GAL LEGAL.BENH BENCH for rule-application have been performed with respect to this answer-guide.
Table I presents an examples of how three generations (corresponding the Alice/Boh example above) would be evaluated under the above metries. The first generation incorrect, because misstates the rule. The second generation correct because it contains no falsehoods.. but performs no analysis because does not articulate inferences. The third generation is both correct and contains analysis, because it has no errors, and explicitly mentions an essential inference (e.g., that bike is a "good").
Incorrect
Correct/no analysis
Correct/yes analysis
The contract is for Alice to sell bike to Bob. The contract is governed by the common law, be- cause goods are governed by the common law.
The contract is for Alice sell her bike to Bob. The contract is governed by the UCC, because the UCC governs all goods.
The contract is for Alice to sell her bike Bob. The contract governed by the UCC, because bike good and goods are governed by the UCC.
Table I: An example of how different generations are evaluated for correctness and analysis.
Interpretation LEGAL evaluates interpretation through tasks which require the LLM parse and understand legal text. Interpretive tasks provide the L.MM with text, and ask the LLM either extract relevant piece information, answer a question, or categorize the text by some property. Interpretive tasks are among the most studied and practically relevant tasks in LEGALBENCH. and many have been taken from actual use-cases. An example of an interpretive task is pead_tudttt. right, which asks the LLM to determine if contractual clause contains an "audit right." An example shown below:
Interpretation example: cuad_ at___udit__ audit_right
Does the clause give party the right to audit the books, records, or physical locations of the counterparty to ensure compliance with the contract?
Clause: "We shall have the right at all times access the information system and to retrieve, analyze, download and use all software, data and files stored or used on the information system." Answer: Yes
inntorical-understaadin LEGALBENCH evaluates thetoricaluunderstaddin understanding through tasks which require
an LLM to reason about legal argumentation and analysis. In these tasks, an LLM is provided with a legal argument (usually excerpted from judicial opinion), and asked to determine whether it performs certain

---

function or has certain property. An example is the must determine if sentence from judicial opinion provides a definition of a term.
task, in which an LLM
sinetorical-uudderstaadnng example: het.iiiiioonllassfiiieee
Does the sentence define term?
Sentence:" To animadvert carried the broader implication of "turn[ing] the attention officially or judi- cially, tak[ing] legal cognizance of anything deserving of chastisement or censure; hence, to proceed by way of punishment censure." Oxford English Dictionary 474 (2d ed. 1989)."
Answer: Yes
We emphasize one aspect of LEGALBENCH: IRAC this work is used as an organizing principle for grouping tasks. On a law exam, a student would be expected to generate an answer which structurally resembles IRAC, where each step builds on the inferences of the previous step [69 (2).. LEGALBENCE tasks, in contrast, cach evaluate single type of legal reasoning, Hence, task like learned_ _anddd_and____ hands_beneff beeeef is only be used to evaluate issue-spotting, and not rule-recall. future work we hope to add tasks which evaluate multiple steps jointly.
LEGALBENCH tasks
Appendix G discusses each task in detail, providing a description of the reasoning that each task evaluates, how task data was constructed, task examples, and evaluation protocols. This section provides overview of LEGALBBENH.
4.1 Construction process
Task sources LEGALBENCH tasks are drawn from three sources. The first source of tasks are existing available datasets and corpora. Most of these were originally released for non-LLM evaluation settings. In creating tasks for LEGALBENCH from these sources, we often significantly reformatted data and restructured the prediction objective. For instance, the original CUAD dataset [60] contains annotations on long-documents and is intended for evaluating extraction with span-prediction models. We restructure this corpora to generate binary classification task for each type contractual clause. While the original corpus emphasized aspects of contracts, our restructured tasks emphasize whether LLMs can identify the distinguishing features of different types of clauses. The second source of tasks are datasets that were previously constructed by legal professionals but never released. This primarily includes datasets hand-coded by legal scholars part of prior empirical legal projects (c.g.. (27). The last category tasks are those that were developed specifically for LEGALBENCH, by the authors of this paper. Overall, tasks are drawn from 36 distinct corpora.
long-document
Collaborative component In August 2022, we published a call for tasks, describing the goals the project and its structure [57]. We publicized the project through mailing lists and legal computational conferences. Submitted tasks were vetted for legal correctness and task validity. Task contributors are drawn from diverse professional backgrounds within the law (e.g., academics, practitioners, computational legal researchers) and constitute the authors of this paper.
Infrastructure LEGALBENCH comes with support designed enable non-law Al researchers to use and study LEGALBENCE tasks. First, each LEGALBENCH task is accompanied by extensive documentation describing how the task is performed, its legal significance, and the construction procedure. The objective of this documentation is to provide AI researchers with a working understanding of the mechanical processes behind each task, for the purposes of better understanding L.M performance. Second, each task is accompanied hy a "base" prompt, which contains task instructions and demonstrations. The base prompt is provided to promote replicability and standardization. We anticipate that future research efforts building off of LEGAL BENCH will identify higher performing prompts/prompt formats. We intended to update the LEGALBENCH GitHub repository with these prompts as they are discovered.
Limitations We note several limitations of the current LEGALBENCE tasks (additional limitations noted Appendix C). First, when this project began, most LLM context-windowl were constrained to few pages of text. As a result, the initial round of LEGAL BENCH tasks does not involve longer documents. We hope to include such tasks in future work, particularly as recent technical developments have resulted in significantly longer
context windows [4] 53 42 107]. Second, LEGAL LEGALBBENCE BENCH's 's tasks focus on legal reasoning questions with objectively correct answers. LEGALBENCH is thus not helpful for evaluating legal reasoning involving degrees of correctness or tasks where "reasonable minds may differ." Third, LEGAL BENCH only considers English language tasks, is skewed towards certain jurisdictions (American law). and certain areas of the law (contracts). Thus, the current iteration of the benchmark limits inferences regarding how LLMs may generalize to legal tasks involving other jurisdictions. As continue solicit and incorporate contributions to LEGALBENCH, hope
7

---

to add tasks addressing these limitations. Finally, LEGALBENCH evaluates IRAC abilities independently. while law exams and other legal work requires lawyers generate oulputs which follow IRAC in multi-hop matter (i.e., each aspect is applied to the same fact pattern).
4.2 Dimensions of variation
Task structure All LeGaLLBEnch tasks contain at least 50 samples, with an average task size 563 samples (Appendix (.4)). These tasks are comparable in size to those used in benchmarking efforts like BigBench [128, HELM [78 or RAFT []. LEGALBENCH tasks also span different formats: multiple-choice questions (35 tasks), open-generation (7 tasks), binary classification (112 tasks), and cualii-laas/mmlli-labee classification tasks).
Reasoning types and legal domains LEGALBENCH provides tasks for each of the reasoning categories discussed above: rulc-recall tasks), issuc-spotting (16 tasks), rule-application (16 tasks), rule-conclusion (16 tasks), interpretation (119 tasks), and rhetorical-unde understanding (10 tasks). Tasks are predominantly drawn from areas of law implicating civil matters, including contracts (58 tasks), civil procedure (8 tasks), evidence law (1 task), and corporate law (58 tasks). The skew towards interpretation tasks and tasks from contract law can be explained by the ubiquity of legal documents from these areas (e.g., contracts, terms-of-service agreements, disclosures, and etc.) and their immediate commercial implications 660 74.
Language variation Legal language is highly heterogeneous, varying in sentence structure, vocabulary, and rhetorical style across different legal areas and document types [58]. This poses a distinct challenge for LLMs, which are extremely sensitive to structure of input text and the vocabulary used (78) LEGALBENCE tasks are drawn from a diverse set of legal language types, thus enabling researchers to study performance variation across different categories of legal text. Specifically, LEGAL BEnCH encompasses tasks with language drawn from plain English (32 tasks), legal opinions (11 tasks), merger agreements (34 tasks), contracts (55 tasks), statutory text (3), and other sources.
5 Results
We use REGALBENCH to conduct three- part study. In the first part (Section 5.2 we evaluate 20 LLMs from 11 different families, four different size points. In the second part (Section 5.3 we show how LEGALBENCH can be used to conduct in-depth evaluations of models. To illustrate, we use LEGAL BENCH to highlight similaritics and differences in the performance of three popular commercial models: GPT-4, GPT-3.5, and Claude-1. In the final part (Section 5.4). we show how LEGALBENCH can support the development of law-specific LLM methods. We focus prompting, and conduct experiments illustrating tradeoff's and challenges with regards guiding LLMs towards certain tasks. Ultimately, our study here serves to illustrate the types of analyses that LEGALBEnCE enables, and highlight potential directions for future work. We summarize the findings of our study here, and provide more details in the Appendix.
5.1 Setup
Models We study 20 LLMs from 11 different families. These are: GPT-3.5 [4] (lext-davinci-003) and
GPT-4 from OpenAl 96]: Claude-1 (v1.3) from Anthropic 3] Incite-Instruct-7B. Incite-Base-7B, and Incite-
Instruct-3B from Together [34 133): OPT-2.7B, OPT-6.7B, and OPT- from Meta [14]: Falcon- Mifcconn-BBInstuuue Instruct
from TH 22 [03]; MPT-7B-88 Sk-Instruct from MosaicML [12]: Vicuna-7B- and Vicuna-1 1annaa-133 13B- 16k from
LMSYS (26 Flan-T5-XL (3B parameters) and Flan- T5-XLL (1IB parameters) from Google B1: LLaMA-
2-7B, and I.I ILLMA-2-133 from Meta [M4 WizardI.M-133 [150; and BLOOM-3b and BI BLOOM-7B from
BigScience [16.. All inference was performed on two-GPU 40GB A100s, using the Manifest library 7 HuggingFace links for each model are provided the Appendix.
We designed a prompt for each task by manually writing instructions for the task, and selecting between zero and eight samples from the available train split to use as in-context demonstration. The number of samples selected depended on the availability of data and the sequence length of samples. For application evaluation, we augmented the prompt with an instruction for the LLM explain its reasoning. We used the same prompt for all models with one exception (Claude-1). LLM outputs were generated using next-token generation at a temperature of 0.0. For classificationneexraatioo tasks, we terminated at new-line 353 token. For rule_ qa and all application tasks except di versity wwwrsst..juiiiiittt..... jur.sdicciion.6666 we generated 150 tokens. For diversity_ jurisdiction 6 we generated 300 tokens.
Prompts
Evaluation Classification tasks are evaluated using "exact-match" (following HELM (8)) Because some tasks contain significant label imbalances, we balanced-accuracy perform normalization on generated outputs to account for differences in bense/casing/punctuuttie (e.g., successor paaccesoorliabble wwllaiiiiityy and ssla_inds iidiiiiiiiaalll. viduallddedd _ee endants) requires the LLM to produce multiple classes or extracted terms per instance. For these, we evaluate using Fl. Rule-application tasks were evaluated manually by law-traine trained individual, who analyzed LLM responses for both correctness and analysis. The
metric. For extraction tasks, we
A few tasks
8

---

Appendix provides more details on our approach to manual grading. All manual evaluation was performed with reference to grading guide, which additionally make available.
5.2 Performance trends
LLM
Issue
Rule
Conclusion
Interpretation
Rhetorical
GPT-4
82.9
59.2
89.9
75,2
79.4
GPT-3.5
60.9
46.3
78.0
72.6
66.7
Claude-1
58.1
57.7
79.5
67.4
68.9
Flan-T5-XXL
66.0
36.0
63.3
64.4
70.7
LLaMA-2-13B
37.7
50.9
54.9
OPT-13B
52.9
28.4
45.0
45.1
43.2
Vicuna-13B-166 13B-16k
34.3
29.4
34,9
40.0
30.1
WizardL.M-13B
24.1
38.0
62.6
50.9
59.8
BLOOM-7B
50.6
24.1
47.2
42.8
40.7
Falcon-7B-Instreen
51.3
25.0
52.9
46.3
44.2
meite-7B-Bas
50.1
36.2
47.0
46.6
40.9
Incite 7B- IB-Instt
54.9
35.6
$2,9
54.5
45.1
LLaMA-2-27
50.2
33.7
55,9
47.7
47.7
54.3
25.9
48.9
42.1
44.3
OPT-6.7B
52.4
23.1
46.3
48.9
42.2
Vicuna-7B-16k
3.9
14.0
35.6
14.0
BLOOM-3B
45.0
36.4
Plan-TS-XL
56.8
31.7
52.1
51.4
67.
Seite-3B-Instrun
51.1
26.9
47.4
49.6
40.2
OPT-2.7B
53.7
22.2
46.0
44.4
39.8
Table Average performance each L.M over different LEGAILBENCH categories. The first black of rows corresponds to large commercial models, the second block corresponds to models in the 11B-13B range, the third block corresponds to models in the 6B-7B range, and the final block corresponds to models in the 2B-3B range. The columns correspond (in order): issue-spotting rule-recall, rule-conclusion, interpretation. each class models (large, 13B, 7B, and 3B), performing model each category of reasoning is underlined.
Table Ba provides the average task performance for all 20 models in five reasoning categories The first block of rows corresponds to large commercial models. the second block corresponds to models in the 11B-131 range. models the 6B-7B range, and the final block corresponds models in the
the third block corresponds 2B-3B range. Table 3 provides the average task performance for the three large models on rule-application. The Appendix provides full results for each model on cach task.
We highlight several observations. First, within LLM families, larger models usually outperform smaller models. For instance, Flan-TP T5- XXL (1IB parameters) outperforms 372 Flan-T', T5 T5-XL (3B parameters) on average across all five reasoning categories, and I.L.AA2--33B outperforms 373 LLaMA-2-7B on average across four reasoning categories. Second,
for LLMs of the size, find considerable differences in performance. For instance, we observe significant gaps performance between Flan-T5-XXII. (11B parameters) 16k (13B parameters), across all reasoning categories. This suggests, unsurprisingly. that the choice of pretraining data, regime of instruction-tuning, and architecture play important role in determining performance, and that certain configurations may be better aligned for LEGALBENCE tasks. Interestingly. such choices may affect which types reasoning categories L.LSs appear to perform well at. For instance, WizardL MM1 M-13B performs worse than all peers on issuc-spotting tasks, best on rule-recall tasks, and nearly matches the performance of the best-performing peer on rule-conclusion tasks. Third, we find evidence that open-source models are
Vicuna- 13B-
LLM
Correctness
Analysis
GPT-4
82.2
79.7
GPT-3.5
58.5
44.2
Claude-v1
61.4
59.0
Table 3: Average performance for the large L.LMs on rule-application tasks.

---

capable of performance that matches or exceeds certain commercial models. For instance. Flan-T5-XXL outperforms GPT-3.5 and Claude-1 383 on calegories (issue-spotting and rhetorical (amooiiclllmmeerstandnnnggg despite the relative gap in parameter count, Notably, the gap between closed and open-source models largest the mole-conclusion category. Amongst LEGALBENCH tasks, rulc-conclusion tasks most like the other types of multi- wwtpsssccppccsseeeiiie reasoning tasks where commercial LLMs have been found to perform well.
5.3 Comparing commercial models
Next, we conduct a more in-depth study performance, focusing on the three commercial models (GPT-4, GPT-3.5, and Claude-1). I). In particular we highlight how LEGALBENCH can provide more rigorous empirical support for anecdotal observations arising out of the legal community's 's use of these models, and explain performance differences between models. The Appendix provides more in-depth analysis.
We highlight two major findings. First, evaluation on LEGAL BENCH reveals that the largest performance difference between GPT-4 and GPT-3.//Claude-1 occurs (on average) for onl-application tasks. On rule application tasks, we observe that GPT-4 outperforms both GPT-3.5 (p < 0.01) and Claude-1 (p < 0.01) on both correctness and analysis. Across LLMss we find that variation in performance across tasks is consistent with subjective impressions of task difficulty. For instance, performance di -ewrstttjuuuiidddeee. task requiring model to determine if amount is greater than S75k and if the plaintiff and defendant from different states) is much higher than performance on siicessor_liabll:. ty harder task requiring a model to identify multiple successor liability exceptions in a fact pattern describing a complex transaction). Second, on the interpretation tasks, we find that on average GPT-4 outperforms GPT-3.5 (p < 0.01), and GPT-3.5 outperforms Claude-1 (p 0.01). Here, the larger API-models are highly performant on tasks which involve binary classification over short clauses. Averaged across the 38 CUAD tasks (contract clauses), for instance, GPT-4, GPT-335, and Claude-1 all have a balanced- calanced-accuracy 2 88%. And on proa (statutory clauses), both GPT-4 and GPT-3.5 have a balanced-accuracy > 90%. Notably, performance degrades on tasks which contain longer text sequences involve multi-class classification. On the Supply Chain Disclosure tasks for instance- in which LLMs must classify disclosures which are 1-2 pages in length-the average balanced--ccuuacy of the large commercial models ranges between 74 75%. And on the MAUD tasks which require answering multiple choice questions about merger deals- the average balanced-accuss of GPT-4 drops to 47.8% accuracy.
(an easy
5.4 Prompt engineering strategies
Finally, we illustrate through series micro-studies. how LEGALBENCE can be used to explore different aspects of prompt-engineering for LLMs in legal settings. We focus on three questions: (1) Can LLMs rely on their latent knowledge of a rule for rule-conclusion tasks? (2) Does simplifying task descriptions to plain language affect performance? (3) Are LLMss sensitive to the choice of in-context demonstrations?
We highlight our findings (full results can be found in the Appendix). With regards to (1), we find considerable variation over studied tasks some rules appear to be sufficiently well known that explaining the rule to the LLM the prompt has a negligible impact on performance. With regards to (2), we find that a languaee prompt significantly outperforms the technical language prompt. by up to 21 points bbalanced-accurace the subset of studied tasks. And with regards to (3), we find that LLMs are highly sensitive to the choice of in-context samples, with performance differences of up 20pts (balanced- accuraccy) These results illustrate that further work on understanding how effectively and efficiently prompt LLMs for legal tasks is needed.
on
6 Conclusion
Our work here describes LEGALBENCH. a collaboratively constructed benchmark of 162 tasks for measuring the legal reasoning capahilities of L.L.s. In future work, hope expand this project, by continuing solicit and collect interesting and useful tasks from the legal community
10

---

Acknowledgments and Disclosure of Funding
We are grateful to the following individuals and groups for feedback on this project: Alex Chao, Amit Haim, Arjun Desai, Armin Thomas, Avanika Narayan, Ben Spector. Brandon Yang. Eric Nguyen, Gautam Machiraju. Javed Qadrud-Din, Jian Zhang, Jonathan Zittrain, Karan Goel, Khaled Saab, Joshua Arp, Krista Opsahl-Ong. Laurel Orr, Lisa Ouellette, Lucia Zheng, Martin Gajek, Mayee Chen, Michael Zhang, Mike Wornww, Pablo Arredondo, Percy Liang, Rishi Bommasani, Roland Vogl, Sabri Eyuboglu, Sarah Hooper, Sergio Servantez, Simran Arora, Tengyu Ma, Tony Kim, Tri Dao and Vishnu Sarukkai. We presented and recieved feedback on earlier versions of this project various forums, including: the Center for Research Foundation Models, the Stanford Regulation and Governance Lab, the New York LLM x Law Hackathon (June 2023), the 2023 Stanford Data Science Conference, the 2023 Stanford CodeX Conference, and the Stanford Generative Al and Foundation Models Workshop. We are grateful to the organizers and attendees of these events for engaging with our work.
We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), 2204926 (Computational Statutory Rea- soning), and 1937301 (RTML); US DEVCOM ARI. under No. W91 W911NF-21- -1--221-2-0211 (Interactive Human-Al Teaming); ONR under No. N00014171222 (Unifying Weak Supervision); ONR N00014-20-1-228 1---4-0000: Under- standing and Applying Non-Fuclidcan Geometry in Machine Learning; N0001 18011421220 2275 (NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba. TSMC, ARM, Hitachi, BASF, Accenture, Eriesson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the Center for Research Foundation Models, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), and members of the Stanford DAWN project: Facebook, Google, and VMWare. We thank Casetext for assistance with evaluating GPT-4, PH is supported by an Open Philanthropy AI Fellowship. The U.S. Govermment is authorized reproduce and distribute reprints for Govermmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions recommendations expressed in this material those of the authors and not necessarily reflect the views, policies, or endorsements, either expressed or implied, NIH, ONR, or the U.S. Govermment.

---

References
[1] Neel Alex, Eli Lifland, Lewis Tunstall, Abhishck Thakur, Pegah Maham, C Jess Riedel, Emmic Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, et al. Raft: A real-world few-shot text classification benchmark. arXiv preprint (xXxx:2119.14077 2021.
[2] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Danicl Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pamnier, and Guilherme Penedo. Falcon-40B: open large language model with ttaee-of-the-art performance. 2023.
[3] Anthropic. Introducing claude. lattps://www http://pww.wtthrop....o. wttp:/cww...cctt.tt.et..mcmctttt httpm//iineeenntoouuuuuiiiie http://///rr/////aaaataot/ocoo/t
claude 2023.
[4] Yonathan A Arbel and Samuel Becher. How smart are smart readers? llms and the future of the no-reading problem. LLMs and the Future of the No-Reading Problem (June 25, 2023), 2023.
[5] Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Omr Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. Ask me anything: A simple strategy for prompting language models. arXiv preprint arXiv:2210.02441, 2022.
[6] Kevin D Ashley. Artificial intelligence and legal analytics: new tools for law practice in the digital age. Cambridge University Press, 2017.
[7] lan Ayres and Alan Schwartz. The no- no-reading problem in consumer contract law. Stan. L. Rev. 66:545, 2014.
[8] Yannis Bakos, Florencia Marotta- Wurgler,, and David R Trossen. Does anyone read fine print? consumer attention to standard-formd contracts. The Journal of Legal Studies, 43(1):1-35, 2014.
[9] Edward Beeching, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm Icaderboard. leaderboard, 2023.
http://gggg....oooo////////nnnna Ilm_
[10] Emily M Bender. Timnit Gebru, Angelina McMillan- Majorr, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fuirness, accountability, and transparency, pages 10-6233,4 2021.
[11] Andrew Blair-Stanck Nils Holzenberger, and Benjamin Van Durme. Shelter check: Proactively finding tax minimization strategies via ai. Tax Notes Federal, Dec, 12, 2022.
[12] Andrew Stane.S Nils Holzenberger, Benjamin Van Durme. Can gpt-3 perform statutory reasoning? arXiv preprint arXiv:2302.06100 2023.
[13] Rishi Bommasami, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint 100.::2008.072555 2021.
[14] Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models Are Few-Shot Learners Advances in Neural Information Processing Systems, 33: 2020.
[15] Bryan. Cave, Leighton. and Paisner. 2023 by ai legislation snapshot. http://wwww.blllaawcoooo wwwp:/wwwwwo/eee////eeeeeee sasseeeett-insightttnee///// www.:/eeeeeaaaaaaaaaaaaaaaalliia
legislation swapshot.humm html, 2023.
[16] Ilias Chalkidis. Chatgpt may pass the har exam soon, has a long way to go for the lexglue benchmark. arXiv preprint wrXiv:2304./22000 2023.
[17] Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. Neural legal judgment prediction in english. arXiv preprint wrXiv:1906.020599) 2019.
[18] Ilias Chalkidis, Manos Fergadiotis, and lon Androutsopoulos. Multieurlex-a multi- munti-lingual and multi-labe label legal document classification dataset for zero-shot cross-lingual transfer. arXiv preprint 2rXiv:2109.00900 2021.
[19] Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and lon Androutsopoulos. Legal- bert: The muppets straight out of law school. arXiv preprint 0rX1:22010.0555,, 2020.
12

---

[20] Ilias Chalkidis, Nicolas Garneau, Catalina Goanta, Daniel Martin Katz, and Anders Spgaard. Lexfiles and legallama: Facilitating english multinational legal language model development. arXiv preprint MrXiv:2305.07500 2023.
[21] Iliss Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. LexGLUE; A Benchmark Dataset for Legal Language Understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310--330. Association for Computational Linguistics, 2022.
[22] Ilias Chalkidis and Dimitrios Kampas. Deep learning in law: early adaptation and legal word embeddings trained on large corpora. Artificial Intelligence and Law, 27(2):171-198, 2019.
[23] Ilias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Sebastian Felix Schwemer, and Anders Søgaard. Fairlex: A multilingual benchmark for evaluating fairness in legal text processing. arXiv preprint wwXii:2203.0722.. 2022.
[24] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
[25] Edward K Cheng, Ehud Guttel, and Yuval Procaccia. Unenforceable waivers. Vanderbilt Law Review, Forthcoming (2023), 2022.
[26] Wei-Lin Chiang. Zhuohan Li. Zi Lin. Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality, March 2023.
[27] Adam S Chilton and Galit A Sarfaty. The limitations of supply chain disclosure regimes. Stan. Int'l L., 53:1, 2017.
[28] Jonathan H. Choi. How to use large language models for empirical legal research. Journal of Institutional and Theoretical Economicy, 2023.
[29] Jonathan H Choi, Kristin E Hickman, Amy Monahan, and Daniel Schwarcz. Chatgpt goes to law school. Available at SSRN, 2023.
[30] Jonathan H. Choi and Daniel Schwarez. Ai assistance in legal analysis: An empirical study. Available SSRN 4533836, 2023.
[31] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fodus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling insmuction-finetnnee language models, 2022.
[32] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova, Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1955.1004 2019.
[33] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers solve math word problems. arXiv preprint arXiv:2/10. 10.11468,, 2021.
[34] Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset. April 2023.
[35] Legal Services Corporation. The Justice Gap: Measuring the Unmet Civil Legal Needs of Low-Income Americans, 2017.
[36] Legal Services Corporation. Eviction laws database: Local dataset. http://www..k...o/nniaaeeeeeeeee 2021.
[37] Junyun Cui, Xiaoyu Shen, Feiping Nie, Zheng Wang, Jinglong Wang, and Yulong Chen. A survey on logal judgment prediction: Datasets, metrics, models and challenges. arXiv preprint arXiv:2204.04852 2022.
[38] Faraz Dadgostari, Mauricio Guim, Peter A Beling, Michael A Livermore, and Daniel N Rockmore. Modeling law search prediction. Artificial Intelligence and Law, 29:3-34 34, 2021.
13

---

[39] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Claxsification, and Recognising Tectual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers, pages 177-190. Springer, 2006.
[40] Xiang Dai, Ilias Chalkidis, Sune Darkner, and Desmond Elliott. Revisiting transformer-based models for long document classification. arXiv preprint arXiv:220400668; 2022.
[41] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344 16359, 2022.
[42] Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models, arXiv preprint arXiv:2212. 14052, 2022.
[43] Yasmin Dawood. Campaign finance and american democracy. Annual Review of Political Science, 18:329 348, 2015.
[44] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04800 2018.
[45] Gregory M Dickinson. A computational analysis of oral argument in the supreme court. Cornell JL & Pub. Pol'y, 28:449, 2018.
[46] Phoebe C Ellsworth. Legal reasoning, In K. J. Holyoak and R. G. Morrison Jr. editors, The Cambridge Handbook of Thinking and Reasoning, pages 685-704. Cambridge University Press, New York, 2005.
[47] David Freeman Engstrom and Jonah B Gelbach. Legal Tech, Civil Procedure, and the Future of Adversarialism. University of Pennsylvania Law Review, 169:1001, 2020.
[48] Epiq.
Pandemics and force majeure: http:///aeeemmiii--aaa----aam-ma -one---mjjeu----o------aaaaaa
How
can ai help you?
2020.
[49] Frank Fagan. From policy confusion to doctrinal clarity: successor liability from the perspective of big data. Va. L. Bus. Rev, 9:391, 2014.
[50] Scan Farhang. The litigation state. The Litigation State. Princcton University Press, 2010.
[51] Yi Feng, Chuanyi Li, and Vincent Ng. Legal judgment prediction: A survey of the state the art. IJCAI. ijcai. org, pages 5461-9 9, 2022.
[52] Jens Frankenreiter and Julian Nyarko. Natural language processing in legal tech. Legal Tech and the Future of Civil Justice (David Engstrom ed.), 2022.
[53] Daniel Y Fu, Elliot L Epstein, Eric Nguyen, Armin W Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. Simple hardware-efficient long convolutions for sequence modeling, arXiv preprint a7X1v22302.06646,, 2023.
[54] Timnit Gebru, Jamie Morgenstem, Briana Vecchione, Jennifer Wortman Vaughan. Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datashoets for datasets. Communications of the ACM, 64(12):8 92, 2021.
[55] Kurt Glaze, Daniel Ho, Gerald K Ray, and Christine Tsang. Artificial Intelligence for Adjudication: The Social Security Administration and AI Governance. In Justin Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew Young and Baobao Zhang, editors, The Oxford Handbook of Al Governance, pages 685-704. Oxford University Press, 2022.
[56] Neel Guha, Mayee F Chen, Kush Bhatia, Azalia Mirhoseini, Frederic Sala, and Christopher Ré. Embroid: Unsupervised prediction smoothing can improve few-shot classification. arXiv preprint XiX.:23307.10331 2023.
[57] Neel Guha, Daniel E Ho, Julian Nyarko, and Christopher Ré. Legalbench: Prototyping a collaborative benchmark for legal reasoning. arXiv preprint arXiv:2209.06/200 2022.
[58] Peter Henderson, Mark S. Krass, Lucia Zheng. Neel Guha, Christopher D. Manning. Dan Jurafisky, and Daniel E. Ho. Pile of law: Learning responsible data filtering from the law and a 256gb open open-source legal dataset, 2022.
14

---

[59] Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint wrXiv:2009.03300 2020.
[60] Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
[61] David Hoffman and Yonathan Arbel. Generative interpretation. Available SSRN 45262/9, 2023.
[62] David A Hoffmann Defeating the empire of forms. Available at SSRN 4334425, 2023.
[63] Nils Holzenberger. Andrew tax law entailment and question answering. arXiv preprint arXiv:2005.055257 2020.
and Benjamin Van Durme. A dataset for statutory reasoning in
[64] Wonscok Hwang, Dongjun Lee, Kyoungyeon Cho, Hanuhl Lec, and Minjoon Sco. A multi-tast task benchmark for korean legal language understanding and judgement prediction. Advances in Neural Information Processing Systems, 35:32537-32551, 2022.
[65] Cong Jiang and Xiaolei Yang. Legal syllogism prompting: Teaching large language models for legal judgment prediction. arXiv preprint arXiv:2307.0832. 2023.
[66] Abhinav Joshi, Akshat Sharma, Sai Kiran Tanikella, and Ashutosh Modi. U-creat: Unsupervised case retrieval using events extraction. arXiv preprint arXiv:2307 05260. 2023.
[67] Ambedkar Kanapala, Sukomal Pal, and Rajendra Pamula. Text summarization from legal documents: a survey, Artificial Intelligence Review, 51:371 402, 2019.
[68] Arav Kapoor, Mudit Dhawan, Anmol Goel, TH Arjun, Akshala Bhatnagar, Vibhu Agrawal, Amul Agrawal, Arnab Bhattacharya, Ponnurangam Kumaraguru, and Ashutosh Modi. Hide: Hindi legal documents corpus. arXiv preprint urXiv:2204.00800 2022.
[69] Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. Gpt-4 passes the bar exam. Available at SSRN 4389233, 2023.
[70] Daniel Martin Katz, Dirk Hartung, Lauritz Gerlach, Abhik Jana, and Michael Bommarito II. Natural language processing in the legal domain. arXiv preprint (tt.:2302.12399 2023.
[71] Noam Kolt. Predicting consumer contracts. Berkeley Technology Law Journal, 37, 2022.
[72] Yuta Koreeda and Christopher Manning. Contractnli: dataset for document-level natural language inference for contracts. arXiv preprint arXtw:21 2021.
[73] Aditya Kuppa, Nikon Rasumov-Rahe, and Mare Voses. Chain of reference prompting helps Ilm think like a lawyer. Generative Al + Law Workshop, 2023.
[74] Kwok- Yan Lam, Victor CW Cheng, and Zee Kin Yeong. Applying large language models for enhancing contract drafting. Proceedings of the Third International Workshop on Artificial Intelligence and Intelligent Assistance for Legal Professionals in the Digital Workspace (LegalAIIA 2023), 2023.
[75] Grant Lamond. Precedent and Analogy Legal Reasoning. Stanford Encyclopedia of Philosophy, 2006.
[76] Sarah B Lawsky. A logic for statutes. Fla. Tax Rev, 21:60, 2017.
[77] Zehua Li, Neel Guha, and Julian Nyarko. Don't use a camnon to kill a fly: An efficient cascading pipeline for long documents. International Conference on Al and Law, 2023.
[78] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga. Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:221 1,009/0011011, 2022.
[79] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Mcasuring how models mimic human falsehoods. arXiv preprint 1rXiv:2109.07955 2021.
[80] Marco Lippi, Przemystaw Palka, Giuseppe Contissa, Francesca Lagioia. Hans- WollsWoo Micklitz, Giovanni Sartor, and Paolo Torroni. Claudette: automated detector of potentially unfair clauses in online terms of service. Artificial Intelligence and Law, 27:117-139, 2019.
[81] William V Luneburg and Thomas M Susman. The lobbying manual: complete guide to federal lobbying law and practice. American Bar Association, 2009.
15

---

[82] Bingfeng Luo, Yansong Feng, Jianbo Xu, Xiang Zhang, and Dongyan Zhao. Learning to predict charges for criminal cases with legal basis. arXiv preprint vrXiv:1707.09166 2017.
[83] Vijit Malik, Rishabh Sanjay, Shouvik Kumar Guha, Angshuman Hazarika, Shubham Nigam, Arnab Bhattacharya, and Ashutosh Modi. Semantic segmentation legal documents via rhetorical roles, arxiv preprint arXiv: 12.0113,, 2021.
[84] Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam, Kripa Ghosh, Shouvik Kumar Guha, Arnab Bhattacharya, and Ashutosh Modi. Ilde for cjpe: Indian legal documents corpus for court judgment prediction and explanation. arXiv preprint orXiv:2105. 13562, 2021.
[85] Dimitris Mamakas, Petros Tsotsi, Ion Androutsopoulos, and Ilias Chalkidis. Processing long legal docu- ments with pre-trained transformers: Modding legalbert and longformer. arXiv preprint arXiv:2211.00972 2022.
[86] Stelios Maroudas, Sotiris Legkas, Prodromos Malakasiotis, and Ilias Chalkidis. Legal-tech open diaries: Lesson learned how develop and deploy light-weight models in the era of humongous language models. arXiv preprint arXiv:2210.130000 33086 2022.
[87] Masha Medvedeva, Martijn Wieling, and Michel Vols. Rethinking the field of automatic prediction of court decisions. Artificial Intelligence and Law, 31(1):195-21) 2023.
[88] Kaiz Merchant and Yash Pande. NIp hased latent semantic analysis for legal text summarization. In 2018 international conference on advances in computing, communications and informatics (ICACCI), pages 1803-1807. IEEE, 2018.
[89] Guilherme Moraes Rosa, Luiz Bonifacio, Vitor Jeronymo, Hugo Abonizio, Roberto Lotufo, and Rodrigo Nogueira. Billions of parameters are worth more than in-domain training data: A case study in the legal case entailment task. arXiv e-prints, pages arXiv-2205, 2022.
[90] John J Nay. Predicting and understanding law-making with word vectors and an ensemble model. PloS one, 12(5)xe0176999, 2017.
[91] John J. Nay. David Karamardian, Sarah B. Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H. Choi, and Jungo Kasai. Large language models as tax attorneys: case study in legal capabilitics emergence, 2023.
[92] Joel Niklaus, Ilias Chalkidis, and Matthias Stürmer. criis-judggmmn--preticttooo A multilingual legal judgment prediction benchmark. arXiv preprint arXiv:2110.00806, 2021.
[93] Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. Lextreme: A multi-lingual and multi-task benchmark for the legal domain. arXiv preprint arXiv:2301.13126 2023.
[94] Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, and Daniel E Ho. Multilegalpile: 689gb multilingual legal corpus. arXiv preprint arXiv:2306. 02066, 2023.
[95] Jonathan A Obar and Anne Deldorf-HHisch The biggest lie on the internet: Ignoring the privacy policies and terms of service policies of social networking services. Information, Communication & Society. 23(1):128 117, 2020.
[96] OpenAI. Gpt-4 technical report, 2023.
[97] Laurel Or.. Manifest. htttps://github.cmm/// htt.b///ww....eeeeeeeeeaaaaaaeae 2022.
[98] Laurel Orr, Megan Leszczynski, Simran Arora, Sen Wu, Neel Guha, Xiao Ling, and Christopher Re. Boot- leg: Chasing the tail with self-supervised named entity disambiguation. arXiv preprint arXiv:2010.1033. 2020.
[99] Anja Oskamp and Mare Lauritsen. Ai in law practice? far, not much. Al & I.., 10:227, 2002.
[100] Adam Pah, David Schwartz, Sarath Sanga, Charlotte Alexander, Kristian Hammond, Luis Amaral, SCALES OKN Consortium, et al. The promise of in an open justice system. Al Magazine, 49917499-74, 2022.
[101] Christos Papaloukas, Ilias Chalkidis, Konstantinos Athinaios, Despina-Athel Athanasia Pantazi, and Mano- lis Koubarakis. Multi-granula legal topic classification on greek legislation. arXiv preprint arXiv:2109. 15298, 2021.
16

---

[102] Shounak Paul, Pawan Goyal, and Saptarshi Ghosh. Lesicin: A heterogeneous graph-based approach for automatic legal statute identification from indian legal documents. In Proceedings the AAAI conference on artificial intelligence, volume 36, pages 11 11119-111444 19149,,4,,.211 2022.
[103] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Aloheidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon I.M: outperforming curated corpora with web data, and weh data only. arXiv preprint arXiv:2336.0111 116, 2023.
[104] Baolin Peng, Michel Galley, Pengcheng Hc, Hao Cheng, Yujia Xic, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu. Weizhu Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv: 12813, 2023.
[105] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances neural information processing systems, 34: I 1054-11070. 2021.
[106] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Schastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: (a0iv:99900000 01066, 2019.
[107] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302. 10866, 2023.
[108] Juliano Rabelo, Randy Goebel, Mi- Young Kim, Yoshinobu Kano, Masaharu Yoshioka, and Ken Satoh. Overview and discussion of the competition on legal information extraction/entailment (colice) 2021. The Review of Socionetwork Strategies, 16(1):1 16(1):111-13:: 11-133, 2022.
[109] Vishvaksenan Rasiah, Ronja Stern, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, Daniel E Ho, and Joel Niklaus. Scale: Scaling up the complexity for advanced language model evaluation. arXiv preprint arXiv:2306.09227, 2023.
[110] Abhilasha Ravichander, Alan W Black, Shomir Wilson, Thomas Norton, and Norman Sadch. Ques- tion answering for privacy policies: Combining computational and legal perspectives. arXiv preprint 1000004, 2019.
[111] Danilo Ribeiro, Shen Wang, Xiaofei Ma. Henry Zhu, Rui Dong, Deguang Kong, Juliette Burger, Anjelica Ramos, William Wang. Zhiheng Huang, al. Street: A multi-tast structured reasoning and explanation benchmark. arXiv preprint 2rXi::3302.067299 06729, 2023.
[112] James Romoser. No, ruth bader ginsburg did not dissent in obergefell and other things chatgpt gets wrong about the supreme court. SCOTUSblog, 2023.
[113] Erik F Sang and Fien De Meulder. Introduction the con1l-2003 shared task: Language-indeppenden named entity recognition. arXiv preprint cs/0306050, 2003.
[114] Jaromir Savelka. Unlocking practical applications in legal domain: Evaluation of gpt for zero-shot semantic annotation of legal texts. arXiv preprint arXiv:230,004417, 2023.
[115] Jaromir Savelka, Kevin D Ashley, Morgan A Gray, Hannes Westermann, and Huihui Xu. Can gpt-4 support amalysis of textual data in tasks requiring highly specialized domain expertise? arXiv preprint arXiv:2306. 13306, 2023.
[116] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvnn, Matthias Gallé, al. Bloom: A 176--parameter open-access multilingual language model. arXiv preprint arXiv:2111.05100. 2022.
[117] Robert E Scott, Stephen Choi, and Mitu Gulati. Contractual landmines. Available at SSRN, 2022.
[118] Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. Multi-lexsum: Real-world summaries of civil rights lawsuits at multiple granularities. arXiv preprint arXiv:2206. 10883, 2022.
[119] Abhay Shukla, Paheli Bhattacharya, Soham Poddar, Rajdcep Mukherjee, Kripabandhu Ghosh, Pawan Goyal, and Saptarshi Ghosh. Legal case document summarization: Extractive and abstractive methods and their evaluation. arXiv preprini arXiv:2210.075: 2022.
[120] Cecilia Silver. Breaking news: Drafting client alerts to prepare for practice. Perspectives: Teaching Legal Research and Writing, 27, 2019.
17

---

[121] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. arXiv preprint arXiv:2206.0460 2022.
[122] Norman Otto Stockmeyer. Legal Reasoning? It's All About IRAC, Mar 2021.
[123] Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.0193 01997,, 2022.
[124] Harry Surden. Artificial intelligence and law: An overview. Georgia State University Law Review, 35:19-22, 2019.
[125] Harry Surden. The ethics of artificial intelligence in law; Basic questions. Forthcoming chapter in Oxford Handbook of Ethics of AI, pages 19-29, 2020.
[126] Harry Surden. Values embedded in legal artificial intelligence. IEEE Technology and Society Magazine, 41(1):66-744 74, 2022.
[127] Mirac Suzgun, Luke Mclas- Kyriazii, Suproteem K Sarkar, Scott Duke Kominers, and Stuart M Shieber. The harvard uspto patent dataset: A large-scale, well-structured, and multi-purpose corpus of patent applications. arXiv preprint arXiv:2207 240000444, 2022.
[128] Mirac Suzgun, Nathan Scales, Nathanacl Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big- big-bench tasks and whether chain-of- solve them, arXiv preprint arXiv:2210.0926. 2022.
[129] MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023. Accessed: 2023-06-22.
[130] Wex Definitions Team. 2022.
http:///at...aar.ccooeele./////a generis
[131] Wex Definitions Team, textualism. http://www..l.....oomlllleeeet/s 2022.
[132] Joel Tito. How ai can improve access justice, 2017.
[133] Together. Releasing 3b and 7b eddajaama-nncie family of models including base, instruction-tura bwtp////l///aaaio//////e///aaaaa chat models. https: / /www together btyz//llg//ee redpa. ama- models- v11, 2023.
[134] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint ar 2Xiv:2307.09288 2023.
[135] Maarten Peter VINK, Luuk Van der Baaren, Rainer Bauböck, Iscult Honohan, and Bronwen Manby, Globalcit citizenship law dataset. 2021.
[136] Eugene Volokh. Chatgpt coming to court, by way of self-represented litigants. The Volokh Conspiracy, 2023.
[137] Brandon Waldon, Madigan Brodsky, Megan Ma, and Judith Degen. Predicting consensus legal document interpretation. Proceedings of the 45th Annual Conference of the Cognitive Science Society, to appear.
[138] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purposs purpose language undcrstanding systems. Advances in neural information processing systems, 32, 2019.
[139] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman, Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1rXiv:1804:07461 2018.
[140] Steven H. Wang, Antoine Scardigli, Leonard Tang, Wei Chen, Dimitry Levkin, Anya Chen, Spencer Ball, Thomas Woodside, Oliver Zhang, and Dan Hendrycks. Maud: An expert-annotated legal nip dataset for merger agreement understanding, 2023.
[141] Xinyi Wang, Wanrong Zhu, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in context learning. arXiv preprint arXiv:2301.11916, 2023.
18

---

[142] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641. 2019.
[143] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903. 2022.
[144] Benjamin Weiser. Here's what happens when your lawyer uses chatgpt. New York Times, 2023.
[145] Hannes Westermann, Jaromir Savelka, and Karim Benyckhlef. Ilmediator: Gpt-4 assisted online dispute resolution. arXiv preprint arw.:: 2023.
[146] Wikipedia. Irac.
[147] Shomir Wilson, Florian Schaub, Aswarth Abhilash Dara, Frederick Liu, Sushain Cherivirala, Pedro Gio- vanni Lcon, Mads Schaarup Andersen, Schastian Zimmeck, Kanthashree Mysore Sathyendra, N Cameron Russell, et al. The creation and analysis of website privacy policy corpus. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1330-1340, 2016.
[148] Lawrence Wrightsman. Oral arguments before the Supreme Court: An empirical approach. Oxford University Press, 2008
[149] Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang. et al. Cail2018: A large-scale legal dataset for judgment prediction. arXiv preprint arXiv: 1807.02477 2018.
[150] Can Xu, Qingfeng Sun. Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng. Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023.
[151] Fangyi Yu, Lee Quartey, and Frank Schilder. Legal prompting: Teaching language model to think like lawyer. arXiv preprint arXiv:2212.01226 2022.
[152] Fangyi Yu, Lee Quartey, and Frank Schilder. Exploring effectiveness prompt engineering for legal reasoning tasks. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13582-13396, 2023.
[153] Diego Zambrano, Neel Guha, Austin Peters, and Jeffrey Xia. Private enforcement in the states. University of Pennsylvania Law Review, forthcoming 2023.
[154) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 2ixi::2205000666 2022.
[155] Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset of 53,000+ Legal Holdings. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, pages 159-168, 2021.
[156] Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. Legal judgment prediction via topological learning. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3540-3549, 2018.
[157] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? arXiv preprint (.X.x:2333.035/2 2023.
[158] Lee B. Ziffer. The robots are coming: Ai large language models and the legal profession. American Bar Association, 2023.
[159] Sebastian Zimmeck, Peter Story, Daniel Smullen, Abhilasha Ravichander, Ziqi Wang. Joel R Reidenberg. N Cameron Russell, and Norman Sadeh. Maps: Scaling privacy compliance analysis to a million apps. Proc. Priv Enhancing Tech., 2019:66, 2019.
19

---

A Outline
For the sake of clarity, we provide brief outline of the Appendix.
Appendix
B
provides more information on the availability of LEGALBENCH.
Appendix discusses limitations of the project and potential negative social impacts.
Appendix provides top-level datashect.
Appendix E
provides an overview of the LEGALBENCH tasks, and contains information on licenses, which datasets have been published before, evaluation protocols, and dataset statisties.
each set of tasks, we provide background on the relevant legal concepts, describe the tasks, discuss how the task was constructed, provide examples of inputs/outputs, and explain their legal significance.
Appendix G provides a description of each task/task- fask/tas--family.
Appendix provides the full-results for evaluated LLMs on all LEGAL BENCH tasks, along with additional mffomaaton about prompting.
B
Availability
The LEGALBENCH
website is
available
http:///aryrraaach..saaaarr
legalbench/
Data can be downloaded from Huggingface here: htt.::///hgginaaaacc..ocaaaattsp
legalbench
The Github repository with evaluation code and prompts is available here: htttps://ittub.. com/
[Hazyleseearch/llgalbboo
Gradehooks guide manual evaluation can be found the website.
C Limitations and social impact
We note several limitations of our work. Legal applications and what constitutes "Ic- gal reasoning" broad. Thus, LEGALBENCH will necessarily be incomplete effort, and important tasks/document typesfreasoning types are not included. To enumerate few examples:
Limitations
LEGALBENCH does not include tasks over long documents. Long documents are significant for legal practice, as writings like contracts, corporate filings, statutory codes, and judicial opinions can be hundreds of pages long (77]
The legal reasoning dimensions identified LEGAL BENCH constitute subset of the possible legal reasoning abilities for which we wish to evaluate LLMs. An example of a reasoning ability which is not currently evaluated in LEGALBENCH would be analogical reasoning grounded in case law.
LEGALBENCH tasks are skewed towards certain legal domains (e.g.. contracts and civil procedure) and others are unrepresented.
LEGALBBENC tasks skew towards US Federal law, and thus may not be representative for studies of other jurisdictions, tasks involving international law.
LEGALBBNCC does not enable evaluation for multilingual, or non-English, legal tasks.
LEGAI.BENCH does evaluate more subjective legal tasks, tasks which contain more ambiguity. These tasks are common the legal field.
We hope to work on these limitations as part of future work. In particular, we would like to expand LEGALBENCH to include other jurisdictions and a broader cross coss-secctinn of legal domains.
Nothing in LEGAL BENCH should be construed as legal advice.
Social impact A potential negative social impact of our work would be if others either (1) construed our work unequivocally endorsing automation in the legal industry. or (2) used performance on LEGAL as the sole justification for AI deployments. We therefore take efforts to mitigate these impacts, noting the following.
As we state in Section I the purpose of our work is not determine whether large language models capable of replacing legal professionals, the types of legal work that should/can be automated, or the broader implications of new technology on the practice of law, Rather, our focus is on developing technical artifacts which better enable stakeholders and affected parties to answer these questions themselves. Rigorous evaluation is essential benchmark, is intended to improve the ability for
the safe and ethical usage of AL. LEGALBENCH,
20

---

stakeholders to conduct evaluations. We additionally note that LEGALBENCH, as tool for research, is not substitute for more in-depth and context-specific evaluation efforts. The deployment of any AI application in the must be accompanied by evaluation on in-domain data, and assessments for ethical and legal compliance.
We finally note that potential negative impact will depend significantly on the task studied and the broader social context. The consequences of mistakes in using LLMs to annotate datasets, for instance, has significantly different consequences from the cost of mistakes when LLMs are used to answer legal aid questions.
D Datasheet
Following recent work, we provide datasheet [4 below. The datasheet below provides general answers to each of the questions, while A provides more in-depth details for cach individual task. In addition, a number of LEGALBENCH tasks have been adapted from previously released datasets, and the datashects accompanying their publication provide further details.
D.1 Motivation
For what purpose was the data set created? Was there a specific task mind? If so, please specify the result type (e.g. unit) to be expected.
LEGAL BENCH was created to evaluate LLMs on legal tasks and better understand their legal reasoning capabili- ties. Recent advances in language modeling techniques have led to the emergence of "large" language models, and spurred interest within the legal community. This has led to questions:
What technical adaptations are necessary to enable LLMs to perform legal tasks? Legal tasks often jargon, involve longer text sequences, jargon, and multi-step reasoning, making them more difficult than traditional NLP tasks.
For which legal tasks current LLMs be trusted perform safely and reliably?
BENCH encompasses many different tasks. The specification for each task and the expected output can be found the full task descriptions (Section[G).
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g. company, institution, organization)?
LEGAL LEGAL BENCH consists of novel datasets (which were created by the authors of this paper), and trans- formed/adapted datasets (which were originally released part of prior research). In Section G we discuss the origins of each dataset.
Who funded the creation of the dataset? If there is associated grant, please provide the name of the grantor and the grant name and number.
LEGALBENCH and its contributors have been generously funded by a range of entities that include the institu- tional affiliations provided for each author, governmental grants, and other sources.
Any other comments?
None.
D.2 Composition
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movics, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.
All LEGALBENCH tasks consist of instances which are text. These include: sentenees, paragraphs, and documents, Some instances are drawn from real world sources of text (c.g., actual contracts, corporate disclosures, judicial opinions, or complaints). Other instances were synthetically crafted. Section G details for each task.
How many instances there in total (of each type, if appropriate)?
Section E provides details for each task.
Does the dataset contain all possible instances is sample (not necessarily random) of instances from larger set? the dataset a sample, then what is the larger set? Is the sample representative of larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/veriffed. it is not representative of the larger set, please describe why not (e.g., to cover more diverse range of instances, because instances withheld or unavailable).
21

---

Nearly every LEGALBENCH task corresponds to sample of population, or entirely synthetie data. Section G contains more detailed description for each dataset. We highlight several broader explanations for the difficulty acquiring complete or representative data which generalizes across lasks:
As prior work on legal benchmarks has noted 58 118 not all legal documents are published or reported. Hence, many are only accessible through special request, or only available in paper. The lack of easily available representative data is a noted challenge in many justice systems [58] 100..
Acquiring legal annotations is exceedingly expensive. The CUAD project, for instance, estimated that a modestly sized dataset of 500 contracts (relative to the standards of NLP) had an estimated cost of S2 million US dollars [60]. As result, it often possible to only annotate small sample of data, even when larger population is available.
What data does each instance consist of? "Raw" data (e.g. unprocessed text or images) or features? In either case, please provide a description.
Instances in LEGALBENCH largely correspond to unprocessed text. Section G contains a more detailed description for each dataset.
Is there label or target associated with each instance? If so, please provide description.
Yes. Labels correspond to: classes, extracted entities, and open-ended generation. Section detailed description of the labels/targets for cach datasct.
G contains a more
Is any information missing from individual instances? If so, please provide description, explaining why this information is missing (e-g., because was unavailable). This does not include intentionally removed information, but might include, e-g., redacted text.
For reused/adapted datasets, we refer readers to the original data sheets which document redactions/missing data. Newly contributed tasks should not be missing information.
Are relationships between individual instances made explicit (e.g.. users' movic ratings, social network links)? If so, please describe how these relationships are made explicit.
Not applicable.
Are there recommended data splits (e.g..trannin, developnennt/vaiicatiionneetee" If so, please provide description of these splits, explaining the rationale behind them.
Yes. Tasks are split into train and test splits. Train splits consist of small random sample of the original dataset (i.e.. between 2-8 instances). We select small training splits order capture the true few-shot setting [105), which a practitioner only has access to a handful of laheled instances. This design choice is also reflected in the structure of the RAFT benchmark 11.
Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.
significant amount of legal data is the product of scanning and OCR. Hence, this data often contains artifacts of these processes, which appear as errant or missing characters.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g.. websites, tweets, other datasets)? If it links to relies on external resources, a) there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); e) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well links other access points, appropriate.
LEGALBENCCL is self-contained.
Does the dataset contain data that might be considered confidential (c.g., data that protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? If so, please provide a deseription.
No. All LEGALBENT BENCH data is derived from public sources or was generated by authors. There is no confidential information in our dataset.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.
No.
Does the dataset relate to people? If not, you may skip the remaining questions in this section.
22

---

LENALBENCH data relates to people to the extent that LEGALBENCH contains tasks which contain language drawn from judicial cases involving individuals, posts by individuals to legal forums (i.e., the Learned Hands Tasks).
Does the dataset identify any subpopulations (e.g.. by age, gender)? so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.
No.
it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.
As LEGALBENCH is drawn entirely from public datasets- which themselves may contain additional information-it is possible to identify the original documents that LEGAL data was drawn from.
Does the dataset contain data that might be considered sensitive in any way (e.g,, data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions union memberships, or locations; financial health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.
The Learned Hands tasks correspond to posts on public forums. In these posts individuals discuss legal questions, and sometimes disclose information that would meet the above definition of "sensitive."
Any other comments?
We note that the data distributions from which some I LEGAL BENCH tasks were drawn-like like judicial cases or legal
been used by prior work published in the NeurIPS Datasets and Benchmarks Track [88 118) These works offer additional information.
D.3 Collection process
How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e-g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-peecch tags, model-based guesses for age language)? If data was reported by subjects indirectly inferred/derived from other data, was the data validated//vriifed?? If so, please describe how.
Data underlying LEGAL BEnCH tasks were collected using different processes, and Section contains a detailed discussion for each task.
What mechanisms or procedures were used collect the data (e-g.. hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?
Please refer to Section G for background on each task.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
Please see the discussion in the Composition section above.
Who was involved in the data collection process (e-g., students, crowdworkers, contractors) and how were they compensated (e.g,, how much were crowdworkers paid)?
Scction G contains detailed discussion for each task.
Over what timeframe the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.
Section G contains detailed discussion each task.
Were any ethical review processes conducted (c-g., by an institutional review board)? so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.
Where applicable, Section G provides information relevant to each task.
Does the dataset relate to people? If not, you may skip the remaining questions in this section.
The dataset relates to people insofar as it draws text from documents which relate to people, or people created.
Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?
Section contains detailed discussion for each task.
23

---

Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots other information) how notice was provided, and provide link other access point or otherwise reproduce, the exact language of the notification itself.
No. Following other works which incorporate data from public judicial sources [8 filings are public, and the individuals implicated in those proceedings are aware of the public nature.
118,, note that judicial
Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.
Individuals whose names and circumstances appear in the original datasets did not separately consent to be part of LEGALBENCH, Again, we note that these documents are generally public, and already accessible wide range of parties, through many different judicial data services.
consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses'? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).
N/A.
Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If please provide description of this analysis, including the outcomes, as well link or other access point to any supporting documentation.
No.
Any other comments? None.
D.4 Preprocessing, cleaning, labeling
Was any prppocesiin/leannnngalllling of the data done (e.g., discretization bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide description. If not, you may skip the remainder of the questions in this section.
No.
D.5
Use
Has the dataset been used for any tasks already? If so, please provide a deseription.
We have used the constructed datasets to evaluate several LLMs.
Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide link or other access point.
LEGALBENCH is available at
thub. http://hhhh.a.oooeee/////errreaa
What (other) tasks could the dataset be used for?
We envision this dataset could be used for the following:
Evaluation of L.IMs.
Finetuning LLMs, either task data directly. self-instruct style generations derived from task data.
Is there anything about the composition of the dataset the way it was collected and prepro- cessed//leanedd/abelel that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e-g., stereotyping, quality of service issues) or other undesirable harms (c.g., financial harms, legal risks) If so, please provide deseription. Is there anything future user could do mitigate these undesirable harms?
We emphasize that RIGALBENCI-HiK all generalized benchmarks can offer only preliminary understanding LLM performance. LEGALBEnCE tasks do not generalize to all legal reasoning tasks or all types legal documents. We thus emphasize that practitioners seeking to deploy LLMs within their own applications should perform their own data collection and validation specific to their use case.
Are there tasks for which the dataset should not be used? If so, please provide description.
These datasets should not be used to predict the legality of real world events, the outcome of lawsuits, or as legal advice.
24

---

these contributions to D.6 Distribution
Will the dataset be distributed under copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide link other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well any fees associated with these restrictions.
Table 4 provides the license that applies to each individual LEGALBENCH task.
Have any third parties imposed [P-bbaeed or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide link or other access point to, or otherwise reproduce, any relevant licensing terms, well as any fees associated with these restrictions.
Yes. Tasks which consist of adapted/transfommed data are released under the same license the original dataset. Table provides these licenses, and Section G provides reference the original dataset for transformed tasks.
Do any export controls other regulatory restrictions apply to the dataset so, please describe these restrictions, and provide link or other access point to, or otherwise reproduce, any supporting documentation.
individual instances?
No.
D.7 Maintenance
Who will be suporringhnstiinnaiitiiinngg the dataset?
Neel Guha will be supporting this dataset.
How the ovner/curator//manger of the dataset be contacted (e.g,, email address)?
Neel Guha can be reached nguha @ cs. sdu. He will be available answer any questions. there an erratum? If so, please provide link other access point.
We have currently not found any, but will make them available on the website.
Willthe dataset be updated (e-g.,to correct labeling errors, add instances, delete instances)? so, please describe how often, by whom, and how updates will communicated to users (e.g., mailing list, GitHub)?
Yes. There will be two types of updates to LEGALBENCH:
First, will update LEGALBENCH to reflect new contributions from the legal community.
Second, we will update LEGAL. BENCH to reflect identified errors the data.
We will strive to make and publicize updates as soon as errors are identified and new tasks are contributed. Neel Guha will be in charge of managing these updates.
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (c.g., individuals in question told thattheir data would be retained for fixed period of time and then deleted)? so, please describe these limits and explain how they will enforced.
N/A.
Will older versions ofthe dataset continue supported/hnstel/maaaaaanne so, please describe how. not, please describe how its obsolescence will communicated to users.
Yes. We will make older versions available on request by email.
others want to extend/augmen//buidd on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there process for communicating/iistibuunu other users?If ?f so, please provide description.
Yes. We encourage members of the legal community contribute tasks. We in the process of formalizing procedures reviewing, validating, and incorporating submissions.
We additionally note that many of the LEGALBENCH tasks are available under permissive licenses, other researchers may thus modify them.
25

---

E
Task overview
E.1 Licenses
LEGAL BENCH tasks subject to different licenses, due the choices of dataset contributors, the license under which the original data was released. Table summarizes the licenses. authors bear all responsibility in case of violation of rights, and confirm the dataset licenses.
License
Tasks
Creative Commons Attribution 4.0
Abercrombie, CUAD Tasks, Citation Predic- tion Tasks, Contract NLI Tasks, Contract QA, Corporate Lobbying. Diversity Tasks, Function of Decision Section, Hearsay, Insurance Policy Interpretation, International Citizenship Ques- tions, J.Crew Blocker, Legal Reasoning Causal- ity, MAUD Tasks, Oral Argument Question Pur- pose, Overruling, Personal Jurisdiction, Private Right of Action, Rule QA. SCALR, Securities Complaint Extraction Tasks, Successor Liabil- ity, Supply Chain Disclosure Tasks, Telemarket- ing Sales Rule, UCC Common Law, Unfair Terms of Service
Curbbttion-onCCommrrid 4.0 International
Canada Tax Court Outcomes, Consumer Con- tracts QA, Textualism Tools
Attibuttion-haarAAlik 4.0 International
Definition Tasks
AuribuiiooVnnCCommerrraaaaaallll 4.0 In- ternational
Learned Hands Tasks
MIT
New York State Judicial Ethics, Privacy Policy
QA, SARA Tasks
Creative
Commons
Attribution-
OPP-115 Tasks
NonCommercial License
Attrbatiion-NnCCmmeeee 3.0 Unported
Privacy Policy Entailment
Table 4: Licenses
E.2 Public availability status
Given that many commercially available LLMs trained the "entirety of the web"-and and little is known how they trained-there are concerns that many benchmarks have inadvertantly become part of the training data for these models. Therefore, this section identifies and organizes LEGALBENCH tasks into three categories:
Previously published tasks, which were derived from datasets that were initially published as part of other works and available on the web for download.
Original but available tasks, which are original creations of the LEGAL.BENCH project but previously made available online.
Original and unavailable tasks, which original creations of the LEGALBENCH project but have been released online.
Table 5 summaiizes the availability status of each of the tasks.
E.3
Reasoning type
Table organizes tasks by LEGALBENCH reasoning type they can used to assess. Table 7 similarily organizes tasks according to reasoning- types recognized in the NLP literature. For each reasoning type, we provide examples of general-domain NLP benchmarks which are similar. For more information on the types of reasoning required for each task, please see the individual task descriptions provided in Appendix G
26

---

Publication status
Tasks
Previously published tasks
CUAD Tasks, Contract NLI Tasks, MAUD Tasks, OPP-115 Tasks, Overruling. Privacy Policy Entailment, Privacy Policy QA, SARA Tasks, Unfair Terms of Service
Original but available tasks
Abercrombie, Diversity Tasks, Hearsay, Inter- national Citizenship Questions. Learned Hands Tasks, New York State Judicial Ethics, Personal Jurisdiction, Private Right of Action, Rule QA
Original and unavailable tasks
Canada Tax Court Outcomes, Citation Predic- tion Tasks, Consumer Contracts QA, Contract QA, Corporate Lobbying, Definition Tasks, Function of Decision Section, Insurance Policy Interpretation, J. J.Crew Blocker, Legal Reason- ing Causality, Oral Argument Question Purpose, SCALR, Securities Complaint Extraction Tasks, Successor Liability, Supply Chain Disclosure Tasks, Telemarketing Sales Rule, Textualism Tools, UCC Common Law
Table 5: Task publication status
LEGALBENCH reasoning type
Tasks
Issue-spotting
Corporate Lobbying, Learned Hands Tasks
Rule-recall
Citation Prediction Tasks, International Citizen- ship Questions, New York State Judicial Ethics, Rule QA
Rule-application
Abercrombie, Diversity Tasks, Hearsay, Per- sonal Jurisdiction, Successor Liability, Telemar- keting Sales Rule, UCC v. Common Law
Rule-conclusion
Abercrombie, Diversity Tasks, Hearsay, Per- sonal Jurisdiction, Successor Liability, Telemar- keting Sales Rule, UCC v. Common Law
Interpretation
CUAD Tasks, Consumer Contracts QA, Con- tract NLI Tasks, Contract QA, Insurance Policy Interpretation, J.Crew Blocker, MAUD Tasks, OPP-115 Tasks, Privacy Policy Entailment, Pri- vacy Policy QA, Private Right of Action, SARA Tasks, Securitics Complaint Extraction Tasks, Supply Chain Disclosure Tasks, Unfair Terms of Service
Rhetorical-understann
Canada Tax Court Outcomes, Definition Tasks, Function of Decision Section, Legal Reasoning Causality, Oral Argument Question Purpose, Overruling, SCALR, Textualism Tools
Table 6: Tasks by LEGALBENCH reasoning type.
E,4 Task statistics
Table D provides statistics for the LEGALBENCH tasks, For each task, we list the number of samples and the average length (in words) of each input. LEGALBENCH encompasses tasks ranging from short (a single sentence)
27

---

NLP reasoning type
Tasks
Knowledge (c.g., MMLU
59]
WikiFact in
Citation Prediction Tasks, International Citizen- ship Questions, New York State Judicial Ethics,
HELM 78
106
Rule QA
Linguistic inference (e.g.. CoLA 142
Canada Tax Court Outcomes, Definition Tasks, Function of Decision Section, Legal Reasoning Causality, Oral Argument Question Purpose, Overruling, Textualism Tools
Topic classification (c.g., RAFT (T]
Corporate Lobbying, Learned Hands Tasks, CUAD Tasks, J.Crew Blocker, OPP-115 Tasks, Private Right of Action, Unfair Terms of Ser- vice
Entailment (e.g., RTE 138
Contract NLI Tasks, Privacy Policy Entailment, Privacy Policy QA, Insurance Policy Interpreta- tion, SARA Tasks
Arithmetic (e.g., GSM8K (33)
Diversity Tasks, SARA Tasks
Multi-step reasoning (e.g., STREET
Abercrombie, Hearsay, Personal Jurisdiction,
Successor Liability, Telemarketing Sales Rule,
UCC Common Law
Document-based QA (e.g., BoolQ 32
Consumer Contracts QA, Contract QA, MAUD Tasks, Supply Chain Disclosure Tasks
Named entity recognition (e.g., CoNLL 2003 113)
Securities Complaint Extraction Tasks
Casual reasoning (e.g., CoPA 138
SCALR
Table Tasks by NL.P reasoning type. For each reasoning type, we provide examples of general- domain NLP benchmarks which are similar. For more information on the types of reasoning required for each task, please see the individual task descriptions provided Appendix G
to longer inputs (two pages of text) (Figure D. instances. All tasks consist of at least 50 instances. A more detailed breakdown is available in Table
The average LEGAILBENCH task contains between 500-600
Task Input Distribution
Number Tasks
200
400
600
800
1000
1200
Average Sequence Length Input (Words)
Figure 1: LEGALBENCH task sizes and input text lengths.
28

---

Size range (samples)
Number of tasks
50- 100
28
100-500
97
500-2000
2000+
Table 8: Number of LEGALBENCH tasks different dataset sizes.
Table 9: Task Statistics
Task
Number of Samples
Mean Sample Length (Words)
abercrombie
100
7.1
canada tax Lattllutt_aaaaaalalllllllllleee outcomes
250
99.2
110
35,9
55
32.4
consumerr_contracts..gg
400
486.8
contract_nli contract_pi_comfiddeniiie oaw.o//ioonnilllyooooaarrrreeeee agreement
aggremmmn
90
73.9
contract nli explicit identification
117
75.7
contract_nli cinncuusionoo_vvrrallllle
147
information
contract nli limited use
216
63.8
contract_nlic plipett________eeeeee Lacpiinniiiooiioeeee polnooolloe _o_licnniinnggggg
170
65.6
contract notice on_ compelled disclosure
150
77.9
contract ali permissible acquirement of
186
66.6
dimilar_informan
contract pewmissible_cop copy
95
60.2
contract_ nli similar information
144
61.0
contract nli catmmissible_post-grreeeee possession
119
81.7
contract return of of_coniidentaal information
74
74.2
contract_nli_ct
178
83.9
contract nli sharing with third-parties
188
78.9
165
64.6
contract ga
88
48.3
corporate_lobbying
500
878.1
cuad_afe affiliate jieeeeeleeeee--eeseeee
204
75.9
cuad affiliate license-licensor
94
99.7
cuad_an aodd_aani-asiimmen
1178
54.6
cuad audit_ rights
1222
53.6
cuad_cap_on__iieee Jiabilliyy
1252
59.8
saadd__hanee_ff_o caieeeooo_oooe control
422
62.8
cuad_comee retiiiiveeeetttttineeeeeeeee
226
67.1
29

---

Table 9 continued from previous page
Task
Number of Samples
Mean Sample Length (Words)
cuad_ codd_covenat__eee not_tttttttttteeeee tuu sue
314
64.6
date date
242
44.5
cuad_excline excluuiiiiyy
768
58.0
cuad_exe cond__expration___ date
882
49.9
cuad governing law
882
46.9
1036
56.8
cuad ip ownership assignment
582
65.3
cuad irrevocable or perpetual license
286
72.1
cuad_joint_ip_owneee cuad
198
59.1
1402
63.5
cuad_lique laad__liuidddddddaaaa dammggs
226
57.2
cuad minimum, commitment
778
57.9
cuad_ quad_most___e
70
68.0
cuad no-solicittittcoooooo of customers
90
61.2
cuad_ no colo-soliiittooo of employees
148
66.6
cuad_ compeee
448
60.6
cuad_ madd-aaaaedseeeemeeeett
106
63.8
cuad_non- non cranontttaasfraale__iiioeoe
548
61.5
cuad_notice_perriae to___tt_ttttoot Lerminate_renovi
228
57.0
cuad post-memtttnnnnnnonnn services
814
67.3
wuad__priceresstritti
52
53.6
cuad renewal term
392
55.2
cuad revenue-profit sharing
780
59.8
cuad_ roff---of-
696
64.0
cuad source code escrow
124
64.4
436
52.0
cuad third party beneficiary
74
42.4
suad__ncapped liabiilit
300
70.1
cuad www.:iiiiccallllyooaaaaaai-iiiia
54
56.8
328
49.0
cuad_ hordd_wrrratyycco wwaranntydduration
326
55.9
1345
41.0
defrnition_extraationn
extraction
695
53.9
diversity_
306
16.2
diversity
306
23.4
diversity_3
306
21.1
diversity 4
306
23.5
diversity
306
28.3
diversity_6 6
306
47.9
function of decision section
374
87.4
hearsay
100
25.3
30

---

Table 9 - continued from previous page
Task
Number of Samples
Mean Sample Length (Words)
Besurance_policy__ee intecc_interpretation
138
87.2
international internationalciiizzensiippppp questions
9310
33.2
rute rule_distinguushing
60
33.8
jerew_blocke blocker
60
167.2
learned hands benefits
72
253.2
gaamed_handse busides_
180
225.3
learned hands consumer
620
246.3
learned hands courts
198
225.6
694
233.7
learned hands divorce
156
240.2
loannd_domesttc.e
180
262.3
learned hands education
62
265.7
716
242.0
learned hands_estates
184
230.9
hlamed_handse __ndd_aaniilly family
2271
258.4
232
286.8
haamed_hands- goousin
4500
254.8
jeanded_hands____ jmmiigratin
140
232.8
438
272.2
learned hands traffic
562
229.6
gggga_rrasonnngccasaaee
59
245.7
maud_ "ability consummate", connnce is
70
688.6
subject to mae carveouts
maud_ consideration
oiew"""""_____ is is_the_sole_______
113
307.5
maud accuracy_ of ff__dannntttal target
176
143.7
r&ws: beas__binndoow__taag s_tandrd
maud_ sedd__acurry__ooo cactuuracy_oottrreeeeeee oofooorgettggeeeeeereeo "general" [&w::
182
142.5
fringdown_timing_.nee _anseer
maud_ ceuuddaccracyy_.c of Be___tagtt_apppe cagraltcaattliiiaiinn
182
142.0
r&w [ow_[outstanding seapssttnniiinnsssssss..... sharesh: bringdoww
standard. answer
maud additional iaadiitionl_matccinnnn.noo maatching_rights_prire for for_modifications_ccoo
159
314.3
maud application buyer_consent. consent requirement (negative interim covenant)
181
85.6
maud_ course)
182
121.3
maud change in law: subject to iissroportionate_impaee _modiiierrr
100
702.6
maud changes in gaap_or gaap_or_other. accounting crrouuting_princpples... subject pripeiples._suueeedd to_
99
703.1
"disprpportionate: impact" _modifirr
maud _aa__oo_______eeeeeeee permitted_ in reureepoone_to intervening_event
101
305.6
31

---

Table 9 continued from previous page
Task
Number of Samples
Mean Sample Length (Words)
maud_com maud__orr_pppa Becid_coo___mmiie corr_pemmitte_... _eithd_hthhhhhh _it___ttthtttt__tt______ _ determination _nnly
101
303.1
maud_cor maad__or_sttaddiee cor_standard (intereeinntn event)
85
326.1
maud_coo cor ctc__ooorr__ttte standard (suueerio_oofiie
101
308.0
maud definitiinn cotiiiiioooooiiiineeeeeee knw.aiddes_nwdddge___. requirement answer
148
243.4
maud definition includes asset deals
147
314.7
maud_definineg senddddddenntiioinnuddeesssstsss
149
313.6
maud fiduciary fiduciary_exception:: board determination_standaaee
180
246.8
maud_ cedducaar__xxcepttinn _oard___
180
245.1
determination Betrttattiooinigee_((_(_ (noeer__oo_oso))
maud_ fts_(mmac_standarr
78
705.
maud wtt.geeeeecooooennee economic_ann financial_
99
704.6
conditions: conditions__sujject_oo "disproportionate.
impact" _modifier
maud includes_"" includes."cooniittet_ "consistent_with_peee practice"
182
122.7
maud_ing ioauudiintial__aa mniial_macchnng_rihhsseeeid (ccrr
159
313.4
maud initial wwwpllmatthhhhiiieeee rightgg__rrg_______e cecllls__erod____ (ftr)
133
336.4
haw_d__iteeennnn__ett_eeeeeee occur_after pocur_affer_sinningg_mmm
148
242.2
maud knowledge definition
168
334.8
maud mraud_iiablliy__standadd htrpaalllltytttttnnaddoooooooooo for_ooosshop_rreeeeeee breach__ by wwg.rrge_noonddoorrrrussennnnn
157
38.1
maud gauud__andemmcc_oooottttpppueeu __clth_______
efteootssstttddddddddd standard
182
122.7
99
707.5
event: event:_ sucnt:__ubjccttto._ torpooooodpipptttittttttt
"mpact"_modiife
maud souud_pandemic. _t___oorr_________eee__eeeeeeeee otther_publiccc______ppooppp health_
99
707.5
wwent__peecific_pge reference to to__andemic--
related Beliloo_ggveemneeaalaaaeeee or measures
705.5
maud specific perfolmmppeooeceeeeee
179
96.8
maud_taie haapaaal__ppodeeeeee length
180
95.1
maud type of consideration
173
128.2
nys_ @ys_judicial_ethies
300
25.7
1oppt155_at__eeeentii oppl
96
31.5
15 data sacuriyyy
1342
38.6
1ppo____d_nnottrree
118
37.1
oppl15_first 1ppll5_first_parrt palllprrrttaaallllllllll cow.eecoooocooooooo use
2094
32.6
oppll15iineeraaiioall.ddd stppiiioonnnnnnosspsspiiiiiiiice audiences
988
52.1
1opl155policy_chamee
439
33.2
1598
35.3
470
32

---

Table 9 continued from previous page
Task
Number of Samples
Mean Sample Length (Words)
oppl15_user, uuse__choice_coneeeee
1554
33.5
oral_arg wuw.aauueeenguuettionnnnoooo questioonpuupooeee purpose
319
50.2
overruling
2400
27.5
personal_ jussoaaljurisdictioon
54
67.8
privacy_policy srrrcyypolccy_eetalmeen entailment
4343
111.9
privacy_policy_ga
10931
41.1
proa
100
42.6
rule qa
50
11.7
scalr
571
275.4
ssla company defendants
1231
310.0
ssla _nd_iiiiidul__deeedddnttt defenddnntss
1015
313.7
ssla plaintiff'
1036
308.4
sara_entailment
276
148.0
sara numeric
100
12222.1
successor successor_liabilitt
50
71.5
supply_ cepoly_chhinnoee dispiiddddssssbbbbeeeee bertteeeeettttsseeeeeeeeeeeeeeee accountability
387
510.0
supply_chain chain disclosure prretice audits
387
508.3
supply chain_ wistiidddsclssbbbbbeeee Baaiiie___et____eeeeeeee bracttcce
386
508.4
certification
copply_chain disclosure best _eet_eeeseeeeee_ee__ practice training
387
508.2
bttpaaindddislouuur_eee_ppppeeed verification
387
507.0
supply_coe chain disclodure_ accountability
386
510.4
387
508.0
hupply_chaing wwweloosreeddseeeeeeo certification
386
509.9
supply chainncchannnaeeeee haary_hhhannidsia biseclourr_ddiscloee__mme
387
506.9
supply_chhep chain wwwclossrseecccooo distoloseeedddlssdd___ verification
387
507.6
telemarketing sales_rule
51
58.4
cetualissm_tool diottoocios
111
151.3
textualism tool plain
169
160.9
ucc_ rrc_v_common_lav
100
20.9
unfair tos
3822
34.3
33

---

F
Evaluation
This section describes metrics and evaluation protocols.
Rule-application To evaluate an LLM's performance
rule-application task, a law-trained expert manually validated each generation. We computed two metrics. The first metric- correctccsssssssss--.s.. corresponds to the proportion of generations which do not misstate the fact pattern, the legal outcome, the rule, contain logical error. Logical errors include arithmetic mistakes, or assertions which are plainly wrong (c.g., that an apple is not a tangible object).
The LLM would incorrectly assert the legal outcome. For instance, an LLM would assert diversity jurisdiction existed, when actually did not.
The LLM would incorrectly assert an intermediate conclusion. For instance, LLM would assert that contract for service, rather than moveable and tangihle good. In
rental agreement for boat this case, the L.LM would fail to realize that a hoat is moveable or tangible good.
The LLM would hallucinate piece of information not explicitly stated in the fact pattern.
The LLM would misstate the content a rule. For instance, the LLM would assert that subprovision a statute barred one type of conduct, when in fact, that conduct was barred by a different subprovision.
The second metric anlaysis corresponds to the proportion of generations which contain the necessary inferences to reach the correct legal conclusion from the provided fact- patteert Thus, it is insufficient for a LLM explanation to merely state that a piece of evidence is hearsay is insufficient: an explanation must reference the qualities of the evidence which make it hearsay. We introduced this measurement after discovering that for tasks, L.MSs often generate explanations which- correct- correct-largely restate the rule being applied, without any reference to the underlying facts. Explanations which are incorrect are automatically deemed to be insufficient on analysis. We compute an overall analysis score for a LLM on a task by measuring the proportion of samples for which the explanation contains sufficient analysis.
To standardize evaluation and enable future work, we have released an "answer guide" for each task used for rule-application, which contains the inferences required for each sample, and deseribes common modes of errors. All evaluations LEGALBENCE for rul-aaplicationn have been performed with respect to this answer-guide
Generation tasks
LEGALBENCH contains the following generation-tase which are evaluated as follows:
rule_qa is question-ansse -nnooer task in which a LLM must generate response to an open-ended question. A law-trained individual evaluated the generations against answer-key for the task, which is available download from the website.
The Securities Complaint Extraction tasks require an LLM to extract the names of different parties from excerpts of securities class-action complaints. Because some samples require the extraction of multiple entities, we evaluate using Fl score.
definition, tettttionntttteee con_eextactiion requires the LLM to identify the term that is being defined in a sentence from Supreme Court opinion. For small number of sentences, any one of multiple terms may We therefore evaluate performance this task using accuracy, and by counting the fraction sentences for which LLM identified permissible term. To account for edge-cases involving word tense, we compare stemmed versions of the answers to stemmed versions of the generation.
constitute the correct
sara_numeric requires an LLM to generate an estimate of the amount of tax that is owed. compute performance here using an accuracy metric, which treats prediction as accurate if within 10% of the true amount.
citation_ oppeinnn_ppennnnoo requires an LLM to predict the name of the case that should be cited for a particular sentence of judicial text. We evaluate by checking if the LLM generation contains the correct case name.
successor__ Satesssr__aabbllee liability requires an L.L identify the multiple possible successor liability excep- tions to a fact pattern. We evaluate using F1.
Classification tasks We evaluate all classification tasks in LEGAL BENCH using exact-match class- balanced-accurray. We do this because a number of LEGAL tasks class-imbalanced.
34

---

G
Task descriptions
This section provides detailed description for each family of tasks.
G.1 Abercrombie
In LEGAL BENCH, the Ahererombie task is denoted as abercrombie.
Background A particular mark (e.g.. name for product service) is only eligible for trademark protection if considered to be distinctive. assessing whether mark distinctive, lawyers and judges follow framework set out in the case Abercrombie & Fitch Co. v. Hunting World, Inc. which enumerates five categories of distinctiveness. These categories characterize the relationship between the dictionary definition of the term used in the mark, and the service product it is being attached to. They are:
Generie: name is generic with respect product service it connotes the basic nature of the product/service, rather than more individualized characteristies of the product. For example, the mark "Salt" for packaged sodium chloride would be generic under Abererombic because "salt" is the common name for sodium chloride. It is also common to think of generic marks as merely referring to the class of goods for which a particular product is species.
Descriptive: name is descriptive it identifies a characteristic or quality of an article or service, such as color, odor, function, dimensions, or ingredients. For example, the name "Sharp" for a television would be descriptive. because it describes plausible characteristic of television (i.e., their sharp image quality).
Suggestive: A name suggestive if suggests, rather than describes. some particular characteristic of the goods or services to which it applies. An essential aspect of suggestive names is that it requires the consumer to exercise the imagination in order to draw conclusion as to the nature of the goods and services. For example, the name "Greyhound" would be suggestive for bus service, because greyhounds are considered be fast, and "fast" is an adjective that could used to describe a bus service.
Arbitrary: A name is arbitrary if it is a "real" word but scemingly "arbitrary" with respect to the product service. For example, the mark "Apple" for software company is arbitrary, because apples are unrelated to software.
Fanciful: A name is fanciful if it is entirely made up. and not found in the English dictionary. For example, "Lanmbe" is fanciful mark, because is madc-up word.
The Ahercrombic spectrum is commonly taught as part of Intellectual Property courses in law school, and students are expected to understand how determine the Ahercromhic classification for particular product/mark combination.
Performing the Abercrombie task requires reasoning about the literal meaning of word and degree of its connection to a particular product/service. It requires having some understanding of the types of words that could plausibly be used to describe particular good/service, and the extent to which those words relate to particular mark. It also requires reasoning as to whether particular word is real English word.
Task
The Abererombic task requires an LLM determine given candidate mark and description of product/service-wine which of the five Abercrombic categories above apply.
Facts
Abercrombie Classification
The mark "Whirlpool" for an oven.
arbitrary
The mark "Compact" for wallets.
descriptive
The mark "Imprion" for a line of sports drinks.
fanciful
The mark "Car" for a line of automobiles.
generic
The mark "Quick Green" for grass seed.
suggestive
Table 10: Task examples.
Abercrombie & Fitch Co. Hunting World, 537 F.2d 4 (2nd Cir. 1976).
35

---

Construction process We manually create dataset to evaluate model's ability to classify mark's s distinctiveness (into of the above 5 categories) with respect product. In writing samples, draw inspiration from similar exercises available in legal textbooks and practice study guides. Hence, the samples provided have definite answer, and are not subject to ambiguity. There is an expectation that law student learning intellectual property would be able to answer these questions to high degree of accuracy.
We create approximately 20 samples for each category of distinctiveness, and randomly select a single sample from each category constitute the train set. The remaining 19 samples (for each category) are assigned to the set (for total of 95 samples).
Class
Number of samples
generic
19
descriptive
suggestive
20
arbitrary
18
fanciful
19
Table 11:
Test set class distribution.
Significance and value Given how casy this task for lawyers with a basic training intellectual property law, it is unlikely that LLMs will be called on to perform this task in the actual practice of law, or that the ability for LLMs to perform this task would alter the way in which lawyers approach IP practice. Instead, the Abererombie task is significant measurement reasoning ability. Because is "simplistic" by the standards human lawyers, provides useful objective measure of reasoning progress LLMs.
36

---

G.2 Canada Tax Court Outcomes
In LEGALBENCH, the Canada Tax Court Outcomes task is also denoted as
Background The Tax Court of Canada hears appeals of government decisions related to taxation The Court's decisions, which are written in natural language, are published on the Court's website, in both French and English | Decisions typically include a section the beginning summarizing the outcome of the appeal. followed by sections describing the factual background and various procedural steps, a section identifying the issues under consideration, sections with legal analysis, and a concluding section. While this is the standard format, judges are free to use other formats if they prefer. Decision length varies depending on the complexity of the litigation, with some decisions heing only a few hundred words, and others being many thousands of words.
Appeals Tax Court of Canada cases are brought by individuals organizations who ask the Court to overtum a government taxation decision. Outcomes of appeals are generally binary: appeals are either granted, which case the government taxation decision is overturned in whole or in part, or appeals are denied in which case the government taxation decision is upheld. Occasionally published decisions will not involve the outcome of an appeal, including where the decision about a procedural step (e.g. the admissibility of particular evidence).
The canada_ couuutotutoooteeeeeeeooo task involves identifying whether excerpt from a Tax Court of Canada decision includes the outcome of the appeal and, if so, what the outcome is. While the task is straightforward, one challenge is that the model must distinguish between outcomes of the appeal as a whole and outcomes of particular aspects of the appeal. Another challenge is that where the excerpt does not include the outcome, the model must avoid predicting the outcome even if the model might plausibly correctly infer the likely outcome from the excerpt provided.
The Canada Tax Court Outcomes task requires an LLM to classify whether excerpt from given decision includes the outcome of the appeal, and whether the appeal was allowed or dismissed. Some excerpts do not include an outcome of the appeal, in which case the model should return 'other'. Where the excerpt includes the outcome and the appeal is allowed in whole or in part, the model should return 'allowed'. Where the excerpt includes an outcome, and the appeal is dismissed the model should return 'dismissed', The model should disregard outcomes that are not about the ultimate outcome of the appeal, such costs awards (i.c.
Task
orders requiring party to pay the other party's 's legal costs).
versions of decisions from 2001 to 2022 by scraping the Tax Court of Canada website 10 We then cleaned and parsed the text to extract excerpts that are most likely contain the outcome of the appeal. For example, many decisions contain brief introductory section describing the outcome of the appeal using specific header, and if the decision contained section with such a header, we excerpted only that section. Where our parsing code could not identify such section, we excerpted the first and last 2,500 characters, because outcomes are generally described at either the beginning or end of decisions. After initially attempting outcome classification on these excerpts using OpenAl's ChatGPT, we selected a quasi-random sample of 250 excerpts (quasi random because we selected these manually, we over- sampled excerpts where the outcome is 'other', and chose some excerpts that were challenging due to factors such as length or unusual format). We manually reviewed outcomes for these excerpts, correcting some that had been miscategorized.
Construction process We obtained the full text of
Two random cases from each class are selected for the training split, while the remainder are used as the test set.
Significance and value Legal scholars frequently gather data about outcomes in large numbers of legal decisions in order to examine patterns in judicial decision-
For example. legal scholar may be interested in comparing outcomes in similar processes across jurisdictions or they might examine whether a legislative change resulted in different outcomes over time. Lawyers and legal information technology companies may also be interested in gathering data on outcomes for the purposes of judicial analytics or to predict future outcomes.
Gathering such data is typically straightforward. It is, for example, a common task assigned to first year law student research assistants who can frequently achieve close to 100% accuracy on such tasks with only minimal training. However, because the data is oflen useful only when gathered on large numbers of decisions, this type of data gathering using human research assistants can be cost prohibitive. If LLMs can obtain high accuracy on these tasks, substantial savings could be achieved which would increase the ability of researchers to pursue new projects
TTax Court of Canada Act, RSC, 1985, T-22 online: http:p///aaae-bbaa.jjiiiee.e..e 2/ //indexx. html s 12. Tax Court of Canada, "Find Decision", online: http::///eccssan...cccco...oo/oo wttpc:/....tttcccc/ooooooooooo/e nav. do
"Ibid. As per the terms of service of the website, we are required to note that the text of the scraped decisions are not the official versions (official versions can be obtained from the website). and that the reproduction of
these cases has not been produced in affiliation with with the endorsement of the Government of Canada.
37

---

Excerpt
Outcome
The appeal is allowed in part and the assess- ment is referred back to the Minister of National Revenue for reconsideration and reassessment to reflect 25% reduction of the tax owed by the appellant and adjustments to the interest and penalties, as agreed to by the respondent. Costs are to be determined after hearing both parties. In all other respects, the assessment is confirmed. Signed this 23rd day of February 2012. "Franois Angers" Angers J.
allowed
IN ACCORDANCE with the Reasons for Judg- ment attached, the appeal from the decision of the Respondent in relation to the income of the Appellant for the purposes of determining his entitlement to the Guaranteed Income Supple- ment under the Old Age Security Act for the payment period from July I, 2014 to June 30, 2015 is dismissed, without costs. Signed at Ot- tawa, Canada, this 24th day of October 2017. R.S. Bocoek Bocock J.
dismissed
(These Reasons for Judgment issued in sub- stitution for the Reasons for Judgment signed on January 22, 2002) Lamarre, [1] These are appeals under the informal procedure against assessments made by the Minister of National Revenue ("Minister") under the Income Tax Act ("Act") for the 1995, 1996, 1997, 1998 and 1999 taxation years. [2] In filing her 1995 income tax return, the appellant claimed a business investment loss of $268,897 8,9 with re- spect investments in eight mortgages held "in trust" for the appellant and her father Henry Sokolowski by Kiminco Acceptance Co. Ltd. (" ("Kiminco"), member of the Glen Coulter group companies. The eight mortgage in- vestments were made in 1987 and 1988 and identified as follows in paragraph 13 the Re- ply the Notice of Appeal: Account/Mortgage Number Ultimate Borrower ... For the Appel- lant: Name: Firm: For the Respondent: Morris Rosenberg Deputy Attorney General of Canada Ottawa, Canada
other
Table 12: Task examples.
Outcome
Number of samples
allowed
101
dismissed
131
other 12
Table 13: Test set class distribution.
38

---

G.3 Citation Prediction Tasks
In the LEGALBENCH, the Citation Prediction tasks are also denoted as citatiion.preticcion..
Background The importance of locating relevant legal materials, "law search" has long been recognized an essential aspect of legal practice. This process involves uncovering case law, statutes, and other materials pertinent legal questions or arguments. As a fundamental aspect legal reasoning, law search plays a crucial role in bridging the gap between the initial translation of behaviors into legal questions and the subsequent interpretation and application of the relevant law.
Legal professionals are often valued for their ability to find and apply the appropriate law to their clients' situations. Given the intricate nature of the contemporary legal domain, the process of law scarch has evolved into complex and nuanced task that demands a comprehensive understanding of the law.
core component of law search is legal relevance. From a sociological perspective, the relevance of legal documents a specific legal question a social fact. This fact is determined by the judgments made by members of the legal community, who must determine which legal materials are applicable to a given question. Relevance relates legal questions to sources of legal authority.
In functional legal communities, law search leads to some degree of convergence over legal materials. Con- vergenec occurs when competent members of legal community, faced with the same legal question, identify the same sources of relevant legal authority. This process is essential to ensuring that the legal system operates consistently, predictably, and coherently.
As a critical process that connects the translation of behaviors into legal questions and the subsequent interpreta- tion and application of the relevant law, law search is indispensable to legal reasoning.
The Citation Prediction task requires reasoning concerning the relationship between the text of judicial opinions and legal propositions. Successful prediction would entail encoding notion of legal relevance and would allow system determine whether legal proposition was not supported by the extant body of law.
version of the evaluation approaches used in 38). There two Citation Prediction tasks. The first Eestatiion_preee cttionoclasssiiiiiiitcoooeaeceae classification requires an LLM to predict whether a given sentence (i.e. legal proposition) is or is not supported by a given case. The second (citation_ prediction_oper requires an LLM to predict a case (by name) that supports provided sentence.
Task The citation task is based
Construction process We collected sample of circuit court opinions published after Jamuary 1, 2023. To the best of our knowledge, most existing LLMs haven't 't been trained on any data generated in 2023. For each opinion, we manually collected sentences which were supported by citation a judicial opinion, where (1) the sentence contained some quotation from the original case, and (2) the sentence was supported by a single cite. We chose sentences which included quotation fragments and were only supported by single cite to avoid sentences which could be supported by broad set of cases. When a sentence is supported by much larger universe of cases, verifying that an LLM answer is incorrect is difficult. We also recorded the circuit for each opinion that we pulled language from. result, we can include the circuit information the prompt, since circuits prefer citing their previous decisions. We collected 55 sentences using this process. For the citation generation task (citation, (citation.predictiini wttp:/ppeedddppppeeeeeeeeoe we ask the L.M to predict the citation given the sentence.
The citation, Ettation_prediction pww.diitioo_cleesiiiicttt.ee classification task is then constructed as follows. We use each sentence- citation pair create two task samples. The first sample corresponds to the sentence and the correct citation (positive label). The second sample corresponds the sentence and randomly selected citation from the remainder of the data (negative label). This generates dataset of sentence-citane cittttionn pairs, two of which are assigned to the training split.
Significance and value Law search is a core function of legal thinking. In addition, the difficulty of identifying relevant law is core barrier in the public's ability usefully access the law. The ability of an LLM to accurately engage in citation prediction would have important practical value in providing access to law, and would also allow the LLM to more reliably support legal statements with relevant authority.
39

---

Input
Citation
Supported?
Exclusions are always strictly construed against the insurer and in favor of the insured.
Nationwide Mut. Ins. Co. V. Cosenza
Yes
The Supreme Court and this court have repeatedly "held that environmental plaintiff's adequately allege injury in fact when they aver that they use the affected area and are per- sons for whom the aesthetic and recreational values of the area will be lessened by the challenged activity."
United States Pearce
No
Table 14: Examples for sitation__rediction, cenddiccionccaasiiiii classification
Input
Citation
In other words, the DJA "creates a means by which rights and obligations may be adjudicated cases involving an actual controversy that has not reached the stage at which either party may seek coercive remedy."
United States Doherty
To be "equivalent to a demotion," the action need not "result in decrease in pay, title, grade; it can be demotion if the new position proves objectively worse-such as being less prestigious or less interesting or providing less room for advancement."
Alvarado v. Tex, Rangers
Table 15: Examples for iitattinn_prdictioon.ooo
40

---

G.4 Clause Classification Tasks
LEGALBENCH includes number of tasks in which the LLM must determine the "type" or "category" of provision/clause in legal document. Specifically:
The Contract QA task (Section[G.4.. in which the L.M is provided with the name of a common type of contractual clause and clause, and must determine if the clause is example of the example type.
38 tasks derived from the CUAD dataset (Section G.4.1 where each task is a binsrycclassification task requiring the LLM to identify whether a clause (from an EDGAR contract) belongs to a certain category (e.g., audit rights clauses) [60].
The J.Crew blocker task (Section G.4.2) 3 in which the LLM must classify whether a clause (from a loan agreement) is a J-Crew blocker provision.
The Unfair Terms of Service task (Section G.4.3). in which the LLM must classify clause (from terms service agreement) to one of eight types, where seven of the types denote clauses that would potentially be considered "unfair" under European law (80).
Significance and value Lawyers spend significant time and energy reviewing legal documents (e.g.. contracts, leases, etc.). Manual review serves an important purpose. allowing parties to identify potentially problematic terms [17]. Parties will sometimes review agreements that have already been signed, in response to changing world events. For instance, the COVID-19 pandemic led many firms to inspect agreements for the existence of force majeure clauses, which ordinarily specify how contractual expectations should be handled in the event of major world crises 488. Because legal documents are long and require legal training understand, the process reviewing often extremely expensive [60]. This, in turn, presents significant access-to-justin concerns. Because most individuals do not have the financial capacity to consult lawyers prior to entering legal agreements, they are oblivious to when those agreements contain predatory, oppressive, unconscionable terms. A rich legal scholarship has noted, for instance, the frequency which legal agreements contain terms that would be invalidated by court (25
The clause classification tasks in LEGALBENCH are thus amongst the most practically useful tasks in LEGAL- BENCH, as they capture an actual current-day use case for LLMs. As the complexity of clause classification depends both on the clause category and document type, LEGAL BENCH tasks span range of clause categories and source documents.
G.4.1 CUAD Tasks
We adapt the CUAD dataset for LEGALBENCH (60] The original dataset consists of 500 contracts, each annotated with up 41 different clause types. These contracts varied significantly in length, ranging from few pages to over one-hundred pages. the original word, [60] studied the ability for BERT-base language models to identify the text spans corresponding to different types of clauses. The principal difficulties were (I) the length of the contract, and (2) the lack of significant training data.
We adapt the CUAD dataset follows. We select 38 of the 41 clause categories. For each selected category. we construct a dataset consisting of (1) clauses in the CUAD contracts which are assigned to that category, and (2) an equal number of clauses randomly sampled from other categories. This produces balanced hinary classification task for clause category, where the purpose is to identify which clauses helong to the respective category. A table with the selected categories, and their descriptions is found below.
Table 16 lists each task, a "description" of the category correspouding to the task, and an example of a clause which meets the category criteria. In accordance with (60), the description is presented as the question posed to the annotators during data labeling. If clause yields affirmative answer with regards to the question, then the label is "Yes", Otherwise the label is "No".
In LEGALBencH the CUAD tasks are denoted as cuad_*
Table 16: CUAD Tasks
Task
Task name: _affddiafftttt_____eeeeeeeeeee__ setttellleeeeeessseee
Description: Does the clause describe license grant a licensee (inel. sublicensor) and the affiliates of such licensee/sublicensor?
Example: [**=]] Valeant hereby grants to Dova a fully paid-up. royalty free, non-transferable, non- exclusive license (with limited right to sub-license to its Affiliates) to any Valcant Property that appears on, embodied contained in the Product materials Product Labeling solely for use in connection with Dova's 's promotion or other commercialization the Product the Territory.
41

---

Table 16 - continued from previous page
Task
Task name: _and__flliate____ affiliate_licnseeeee license-licenso
Description: Does the clause describe license grant by affiliates of the licensor or that includes intellectual property of affiliates of the licensor?
Example: "Company Licensed Know-How means all Know-How owned by any Company Entity as of the Effective Date and used or held for use in the Arizona Field as of the Effective Date, SuujteddSuujeee to the terms and conditions of this Agreement, the Company hereby grants to Seller a perpetual, non- exclusive, royalty-free free license in, and under the Company Licensed know-How for use in the Arizona Field throughout the world.
Task name: cuad anti-assignment
Description: Does the clause require consent or notice of a party if the contract is assigned to a third party? Example: Except as otherwise set forth herein, neither party shall transfer, assign or cede any rights or delegate any obligations hereunder, in whole or in part, whether voluntarily or by operation of law, without the prior written consent of the other party, which consent may he withheld the other party's reasonable business discretion; provided, however, that either party may transfer this Agreement without prior written consent of the other to an Affiliate of such party, or to the surviving party in merger or consolidation, or to purchaser of all or substantially all of its assets.
Task name: cuad audit rights
Description: Does the clause give a party the right to audit the books, records, or physical locations of the counterparty to ensure compliance with the contract? Example: For avoidance of doubt, all audits under this Section shall be conducted solely by an independent public accountant as described in the foregoing sentence.
Task name: cuad_ saadd_ap_oor sas_ooo__aiiiiii liability
Description: Does the clause specify a cap on liability upon the breach of a party's obligation? This includes time limitation for the counterparty to bring claims or maximum amount for recovery.
Example: EXCEPT FOR LIABILITIES UNDER SECTION 7.2 [Indemnity], NEITHER PARTY'S AG- GREGATE LIABILITY ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT OR THE TRANSACTIONS CONTEMPLATTO LLTTED HEREBY, WHETHER IN CONTRACT, TORT (INCLUDING NEGLIGENCE), WARRANTY OR OTHERWISE, SHALL EXCEED [***].
Task name: cuad change of control
Description: Daes the clause give one party the right to terminate or is consent or notice required of the counterparty if such party undergoes change of control, such merger, stock sale, transfer of all or substantially all of its assets business, or assignment by operation of law?
Example: Notwithstanding the foregoing, any Party to this Agreement (or any of its successors or permitted assigns) (a) shall enter into a consolidation or merger transaction in which such Party is not the surviving entity and the surviving entity acquires or assumes all or substantially all of such Party's 's assets, (b) shall transfer all or substantially all of such Party's 's assets any Person or (c) shall assign this Agreement to such Party's ss Affiliates, then, each such case, the assigning Party (or its successors or permitted assigns, as applicable) shall ensure that the assignee or successor- in in-interest expressly assumes in writing all of the obligations of the assigning Party under this Agreement, and the assigning Party shall not be required to seek consent, hut shall provide written notice and evidence of such assignment, assumption or succession to the non-assigning Party.
Task name: cuad competitive cenppetiiveeresttrtttioono exception Description: Does the clause mention exceptions or carveouts to Non-Compete, Exclusivity and No-Solicit Customers?
Example: Notwithstanding the foregoing, Excite may make available opportunities on the Excite Site to purchase Music Products from parties other than Sponsor if such Music Products are not available from Sponsor so long as, prior entering into arrangements to make available opportunities to purchase Music Products from parties other than Sponsor, Excite notifies Sponsor of interest in the Music Products and gives Sponsor thirty (30) days to make the desired Music Products available through the Sponsor Site.
Task name: cuad_ cored__oeean_tteeeeeee to_suc Description: Is a party restricted from contesting the validity of the counterparty's 's ownership of intellectual property or otherwise bringing claim against the counterparty for matters unrelated to the contract?
Example: In connection with any reference the Trademarks, Distributor shall not in any manner represent that has an ownership interest in the Trademarks or registration(s) thereof, and Distributor acknowledge that no action by it or on its behalf' shall create in Distributor's 's favor any right, title, or interest in or to the Trademarks.
Task name: _xcclusvvyt
42

---

Table 16 - continued from previous page
Task
Description: Does the clause specity exclusive dealing commitment with the counterparty? This includes a commitment procure all "requirements" from one party of certain technology, goods, services prohibition on
prohibition licensing or selling technology, goods or services to third parties, collaborating or working with other parties), whether during the contract or after the contract ends (or both), Example: Bosch hereby grants to Client the exclusive rights to sell and distribute the Product, subject to the Territory as set forth below, to certain select companies in the Automotive Industry, each of which shall be approved by Bosch in writing requested by the Client case by case basis.
Task name: cuad_ _insurancee
Description: Is there a requirement for insurance that must be maintained by one party for the benefit of the counterparty?
Example: Throughout the entire Term, you must maintain such types of insurance, in such amounts, as we may require.
Task name:
Description: Does intellectual property created by one party become the property of the counterparty, either per the terms of the contract or upon the occurrence of certain events?
Example: Upon written request ArTara, University will assign the IND to ArTara.
Task name: prau__irvvocable_.. Brr___rr__ppppeee or_perpetual Lipeeueallicense Description: Does the clause specify a license grant that is irrevocable or perpetual??
license
Example: Subject to the terms and conditions of this Agreement, as of the Distribution Date, SpinCo hereby grants to Nuance and the members of the Nuance Group worldwide, non-exclusive, fully paid-up. perpetual and irrevocable, transferable (suhject ARTICLE VIII), sublicensable (subject to Section 4.01(g)) license install, access, use, reproduce, perform, display, modify (including the right to create improvements and derivative works), further develop, sell, manufacture, distribute and market products and services based on, using or incorporating the SpinCo Shared Technology Assets within the Nuance Field of Use, together with natural extensions and evolutions thercof.
Task name: quad_joint_jp joint_ipoowoneeeeeeeeeeeeeeeeeee ip_ownership
Description: Does the clause provide for joint or shared ownership of intellectual property between the parties to the contract?
Example: If the Domain Name is deemed a combination mark, neither party shall use the Domain Name for any purpose except as expressly provided herein or attempt to register the Domain Name, and the parties will jointly cooperate on any enforcement action of infringement of the Domain Name.
Task name: cuad_ Jicense__eeeeeeeeee _gratt
Description: Does the clause contain license granted by one party to its counterparty? Example: Neoforma hereby grants VerticalNet a non-exclusive, non-transferable, royalty- free, right and license to link to the Neoforma Sites through a Neoforma Link.
Task name: cuad_ liquidated @guuddateddaaaaees
Description: Does the clause award cither party liquidated damages for breach or a fee upon the termination of a contract (termination fee)?
Example: You and each of your principals agree that the liquidated damages provision does not give adequate remedy at law for any default under, or for enforcement of, any provision this Agreement other than the Royalty Fee sections.
Task name: cuad_ miinnumm_commimmmn
Description: Does the clause specify a minimum order size or minimum amount or units pertime period that party must buy from the counterparty?
Example: If the Quarterly Average Sales Force Size is less than [***] Sales Representatives for an applicable Calendar Quarter, then calculating the promotion fee due under Section 6.1.1, the Applicable Percentage for such Calendar Quarter shall be reduced to new percentage equal to [+++
Task name: cuad_most most favored nation
Description: Does the clause state that if a third party gets better terms on the licensing or sale of technol- ogy/goods//ervices descrihed in the contract, the buyer of such technology/gooddssrrvies under the contract shall be entitled to those better terms?
43

---

Table 16 - continued from previous page
Task
Example: Eutectix agrees that in the event any Licensed Products shall be sold (1) to any Affiliate (as defined herein), (2) to corporation, firm, association with which, individual with whom Eutectix its stockholders Affiliates shall have any agreement, understanding, or arrangement (such as, among other things, an option to purchase stock, or an arrangement involving a division of profits or special rehates or allowances) without which agreement, understanding, or arrangement, prices paid by such a corporation, firm, association or individual for the Licensed Products would be higher than the Net Sales Price reported by Eutectix, if such agreement, understanding, arrangement results in extending such corporation. firm, association, or individual lower prices for Licensed Products than those charged to outside concerns buying similar products in similar amounts and under similar conditions, then, and in any such events, the royalties to be paid hereunder in respect of such Licensed Products shall be computed based on an assumed or deemed Net Sales Price equal to those charged to such outside concerns.
Task name: cuad no-solicit of customers
Description: Docs the clause restrict party from contracting or soliciting customers or partners of the counterparty, whether during the contract or after the contract ends (or both)? Example: During the Term of this Agreement, and for a period of one year thereafter, except as expressly provided in this Agreement, PlanetCAD shall not market any services to Customers without the prior written approval of Dassault Systemes.
Task name: cuad mond_nno-oliittt... of employees Description: Does the clause restrict party's soliciting or hiring employees and/or contractors from the counterparty. whether during contract or after the contract ends (or both)? Example: You covenant that during the term of this Agreement, except as otherwise approved in writing by us, you will not, either directly or indirectly, for yourself, or through, on behalf of, or in conjunction with any person, persons, partnership, corporation company:<omiteeds who is that time employed by us, our aftiliates, by any other franchisee of ours, or otherwise directly indirectly induce or seek to induce such person to leave his or her employment thereat.
Employ or seek to employ any person
Task name: cuad_non-comt non-compete
Description: Does the clause restrict the ability of party to compete with the counterparty or operate in a certain geography or business or technology sector? Example: Agent may not offer promote competitive products without the consent of Kallo.
Task name: cuad_nop _nn__nnoddo disparagement Description: Does the clause require a party not disparage the counterparty? Example: The Company shall not tarnish or bring into disrepute the reputation of or goodwill associated with the Seller Licensed Trademarks or Arizona.
Task name: cuad non- -on-transferable license
Description: Does the clause limit the ability of a party to transfer the license being granted to a third party? Example: Subject to the terms and conditions of this Agreement, Licensor hereby grants to Licensee, and Licensee hereby accepts from Licensor, a personal, non-exclusive, royalty-free right and license to use the Licensed Mark solely and exclusively as component of Licensec's 's own corporate name and in connection with marketing the investment management, investment consultation and investment advisory services that Investment Advisor may provide to Licensee.
Task name: cuad post-termination services Description: Does the clause subject party to obligations after the termination or expiration of a con- tract, including any ues-tterminaton transition, payment, transfer of IP, wind-down last-buy, similar commitments?
Example: Cisco agrees to repurchase Product in Distributor's 's inventory within [=****] days following the effective date of termination or expiration.
Task name: cuad _uad_price restrictions
price_restrictions
Does the clause place restriction on the ability of a party to raise reduce prices of technology, goods, or services provided? Example: The prices set forth in Section 2.4(a) shall be subject to adjustment annually on the first day of each Product Year beginning in the calendar year 2000 and on the first day of cach succeeding Product Year for the remainder of the Term and all renewals of this Agreement in proportion to the increase or decrease in the Consumer Price Index (CPI) as compared to the CPI as it existed on the first day of the Term of this Agreement.
Description:
Task name: Description: Does the clause require a party to share revenue or profit with the counterparty for any technology, goods, or services?
eevenuc-profit
44

---

Table 16 - continued from previous page
Task
Example: In consideration for the licenses granted to Corio pursuant to Scction 2 (exeept Section 2.5) of this Agreement, Corio shall pay the revenue sharing fees specified in EXHIBIT B hercto.
Task name: cuad_ roft-roforrofn
Description: Does the clause grant party right of first refusal, right of first offer right of first negotiation to purchase, license, market, or distribute equity interest, technology, assets, products or services? Example: If Licensee shall have exercised such right, the closing shall be held at the corporate offices of Licensee on the closing date specified in the Offering Notice or the date that is nincty (90) days after the date of Licensee's 's notice of its exercise of such right, whichever is later.
Task name: cuad source code escrow
Description: Does the clause require one party to deposit its source code into escrow with third party. which can be released the counterparty upon the occurrence of certain events (bankruptcy, insolvency, etc.)?
Example: With cach delivery of Software to Bank of America hereunder, Supplier shall deliver to Bank of America the Source Code for all Software and for all Updates, Upgrades and new releases of the Software.
Task name: cuad termination for convenience
Description: Does the clause specify that one party can terminate this contract without cause (solely by giving a notice and allowing waiting period to expire)?
Example: Customer may terminate this Agreement during the Term upon at least one (1) years' written notice to M&I, provided that Customer pays M&T an early termination fee ("Termination for Convenience Fee") in amount equal to REDACTED of the Estimated Remaining Value.
Task name: cuad third party beneficiary
Description: Does the clause specify that that there non-contracting party who is beneficiary all of the clauses the contract and therefore can enforce its rights against contracting party? Example: Such covenants must on a form that we provide, which form will, among other things, designate us as third party beneficiary of such covenants with the independent right to enforce them.
Task name: cuad uncapped liability
Description: Does the clause specify that a party's liability is uncapped upon the breach of its obligation in the contract? This also includes uncap liability for particular type of breach such IP infringement breach of confidentiality obligation
Example: Subject to the foregoing as wen as Mobimagic's 's obligations under this Agreement, Mobimagic shall not in any manner be held or be responsible or liable for any unforeseen contingency, claims, liabilities, demands. losses, damages or expenses arising due to absence of which shall he the sole responsibility of PC Financial
storage or retention of any PC Financial data
Task name: cuad unlimited-all all-you-can conuucca-aat---ee eat-license Description: Does the clause grant one party an "enterprise." "all you can eat" or unlimited usage license? Example: Subject to the terms and conditions of this Agreement, Commerce One hereby grants to Corio a tec-bearing, perpetual and irrevocable, nonexclusive, nontransferable (except in accordance with Section 14.1 of this Agreement), right and license in the Territory to<omitted<ivv) sublicense an unlimited number of Customers access and use the Software and MarketSite.net Service only through the installation Corio servers;
Task name: quad_volume wtt__voooeeeerttttiiiiiee restriction
Description: Does the clause specify a fee increase or consent requirement, etc. one party's use of the productt/eervices exceeds certain threshold?
Example: Make himself available for four (4) sessions for production of photographs, radio, television, video or other multi-media programming for Bizzingo's 's advertising or promotional materials, with each such session not exceeding eight (8) hours.
Task name: cuad effective date
Description: Does the clause specify the date upon which the agreement becomes effective? Example: This JV Agreement shall hecome effective on the signing date and shall have duration of years, extendable for a further * years, unless notice of non- renewal is sent one year before the natural expiry date. cotetcomit April 2020
Task name: cuad_ expiration_date
Description: Does the clause specify the date upon which the initial term expires?
Example: This Agreement shall be effective as of the Effective Date and shall continue in effect through December 31, 2021 and any Renewal Term (the "Term") ") unless terminated earlier as set forth herein.
Task name: cuad governinggggggngg law Description: Does the clause specify which state/country's 's law governs the contract?
45

---

Table 16 - continued from previous page
Task
Example: This Agreement shall be governed by and construed under the laws of the State of California, excluding conffict of laws provisions and excluding the 1980 United Nations Convention on Contracts for the International Sale of Goods.
Task name: sotodd_otte_____ notice_period______e to terminate renewal
Description: Does the clause specify a notice period required to terminate renewal?? Example: Unless either party gives written notice to terminate this Agreement at least six (6) months prior to the end of said Initial Term, this Agreement shall continue on a year to year basis ("Extended Term(s)") until terminated by either party by giving written notice of termination thercof to the other party at least six (6) months prior to the end of the then current Extended Term.
Task name: cuad renewal term Description: Does the clause specify a renewal term?
Example: This Agreement shall commence on the Effective Date and, unless sooner terminated in accordance with its terms, including by Ginkgo pursuant to Section 7.3 (Buy-Down Election) or extended by the mutual written agreement of the Parties, shall continue until the Intended End of Term (such time period, as may be extended pursuant to this Section 13.3.1 (Term General). the "Term"); provided that, iomttmitte the expiration of the Intended End of Term, Ginkgo has paid the Minimum Cumulative Purchase Commitment, but will not have paid to BLI the Full Purchase Target, then the Term of this Agreement shall automatically
extend for an additional [***] ([ [*** I) year period from the date of the expiration of the then- -en-Innedddd End of Term that, among other things, BLI may potentially receive the benefit of the Full Purchase Target and Ginkgo may receive the continuing benefit royalty- free licenses.
Task name: cuad_ warraaty_ddrrnn duration
Description: Does the clause specify duration of any warranty against defects or errors in technology. products, or services provided under the contract?
Example: Airspan warrants that, following repair replacement, the repaired replaced Equipment or Software by Airspan shall be free from defects in materials and faulty workmanship and that the Software will conform in all material respects Airspan's 's published specifications therefor for nincty (90) days from date of shipment from Airspan to Distributor or until the end the Initial Warranty Period, whichever is longer.
G.4.2 J.Crew Blocker
LEGALBENCH, the J.Crew Blocker task is denoted as jcrev-blocker.
Background Loan agreements often contain restrictive covenants that place limits on borrower's activities to protect the lender's interests. One such restrictive covenant that has become popular in recent years is the "J.Crew blocker" provision. This provision was created in response to actions taken by the retailer J.Crew in 2016. J.Crew transferred valuable intellectual property assets out of the collateral pool for its existing loans by moving them into new unrestricted subsidiary. This subsidiary was then able to use the IP assets collateral to obtain new financing.
The J.Crew blocker provision aims to prevent this type of activity by prohibiting borrowers from transferring IP assets out of the reach of existing lenders. There are two key components to a J.Crew blocker:
1. A prohibition on transferring IP assets unrestricted subsidiaries. This prevents the borrower from moving assets outside the scope of lender restrictions.
A requirement to obtain lender consent for any transfers to subsidiaries. This gives lenders oversight and control over how IP assets are distributed within the corporate group.
The presence of a robust J.Crew blocker in loan agreement is designed keep material assets within the collateral pool, and therehy protect lenders from horrowers' attempts secure additional debt through unexpected transfers of IP. For this reason, J.Crew blocker provisions have been widely adopted leveraged loan agreements.
Task The J.Crew blocker task requires determining whether a given provision a loan agreement qualifies as a J.Crew blocker. To make this determination, the provision must he analyzed to assess whether it contains:
1. A prohibition on transferring IP assets to unrestricted subsidiaries
AND/OR
2. A requirement obtain lender consent for IP transfers to any subsidiary.
46

---

If the provision includes one or both of these components, it can be classified a J.Crew blocker. If not, the provision does not meet the criteria.
Construction process The dataset for this task constructed by legal experts extracting real examples of provisions from public loan agreements. Each example was labeled either meeting the criteria for a J. J.Crew blocker or not. The dataset contains 60 total examples, organized into two columns: "Text" (containing the clause in question) and "Label" (indicating whether the clause is a J.Crew Blocker provision). Each clause was analyzed and classified as J.Crew Blocker provision ("Yes") not ("No"). The construction process involved manually reviewing and annotating these samples, ensuring that each clause accurately categorized. This process, carried out by legal experts, provides definitive answers to each sample, eliminating ambiguity.
Significance and value The ability to identify J.Crew blocker provisions important for both lenders and borrowers in leveraged finance. For lenders, it helps ensure key protections are included in loan agreements. For borrowers, it provides insight into restrictions being placed on their activities. Given the widespread adoption of J.Crew blockers, this is task that requires proficiency to actively participate in the leveraged loan market. The important measure of L.M''s 's ability understand and apply legal concepts, particularly those related to secured lending and intellectual property law. also tests the LLM's capacity analyze and interpret legal provisions. Given the increasing complexity and sophistication of financial transactions, the ability accurately identify and understand such provisions is valuable skill for any LLM. This task, therefore. provides useful measure of progress for LLMs their understanding and interpretation of complex legal clauses.
task serves
47

---

Clause
J.Crew
Blocker
Provi-
sion?
provided that (i) immediately before and after such designation, no Event of Default shall have occurred and be continuing, (ii) in the case of the designation of any Subsidiary as an Unrestricted Subsidiary, such designation shall constitute an Investment in such Unrestricted Subsidiary (calculated as an amount equal to the sum of (x) the fair market value of the Equity Interests of the designated Subsidiary and any of its Subsidiarics that are owned by Holdings or any Restricted Subsidiary, immediately prior to such designation (such fair market value to be calculated without regard to any Obligations of such designated Subsidiary or any of its Subsidiaries under the Guaranty Agreement) and (y) the aggregate principal amount of any Indebtedness owed by such Subsidiary and any of its Subsidiaries to Holdings or any the Restricted Subsidiaries immediately prior to such designation, all calculated, except as set forth in the parenthetical to clause (x) above, on a consolidated basis in accordance with U.S. GAAP), and such Investment shall be permitted under Section 10, 10,05, (ii) no Subsidiary may be designated an Unrestricted Subsidiary if any of its Subsidiaries is Restricted Subsidiary for the purpose any Refinancing Notes Indenture, any Permitted Pari Passu Notes Document, any Permitted Pari Passu Loan Documents, any Permitted Junior Notes Document or other debt instrument, with a principal amount in excess of the Threshold Amount, (iv) following the designation of an Unrestricted Subsidiary as Restricted Subsidiary. Holdings shall comply with the provisions of Section 9.12 with respect such designated Restricted Subsidiary, (v) no Restricted Subsidiary may be a Subsidiary of an Unrestricted Subsidiary (and any Subsidiary of an Unrestricted Subsidiary that is acquired or formed after the date of designation shall automatically be designated as an Unrestricted Subsidiary) and (vi) in the case of the designation of any Subsidiary as an Unrestricted Subsidiary, each of (x) the Subsidiary to be designated and (y) its Subsidiaries has not, at the time of designation, and does not thercafter, create, incur, issue, assume, guarantec or otherwise become directly or indirectly liable with respect to any Indebtedness pursuant to which the lender has recourse to any of the assets of Holdings or any Restricted Subsidiary (other than Equity Interests in an Unrestricted Subsidiary).
No
provided, that (i) immediately before and after such designation, Event of Default exists (including after giving effect the reclassffication Investments in, Indebtedness of and Liens on the assets of, the applicable Restricted Subsidiary or Unrestricted Subsidiary), (ii) as of the date of the designation thereof, no Unrestricted Subsidiary shall own any Capital Stock in any Restricted Subsidiary of the Borrower or hold any Indebtedness of or any Lien any property of the Borrower its an Unrestricted Subsidiary
Yes
Restricted Subsidiaries and (iii) no subsidiary may be designated it owns intellectual property that is material to the business of the Borrower and its Restricted Subsidiaries, taken as a whole (such intellectual property, Material Intellectual Property), at the time of designation, other than in connection with transactions that have a bona fide business purpose long such transactions are undertaken facilitate financing or Restricted Payment or undertaken in connection with a liability management transaction. Notwithstanding anything contained in this Section 6.05 to the contrary, in no event shall (a) the Borrower or any Restricted Subsidiary be permitted to make or own any Investment in the Holdings direct or indirect equityholders constituting Material Intellectual Property (other than pursuant a bona fide transition service similar arrangement in the same manner as other customers, suppliers commercial partners of the relevant transferee generally) or (b) any Restricted Subsidiary transfer ownership of, or license on an exclusive basis, any Material Intellectual Property to any Unrestricted Subsidiary, other than in connection with transactions that have bona fide business purpose and so long as such transactions not undertaken to facilitate financing Restricted Payment undertaken in connection with a liability management transaction. Notwithstanding anything contained in this Section 6.06 to the contrary, in no event shall (a) the Borrower or any Restricted Subsidiary make any Disposition of Material Intellectual Property to Holdings direct or indirect equityholders (other than pursuant to bona fide transition service or similar arrangement or in the same manner other customers, suppliers commercial partners of the relevant transferee generally) or (b) any Restricted Subsidiary make any Disposition, constituting either a transfer ownership or an exclusive license, of any Material Intellectual Property to any Unrestricted Subsidiary, other than in connection with transactions that have a bona fide business purpose and so long as such transactions are not undertaken facilitate financing Restricted Payment or undertaken in connection with a liability management transaction.
permitted to
Table 17: Examples from jarew_ blocker.
48

---

G.4.3 Unfair Terms of Service
LEGALBENCH, the Unfair Terms Service task is denoted as unfair_too tos.
Background An array of recent work has found that consumers rarely read terms of service agreements [95 62.. As result, consumers regularly sign agreements or contracts containing provisions that (I) they lack awareness of, and/or (2) would consider "unfair" or "predatory." Reasons for this phenomenon include the sheer amount of time would take read every agreements, and the lack of actual recourse on an individual basis.
of service agreement, the obtuse language of these
With reference to European consumer law, 80) identify eight categories of clauses terms-af-service agreements which could be considered "potentially unfair":
Arbistration: clauses which mandated that all disputes between the parties would be resolved through arbitration.
Unilateral change: clauses which allow the provider to modify terms service and/or the service itself.
Content removal: clauses which give the provider right modify/delete user's content
Jurisdiction: clauses which specify a jurisdiction in which claims must be brought, regardless where the user lives.
Choice of law: clauses which specify the country's law which governs disputes arising under the contract, regardless where the user lives.
Limitation of liability: clauses which limit the liability of the service provider.
Unilateral termination: clauses which empower the service provider terminate/suspend the service at their discretion.
Contract by using: clauses which stipulate that consumer is bound by the terms of service simply by using the service.
A more detailed description of these categories be found in 80
Task The Unfair Terms of Service task requires LLM determine given clause from terms service agrecement-whethee belongs to one of the above eight categories,
so, which one.
use the version of data available in (21). which takes subset from Unlike (21 which frames the task as distinguishing "fair" from "unfair" clauses we cast the task as 8-way multiclassification task across the original categories identified in [80].
Construction process
Significance and value Unlike the CUAD and J.Crew Blocker task, the Unfair TOS task evaluates a LLM's ability perform multiclass clause classification across highly imbalanced dataset.
49

---

Clause
Clause type
you also acknowledge that variety of evernote actions may impair prevent you from accessing your content or using the service at certain times and/or in the same way. for limited periods permanently and agree that evernote has no responsibility or liability as a result of any such actions results, including, without limitation, for the deletion of, or failure to make available to you, any content.
Arbitration
if you do not terminate your agreement before the date the revised terms become effective, your continued access to or use of the airbnb platform will constitute acceptance of the revised terms.
Choice of law
may at any time and from time to time, in our sole discretion, change the fees and charges, or add new fees and charges, in relation any of the products.
Content removal
you and academia.edi agree that any dispute, claim or controversy arising out of or relating to these terms or the breach, termination, enforcement interpretation validity thereof or the use of the site or services (collectively arbitration except that each party retains the right: (i) (i) to bring an individual action in small claims court and ( to seek injunctive or other equitable relief in a court of competent jurisdiction to prevent the actual threatened infringement, misappropriation of a party 's copyrights, trademarks trade secrets patents or other intellectual property rights the action described in the foregoing clause (i) ( an ip protection action" ")
Contract by using
disputes ) will be settled by binding
violation
if we find that any shared content in your account violates our terms of Jurisdiction
service including by violating another person intellectual property or privacy rights) ), we reserve the right to un-share or take down such content.
you live in the european union : you agree that the laws ireland, excluding conflict of laws rules, shall exclusively govern any dispute relating to this contract and/or the services.
Limitation liability
oculus does not endorse or guarantee the opinions, views, advice or Other recommendations posted or sent by users.
you object to the changes nintendo reserves the right to terminate this agreement any portion of it upon reasonable notice and you will have to register again if you wish to continue using the nintendo account service under the terms and conditions
Unilateral change
unless you and we agree otherwise arbitrate above is found not to apply to you or to particular claim or dispute, either as result of your decision to opt out of the agreement to arbitrate or result of decision by the arbitrator or court order, you agree that any claim or dispute that has arisen or may arise between you and ebay must be resolved exclusively by state or federal court located salt lake county, utah
the event that the agreement
Unilateral termination
Table 18: Examples from unfair_tor
G.4.4 Contract QA
In LEGALBENCH, the Contract QA task is denoted as contract_qa.
Background Each of the above tasks evaluates the capacity for LLMs to learn to recognize single type of clause, given description of that clause and/or examples of it. The Contract QA task generalizes this across
50

---

Class
Number of samples
Other
3454
Contract by using
15
Choice of law
38
Content removal
53
Unilateral change Arbitration
70
98
Limitation of liability
28
Unilateral termination
32
Jurisdiction
25
Table 19: Test set class distribution.
multiple clause types, evaluating an LLM's ability to recognize legal provisions that are not described in the prompt.
Task Each sample in the dataset consists of (1) contract clause, and (2) question asking if the clause is an example of provision type (e.g., "Is this severability clause?"). Across the datasct, the questions correspond 22 different legal provisions. Questions and provisions are paired such that for each provision type, the LLM presented with two clauses that are an example of the type, and two clauses which are not.
Construction Process The data was manually extracted from a set of sample agreements contributed by
LegalTech vendor and from public sources. represents a variety of contracts, such as:
Vendor Partner Data Protection Agreements (DPA)
Master Services Agreements (MSA)
Licensing Terms
BIPA consents
Clause
Question
Answer
This Agreement shall be gov- erned by and construed ac- cordance with the laws of the State of New York, without giving effect to any choice of law or conflict of law provi- sions.
Does the clause discuss BIPA consent?
No
a dispute arises between the parties under this Agree- ment that cannot be resolved through good faith negotia- tions within reasonable pe- riod of time, such dispute shall be escalated to an executive of- ficer of each party for resolu- tion. such executive officers unable to resolve such dis- pute within a reasonable pe- riod of time after escalation, either party may pursue any available legal remedies.
Does the clause discuss how Yes disputes may be escalated?
Table 20: Examples for contract_ga.
51

---

G.5 Consumer Contracts QA
In LEGALBENCH, the Consumer Contracts QA task is denoted consumer_contratta_gg..
Background Consumer contracts govem many economic and social activities, ranging from retail purchases and online search social media and entertainment. These contracts affect consumers' access services, control terms of payment,
determine remedies available when consumers' rights are violated. Despite the importance of these legal agreements, consumers typically lack the time, expertise, and incentive to properly examine how consumer contracts impact their rights and interests. This issue is known as the "no- reading" problem & F M.LMs may offer solution. By reading consumer contracts and explaining their legal ramifications, LLMS could enable consumers better understand and exereise their legal rights in many everyday contexts.
Task
The Consumer Contracts QA task, first introduced 71, aims to examine the degree to which an LLM can understand certain consumer contracts. Specifically, the task is comprised of 200 yes/no legal questions relating to the terms of service of popular websites. Examples of questions are provided in the table below.
In addition to the original 200 questions, the task includes alternatively worded version of all 200 questions. While cach question's content substantially the same across both versions of the question, the alternatively worded questions are, by design, less readable, that more difficult for human read. Comparing performance across the original questions and the alternatively worded questions can help assess an LLM's brittleness in performing the task at hand. An example is provided in the table below:
Construction process The task was introduced in [71] To construct the dataset, an attomey drafted 200 yes/no questions relating to the terms of service of the 20 most-visited U.S. wehsites (10 questions per document), well an alternatively worded version of all 200 questions. The questions relate wide range of legal issues arising the terms of service, including eligibility access services, payment for services, limitations of liability, intellectual property rights, and dispute resolution procedures. Answers to all questions can be obtained from the applicable terms of service.
Significance and value Given the ubiquity of consumer contracts, LLMs capable of reading these documents and communicating their contents to consumers might offer significant benefits. These benefits, however, are contingent on model's accuracy and reliability. LLMs that misinterpret the provisions of consumer contracts may hinder consumers' ability to understand exercise their contractual rights. The Consumer Contracts QA task is a preliminary attempt at evaluating the ability of LLMs to read certain consumer contracts.
52

---

Content Removal and Disabling or Terminating Your Account We can remove any content information you share on the Service if we helieve that it violates these Terms of Use, our policies (including our Instagram Community Guidelines), so by law. We can refuse to provide or stop providing all or part the Service to you (including terminating or disabling your your access to the Facebook Products and Facebook Company Products) immediately to protect our community or services, or if you create risk or legal exposure for us, violate these Terms of Use policies (including our Instagram Community Guidelincs), if you repeatedly infringe other people's s intellectual property rights, or where we are permitted required to do so by law. We can also terminate or change the Service, remove or block content or information shared on our Service, or stop providing all or part of the Service if we determine that doing so is reasonably necessary to avoid or mitigate adverse legal or regulatory impacts us. If you believe your account has been terminated in error, you want to disable permanently delete your account, consult our Help Center. When you request delete content or your account, the deletion process will automatically begin no more than 30 days after your request. may take up to 90 days to delete content after the deletion process begins. While the deletion process for such content is being undertaken, the content is no longer visible to other users, but remains subject to these Terms of Use and our Data Policy. After the content is deleted, it may take us up to another 90 days remove it from backups and disaster recovery systems. Content will not deleted within 90 days of the account deletion or content deletion process beginning in the following situations:
Contract:
we are permitted required to do
where your content has been used by others in accordance with this license and they have not deleted it (in which case this license will continue apply until that content is deleted); where delction within 90 days is not possible due technical limitations of our systems, in which case, we will complete the delction as soon as technically fcasible; or where delction would restrict our ability to: investigate or identify illegal activity or violations of our terms and policies (for example, to identify or investigate misuse of our products systems); protect the safety and security of our products, systems, and users; comply with a legal obligation, such as the preservation of evidence; or comply with a request of judicial or administrative authority, law enforcement or government agency; in which case, the content will be retained for no longer than is necessary for the purposes for which it has been retained (the exact duration will vary on case-by case-by-case by-case basis).
If you delete or we disable your account, these Terms shall terminate an agreement between you and us, but this section and the section below called "Our Agreement and What Happens if We Disagree" will still apply even after your account is terminated, disabled, or deleted.
Question: According to the terms, 30 days after Ive asked to delete content, can other users see that content?
Answer: No
Contract: 16. Termination You may terminate these Terms at any time and for any reason by deleting your Account and discontinuing use of all Services. If you stop using the Services without deactivating your Account, your Account may be deactivated due to prolonged inactivity. We may suspend or terminate your Account, moderator status, or ability to access or use the Services at any time for any or no reason, including for violating these Terms or our Content Policy.
The following sections will survive any termination of these Terms or of your Account: 4 (Your Content). 6 (Things You Cannot Do), 10 (Indemnity).. 11 (Disclaimerss) 12 (Limitation of Liability), 13 (Goveriigg Law and Venue), 16 (Termination), and 17 (Miscellancous).
17. Miscellancous These Terms constitute the entire agreement between you and us regarding your access to and use of the Services. Our failure to exercise or enforce any right or provision of these Terms will not operate as waiver of such right or provision. If any provision of these Terms is, for any reason, held to be illegal, invalid, unenforceable, the rest of the Terms will remain in effect. You may not assign or transfer any of your rights or obligations under these Terms without our consent. We may freely assign any of our rights and obligations under these Terms.
Question: Will certain terms remain in force notwithstanding users termination of the service?
Answer: Yes
Table 21: Task examples.
Original wording
Alternative wording
Am allowed be paid for writing a Wikipedia article, assuming disclose who's 's paying me?
Are Wikipedia contributors permitted receive payment in respect of their contributions, pro- vided they disclose the identity of the person or institution providing such payment?
Table 22: Example of reworded question.
53

---

G.6 Contract NLI Tasks
In LEGALBENCH, the Contract NLI tasks are denoted as contract_nli_*
Task The Contract NLI tasks require LLM-given an excerpt of a contract and an assertion about the legal effect of that excerpt-1o determine whether the assertion is supported or unsupported by the excerpt.
Construction process These tasks constructed by transforming data released by 72. The original dataset consists 607 contracts and 17 assertions (e.g., "Receiving Party shall not disclose the fact that Agreement was agreed or negotiated "). Each contract is labeled for each assertion as supporting, negating, or not mentioning the assertion. Please refer to the original paper for details on annotation.
We restructure this dataset for a short-context L.M setting. Specifically, we treat each assertion as separate task, where the objective is to determine whether a contract excerpt is supportive (or not) of the assertion. For each instance where contract is supportive of an assertion, 722 has annotated the excerpt of the contract that is supportive. When creating task, we use the supportive excerpts for the assertion from the test set as positive instances. To generate negative instances, we combine excerpts where the assertion is contradicted with random sample of excerpts associated with other assertions. We treat both groups of excerpts instances which are "unsupportive" of the assertion. We transform the assertion into Ycs/No question, where the LLM is asked to determine if clause satisfies the assertion.
Table lists each task, the assertion associated with the task, and an example of excerpt which supports the assertion.
Table 23: ContractNLJ Tasks
Task
Task name: contract_ @ictuu_ooooooe confideniialilffommtiin
Question: Identify if the clause provides that the Receiving Party shall destroy return some Confidential Information upon the termination of Agreement.
Example: Upon receipt by the Recipient of a written demand from the Disclosers: 8.1.1 the Recipient must return or procure the return to the Disclosers or, as the Disclosers may require, destroy or procure the destruction of any and all materials containing the Confidential Information together with all copies; 8.1.2 if the Disclosers requires, the Recipient must provide the Disclosers with certificate or such other evidence as Disclosers may reasonably require duly signed or executed by an officer of the Recipient confirming that the Recipient has complied with all of its obligations under this Agrecment including about return, destruction and deletion of Confidential Information and media; 8. 1.3 the Recipient must delete or procure the deletion of all electronic copies of Confidential Information; and 8. 1.4 the Recipient must make, and procure that the Authorised Persons shall make, no further Use of the Confidential Information.
Task name: contract nli no licensing Question: Identify if the clause provides that the Agreement shall not grant Receiving Party any right to Confidential Information.
Example: No license to the receiving party under any trade secrets or patents or otherwise with respect to any of the Proprietary Information is granted or implied by conveying proprietary Information or other information such party, and none of the information transmitted exchanged shall constitute any representation, warranty, assurance, guaranty or inducement with respect to the infringement patents or other rights of others.
Task name: contract_nie pli confidentialityy of_agreement Question: Identify if the clause provides that the Receiving Party shall not disclose the fact that Agreement agreed or negotiated.
Example: In addition, except as permitted herein, Recipient shall disclose the fact that the parties exchanging Confidential Information and having discussions. connection therewith, is agreed that no public release or disclosure of any contemplated transaction shall be made except by a mutually agreed disclosure except that each party may make such disclosure if advised by its outside securitics counsel in writing that such disclosure is required; PROVIDED, HOWEVER, that in such event such party will notify the other party that it intends, as a preliminary matter, to take such action and the outside securities counsel of such party shall first discuss the mater with the outside securities counsel of the other party before any definitive decision is made on the disclosure.
Task name: contract nli explicit_iieeeeee identification Question: Identify if the clause provides that all Confidential Information shall be expressly identified by the Disclosing Party.
54

---

Table 23 - continued from previous page
Task
Example: 1. As used herein, the term "Proprietary Information" refers to any and all Information of a secret nature which is applicable to or related In any way to (i) the business, present future, of the Disclosing Party, (ii) the research and development or investigations of the Disclosing Party or (iii) the business of any customer the Disclosing Party; provided, in each case, that such information is delivered to the Receiving Party by the Disclosing Party and (a) is marked or identified in writing as "Confidential", (b) if verbal or visual disclosure. is identified as "Confidential" in writing within ten (10) business days of such disclosure,
confidential, proprictary,
Task name: contract_nie pli aal_ssuvival_oo of latl__of_obogggee oolligations
Question: Identify if the clause provides that ome obligations of Agreement may survive termination of Agreement.
Example: b. This Agreement shall be valid when signed by duly authorised representatives of the Parties and shall be binding on each Party for 10 (ten) years as from the date of signature of the last signatory, even if at the end of the negotiations data sharing agreement is not signed between the Partics, until such time as the Information enters into the public domain.
Task name: contract nli permissible development of similar information Question: Identify if the clause provides that the Receiving Party may independently develop information similar to Confidential Information.
Example: "Confidential Information" of a disclosing party ("Discloser") means the following, regardless of its form and including copies made by the receiving party ("Recipient"), whether the Recipient becomes aware before or after the date of this Agreement: except where that information is: Independently developed by the Recipient without use, directly or indirectly of Confidential Information received from the Discloser.
Task name: contract nli permissible post-agreement possession Question: Identify if the clause provides that the Receiving Party may retain some Confidential Information even after the return or destruction of Confidential Information.
Example: 9. Upon the Disclosing Party's 's written request, the Receiving Party shall (at the Receiving Party's election) promptly return or destroy (provided that any such destruction shall be certified by duly authorized Representative of the Receiving Party) all Confidential Information of the Disclosing Party and all copies, reproductions, summaries, analyses or extracts thereof or based thereon (whether in hard-copy form or an intangible media, such as electronic mail or computer files) in the Receiving Party's possession in the possession any Representative of the Receiving Party; provided, however: (i) that a legal proceeding has been instituted to seek disclosure of the Confidential Information, such material shall not be destroyed until the proceeding is settled or final judgment with respect thereto has been rendered; (ii) that the Receiving Party shall not, in connection with the foregoing obligations, be required to identify or delete Confidential Information held electronically in archive back-up systems in accordance with general systems archiving or hackup policies; and (iii) that the Receiving Party shall not be obligated to return or destroy Confidential Information of the Disclosing Party to the extent the Receiving Party is required to retain copy pursuant applicable law, and further provided that the Receiving Party will not, and the Receiving Party will use reasonable measures to cause its employees to, access such Confidential Information so archived backed-up.
Task name: contract_ lii__nniijine inclusion_of -errballll__ verballyycoovveyddd conveycd_information information
Question: Identify if the clause provides that Confidential Information may include verbally conveyed information.
Example: I acknowledge that The Business Partnership has provided, and/or has agreed to provide in the future, to information of a confidential or proprietary nature (the Confidential Information) Contidential Information shall mean any information or data relating to any clients of The Business Partnership business or affairs disclosed whether in writing, orally or by any other means.
Task name: contract nli _iharrng_ww... siit_wttttthhtettt phatrd-prrteess
Question: Identify if the clause provides that the Receiving Party may share some Confidential Information with some
(including consultants. agents and professional advisors). Example: Receiving Party shall carefully restrict access Sensitive Information to employees, contractors and third parties as is reasonably required and shall require those persons to sign nondisclosure restrictions least as protective as those in this Agreement.
Task name: contract nli permissible copy Question: Identify if the clause provides that the Receiving Party may create a copy of some Confidential Information in some circumstances.
55

---

Table 23 - continued from previous page
Task
Example: If any party makes copies of the Confidential Information of the other party, such copies shall also constitute Confidential Information and any and all confidential markings such documents shall be maintained.
Task name: contract_ aaii_notice__o on_ comopeeleed disclosure
Question: Identify if the clause provides that the Receiving Party shall notify Disclosing Party in case Receiving Party is required by law, regulation or judicial process to disclose any Confidential Information. Example: If the Receiving Party or its Representatives are requested or required in any judicial, arhitral administrative proceeding or by any governmental or regulatory authority to disclose any Evaluation Material (whether by deposition, interrogatory, request for documents, subpoena, civil investigative demand, or otherwise). or the Receiving Party is so requested or required to disclose any the facts disclosure of which is prohibited under paragraph (3)e) of this Agreement, the Receiving Party shall give the Furnishing Party prompt notice of such request so that the Furnishing Party may seek appropriate protective order other appropriate remedy and/or waive compliance with the provisions of this Agreement, and, upon the Furnishing Party's 's request and at the Furnishing Party's expense, shall reasonably cooperate with the Furnishing Party in seeking such an order. (d) Notice If either Party proposes to make any disclosure in reliance on clause (i) above, the disclosing Party shall, to the extent practicable, provide the other Party with the text of the proposed disclosure as far in advance of its disclosure is practicable and shall in good faith consult with and consider the suggestions of the other Party concerning the nature and scope of information proposes to disclose. Notwithstanding the foregoing, Party may make such public announcement or public statement if in the opinion of such Party's 's outside counsel or General Counsel, such public announcement or public statement is necessary to avoid committing violation of law or of any rule regulation of any securities association, stock exchange national securities quotation system on which such Party's 's securities are listed trade. In such event, the disclosing Party shall use its reasonable best efforts to give advance notice to the other Party and to consult with the other Party on the timing and content of any such public announcement or public statement.
Task name: contract nli permissible acquirement of similar information Question: Identify if the clause provides that the Receiving Party may acquire information similar to Confidential Information from a third party.
Example: For the purposes of this Agreement, the term "Confidential Information" " shall mean all trade secrets and confidential or proprictary information (and any tangible representation thereof) owned, possessed or used in connection with The Company Business or by the Buyer Parties and its Affiliates; provided, however, that "Confidential Information" does not include information which is or becomes generally available to the public other than as result of disclosure by a Seller Party..
Task name: contract_ miiishhringgwv tharing_with____ employees Question: Identify if the clause provides that the Receiving Party may share some Confidential Information with some of Receiving Party's 's employees.
Example: We and our representatives will keep the Evaluation Materials completely confidential; provided, however. that (i) any of such information may be disclosed to those of our directors, officers, employees, agents, representatives (including attomeys, accountants and financial advisors). lenders and other sources of financing (collectively, "our representatives" ") who we reasonably determine need to know such information for the purpose of evaluating a Possible Transaction between us and the Company (it being understood that our representatives shall be informed by us of the confidential nature of such information and shall be directed by us, and shall each agrec treat such information confidentially) and
Task name: contract nli _ij_iimmedd_____ use Question: Identify if the clause provides that the Receiving Party shall not any Confidential Information for any purpose other than the purposes stated in Agreement.
Example: 2. A Receiving Party agrees: 2.1.2. to use the Confidential Information of the other solely in, and to the extent necessary for the Purpose and not to copy or use any Confidential Information of the other save to the extent necessary for the Purpose;
Significance and value The Contract NLI tasks evaluate an LLM's capacity reason over the rights and obligations created by contract. The ability to perform this skill is essential to many types of legal work.
56

---

G.7 Corporate Lobbying
In LEGALBENCH, the Corporate Lobbying task is denoted corporate_1obbying
Background A significant amount of effort is devoted to identifying developing sources of law which implicate client issue interests. Examples of such sources include: legislative bills, proposed regulations, or in progress litigation. Identifying these sources serves multiple purposes. From scholarly standpoint. researchers often aggregate sources into issue-focused databases, enabling them to identify emerging trends or patterns across different sources [36]. From an advocacy standpoint, identifying sources allows affected groups better understand how their rights obligations may be affected, and how to focus efforts interacting with
courts, legislatures, and other govemmental bodies 15 81 43
Task The Corporate Lobhying task requires L.MM determine whether proposed Congressional bill may be relevant a company based on company's self-description in SEC 10K filing. The following information about a bill and a company are available:
The title of the bill.
A summary of the bill.
The of the company.
A doscription of the company.
We expect higher accuracy of LLM predictions if we were to provide the model with more data about bill, especially if we provide with more data about a company. Proprietary applications of this approach could leverage significant internal company data. More expensive deployments could leverage the full text of the bill
Construction process
This data was manually labeled. This work was an extension of the rescarch described
in 90
Significance and value Determining whether particular bill is relevant for company requires (1) identifying the legal consequences the bill, and (2) whether those consequences relevant to company's business model, structure, or activities. As discuss above, this type prognostication is common legal practice. For instance, law firms regularly publish "client alerts" which seek to keep clients updated on new legal developments [120].
Class
Number of samples
No Yes
345
145
Table 24: Test set class distribution
57

---

Field
Text
Bill Title
A bill provide standards relating to airline travel by Federal employees for official business.
Bill
Sum-
Fly Smart Act
mary
This bill establishes standards for airline travel by federal employces for official business, including general requirement to use coach-class accommodations and a ban on military aircraft for domestic official travel. It allows use of first-class and business class for federal employees under certain cireumstances, such accommodate disability or special need or because of exceptional security circumstances
Company Name
Alaska Air Group, Inc.
Company Descrip- tion
Virgin America has been member of Air Group since it was acquired in 2016. In 2018, Virgin America and Alaska combined operating certificates to become a single airline, and legally merged into a single entity. The Company also includes McGee Air Services, an aviation services provider that was established as a wholly-owned subsidiary of Alaska in 2016. Together with our regional partner airlines, we fly to 115 destinations with over 1,200 daily departures through an expansive network across the United States, Mexico, Canada, and Costa Rica. With global airline partners, provide our guests with a network more than 900 destinations worldwide. Our adjusted net income was $554 million, which excludes merger-related related costs, special items and mark-to to to-market fuel hedge adjustments. Refer to "Results of Operations" in Management's 's Discussion and Analysis for our reconciliation of Non-GAAP measures to the most directly comparable GAAP measure. Mainline- includes scheduled air transportation on Alaska's Boeing Airbus jet aireraft for passengers and cargo throughout the U.S., and in parts of Canada, Mcxico, and Costa Rica. other third-party carriers' scheduled air transportation for passengers across shorter distance network within the U.S. under capacity purchase agreements (CPA). Horizon includes the capacity sold to Alaska under CPA. Expenses include those typically borne by regional airlines such crew costs, ownership costs and maintenance costs. We believe our success depends on our ability to provide safe transportation, develop relationships with guests by providing exceptional customer service and low fares, and maintain a low cost structure to compete effectively. In 2018 we focused much of our energy on the integration of Virgin America, completing over 95% of our integration milestones. In January 2018, Alaska and Virgin America received a Single Operating Certificate (SOC) from the Federal Aviation Administration (FAA), which recognizes Alaska and Virgin America as one airline. In April 2018, we transitioned to a single Passenger Service System (PSS), which allows us to provide one reservation system, one website and one inventory of flights to our guests. This transition to single PSS enables unlock many of the revenue synergies expected from the acquisition. and to provide consistent branding
guests all airport gates, ticketing, and check-in areas. The two most important milestones we have yet to complete include combining the maintenance operations of Boeing and Airbus, and reconfiguring our Airbus fieet. In 2018, we painted 33 Airbus aireraft with the Alaska livery and we are in process of reconfiguring all Airbus aircraft to achieve a cabin experience for our guests that is consistent with Boeing fleet. In early 2019, we will also complete the integration of our crew management systems and aim reach a collective bargaining agreement with our aircraft technicians, the last remaining labor group that has not yet reached a joint collective bargaining agreement. With the integration largely behind us, we remain committed to our vision to become the favorite airline for people on the West Coast. The acquisition of Virgin America positioned us the fifth largest airline in the U.S., with unparalleled ability serve West Coast travelers. evolving needs by offering a relevant network and schedule, upgrading our onhoard offerings, and retaining our unique West Coast vibe. Some of the more notable product enhancements underway include adding high-speed satellite connectivity to our entire Boeing and Airbus fleets, updating and expanding our airport lounges, and working with the Port of Seattle to open state-of-the- art gate North Satellite Concourse at Sea-Tac Airport, including a 15,000 square-foot flagship lounge. We have also introduced new food and heverage menus, which include more fresh, local, and healthy offerings including salads, protein plates, and fresh snacks, as well as new heverage offerings, including craft beers, juices and an updated wine selection. We are also active in the communities serve and strive to be industry leader in environmental and community stewardship.
Table 25: An example of a relevant bill for corporate_lobbying
58

---

G.8 Definition Tasks
In LEGALBENCH, the Definition Tasks are denoted as definitiin,classiificaa and definition_ extraction.
Background Judicial opinions regularly involve definition, assigning particular meaning words phrases (Let us define words and phrases as "terms"). Definition of terms can occur when judges introduce or discuss legal concepts (e.g. parol evidence), and it frequently occurs when judges interpret terms in legal texts, This can include language from past judicial opinions and language appearing in legal texts like contracts, statutes, and the Constitution. Historically, interpreters have often evaluated the definition(s) of individual words. For example, in interpreting the meaning of "keep and bear arms" in the Second Amendment, courts consider the definition(s) of individual words (like "bear"), This approach-focusing on terms' definitions- has only increased in recent decades with the rise of textualist approaches to constitutional and statutory interpretation.
Judicial opinions define a wide range of terms, including ordinary terms, legal terms, and scientific terms. They also appeal to wide range of defining sources, including ordinary dictionaries. legal dictionaries, and legal texts, For an example of the last, consider statutory definitions: U.S.C. I offers generally applicable definitions of many frequent statutory terms.
is useful for lawyers to identify when definition occurs (definition classification), as well which terms have been defined (definition extraction). These tasks might seem simple at first. There are some intuitively plausible indicators of definition classification and extraction. For example, defined terms often (but not always) appear in quotation marks or near citation to dictionary.
However, these tasks are not entirely straightforward. Indicators like quotation will not lead to perfect definition classification and extraction. Consider for example, this sentence from the dataset related the definition of "confidential": The term "confidentaa" meant then, as does now, "private" " or "secret." " Webster's Seventh New Collegiate Dictionary 174 (1963) ) [ As another example from the dataset, consider this definition of "brought": But a natural reading of $ 27's text does not extend so far. "Brought" in this context means "commenced." Black's 's Law Dictionary 254 (3d ed. 1933] Other examples exclusively quote the definition, rather than defined terms: Stare decisis ("to stand by things decided") is the legal term for fidelity to precedent. Black's Law Dictionary 1696 (1Ith ed. 2019) In all of these examples, the presence a dictionary would not indicate which term is extracted. In other examples, there is no dictionary cited: there is not a perfect correlation between dictionary citation and classification of sentence as defining one [1]
Tasks
The Definition Classification task requires an LLM to determine- given excerpt from Supreme Court opethon- the excerpt is defining any term (Yes/No). The Definition Extraction task requires an LLM determine given an excerpt from a Supreme Court phinion-whide term the excerpt is defining (Open-endee response).
An original code- dataset was constructed to study how the Supreme Court relies on dictionaries over time. Any case citing a dictionary was included in the dataset, and human coders identified relevant excerpts that defined terms and which terms were defined.
Construction process
That dataset has been repurposed for the task here. For the definition extraction task, the original dataset includes the relevant information (excerpts, with the defined term coded separately).
For the definition classification task, the original dataset includes examples of language defining terms. To create a set of non-defining language, Neel Guha randomly selected similarly long excerpts of text from the same Supreme Court opinions. Kevin Tobia analyzed those randomly selected excerpts, identifying any that include definitions (for removal). The resulting dataset has 691 sentences which define sentences, and 646 sentences which do not.
This is not particularly difficult task for human lawyers, and it is unlikely that LLMs would replace lawyers as experts in this process. However, it is possible that LLMs successful in these tasks could provide beneficial legal research roles (c.g. quickly identifying all prior definitions of specific term in a particular jurisdiction).
Significance and value
Moreover, the definition extraction task serves as useful test of LLMs abilities, given the task's open-ended nature. The task is not limited to a small set of possible answers (e.g. Yes, No). Rather, it requires identifying which term of all terms in excerpt is defined. Most of these choices will admit of over ten possible answers (i.e. excerpts of over ten words). Moreover, there great variety in the language used across the examples. There are hundreds of possible answers, across all items.
1 Food Mktg. Inst. v. Argus Leader Media, 139 Ct. 2356, 2363 (2019).
12 Merrll Lynch. Pierce. Fenner & Smith Inc. v. Manning. 136 S. Ct. 1562, 1568 (2016). June Medical Services L.L.C. v. Russo, 140 Ct. 2103, 2134 (2020).
"E.g. "And "remuneration" means "a quid pro quo," "recompense" or "reward" for such services. 1d., at 1528." BNSF Ry. Co. Loos, 139 S. Ct. 893, 905 (2019).
59

---

Sentence
Definition sentence?
The risk of that consequence ought to tell us that something is very wrong with the Court's analysis.
No
This term has long referred to class of ex- penses commonly recovered in litigation to which attorney's fees did not traditionally be- long. See Black's Law Dictionary 461 (1891) (defining "expensae litis" to mean "generally allowed" costs); Bouvier, Law Dictionary 392 (1839) (defining the term to mean the "costs which are generally allowed to the successful party"); :); id., at 244 (excluding from the defini- tion "costs" " the "extraordinary fees [a party] may have paid counsel").
Yes
Table 26: Examples for derinttion_caasificttiioo
Sentence
Defined term
The term "plaintiff" is among the most com- monly understood of legal terms of art: It means a "party who brings civil suit in a court of law." Black's Law Dictionary 1267 (9th ed. 2009) see also Webster's 's Third New International Dictio-
plaintiff
nary 1729 (1961)"
The ordinary understanding of law enforcement includes not just the investigation and prosecu- tion of offenses that have already been commit- ted, but also proactive steps designed to prevent criminal activity and to maintain security.
law enforcement
Table 27: Examples for definition, ext.ratiion.............
60

---

G.9 Diversity Jurisdiction
In LEGALBENCH, the Diversity Jurisdiction tasks are denoted diversity_*.
Background Diversity jurisdiction is one of two ways in which federal court may have jurisdiction over lawsuit pertaining state law. Diversity jurisdiction exists when there (1) complete diversity between plaintiff's and defendants, and (2) the
(AiC) greater than $75,000.
"Complete diversity" requires that there no pair plaintiff and defendant that are citizens of same state. However, it acceptable for multiple plaintiff's to be from the same state, or for multiple defendants to be from the same state.
The AiC requirement allows for certain forms of aggregation. Specifically, if plaintiff A asserts two independent claims against defendant B, the value of the claims may be added together when considering if the AiC: requirement is met. However, plaintiff may aggregate the value of claims against separate defendants, and two plaintiff's may not aggregate claims against same defendant.
Tasks We define different tasks, each of which tests the diversity jurisdiction rule under different pattern of facts. The diversity jurisdiction tasks are:
diversity_.. The fact patterns consists of one plaintiff, one defendant, and one claim per plaintiff- defendant pair.
diversity.2 22: The fact patterns consists of one plaintiff, two defendants, and one claim per plaintiff- defendant pair.
diversity.5 The fact patterns consists of one plaintiff, one defendant, and two claims per plaintiff- defendant pair.
diversity_44 The fact patterns consists two plaintiff's, one defendant, one claim per plaintifT- defendant pair.
diversity_5 5: The fact patterss consists of two plaintiffs, defendant, and two claims per plaintiff- defendant pair.
diversity_6; The fact patterns consists of two plaintiffs, two defendants, and two claims per plaintiff-defendant pair.
Construction process We programmatically construct dataset to test the diversity jurisdiction. We generate
randomness over the names of the parties, the claims, and the amounts.
Significance and value It is extremely unlikely LLMs would ever be used to evaluate diversity jurisdiction practical settings. However, because the task is considered extremely simplistic- and that first year law students are expected perform perfectly it offers useful evaluation benehmark for LLMs. The structure of the task is potentially non con-trivial for LLMs, as it requires identifying the relationships between parties (i.c., who are plaintiffss and defendants), understanding which claims may be aggregated, and computing whether the aggregated amounts meet the AiC requirement.
61

---

Task
Facts
Diversity Jurisdic-
tion?
diversity_
Oliver is from Oregon, William from Oregon. Oliver sues William for defamation for $3,000.
diversity_
James is from South Dakota. Sophia is from Virginia. James sues negligence for $9,010,000.
Sophia for Yes
1
diversity_
Benjamin from South Carolina. Amelia from Indiana. Mia from South Carolina. Benjamin sues Amelia and Mia each for wrongful eviction for $22,000.
No
2
diversity_
James is from Colorado. Elijah is from West Virginia. Theodore from Washington.
Yes
2
James sues Elijah and Theodore each for negligence for $2,864,000.
diversity_
Ava is from Rhode Island. Theodore from Rhode Island. Ava Theodore for No
3
securities fraud for $70,000 and trespass for $6,000.
diversity_
Charlotte is from Colorado. Harper is from Oklahoma. Charlotte sues Harper for Yes breach of contract for $74,000 and securities fraud for $88,000.
3
diversity_
Harper is from New Jersey. Benjamin is from Colorado. Isabella is from Colorado. Harper and Benjamin both Isabella for breach of contract for $6,165,000
No
4
diversity
Noah is from Indiana. Sophia is from West Virginia. Benjamin from Montana.
Yes
4
Noah and Sophia both sue Benjamin for defamation for $3,996,000.
diversity_
Noah is from Idaho. Elijah is from Connecticut. Theodore is from Wyoming, Noah and Elijah both sue Theodore for medical malpractice for $57,000 and legal malpractice for $16,000.
No
5
diversity_
Charlotte from Oregon. Mia is from Virginia. Elijah is from Tennessee. Charlotte Yes and Mia both sue Elijah for trademark infringement for $57,000 and medical mal- practice for $20,000.
5
diversity
Lucas is from South Dakota. Amelia is from New Hampshire. Benjamin is from No South Dakota. Benjamin from South Dakota. Lucas and Amelia both sue Benjamin for negligence for $16,000 and wrongful eviction for $76,000, Lucas and Amelia both suc Olivia for medical malpractice for $3,000 and breach of contract for $76,000.
diversity_
Emma is from Kansas. Noah is from Delaware. Elijah is from South Dakota. Elijah is from New Jersey. Emma and Noah both sue Elijah trademark infringement for $4,000 and trespass for $85,000, Emma and Noah both sue Liam for negligence for $10,000 and defamation for $67,000.
Yes
Table 28: Examples for the Diversity Tasks.
62

---

G.10 Function of Decision Section
In LEGALBENCH, the Function of Decision Section task is denoted as function_of_dectioo sss_ddcisson_secction
Background In common-law legal systems, written judicial decisions serve two functions. First, they resolve the dispute that litigants brought before the court and explain the reason for the court's decision. Second, they become new law, binding on future parties and future courts should another case arise that presents sufficiently similar facts.
Because judicial decisions not only describe the law, but are themselves the law, lawyers in common- legal systems must be able to read and digest case law to extract key legal principles and apply those principles to their own cases. This skill takes time and practice to develop.
Importantly, not every word in a judicial decision is binding, only the facts and reasoning that were required for the court reach its decision. Thus, lawyers must distinguish important from trivial facts across numerous past decisions before they can conclude what the law on a particular issue is. One the most foundational case-reading skills is the ability to review a legal decision and identify the function that each section of the decision serves. In the American legal education system, this skill is taught beginning in the first year of law school, often by encouraging students to identify the function of each section of decision. A typical classification scheme follows:
Facts: A section of the decision that recounts the historical events and interactions between the parties that gave rise to the dispute.
Procedural History: A section of the decision that describes the parties' prior legal filings and prior court decisions that led up to the issue to be resolved by the decision.
Issue: section of the decision that describes legal or factual issue be considered by the court.
Rule: A section of the decision that states legal rule relevant to resolution of the case.
decision that evaluates an issue before the court by applying governing legal principles to the facts of the case
Analysis: A section
Conclusion: A section of the decision that articulates the court's conclusion regarding question presented to it.
Decree: A section of the decision that announces and effectuates the court's resolution of the parties' dispute, for example, granting or denying party's 's motion or affirming, vacating, reversing, or remanding lower court's decision.
Identifying the function of sections within judicial decisions is a fundamental skill for lawyers in common-law legal systems. Without it, precedent- based legal reasoning would he impossible.
Task The Function of Decision Sections task requires L.MM to determine-given givenn one-paragraph excerpt of legal which of the seven functions above that paragraph serves in the context of the entire decision.
Construction process We created dataset of paragraphs from legal decisions, classified into of the seven functions above. Paragraphs were taken from decisions West Publishing's 's fourth Federal Reporter series, which publishes the decisions of the United States Courts of Appeals. To avoid selection bias and achieve a degree of randomness, paragraphs were selected from sequential decisions, in the order they appeared, spanning all areas of civil and criminal law that fall within the jurisdiction of the federal courts.
Significance and value Beginning law students may initially have trouble identifying the function of particular section within a judicial opinion, but it quickly becomes simple task. LLMs would not be called on to perform this task the actual practice of law, but because it foundational legal reasoning skill, it provides a useful measure of reasoning progress for LLMs.
Class
Number of samples
Facts
49
Procedural History
58
Issue
51
Rule
56
Analysis
56
Conclusion
50
Decree
47
Table 29: Test set class distribution.
63

---

Excerpt
Function
The Commission's 's notice and orders, however, are to the contrary. From the very outset, the Commission has made clear that the Governance Order was no more than a call for proposal that would then be subject to further notice, comment, and revision.
Analysis
Donna and Hurley contend that the Supreme Court's decision in Honeycutt v. United States, U.S. -, 137 S. Ct. 1626, 198 L.Ed.2d 73 (2017), should be applied retroactively to invalidate the forfeiture judgments against them.
Conclusion
For the reasons stated, affirm the district court's ' judgment.
Decree
"The Game of Life" is a classic family board game, introduced in 1960 by the Milton Bradley Company to great success. This case involves long-running dispute between Rueben Klamer, a toy developer who came up with the ini- tial concept of the game, and Bill Markham, a game designer whom Klamer approached to design and create the actual game prototype. Eventually, their dispute (which now involves various assignees, heirs, and successors-in innsssor--i--nteeest) reduced one primary issue: whether the game qualified as a "work for hire" under the Copyright Act of 1909. If it did, Markham's 's successors-in-interss would not possess the termination rights that would allow them to reassert control over the copyright in the game. After considering the evidence produced at a bench trial, the district court concluded that the game was, indeed, such who all trace their interest in the game to Markham,
Facts
work. Maainifffappeellann challenge that determination. We affirm.
Officers of the Puerto Rico Police Department watched Julio Casiano-Santanl ("Casiano")) engage in drug deal. They arrested him, recovering loaded pistol and three bags of crack cocaine from the scene. Casiano was charged with possession of a firearm in furtherance of drug trafficking crime, 18 U.S.C. 924(c)(1)(A)() (i), two counts of possession with intent to distribute controlled substances, 21 U.S.C. $ (41(a)(1) and (b)(1)(C), and possession of firearm by convicted felon, 18 U.S.C. $ 922(g)(1).
Issue
On remand, the district court held new sentencing hearing, in which Lawrence allocuted. Resentencing Transcript at 11-12, United States v. Lawrence, No. 03-cr-00922CCKK (D.D.C. Oct. 5, 2009), ECF No. 103. Lawrence told the court that, while incarcerated, he had "been trying to do the right things as far as & * * becoming a man can provide for my son, he's 11 and very big." Id. Lawrence's s mother was "getting old" and does "the best that she can but his son had "health issues as far as * weight gain and lot of other things." Id. at 12. Lawrence explained that he "just want[ed] a chance to be father" to his son, and that he "was just hoping that it's possible that * * * can get out in his life before # the streets Id. He said he wanted to "be a productive citizen[.]]" and noted that he "read the Bible" and "attended church, school, [and] college." 1d. He admitted that he had "gotten into some altercations," but "not because wanted to, but it's prison, and you know, there's 's all types of people in prison." ld. While "making no excuses" for his actions, he said he "was just hoping the Court would have leniency" his "particular case." ld.
Procedural History
or anything that maybe have done affect him[.]"
The border between interpretation and bare consultation can be hazy and, there- fore, "difficult to plot." Lawless, 894 F.3d at 18 (citing Livadas, 512 U.S. at 124 n. 18,8 114 S.Ct. 2068). This case, however, does not closely approach the border: on their face, Rose's 's state-law claims require more than bare consultation of the CBA. They substantially depend on construing the terms of the agreement (the CBA) that RTN and the Union negotiated. We explain briefly.
Rule
Table 30: Examples for function of dee.isionnscctioon
64

---

G.11 Hearsay
In LEGAL, the hearsay task is denoted as hearsay.
Background The Federal Rules of Evidence dictate that "hearsay" evidence is inadmissible at trial. Hearsay defined as "out-of-court statement introduced prove the truth of the matter asserted." In determining whether a piece evidence meets the definition of hearsay, lawyers ask three questions:
1. Was there statement? The definition of statement is broad, and includes oral assertions, written assertions, and non-verbal conduct intended to communicate (i.e. assert) a message. Thus, for the purposes the hearsay rule, letters, verbal statements, and pointing all count as statements.
2. Was it made outside of court? Statements not made during the trial or hearing in question count as being out-of-court.
3. Is it being introduced to prove the truth of the matter asserted? A statement is introduced to prove the truth of the matter asserted if its truthfulness is essential to the purpose of its introduction. Suppose that trial, the parties were litigating whether Alex was soccer fan. Evidence that Alex told his brother like soccer," would objectionable on hearsay grounds. as (1) the statement itself asserts that Alex likes soccer, and (2) the purpose of introducing this statement is to prove/disprove that Alex likes soccer. In short, the truthfulness of the statement's ' assertion is central to the issue being litigated. However, consider if one of the parties wished to introduce evidence that Alex told his brother. "Real Madrid is the greatest soccer team in the world." This statement would not be hearsay. It's assertion- that Real Madrid is the greatest soccer team in the world is unrelated to the issue being litigated. Here, one party is introducing the statement not to prove what the statement says, but to instead show that particular party (i.c. Alex) was the speaker of the statement.
Task Given a legal issue and a piece of prospective evidence, the LLM must determine whether the evidence constitutes hearsay under the above test.
We note that in practice, many pieces of evidence which are hearsay are nonetheless still admissible under one of the many hearsay exception rules. We ignore these exceptions for our purposes, and leave the construction of benchmarks corresponding these exceptions for future work.
We create the hearsay dataset by hand, drawing inspiration from similar exercises available in legal casehooks and online resources. The dataset consists of slices, where each slice tests different aspect of the hearsay rule. We randomly select sample from each slice to be in the train set. The remainder of the slice constitutes the test set (for a total of 95 samples). The slices (with test set counts) are:
Construction process
Statement made court = 14): Fact patterns where the statement in question made during the course testimony at trial. Thus, the statement is not hearsay.
Non-assertive conduct (n 19); Fact patterns where the evidence docs not correspond a statement. Hence, the hcarsay rule is inapplicable.
Standard hearsay (n = 29): Fact patterns where there is an oral statement, it is said out of court, and it is introduced to prove the truth of the matter asserted. Thus, these fact patterns correspond to hearsay.
Non-verbal hearsay (n 12): Fact patterns where the statement is hearsay, but made writing or through assertive conduct (e.g. pointing).
Not introduced prove truth (n 20): Fact patterns where an out-of-court statement is introduced to prove something other than what asserts.
Significance and value The hearsay rule is commonly taught in law school as part of Evidence. Law students are expected to understand the rule, and how to apply it. The hearsay task is interesting for LLM evaluation because emphasizes multi-step reasoning the test for hearsay encompasses several different steps, where each step differs in difficulty. These include:
Event detection: The LLM must determine whether the fact pattern mentions statement being made.
Spatial reasoning: The LLM must determine whether the statement was made inside a court room.
Argument extraction: The LLM must determine what the statement is asserting.
Argument relevance: The L.M must finely determine whether the assertion is relevant to the issue being litigated.
65

---

Facts
Hearsay?
On the issue of whether Will knew that the com- pany intended to announce its drug trials had been cancelled, the fact that he told the jury that "he didn't i know the first thing about how medicines worked."
No
On the issue of whether Gerald was alive im- mediately after being attacked by Kathryn, Ger- ald's statement, "I was attacked by Kathryn."
No
On the issue of whether Susan was familiar with No Shakespeare, the fact that she had once played the role of Macbeth and recieved standing ovation after her monologue.
To prove that the insured under a life policy is dead, his wife offers a death certificate.
Yes
the issue of whether Albert bought knife, Angela testified that he shook his head when she asked him.
Yes
On the issue whether the brakes were faulty, Amy testifies that she heard Arthur claim that he thought something was wrong with the car.
Yes
Table 31: Examples for hearsay
66

---

G.12 Insurance Policy Interpretation
In LEGALBENCH, the Insurance Policy interpretation.
Interpretation
task is denoted as insurance_policyy
Background Insurance disputes often arise when parties disagree on whether claim covered under certain insurance policy. To study such disagreements in interpretation, researchers at Stanford recruited crowdsource workers to review a pair of an insurance policy and a claim and respond whether they believe the claim is covered. A policy-claim pair whose applicability the workers disagree with each other on suggests ambiguity in the policy text.
Task
The Insurance Interpretation task requires I.I.M to review pair of an insurance policy and claim and determine whether the policy clearly covers the claim, clearly does cover it, if is unclear whether it covers or not.
Construction process The clause-claim pairs are manually constructed before being reviewed by crowd- source workers [137]. To convert the numbers of Covered/Not Covered/Can't Decide responses to discrete labels. we first calculate the 95% multinomial confidence interval of the proportion of each respouse. We then choose the label for which the confidence interval lower bound is greater than equal to .5. If no label has lower bound 2 .5, we classify the policy-claim pair "It's ambiguous." " This conversion process ensures that individual crowdsource workers do not arbitrarily sway the labels. Examples for cach label can be found in Table 32
Policy: Harper's insurance covers damage from "House Removal," which includes "damage to belongings that occurs while being stored by professional removal contractors."
Claim: Harper is moving to a new home on the other side of town. Because her old home has already sold and her new home is not yet ready for her to move in, she checks into a hotel and asks professional moving company to store some of her belongings at the company warehouse, A couple days before she is set to move in, the warehouse floods, which ruins the items that the movers were storing for Harper. Harper files a claim with her insurance company for the damage to her belongings.
Label: Covered
Policy:
Denise's insurance covers damage from "House Removal," defined as "damage to belong- ings caused while being removed by professional removal contractors from the home."
Claim: Denise is moving to a new home on the other side of town. She asks her uncle, a retired professional mover, to help move her belongings
of her current home. During the move, her unele's truck is involved in a minor accident that damages several pieces of her furniture and other belongings. Denise files a claim with her insurance company for the damage to her belongings. Label: Not Covered
Policy: Jason has insurance coverage against loss and damage from "Identity Theft," which excludes "identity theft connected with the policyholder's 's business."
Claim: Jason is successful car salesman. One day, while Jason is at home, hackers manage to infiltrate Jason's 's home WiFi network. The hackers steal Jason's 's social security number and open a number of fraudulent lines of credit in his name. To resolve the fraud, Jason must spend thousands of dollars in legal fees. Jason files a claim with his insurance company for his losses. Label: It's ambiguous.
Table 32: Examples for insurance, insurance_interprettan interpretation.
Significance and value The ability determine whether insurance claim is covered under given policy can significantly reduce claim processing time. It also shine light potential ambiguity existing policies. Additionally, this task represents one the rare benchmarks where an LLM is required to predict laypcople's logal interpretations, as we retrieve the ground truth labels based on crowdsourced responses.
67

---

G.13 International Citizenship Questions
In LEGALBENCH, the International Citizenship Questions task is denoted as anternationallcctiizann
questions.
The GLOBALCIT Citizenship Law Dataset is valuable resource that comprehensively categorizes citizenship acquisition and methods in 190 countries. It enables cross-country comparisons and offers insights into global trends in citizenship laws and examines 28 different ways in which citizenship can be acquired, as well as 15 ways laws allow citizenship to be lost. The original dataset is formulated as a tabular survey dataset. We change this survey format into Yes/No questions about specifie countries and their laws as of 2020 resulting in 9300 question- answer pairs.
Background
Task
The model must answer yes/no questions about global citizenship law.
Question
Answer
Consider the country of Central African Repub- lic. Does the country provide for acquisition of citizenship by person who is the public service (including military service) and, if so, under which conditions?
No
Consider the country of Bolivia. Does the coun- try provide for involuntary loss of citizenship by a person who is adopted by or in legal guardian- ship of citizen of another country and, if so. under which conditions?
No
Consider the country of Denmark. Which resi- dence conditions does the country provide for residence-basee based acquisition?
Yes
Consider the country of Germany. Does the country require the demonstration of civic knowledge cultural integration for residence- based acquisition?
Yes
Table 33; Examples for international iiderationn_eiizzzeehpppuuseosor
Contruction process download the GLOBALCIT Citizenship Law Dataset 135 and craft script that
converts the tabular survey data into yes/no questions, appending information about the country and the time at which the survey was created.
Significance and value Understanding knowledge about the law globally is important to evaluate. To successfully answer legal reasoning questions globally language models must be able retrieve rule and then reason about it.
68

---

G.14 Learned Hand Tasks
In LEGALBENCH, the Learned Hand tasks are denoted as learned saarndd_hand_..
Background A person may experience problems in many areas of their lives - their family, work, finances, housing, education, driving, and more- which legal professionals would recognize as being 'legal issues'. The person may not know that problem with credit card company, a landlord, a spouse, an employer is 'legal issue', or what terminology or categorization a lawyer would use make sense
The designation of legal issue means that a person may benefit from getting specialized guidance from legal professional to resolve this problem because they can guide them on their rights, liabilities, possible options, procedures, and specialized legal tasks. Not all people may want to pursue legal services to resolve a legal issue. But legal issue-spotting can help them both make sense of the problem they are experiencing, and what services and laws might be available if they wish to make use of them.
Legal professionals typically carry out issue-spotting during an intake process. They receive a person's verbal or written description of the situation they are in. Then the professional identifies the main legal issues that are apparent in this situation, starting at the main top-level categories and then sometimes proceeding to identify more specific sub-categories of legal issues. For example, the professional may identify that a person's 's situation involves legal issue with the top-level category of 'housing' and specific sub-categories of "possible eviction for non non-paynn payment of rent' and 'poor living conditions of their rental".
The professional may identify multiple overlapping legal issue categories one situation. For example, the professional may identify that a person has a housing law issue and a family law issue their landlord is threatening to evict them because of the police being called to the rental home because of a domestic violence incident.
The main categories of legal issues that professionals would identify in people's 's situations are:
Benefits: A situation would have benefits issue if it involves the person attempting to resolve a problem with applying for, receiving, discontinuing public benefits and social services from the govermment. This could include benefits that support them regarding food, disability, old age, housing, health, unemplayment, child care, or other social needs.
Business: A situation would have business issue if the person is running small business or nonprofit, and encounters problem around incorporation, licenses, taxes, regulations, bankruptcies, natural disasters. This category not meant apply to larger corporate legal issues, but rather the kinds of business problems that an individual might bring to a legal professional for help.
Consumer: A situation would have a consumer legal issue if the person was dealing with problems around debt and money, insurance, consumer goods and contracts, taxes, or small claims about the quality of service
Courts: situation would be categorized as courts issue the person is dealing with a problem around how to interact with the court system or with lawyers more broadly. This may involve the person attempting to follow legal procedures, court rules, or filing requirements, it may involve them attempting to hire, manage, address lawyers.
Crime: situation would have crimes issue if the person is dealing with the criminal justice system as a defendant, victim, or family member. They may be experiencing problems around being investigated, searched, or charged with a crime, or going to criminal trial and prison, or being a victim of a crime.
Divorce: situation would be categorized as a divorce issue if person is dealing with a separation, divorce, or annulment while splitting with a spouse or partner. The problem may involve separation, spousal support, splitting money and property, child support, visitation, or following the related court process.
Domestic Violence: A situation would have domestic violence issue if the person is dealing with abuse with partner, family member, or other intimate acquaintance. The situation may involve understanding rights and laws related to domestic violence, getting protective orders, enforcing them, reporting abuse, and dealing with collateral consequences to housing, finances, employment, immigration, and education.
Education: situation has an education issue the person is dealing with a problem around school for themselves or a family member. The situation may involve accommodations for special needs, discrimination. student debt, discipline, or other issues in education.
Employment: A situation would be identified having an employment issue if the person has a problem with job, including during the application process, during the job, or after ending employment. Problems may include discrimination, harassment, payment, unionizing, pensions, termination, drug testing, background checks, worker's compensation, classification as a contractor, or more.
69

---

Estates: A situation would have an estates issue if a person is dealing with an estate, wills, or guardianship. This may include issues around end-of-life planning, health and financial directives. trusts, guardianships, conservatorstips, and other estate issues that people and family deal with.
Family: A situation would have a family law issue if person is dealing with an issue involving a family member. This may include issues around divorce, child custody, domestic violence, adoption, paternity, name change, and other family issues.
Health: A situation would be categorized as health law issuc if the person is dealing with problems around accessing health services or protecting their rights in medical settings. This may involve problems with accessing health care, paying for care, getting public benefits for care, privacy of medical records, problems with quality of care, other issues.
Housing: situation would have housing law issue a person is dealing with problems around the housing where they live, or that they own. These include problems with renting home, eviction, living conditions, discrimination, foreclosure, post-disaster housing, housing assistance, and more.
Immigration: situation would have an immigration issue if person is not a full citizen the US and is dealing with problems related to their status. This may include understanding visa options, working as an immigrant, political asylum, border searches, deportation, human trafficking, refugees, immigration court hearings, and more.
Torts: A situation would be categorized as having a torts issue the person is dealing with an accident or conflict with another person that involves some perceived harm. These problems may include car accident, conflicts with neighbors, dog bites, bullying, harassment, data privacy breaches, being sued, suing someone elsc.
Traffic: A situation would have a traffic law issue if the person is experiencing a problem with traffic, parking, or car ownership. This might include problems with getting ticketed, getting or reinstating a driver's 's license, car accidents, purchasing a car, repossession, and more.
This set of categorics is commonly used by legal professionals as they triage potential clients. The LIST Taxonomy from Stanford Legal Design Lab has formalized these categories into a machine-readable taxonomy. available at htttp::// httpa://taxooooy/ 1egal/ The LIST taxonomy builds on the taxonomies built by legal aid groups, like the Legal Services Corporation categories list that most legal aid groups use to encode their matters, signifying what issucs they helped clients with [5] LIST also builds off of the legal aid community' 's National Subject Matter Index, which was more extensive list of categories to further assist legal aid groups in tracking the issues they helped people with [6
Performing the Legal Assue-Spottinn task requires parsing through informal wording and structures that person may use to convey the situation they are struggling with. Typically the person is writing this narrative down in an informal, quick manner (like in an online intake form on a legal services website) or speaking aloud (like on an intake hotline, or during an in-person interview). The narratives are not typically structured into concise order. They use informal terminology rather than legal terms.
Task The Legal Issuc-spotting task requires an LLM to consider a person's 's narrative about their situation. The LLM must use this narrative to determine which legal issue category (or categories) apply to the person's 's situation.
Construction process There is crowdsourced dataset, created via the online labeling game Learned Hands, that has established when and how these legal issue categories apply to people's 's narratives. The narratives are drawn from the subreddit r/legaladvice, in which people share several lines or paragraphs about the situation they are dealing with, which they think might involve legal issues. The moderators of the subreddit are active in managing activity, so the posts do not contain personal identifying information off-topic postings.
The Stanford Legal Design Lab and Suffolk LIT Lab built the Learned Hands game so that law students and lawyers could read narratives one by one, and then answer series of yes-no-skip questions about what legal issue seems to be present. Once there are sufficient consistent votes for a certain label to apply, or to not apply, to a narrative, then the label is finalized. A narrative may have more than one label, as mentioned above.
Significance and value The legal issue categorization helps the professional triage the person to the right services, resources, and procedures that can assist them in resolving the legal issue. If an L.M is able to identify legal issues in people's 's informal narratives, this demonstrates an ability to perform a key task in people's justice journeys. Issue-spotting by LLM may be able to help person who just starting explore whether how they should engage with legal services, courts, or exercising their rights.
The issuc-spotting task may be provided online, when people are visiting legal help websites and trying to find what guide, form, or service would best help them with their problem. Or it may be integrated into the intake
www.:wwww.cc................. http://wwwww....oooiooooo/o///// hot.ss.ggorriiammmaattteeeeeeeee taatttte//ggran /grg///egaannnnnneeee../e lsc-reporting prceri/iieettttee.e.eeeeeeeeerrr handhoooo 2017
See the LSC's list at
"eee the NSMI database at htt.a//psmmm.nttttoooroo..
70

---

process that paralegals or justice workers carry out over hotlines or in-person. to speed up the often lengthy intake process.
71

---

G.15 Legal Reasoning Causality
In LEGALB the Legal Reasoning Causality task is denoted as legal_ressonig__cusaliiy
In many legal domains, systematic evidential barriers hinder the substantiation of causal claims through direct evidence. To address these shortcomings, courts have regognized the power of statistical evidence establishing causation in various contexts, such as product liability " medical malpractice discrimination " and more. For instance, when pursuing a labor discrimination claim, the plaintifff must establish that her protected trait was the underlying reason for the alleged discriminatory decision (e.g., firing or not hiring). However, direct evidence of discriminatory intent rarely exists, so is often nearly impossible to refute the possibility that other (legitimate) differences between two employees or candidates were the cause for favoring one over the other. In such cases, litigants can and often do try to substantiate a causal link between the plaintiff's s group lifioiation and the defendant's behavior through statistical analysis. For instance, plaintiff's might send fictitious resumes that differ only by the suspected demographic characteristic 20 akin to field experiment. Likewise, statistical analysis of observational data that controls for the major factors affecting the employment practice can be used to demonstrate whether specific social group suffers from inferior outcomes (relative to some control group) vis-à-vis particular employer, landlord, or lender that engages with a sufficiently large number of employees or customers
Background
Task The "causal reasoning" task requires an LLM to determine whether the court's 's reasoning regarding the finding of whether causal link exists between the plaintiff's 's protected trait and the allegedly discriminatory decision relied on either statistical or direct- probbtive evidence. It requires understanding the types of words that are used to describe statistical evidence in any given context (regression, correlation, variables, control, and more), and the extent to which those words relate to substantiating a finding of causality (as opposed to other legal components).
Construction process We manually created a dataset of fifty-nine excerpts from court decisions in lawsuits alleging labor market discrimination filed in US Federal District Courts. First, fifty-nine court decisions involving claims of labor discrimination were identified using the LexisNexis database. Second, the passages in which the finding of causality appeared were identified and extracted. Third, we coded the passages as either relying on statistical evidence (c.g., regression analysis, findings of correlation, ctc.) or on direct evidence (c.g., witnesses, documents, etc.).
We selected two random samples from each class to use as part of the train split.
Significance and value The potential of LLMs to identify different types of legal reasoning in general, and the finding of causality in particular, has implications both for the legal profession and for the academic study of law and judicial decision making. First, algorithmic tools are gradually being utilized by lawyers to assist them in preparing for litigation. Specifically, given the heterogeneity among judges, a key element of a successful litigation strategy is lawyer's ability to construct their arguments based on the specific inclinations of the judge assigned to the case. Gaining accurate understanding of judges' unique mode of reasoning (including, e.g., the types of evidence they tend rely on), based their prior decisions, is crucial for winning any lawsuit. Second, datahases consisting of court decisions are the most common source for studying the law and judicial decision-maker technical information, such as the names of the parties and the judge(s). the legal area of the case, and the like. The essential part of any judicial opinion- the legal reasoning is typically treated that could classify the various types legal reasoning - c.g., what evidence is used to establish causation can facilitate studying judicial decision- making in ways currently not feasible at large scales.
to
decision-making.
in legal academia. However, these databases are typically limited to rather
black box. An LLM
"See, e.g, Neurontin v. Pfizer, 712 1st Cir. 52 (2013).
1*O'Neal v. St. John Hosp. & Med. Ctr, 487 Mich SC, 485 (2010).
19ee, e.g., International Broth. of Teamsters U.S., 431 U.S. 324 ('Ti]n many cases the only available avenue of proof is the use of racial statistics to uncover clandestine and covert discrimination by the employer union involved")); Bazemore v. Friday, 478 U.S 385 (1986); Marcus Jones v. Lee Way Motor Freigh. 431 10 cir 245 (1970) ("In racial discrimination cases, statistics often demonstrate more than the testimony of many witnesses").
Havvns Realty Corp. v. Coleman, 455 U.S. 363, 374-75, 71 L. Ed. 2d 214, 102 S. Ct. 1114 (1982).
21 See Bazemore v. Friday, 478 U.S 385 (1986) ("although it need not include every conceivable factor. Given the frequency of employment discrimination litigation in the contemporary United")).
72

---

Excerpt
Relies on statistical evidence?
However, a review of the "over base level" numbers of the four comparators and Escalera in core endourology reflects significant differences in the severity of losses between the comparators and Escalera during the January through June 2015 time period. Escalera's 's utilization of different time periods for each comparator within 2015 is not appropriate when examining the team managens' performances given Bard Medical's s Solo/Skylite production products. Using the same time frame for each comparator, the record reflects that between January through June of 2015, Kunzinger was $55, $55,626, 255,26689 89 below base, Santoro was $160,651.77 above base, Peters was $20,070.56 above base, and Martin was $79.932.38 above base. (Outley Dep. Exs. 3. 12, 14. 16.) These numbers demonstrate that the "losses" experienced by the comparators during the same time period as Escalera are not substantially identical. Escalera's loss of base was $68,79 799.06 more than [$*25] the closest comparator he identified. Additionally, comparing the "over base level" numbers of the comparators and Escalera between January through October 2015 reflects that at the time Escalera was terminated he had suffered significantly more loss over base than his iden- tified comparators: Escalera was [*805] $1 77,,92..44 below base, Kunzinger was $101,132.66 60 below base.3 Santoro was $110,078.73 above base, Peters was $31,876.80 80 below base, and Martin was $1,61 below base. Because of these significant differences in losses, no reasonable jury could find that these four comparators and Escalera are similarly situated in all relevant respects.
No
Equally without evidentiary significance is the statistical analysis of the list of 17; indeed, the analysis was not even admissible under HN4 the standard of Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579, 125 L. Ed. 2d 469, 113 Ct. 2786 (1993). governing the admissibility of expert testimony. which requires the district judge to satisfy himself that the expert is being as careful as he would be in his regular professional work outside his paid litigation consulting. E.g., Braun v. Lorillard Inc., 84 F.3d 230, 234-35 35 (7th Cir. 1996); Rosen v. Ciba- Saiby Corp., 78 F.3d 316, 318 (7th Cir. 1996): Daubert V. Merrell Dow Pharmaceuticals, Inc., 43 F.3d 1311, 1316-19 19 (9th Cir. 1995): ef. Mid-State Fertilizer Co. v. [**7] Exchange National Bank, 877 F.2d 1333, 1339 (7th Cir. 1989). Although the expert used standard statistical methods for determining whether there was a significant correlation between age and retention for the 17 persons on the list, see Michael O. Finkelstein & Bruce Levin, Statistics for Lawyers 157 (1990) (Fisher's exact test), the omission of Sebring and Shulman from the sample tested was arbitrary. The expert should at least have indicated the sensitivity of his analysis to these omissions. More important is the expert's 's failure to correct for any potential explanatory variables other than age. Completely ignored was the more than remote possibility that age correlated with a legitimate job-related qualification, such as familiarity with computers. Everyone knows that younger people are average more comfortable with computers than older people are, just as older people are on average more comfortable with manual-shift cars than younger people are. Three weeks of training might go some distance toward closing the computer-litrracy gap, yet it would be more surprising than otherwise if so short a period of training could close the gap completely. The expert could easily [**8] have inquired about the feasibility of ascertaining through discovery the history of the use of computers by each of the employees on the list of 17.
Yes
Table 34: Examples for ooggllrrosonnngcccusslli
73

---

Class
Number of samples
Yes
31
No
24
Table 35: Test set class distribution.
74

---

G.16 MAUD Tasks
In LEGALBENCIL, the MAUD tasks denoted as maud_s.
Background We adapt the Merger Agreement Understanding Dataset (MAUD) for LEGALBENCH. MAUD consists of over 37,000 expert-annotaled examples for set of legal reading comprehension tasks based on the American Bar Association's s 2021 Public Target Deal Point Study. the Study, lawyers review merger agreements and identify key legal clauses ("deal points") within those contracts. The lawyers then specity the nature of the clauses by answering predetermined set of questions that cover wide range of topics. including conditions to closing, the definition of material adverse effect, and remedies breach of contract. MAUDD' multiple-choice format, according [40,, assesses an L.M''s ability interpret the meaning of specialized legal language.
Task The tasks take advantage of MAUD's 's reading comprehension component. They require an L.M LLM--iven a key legal clause and a set of descriptions for the clause to choose the option that best describes the clause.
Construction process These tasks are constructed by transforming the abridged dataset released by [140]. The abridged dataset contains 14,928 examples with deal points extracted from 94 merger agreements covering 92 multiple-choice questions. We narrow down to 57 questions by filtering out the ones with fewer than 50 examples. Each example consists of the text a deal point, the question, options, and the answer key.
We create translations that map questions into numan-readable multiple-choice prompts. For instance, the prompt for the question "Accuracy of Target 'General' R&W: Bringdown Timing" is "When are representations and warranties required to be made according to the bring down provision?" It is then followed by an enumeration of the options for the LLM to choose among.
We focus on MAUD's 's abridged examples because are interested in assessing LLM's legal reading compre- hension capability rather than its ability extract relevant segment given a complete deal point. Additionally, imputs of examples from main dataset, which contains complete deal point texts, are oftentimes longer than what an average open open-source LLM could ingest at once, rendering them unsuitable for benchmarking purposes.
The table below lists the question and options for each MAUD-based LEGALBENCH task along with an example input-answer pair.
Table 36: MAUD Tasks
Task
Task name: maud_type_ type_
consideration
Question: What type of consideration is specified in this agreement?
Options: A: All Cash; B: All Stock: C: Mixed Cash/Stock; D: Mixed Cash/Stock: Election
Example: each Share <omitted> shall be converted into the right to receive the Offer Price in cash, without interest (the "Merger Consideration") minus any withholding of Taxes required by applicable Laws in accordance with Scction 3.6(d) (Page 20)
Answer:
Task name: maud_ accuracy_off of wwwpscyyoootttttttteeeee Question: When are representations and warrantics required be made according to the bring down provision?
http://bbbnddoontitiiee answer
Options: A: At Closing Only; B: At Signing & At Closing
Example: Section 7.2 Conditions to Obligations of Parent and Acquisition Sub to Effect the Merger. The obligations of Parent and Acquisition Sub to effect the Merger are, addition to the conditions forth in Section 7.1, further subject to the satisfaction or (to the extent not prohibited by Law) waiver by Parent at or prior to the Effective Time of the following conditions: (a) each of the representations and warranties of the Company contained in this Agreement, without giving effect to any materiality or "Company Material Adverse Effect" or similar qualifications therein, shall be true and correct as of the Closing Date, except for such failures to be true and correct as would not, individually or in the aggregate, have Company Material Adverse Effect (except to the extent such representations and warranties are expressly made as of specific date, in which case such representations and warranties shall be so true and correct as of such specific date only); (Page 67)
Answer: A
Task name:
haa.leecorrrooooo.....ee..eaaaaa (orrriiatttooottooottttttttnnnn (wutstanding.... _sarees)_________
bringdoon_standard standard_answer
Question: How accurate must the capitalization representations and warranties be according to the bring down provision?
Options: A: Accurate in all material respects; B: Accurate in all respects; C: Accurate in all respects with thlow-threshold carveout; D: Accurate in all respects with de minimis exception
75

---

Table 36 - continued from previous page
Task
Example: Conditions to the Offer
Notwithstanding any other term of the Offer or this Agrecment the contrary. Merger Sub will not be required to accept for payment or, subject to any applicable rules and regulations of the SEC, including Rule 14e-((c) under the Exchange Act (relating to Merger Sub's 's obligation to pay for or return tendered Shares promptly after the termination or withdrawal of the Offer), to pay for any Shares tendered pursuant to the Offer, and may delay the acceptance for payment of or, subject to any applicable rules and regulations of the SEC, the payment for, any tendered Shares, and (subject to the provisions of this Agreement) may terminate the Offer and not accept for payment any tendered Shares, at any scheduled Expiration Date (as it may have been extended pursuant to Section 2.1 this A Agreement) if <omitted> (ii) any of the additional conditions set forth below are not satisfied or waived in writing by Parent at the Expiration Time:
<omitted> (d) Representations and Warranties. Each of the representations and warranties set forth in:
<omitted> (iv) this Agreement (other than those set forth in the foregoing clauses (i), (ii) and (iii) of this
clause (d) of Annex D. without giving effect to any "materiality" or "Material Adverse Effect" qualifiers or
qualifiers of similar import set forth therein, shall be true and correct as of the consummation of the Offer as though made as of the consummation of the Offer (Page 107)
Answer: D
Task name: maud_ accuracy_ _f____f________ fundamental, taapameettaattagettttta Swapgee_rrsb__rre_e www._bbrinddo_sseeee sttadaard Question: How accurate must the fundamental representations and warranties he according to the bring down provision?
Options: A: Accurate at another materiality standard (e.g.. hybrid standardy; B: Accurate in all material
respects; C: Accurate in all respects
Example: (b) Additional Conditions to Obligation of Parent and Merger Sub. <omitted> the representations
and warranties of the Company set forth in Artiele 3 shall be true and correct <omitted> and of the Closing as if made at and as of such time (Page 11)
Answer: A
to tt__ae______eeeeeeeeepeeeee__e__ carveouts Question: Is the "ability to consummate" concept subject to Material Adverse Effect (MAE) carveouts?
Task name: maud_
Options: A: No; B: Yes
Example: "Material Adverse Effect" means, with respect Huntington, TCF or the Surviving Corporation, as the case may be, any effect, change, event, circumstance, condition, occurrence or development that, either individually or in the aggregate, has had or would reasonably be likely to have a material adverse effect on (i) the business, properties, assets, liabilities, results of operations or financial condition of such party and its Subsidiaries. taken as whole (provided. however, that, with respect to this clause (i), Material Adverse Effect shall not be deemed to include the impact of (A) changes, after the date hereof, U.S. generally accepted accounting principles ( ("GAAP") or applicable regulatory accounting requirements, (B) changes, after the date hereof, in laws, rules or regulations (including the Pandemic Mcasures) of general applicability to companies in the industrics in which such party and its Subsidiaries operate, or interpretations thereof by courts or Governmental Entiries, (C) changes, after the date hereof, in global, national or regional political conditions (including the outbreak of war or acts of terrorism) or in economic or market (including equity, credit and debt markets, as well as changes in interest rates) conditions affecting the financial services industry generally and not specifically relating to such party or its Subsidiaries (including any such changes arising out of the Pandemic any Pandemic Mcasures), (D) changes, after the date hereof, resulting from hurricanes, earthquakes, tornados, floods or other natural disasters or from any outbreak of any disease or other public health event (including the Pandemic), (E) public disclosure of the execution of this Agreement, public disclosure or consummation of the transactions contemplated hereby (including any effect on party's 's relationships with its customers or employees) (it being understood that the foregoing shall not apply for purposes of the representations and warranties in Sections 3.3(b), 3.4, 4.3(b) 4.4) actions expressly required by this Agreement or that are taken with the prior written consent of the other party in contemplation of the transactions contemplated hereby, or (F) a decline in the trading price of a party's 's common stock or the failure. in and of itself, to meet earnings projections or internal financial forecasts, but not, in either case. including any underlying causes thereof; except, with respect to subclauses (A), (B), (C) or (D), the extent that the effects of such change are materially disproportionately adverse the business, properties, assets, liabilitics, results of operations or financial condition of such party and its Subsidiaries, taken as whole, as compared to other companies in the industry in which such party and its Subsidiaries operate) or (ii) the ability of such party to timely consummate the transactions contemplated hereby. (Page 18) Answer: A
Task name: maud_ fls_ (mas__ma________ (maae__staddaa_ sstandard
Question: What is the Forward Looking Standard (FLS) with respect to Material Adverse Effect (MAE)?
Options: A: "Could" (reasonably) he expected to; B: "Would"; C: "Would" (reasonably) he expected to; D:
No; E: Other forward-looking standard
76

---

Table 36 - continued from previous page
Task
Example: "Material Adverse Effect" means, with respect to BancorpSouth, Cadence or the Surviving Entity, the case may be, any effect, change, event, circumstance, condition, occurrencc development that, either individually or in the aggregate, has had would reasonably be expected have material adverse effect on (i) the business, properties, assets, liabilities, results operations or financial condition such party and its Subsidiaries taken as a whole (provided, however, that, with respect to this clause (i), Material Adverse Effect shall not be deemed to include the impact of (A) changes, after the date hereof, in U.S. generally accepted accounting principles ("GAAP") applicable regulatory accounting requirements. (B) changes, after the date hereof, in laws, rules or regulations (including the Pandemic Measures) general applicability to companies the industrics which such party and its Subsidiaries operate, or interpretations thereof by courts or Governmental Entities (as defined below), (C) changes, after the date hercof, in global, national or regional political conditions (including the outbreak of war acts of terrorism) or in cconomic or market (including equity, credit and debt markets, as well changes in interest rates) conditions affecting the financial services industry generally and not specifically relating to such party or its Subsidiaries (including any such changes arising out of a Pandemic or any Pandemic Measures). (D) changes, after the date hereof, resulting from hurricanes, carthquakes, tornados, floods or other natural disasters or from any outbreak of any disease or other public health event (including Pandemic), (E) public disclosure of the execution of this Agreement, public disclosure or consummation of the transactions contemplated hereby (including any effect on party's s relationships with its customers or employces) or actions expressly required by this Agreement or that are taken with the prior written consent of the other party in contemplation of the transactions contemplated hereby, or (F) decline in the trading price of a party's 's common stock or the failure, in and of itself, to meet earnings projections or internal financial forecasts (it being understood that the underlying causes of such decline or failure may taken into account in determining whether a Material Adverse Effect has occurred), except to the extent otherwise excepted by this proviso); except, with respect to subclauses (A), (B), (C), or (D) to the extent that the effects of such change are materially disproportiionttly adverse to the business, properties, assets, liabilities, results of operations taken as whole, as compared to other companies in the industry in which such party and its Subsidiaries operate), or (ii) the ability of such party to timely consummate the transactions contemplated hereby. (Page 19)
financial condition of such party and its Subsidiaries,
Answer: C
Task name:
maud_general wateerraleecoomccc.o Catpaoooiiianndddnaaaaaaaaaaaala
to_
disproportionate wtt___iaaatttototata.
Question: Do changes caused by general economic and financial conditions that have disproportionate
impact qualify for Material Adverse Effect (MAF)?
Options: A: No; B: Yes
Example: "Material Adverse Effect" means, with respect to Huntington, TCF or the Surviving Corporation, the case may be, any effect. change. event, circumstance, condition, occurrence development that, either individually or the aggregate, has had or would reasonably likely to have material adverse effect on (i) the business, properties, assets, liabilitics, results of operations or financial condition of such party and its Subsidiaries, taken as whole (provided, however, that, with respect to this clause (i), Material Adverse Effect shall not be deemed to include the impact of (A) changes, after the date hereof, in U.S. generally accepted accounting principles ("GAAP") or applicable regulatory accounting requirements, (B) changes, after the date hereof, laws, rules or regulations (including the Pandemic Measures) of general applicability to companies in the industries in which such party and its Subsidiaries operate, or interpretations thereof by courts or Governmental Entities, (C) changes, after the date hereof, in global, national or regional political conditions (including the outbreak of war acts of terrorism) or in economic or market (including equity, credit and debt markets, as well as changes interest rates) conditions affecting the financial services industry generally and not specifically relating to such party or its Subsidiaries (including any such changes arising out of the Pandemic or any Pandemic Mcasures), (D) changes, after the date hereof, resulting from hurricanes, carthquakes, tornados, floods other natural disasters from any outbreak of any disease or other public health event (including the Pandemic). (E) public disclosure of the execution of this Agreement, public disclosure or consummation of the transactions contemplated hereby (including any effect on party's s relationships with its customers or employees) (it being understood that the foregoing shall not apply for purposes of the representations and warranties in Sections 3.)(b) 3.4, 4.3(b) or 4.4) or actions expressly required by this Agreement that are taken with the prior written consent of the other party in contemplation of the transactions contemplated hereby, or (F) decline in the trading price a party's common stock or the failure, in and of itself, to meet earnings projections or internal financial forecasts, but not, in either case, including any underlying causes thereof; except, with respect to subclauses (A), (B), (C) or (D), to the extent that the effects of such change are materially disproportionately adverse to the business, properties, assets, liabilities, results of operations or financial condition of such party and its Subsidiaries. taken whole, as compared to other companies in the industry in which such party and its Subsidiaries operate) or (ii) the ability of such party to timely consummate the transactions contemplated hereby. (Page 18)
77

---

Table 36 - continued from previous page
Task
Answer: A
to_ cecittttt____ppppp tto.disproportionattet lacpoonntteenittteeeeeeee impact_modifier modifier Question: Do changes in law that have disproportionate impact qualify for Material Adverse Effect (MAB)?
Task name: maud_ change_in latlllllvaabeeee
Options: A: No; B: Yes
Example: "Material Adverse Effect" means, with respect to SVB Financial, Boston Private or the Surviving Corporation, as the case may be, any effect, change, event, circumstance, condition, occurrence or devel- opment that, either individually or in the aggregate, has had or would reasonably he expected to have a material adverse effect (i) the business, properties, assets, liabilities, results of operations financial condition of such party and its Subsidiaries taken as whole (provided, however, that, with respect this clause (i), Material Adverse Effect shall not deemed include the impact (A) changes, after the date hereof, in U.S. generally accepted accounting principles ("GAAP") or applicable regulatory accounting requirements, (B) changes, after the date hereof, in laws, rules regulations of general applicability to companics in the industries in which such party and its Subsidiaries operate, interpretations thereof by courts Govermmental Entities, (C) changes, after the date hereof, global, national or regional political conditions (including the outbreak of war or acts of terrorism) or in cconomic or market (including equity, credit and debt markets, as well as changes in interest rates) conditions affecting the financial services industry generally and not specifically relating to such party or its Subsidiaries, (D) changes, after the date hereof, resulting from hurricanes, earthquakes, tornados, floods or other natural disasters from any outhreak of any disease or other public health event (including Pandemic Measures), (E) public disclosure or consummation of the transactions contemplated hereby or actions expressly required by this Agreement or that are taken with the prior written consent of the other party contemplation of the transactions contemplated hereby (it heing understood and agreed that this clause (E) shall apply with respect any representation warranty that is intended address the consequences of the execution, announcement or performance of this Agreement or consummation of the Merger) or (F) the failure, in and of itself, to meet earnings projections or financial forecasts, but not including the underlying causes thereof; except, with respect to subclause (A), (B), (C) or (D), to the extent that the effects of such change are disproportionately adverse the business, properties, assets, liabilities, results of operations financial condition such party and Subsidiaries, taken as whole, compared to similar companies the industry in which such party and its Subsidiaries operate); or (ii) the ability of such party to timely consummate the transactions contemplated hereby. (Page 20)
COVID- pandemic and implementation of the
Answer:
Task name:
maud_ cwanges.na........ www.iiinnnnggggaeeeeeeeeeaae or_ other www.::/wmtiiigg.ricciissssssssse subjecttto
disproportionate pttoontttemaattttteeeeeeeeeee impact_modiiieer mmooiiiiiirrrr
Question: Do changes in GAAP or other accounting principles that have disproportionate impact qualify for Material Adverse Effect (MAE)?
Options: A: No; B: Yes
78

---

Table 36 - continued from previous page
Task
Example: "Company Material Adverse Effect" shall mean any state of facts, circumstance, condition, event, change, development, occurrecce, result, effect, action or omission (each, an "Effect") that, individually in the aggregate with any one or more other Effects, (i) results in material adverse effect the business, financial condition or results of operations of the Company and its Subsidiaries, taken as a whole or (ii) prevents, materially impairs, materially impedes or materially delays the consummation of the Merger and the other transactions contemplated hereby on or before the End Date; provided, however, that with respect clause (i) only, Effect to the extent resulting arising from any of the following, shall, to such extent, be deemed to constitute, or be taken into account in determining the occurrence of, a Company Material Adverse Effeet: (A) general cconomic, political, business, financial or market conditions; (B) any outbreak, continuation or escalation of any military conflict, declared or undeclared war, armed hostilitics, or acts of foreign or domestic terrorism; (C) any pandemic (including the SARS-CoV-2 virus and COVID-19 discase), epidemic, plague, or other outbreak of illness public health event, hurricane, flood, tomado, earthquake or other natural disaster or act of God; (D) any failure by the Company or any of its Subsidiaries to meet any internal or external projections or forecasts or any decline in the price or trading volume of Company Common Stock (but excluding, in each case, the underlying causes of such failure or decline, as applicable, which may themselves constitute or be taken into account in determining whether there has been, or would be, a Company Material Adverse Effect); (E) public announcement or pendency of the Merger and the other transactions contemplated hereby; (F) changes in applicable Legal Requirements; (G) changes GAAP or any other applicable accounting standards; or (H) any action expressly required to he taken by the Company pursuant to the terms of the Agreement or at the express written direction consent of Parent; provided, further, that any Effect relating
arising out of resulting from any change or event referred in clause (A), (B). (C). (F) or (G) above may constitute, and be taken into account in determining the occurrence of, a Company Material Adverse Effect to the extent that such change or event has a disproportionate impact (but solely to the extent of such disproportionate impact) on the Company and its Subsidiaries as compared to other participants that operate in the industry in which the Company and its Subsidiaries operate. (Pages 87-88)
Answer:
Task
wttpaaiiio_oottoppppaepeeeee www.hhaatteeeeeeeeeeeeeeeeeeeeee owent.spoccf.ccrreeeeeeee. pandemic c- seaatted_gvvermmeeetee rrrpottsssrrrssssss____roo saarssessorr_essuuue
name:
Question: Is there specific reference to pandemic-relaan governmental responses or measures in the clause that qualifies pandemies other public health events for Material Adverse Effect (MAE)? Options: A: No; B: Yes
Example: "Company Material Adverse Effect" shall mean any state facts, circumstance, condition, event, change, development, occurrence, result, effect, action or omission (cach, an "Fffect") that, individually or in the aggregate with any one or more other Effects. (i) results in material adverse effect on the business, financial condition results of operations of the Company and its Subsidiaries, taken whole or (ii) prevents, materially impairs, materially impedes or materially delays the consummation of the Merger and the other transactions contemplated hereby on or before the End Date; provided, however, that with respect to clause (i) only, no Effect to the extent resulting or arising from any of the following, shall, to such extent, be deemed to constitute, be taken into account in determining the occurrence of, Company Material Adverse Effect: (A) general economic, political, business, financial market conditions; (B) any outbreak, continuation or escalation of any military conflict, declared or undeclared war, armed hostilities, or acts of foreign or domestic terrorism; (C) any pandemic (including the SARS-CoV-2 virus and COVID-19 disease), epidemic. plague, or other outbreak of illness or public health event, hurricane, flood, tornado, earthquake other natural disaster act of God; (D) any failure by the Company any of its Subsidiaries to meet any internal or external projections or forecasts or any decline in the price or trading volume of Company Common Stock (but excluding, each case, the underlying causes of such failure or decline, as applicable, which may themselves constitute or be taken into account in determining whether there has been, or would be, a Company Material Adverse Effect);; (F) the public announcement pendency of the Merger and the other transactions contemplated hereby; (F) changes in applicable Legal Requirements; (G) changes in GAAP any other applicable accounting standards; or (II) any action expressly required be taken by the Company pursuant to the terms of the Agreement or at the express written direction or consent of
Parent; provided, further, that any Effect relating to or arising out of or resulting from any change or event referred to in clause (A), (B), (C), (F) or (G) above may constitute, and be taken into account in determining the occurrence of, Company Material Adverse Effect to the extent that such change or event has disproportionate impact (but solely to the extent of such disproportionate impact) on the Company and its Subsidiaries as compared to other participants that operate in the industry in which the Company and its Subsidiaries operate. (Pages 87-88) Answer: A
Task name: paauudd_anemic___em orttoorr_tttttuteeeeeeeee www.er hww.bbliiihhhhtheeeeeeeeeeeeee event ttw.eeettttoddispppprrrrrrrrrrre wwwp::/ww.....aa........o.ooooe. impact_modee modifier
79

---

Table 36 - continued from previous page
Task
Question: Do pandemies or other public health events have to have disproportionate impact to qualify for Material Adverse Effect (MAE)?
Options: A: No; B: Yes
Example: "Material Adverse Effect" means, with respect to BancorpSouth, Cadence or the Surviving Entity, as the case may be, any effect, change, event, circumstance, condition, occurrence or development that, either individually or in the aggregate, has had or would reasonably be expected to have a material adverse effect on (i) the business, properties, assets, liabilities, results of operations financial condition of such party and its Subsidiaries taken as whole (provided, however, that, with respect to this clause (i), Material Adverse Effect shall not deemed to include the impact of (A) changes, after the date hercof, in U.S. generally accepted accounting principles ("GAAP") or applicable regulatory accounting requirements, (B) changes, after the date hereof, in laws, rules or regulations (including the Pandemic Mcasures) of general applicability companies in the industries in which such party and its Subsidiaries operate, interpretations thereof by courts Governmental Entities (as defined below). (C) changes, after the date hereof, in global, national or regional political conditions (including the outbreak of war or acts of terrorism) or in economic or market (including equity, credit and debt markets, as well as changes in interest rates) conditions affecting the financial services industry generally and not specifically relating to such party its Subsidiaries (including any such changes arising out of a Pandemic or any Pandemic Measures), (D) changes, after the date hereof, resulting from hurricanes, earthquakes, tornados, floods or other natural disasters or from any outbreak of any disease or other public health event (including a Pandemic), (E) public disclosure of the execution of this Agreement, public disclosure consummation of the transactions contemplated hereby (including any effect actions expressly required by this Agreement or that
party's s
relationships with its customers employees) taken with the prior written consent of the other party in contemplation of the transactions contemplated hereby, or (F) decline in the trading price of a party's 's common stock or the failure, in and of itself, to meet carnings projections or internal financial forecasts (it being understood that the underlying causes of such decline failure may be taken into account in determining whether Material Adverse Effect has occurred), except to the extent otherwise excepted by this proviso); except, with respect to subclauses (A), (B), (C), or (D) to the extent that the effects of such change are materially disproportionately adverse to the business, properties, assets, liabilities, results of operations or financial condition of such party and its Subsidiaries, taken whole, compared to other companies in the industry in which such party and its Subsidiaries operate), (ii) the ability of such party to timely consummate the transactions contemplated hereby, (Page 19)
Answer:
Task name: maud_ relationalllll... laalationa___anuuaee language, (mae) _applie___________p__eeppep_____
Question: What carveouts pertaining to Material Adverse Rffect (MAF) does the relational language apply to?
Options: A: All MAE carveouts; B: No; C: Some MAE carveouts
80

---

Table 36 - continued from previous page
Task
Example: "Material Adverse Effect" means, with respect to Huntington, TCF or the Surviving Corporation, the case may be, any effect, change, event, circumstance, condition, occurrence development that, in the aggregate, has had would reasonably be likely have material adverse effect on (i) the business, properties, assets, liabilities, results of operations or financial condition of such party and its Subsidiaries, taken as whole (pravided, however, that, with respect to this clause (i), Material Adverse Effect shall not be deemed to include the impact of (A) changes, after the date hereof, in U.S. generally accepted accounting principles ("GAAP") applicable regulatory accounting requirements, (B) changes, after the date hereof. laws, rules or regulations (including the Pandemic Measures) of general applicability to companies in the industrics in which such party and its Subsidiarics operate, or interpretations thereof by courts or Govermmental Entities, (C) changes, after the date hereof, in global, national or regional political conditions (including the outbreak of war acts of terrorism) or in cconomic or market (including equity, credit and debt markets, as well changes in interest rates) conditions affecting the financial services industry generally and not specifically relating to such party or its Subsidiaries (including any such changes arising out of the Pandemic or any Pandemic Measures), (D) changes, after the date hereof, resulting from hurricanes, carthquakes, tornados, floods or other natural disasters or from any outbreak of any disease or other public health event (including the Pandemic), (E) public disclosure of the execution of this Agreement, public disclosure or consummation of the transactions contemplated hereby (including any effect on party's s relationships with its customers or employees) (it being understood that the foregoing shall not apply for purposes of the representations and warranties in Sections 3.3(b), 3.4, 4.3(b) or 4.4) or actions expressly required by this Agreement that are taken with the prior written consent of the other party in contemplation of the transactions contemplated hereby, or (F) decline in the trading price a party's common stock or the failure. and of itself, to meet earnings projections or internal financial forecasts,
either individually
not, either case, including any underlying causes thereof; except, with respect to subclauses (A), (B). (C) or (D), to the extent that the effects of such change are materially disproportionately adverse to the business, properties, assets, liabilities, results of operations or financial condition of such party and its Subsidiaries, taken whole, compared to other companies the industry in which such party and its Subsidiaries operate) or (ii) the ability of such party timely consummate the transactions contemplated hereby. (Page 18) Answer:
Task name: maud_ www.kkogeewdeeeeee leaeeddeeeeeiiiiie definition
Question: What counts as Knowledge?
Options: A: Actual knowledge; B: Constructive knowledge
Example: provided, however, that with respect clause (i) only, Effect to the extent resulting or arising from any the following, shall <omitted> deemed to constitute <omitted> Company Material Adverse Effect (Pages 87-88)
Answer: B
Task name: maud_ buyer_ cerverrconsett__ee rew.uttterrrrrreettttmmmm (ordinary_ course)
Question: In case the Buyer's consent for the acquired company's ordinary business operations is required, are there any limitations the Buyer's right to condition, withhold, or delay their consent?
Options: A: Yes. Consent may not be unreasonably withheld, conditioned or delayed.; B: No.
81

---

Table 36 - continued from previous page
Task
Example: Section 5.1 Interim Operations of the Company and Parent.
(a) From the date of this Agreement and until the Effective Time or the earlier termination of this Agreement in accordance with its terms, except as (v) otherwise expressly contemplated by this Agreement, (w) set forth in the applicable subsection of Section 5.1 of the Company Disclosure Letter (it being agreed that disclosure of any item in any subsection of Section 5.1 of the Company Disclosure Letter shall be deemed disclosure with respect to any other subsection of Section 5.1 of the Company Disclosure Letter only to the extent that the relevance of such item such subsection is reasonably apparent its face). (x) required by applicable Law, (Y)() required to comply with COVID-19 Measures or otherwise taken (or not taken) by the Company or any of its Subsidiarics reasonably and in good faith to respond to COVID-19 or COVID-19 Mcasures or (B) taken (or not taken) by the Company or any of its Subsidiaries reasonably and in good faith to respond to any other extraordinary event that was not rcasonably foresccable as of the date of this Agreement and occurring after the date of this Agreement that is outside of the control of the Company or its Affiliates and is outside of the ordinary course of business of the Company and its Subsidiaries and Joint Ventures (and is not related to a Company Takeover Proposal); provided that prior to taking any actions in reliance on this clause (y), which would otherwise be prohibited by any provision of this Agreement, the Company will use commercially reasonable efforts to provide advance notice and consult with Parent (if reasonably practicable) with respect thereto or (z) consented to in writing by Parent (which consent shall not be unrcasonably withheld, conditioned or delayed), the Company shall, and shall cause each of its Subsidiaries to, use its commercially reasonable efforts to conduct its business in all material respects in the ordinary course of business consistent with past practice and in compliance in all material respects with all material applicable Laws, and shall, and shall cause each of its Subsidiaries to, use its commercially reasonable efforts to preserve intact its present business organization. keep available the services its directors, officers and employees and maintain existing relations and goodwill with customers, distributors, lenders, partners (including Joint Venture partners and others with similar relationships), supplicrs and others having material business associations with it or its Subsidiaries; (Pages 40-41) Answer:
Task name: maud_ includes,
pancludds_ccosisstere consistent_...... wwtthppsst. caa__practice
Question: Does the wording of the Efforts Covenant clause include "consistent with past practice""
Options: A: No; B: Yes
Example: 5.2 Operation of the Acquired Corporations' Business. (a) During the Pre- Pre-Closing Period, except (w) required or otherwise contemplated under this Agreement Legal Requirements, (x) with the written consent of Parent (which consent shall not he unreasonably withheld, delayed or conditioned, and provided that no consent shall he required if the Company reasonably helieves after consulting with outside legal counsel that seeking such consent would violate Antitrust Law), (y) for any action required to be reasonably taken, or omitted to be taken, pursuant to any COVID-19 19 Measures or which is otherwise required or reasonably taken, omitted to be taken, in response COVID- 19 any other pandemic, epidemic or disease outbreak, as determined by the Company in reasonable discretion, or (z) as set forth in Section 5.2 of the Company Disclosure Schedule, the Company shall, and shall cause each Acquired Corporation to, use commercially rcasonable efforts to conduct its business and operations in the ordinary course in all material respects (Page 41)
prohibited required by applicable
Answer: A
Task name: maud_ ordinary. aettinarr_couuuee _ew_ureurreeeeeeeeeeeeeeeeeeeeee wwwurroeeeoorrrereeeeeee Question: What is the efforts standard?
Options: A: Commercially reasonable efforts; B: Flat covenant (no efforts standard); C: Reasonable best efforts
Example: "Ordinary Course of Business" means, with respect to any Person, the conduct of such Person's 's business that consistent with the past practices of such Person prior to the date of this Agreement and taken the ordinary course of normal, day-to-day operations of such Person, but excluding any conduct that would reasonably be expected to violate applicable Law in any material respect. <omitted> 7.1. Interim Operations. (a) The Company shall. and shall cause each of its Subsidiaries to, from and after the date of this Agreement until the earlier of the Effective Time and the termination of this Agreement pursuant to Article IX (unless Parent shall otherwise approve writing (such approval not be unrcasonably withheld, conditioned or delayed), and except as otherwise expressly required by this Agreement or as required by a Governmental Entity or applicable Law and any Material Contract in effect prior to the date of this Agreement), conduct its business in the Ordinary Course of Business (Page 66)
Answer:
Task name:
maud_ aed._aappccatiiooo...
covenant)
Question: What negative covenants does the requirement of Buyer consent apply to?
Options: A: Applics only to specified negative covenants; B: Applies to all negative covenants
82

---

Table 36 - continued from previous page
Task
Example: Except (w) with respect to the Specified Exceptions (other than as applied to Section (a),
Section 5.)(b), Section 5.1 (x) 25
permitted by this Agreement, (y) as set forth in Section 5.1 of the Company Disclosure Schedule, or (z) with the Parent's 's consent (which shall not unreasonably withheld, conditioned or delayed), during the Pre-Closing Period the Company shall not, and shall not permit any of its Subsidiaries to, directly or indirectly, do any of the following: (Pages 29-30)
otherwise expressly contemplated
Answer:
Task name: maud_ Fiduccary__eceepiion. exc.ppiion__teoeeeeee board, determinntiion iatpattiiinstttrrdeerraa
Question: Under what circumstances could the Board take actions on different acquisition proposal notwithstanding the no-shop provision?
Options: A: If failure to take actions would lead to "breach" of fiduciary duties; B: If failure to take actions would be "inconsistent" with fiduciary duties: C: If failure to take actions would lead to "reason- ably likely/expected breach" of fiduciary duties; D: If failure to take actions would lead to "reasonably likely/expected
inconsistent" with fiduciary dutics; E: If failure take actions would Icad "reasonably likelylexpected violation" of fiduciary duties; F: If taking such actions is "required to comply" with fiduciary duties; G: If failure to take actions would lead to "violation" of fiduciary duties; H: Under no circumstances could the Board do so.; I: Other circumstances.
Example: Section 5.4 No Company Solicitation. <omitted> (b) Notwithstanding anything in Section 5.4(a) the contrary, until the Company Stockholder Approval is obtained, the Company receives a bona fide written Alternative Acquisition Proposal made after the date hereof that does not result from a material breach of this Section 5.4, and the Company Board determines in good faith (after consultation with outside legal counsel and nationally recegnized financial advisor) that such Alternative Acquisition Proposal is, or could reasonably be expected to lead to, Superior Acquisition Proposal, (i) the Company may negotiate and enter into an Acceptable Confidentiality Agreement with the Person making such Alternative Acquisition Proposal; provided, that the Company shall promptly (and in no event later than twenty-four (24) hours after execution thereof) deliver a copy of such Acceptable Confidentiality Agreement to Parent, (ii) following entry into such Acceptable Confidentiality Agreement by the Company, the Company and its Representatives may provide information (including nonpublic information) subject such executed Acceptable Confidentiality Agreement; provided, that any nonpublic information provided to such Person, including if posted to an electronic data room, shall be provided to Parent prior to or substantially concurrently with the time it is provided to such Person, and (iii) the Company and its Representatives may engage in discussion negotiations for such Alternative Acquisition Proposal with such Person and its Representatives. (Page 59) Answer:
Task name: maud_ ettuuiiirryeeepttiiooooo
Seeardddeeeriinee
teriggrrrrrrrrrrr (no_ (no_shop)
Question:
What type of offer could the Board take actions on notwithstanding the no-shop provision? Options: A: Acquisition Proposal only; B: Superior Offer, or Acquisition Proposal rcasonably likely/cxpected
result in a Superior Offer
Example: SECTION 5.02. Acquisition Proposals. <omitted> (c) Information Exchange; Discussions or Negotiation. Notwithstanding anything to the contrary contained in Section 5.02(a). prior to obtaining the Company Requisite Vote, in the event that the Company, any of its Subsidiaries or its or their Representatives receive from any Person, after the date of this Agreement, Proposal that did not result from breach of this Section 5 5.02, and that the Company Board determines in good faith, after consultation with its financial advisors and outside legal counsel, is, or is reasonably likely to lead to, Superior Proposal, the Company may (i) furnish or provide information to the Person making such Acquisition Proposal and its Representatives pursuant to an Acceptable Confidentiality Agreement; provided, however, that the Company shall as promptly as is reasonably practicable (and in any event within one (1) Business Day) make available Parent and Merger Sub any written material non-public information concerning the Company or its Subsidiaries that is provided to any Person pursuant to this Section (i)),cK(i, to the extent such information was not previously made available to Parent, Merger Sub or their Representatives, and (ii) engage in discussions and negotiations with such Person and its Representatives with respect to such Acquisition Proposal. (Page 35)
unsolicited, bona fide written Acquisition
Answer:
Task name: maud_ cwr_eoorr_roeeeee potr__emmiteed_eem with bwwrdbaarddaaoee fiduciary.y..o www.deeeeeetiiinnncmmm Question: Is Change of Recommendation permitted as long as the board determines that such change required to fulfill its fiduciary obligations?
www.fcciiia...ertemiaaatoooooooo only
Options: A: No; B: Yes
83

---

Table 36 - continued from previous page
Task
Example: SECTION 5.3 No Solicitation by the Company; Company Recommendation. <omitted> (d) <omitted> Notwithstandiigg the foregoing any other provision of this Agrcement to the contrary, prior the time the Company Stockholder Approval obtained (but not thereafter), the Company Board the Company Special 41
Committee may make a Company Adverse Recommendation Change if either (x) in the case of a Company Adverse Recommendation Change made in response to a Company Acquisition Proposal, the Company Board the Company Special Committee has determined in good faith, after consultation with its outside financial advisors and outside legal counsel, Superior Proposal and that failure to take such action would reasonably expected to be inconsistent with the directors' fiduciary dutics under applicable Law or (y) in the case of Company Adverse Recommendation Company Intervening Event, the Company Board or the Company Special
such Company Acquisition Proposal constitutes a Company
Change made in response Committee has determined in good faith, after consultation with its outside financial advisors and outside legal counsel, that, as a result a Company Intervening Event, failure to take such action would reasonably be expected to be inconsistent with its fiduciary duties under applicable Law; (Pages 46-47) Answer: A
Task name: maud_ cor cor_standard, (superior_ _offerr
Question: What standard should the board follow when determining whether to change its recommendation connection with superior offer?
Options: A: "Breach" of fiduciary duties; "Inconsistent" with fiduciary duties; C: "Reasonably likely/expected breach" of fiduciary duties; D: "Reasonably likely/expected to be inconsistent" with fiduciary duties; E: "Reasonably likely/expected violation" of fiduciary duties; F: "Required to comply" with fiduciary duties; G: "Violation" of fiduciary duties; H: More likely than not violate fiduciary duties; I: None; J: Other
specified standard
Example: Scction 5.2. No Solicitation. <omitted>
(c) Notwithstanding anything to the contrary contained in this Agreement, at any time prior to obtaining the Company Stockholder Approval, the Company Board may make Change in Recommendation in response unsolicited bona fide written Acquisition Proposal cause the Company enter into Alternative Acquisition Agreement concerning an Acquisition Proposal, each case only (i) such Acquisition Proposal or Superior Proposal did not result from a breach of Section 5.2(a); (ii)the Company Board (or a committee thereof) determines in good faith (A) after consultation with the Company's outside legal counsel and Independent Financial Advisor, that such Acquisition Proposal constitutes Superior Proposal and (B) after consultation with the Company's outside legal counsel, that in light of such Acquisition Proposal, a failure to make a Change in Recommendation or cause the Company to enter into such A Alternative Acquisition Agreement would be inconsistent with the Company Board's 's fiduciary obligations to the Company's 's stockholders under the DGCL; (Page 27)
Answer: B
Task name: maud_ cww__oerrriteeeeeeeeeeoooe permitted in_ response. to winterveeinngge..m event Question: Is Change of Recommendation permitted in response an intervening event? Options: No; B: Yes
84

---

Table 36 - continued from previous page
Task
Example: 6.1 No Solicitation. <omitted>
Notwithstandding the foregoing or anything to the contrary set forth in this Agreement (including the provisions of this Section 6. ), at any time prior receipt of the Company Stockholder Approval, the Company Board may effect a Company Board Recommendation Change in response to a Superior Proposal or an Intervening Event if: (i) the Company Board shall have determined in good faith (after consultation with outside counsel and outside financial advisor) that the failure to effect Company Board Recommendation Change would be reasonably likely be inconsistent with its fiduciary obligations under applicable law; (ii) so long as the Company and its Subsidiaries are not in material breach their obligations pursuant to this Section 6.1 with respect to an Acquisition Proposal underlying such Company Board Recommendation Change; (iii) the Company has notified the Parent in writing that it intends to effect a Company Board Recommendation Change, describing in reasonable detail the reasons for such Company Board Recommendation Change (a "Recommendation Change Notice") (it being understood that the Recommendation Change Notice shall not constitute a Company Board Recommendation Change or a Trigger Event for purposes of this Agreement); (iv) if requested by the Parent, the Company shall have made its Representatives available to negotiate (to the extent that Parent desires to so negotiate) with the Parent's 's Representatives any proposed modifications to the terms and conditions of this Agreement during the three (3) Business Day period following delivery by the Company to the Parent of such Recommendation Change Notice; and (v) the Parent shall have delivered to the Company written, binding and irrevocable offer to alter the terms or conditions this Agreement during such three (3) Business Day period, the Company Board shall have determined in good faith (after consultation with outside counsel), after considering the terms of such offer by the Parent, that the failure to effect a Company Board Recommendation Change would still be reasonably likely be inconsistent with its fiduciary obligations under applicable law; provided, however. that in the event of any material revisions to an Acquisition Proposal underlying a potential Company Board Recommendation Change, the Company will be required to notify Parent of such revisions and the applicable three (3) Business Day period deseribed above shall be extended until two (2) Business Days after the time Parent receives notification from the Company of such revisions. (Page 34) Answer:
Task name: maud_ cor_ ctr__oorrrsta__e standard_ (intervening.. egentett)
Question: What standard should the board follow when determining whether to change its recommendation in response to an intervening event?
Options: A: "Breach" of fiduciary duties; B: "Inconsistent" with fiduciary duties; C: "Reasonably likely/expected breach" of fiduciary duties; D: "Reasonably likely/expected to he inconsistent" with fiduciary duties; E: "Reasonably likely/expected violation" " fiduciary duties; F: "Required to comply" with fiduciary duties; G: "Violation" of fiduciary duties; H: More likely than not violate fiduciary duties; I: Other specified standard
Example: 6.3 Shareholders' Approval and Stockholder Approval. <omitted> (c) <omitted> if the Board of Directors <omitted> the Company, after receiving the advice of its outside counsel and, with respect to financial matters, its outside financial advisors, determines in good faith that it would more likely than not result in a violation of its fiduciary duties under applicable law to make or continue to make the Parent Board Recommendation the Company Board Recommendation, as applicable, such Board of Directors may <omitted> submit this Agreement to its shareholders stockholders, respectively, without recommendation (which, for the avoidance of doubt, shall constitute Recommendation Change) (Page 57) Answer: I
Task name: maud_ ainitialmattciiiiiieieeaee
(cor)
Question: How long is the initial matching rights period in case the board changes its recommendation?
Options: A: 2 business days or less; B: 3 business days; C: 3 calendar days; D: 4 business days; E: 4 calendar days; F: 5 husiness days; G: Greater than 5 business days Example: 6.3 No Solicitation by Golden. <omitted> in response to a <omitted> Golden Competing Proposal <omitted> the Golden Board may effect a Golden Change of Recommendation; provided. however, that such a Golden Change of Recommendation may not be made unless and until: <omitted>; provided that in the event of any material amendment material modification to any Golden Superior Proposal <omitted> Golden shall be required to deliver a new written notice Labrador and to comply with the requirements of this Section 6.3 6.3(c)iv) with respect to such new written notice, except that the advance written notice obligation set forth in this Section 6i3(e) shall be reduced to two Business Days (Pages 34-35)
Answer: D
Task name: maud_ addi additional cerlionnnnatcchiiiee
feiggtss_rrrdddppe wwrooodoooooooooooooeeoeeoeeeeee www.mmodiiiaa.. (cor) Question: How long is the additional matching rights period for modifications in case the board changes its recommendation?
Options: A: 2 business days or less; B: business days; C: 3 days; D: business days; E: 5 business days; F: > 5 business days; G: None
85

---

Table 36 - continued from previous page
Task
Example: Section 5.4 Non- Soli-oiiittaiin. <omitted>
(b) <omitted> Notwithstanding the foregoing, at any time prior to obtaining the East Stockholder Approval, and subject East's compliance in all material respects all times with the provisions of this Section 5.4 and Section 5.3, in response to a Superior Proposal with respect to East that was not initiated, solicited, knowingly encouraged or knowingly facilitated by East or any of the East Subsidiaries or any of their respective Representatives, the East Board may make an East Adverse Recommendation Change; provided, however, that East shall not be entitled to exercise its right make East Adverse Recommendation Change response to a Superior Proposal with respect to East (x) until three (3) Business Days after East provides written notice to Central (an "Fast Notice") advising Central that the East Board or a committce thereof has received a Superior Proposal, specifying the material terms and conditions of such Superior Proposal, and identifying the Person group making such Superior Proposal, (y) if during such three (3) Business Day period, Central proposes any alternative transaction (including any modifications the terms of this Agreement), unless the East Board determines in good faith (after consultation with East's financial advisors and outside legal counsel, and taking into account all financial, legal, and regulatory terms and conditions of such alternative transaction proposal, including any conditions to and expected timing of consummation, and any risks of non- consummaaiinn of such alternative transaction proposal) that such alternative transaction proposal is not at least as favorable to East and its stockholders as the Superior Proposal (it being understood that any change in the financial or other material terms of Superior Proposal shall require a new East Notice and a new two (2) Business Day period under this Section 5.4(b)) and (x) unless the East Board, after consultation with outside legal counsel, determines that the failure to make an East Adverse Recommendation Change would be inconsistent with its fiduciary duties. (Page 76)
Answer:
Task name: maud_ definition includes____.. sttck_ddaal
Question: What qualifies as superior offer terms of stock deals?
Options: A: "All or substantially all"; B: 50%; C: Greater than 50% but not "all or substantially all"; D: Less
than 50%
Example: 5.4 No Solicitation by the Company; Other Offers. <omitted> the Company shall not be entitled to: (i) make a Change in Company Board Recommendation <omitted> unless: <omitted> the Company shall have first provided prior <omitted> notice to Parent that it is prepared to <omitted> make a Change in Company Board Recommendation (a "Recommendation Change Notice") <omitted> Any material changes with respect to the Intervening Event <omitted> or material changes the financial terms of such Superior Proposal <omitted> shall require the Company provide to Parent new Recommendation Change Notice <omitted> and new three (3) Business Day period. (Pages 45-46)
Answer: C
Task name: maud_ douudddefiiiiion.nc includes_ Seatludss_sss_______ _as_ss_dsseal___s
Question: What qualifies as superior offer in terms of asset deals?
Options: A: "All or substantially all"; B: 50%; C: Greater than 50% but not "all substantially all"; D: Less
than 50%
Example: Section 5.4 Acquisition Proposals. <omitted> (d) <omitted> following receipt of a <omitted> Acquisition Proposal comitted> that the Company Board determines <omitted> constitutes a Superior Proposal, the Company Board may <omitted> make an Adverse Recommendation Change <omitted> if <omitted> (i) (A) the Company shall have provided to Parent <omitted> notice, <omitted> (it being understood and agreed that any amendment the financial terms or any other material term or condition of such Superior Proposal shall require a new notice and an additional three Business Day period) (Pages 44-45) Answer: B
Task name: maud_ financial financial_point_co _fof______________ view_iieeeeeee satps_i__he__eeeee wttee_eoeecccsideeaaeeeeeee consideration
Question: Is "financial point of view" the sole consideration when determining whether an offer is superior?
Options: A: No; B: Yes
Example: 5.4 No Solicitation by the Company: Other Offers. <omitted> the Company shall not be entitled to: (i) make a Change in Company Board Recommendation <omitted> unless: <omitteds the Company shall have first provided prior <omitted> notice to Parent that it is prepared to <omitted> make a Change in Company Board Recommendation (a "Recommendation Change Notice") <omitted> Any material changes with respect the Intervening Event <omitted> or material changes the financial terms of such Superior Proposal <omitted> shall require the Company to provide to Parent new Recommendation Change Notice <omitted> and new three (3) Business Day period. (Pages 45-46)
Answer: A
Task name: maud_ definition containscccnnnceeeee cwntains_kkovleeeeeeooooooo Jww..eedge_reguireenttttt requirement_ -_ansver Question: What is the knowledge requirement in the definition of "Intervening Event"?
86

---

Table 36 - continued from previous page
Task
Options: A: Known, but consequences unknown or not reasonably foresecable, at signing; B; Known, but consequences unknown, at signing; C: Not known and not reasonably foresccable at signing; D: Not known signing
Example: "Acquisition Proposal" means any inquiry, proposal or offer from any Person or group of Persons other than Parent or one of its Subsidiaries made after the date of this Agreement relating to (A) a merger, reorganization, consolidation, share purchase, share exchange, business combination, recapitalization. liquidation, dissolution, joint venture, partnership, spin-off, extraordinary dividend similar transaction involving the Company or any of its Subsidiaries, which structured to permit such Person or group of Persons to, directly or indirectly, acquire beneficial ownership of 20% or more of the outstanding equity securities of the Company, or 20% or more of the consolidated net revenues, net income or total assets of the Company and its Subsidiaries, taken a whole (B) the acquisition in any manner, directly or indirectly, of over 20% of the equity securities or consolidated total assets of the Company and its Subsidiaries, in each case other than the Merger and the other transactions contemplated by this Agreement. <omitted> "Superior Proposal" means any bona fide written Acquisition Proposal (A) on terms which the Company Board determines in good faith, after consultation with its outside legal counsel and financial advisors, to be more favorable from a financial point of view to the holders of Shares than the Merger and the other transactions contemplated by this Agreement, taking into account all the terms and conditions of such proposal and this Agreement and (B) that the Company Board determines in good faith is capable of being completed, taking into account all financial, regulatory, legal and other aspects of such proposal: provided, that for purposes of the definition of "Superior Proposal," the references to "20% in the definition of Acquisition Proposal shall be deemed to be references to "50%." " (Page 47)
Answer:
Task name: maud_ intervening.eveeeee -eeuuttrrrruuredd___e tto_tto_oooooe oee_ccurr___ after_ attaarffteraannnaa wwinnnggnnn..................... -_. Baigiingg-_anserr answer Question: Is an "Intervening Event" required to occur after signing?
Options: A: No. It may occur or arise prior to signing.; B: Yes. It must occur or arise after signing. Example: "Superior Proposal" shall mean, with respect to party hereto, any <omitted> Acquisition Proposal with respect to such party made by third party acquire, directly indirectly, pursuant tender offer, exchange offer, merger, share exchange, consolidation or other business combination, (A) all or substantially all of the assets of such party and its Subsidiaries, taken as a whole, (Page 120)
Answer: A
Task name: maud_ initial initiillmatchinngeeee Baaiinn__igghh___eeeee_ee
(ftr)
Question: How long is the initial matching rights period in connection with the Fiduciary Termination Right (FTR)?
Options: A: 2 business days or less; B: business days; C: calendar days; 4 business days; E: calendar days; 5: 5 business days; G: 5 calendar days: H: Greater than business days
Example: SECTION' 5.3 No Solicitation by the Company; Company Recommendation. <omitted> (d) <omitted> provided, however, that the Company Board and the Company Special Committce shall not, and shall cause the Company not to, make a Company Adverse Recommendation Change in connection with a Company Superior Proposal unless (I) the Company has given Parent at least four (4) Business Days' prior written notice of its intention to take such action (which notice shall reasonably describe the material terms of the Company Superior Proposal or atlach the agreement and all material related documentation providing for such Company Superior Proposal), (II) the Company has negotiated, and has caused its Representatives negotiate, good faith with Parent during such notice period, to the extent Parent wishes to negotiate, to enable Parent to propose in writing a binding offer to effect revisions to the terms of this Agreement such that it would cause such Company Superior Proposal to no longer constitute a Company Superior Proposal, (II) following the end of such notice period, the Company Board or the Company Special Committee shall have considered in good faith any such binding offer from Parent, and shall have determined that the Company Superior Proposal would continue to constitute a Company Superior Proposal the revisions proposed in such binding offer were to be given effect and (IV) in the event of any material change to the material terms of such Company Superior Proposal, the Company shall, in each case, have delivered to Parent an additional notice consistent with that described in clause (I) above and the notice period shall have recommenced, except that the notice period shall be at least two (2) Business Days (rather than the four (4) Business Days otherwise contemplated by clause (I) above); (Page 47)
Answer: D
Task name: maud_ tail fesll_period_eeee length Question: How long is the Tail Period?
Options: A: 12 months longer; B: Other; C: within 12 months; D: within 6 months; E: within 9 months Example: Section 7.3 Termination Fees. <omitted> (b) <omitted> if <omitted>> Parent or the Company terminates this Agreement <omitted> (iii) comitted> the Company shall have consummated an Alternative Acquisition Proposal or entered into an Alternative Acquisition Agreement for any Alternative Acquisition Proposal <omitted> which Alternative Acquisition Proposal is ultimately consummated (Page 80)
87

---

Table 36 - continudd from previous page
Task
Answer:
Task name: maud_ ctedd_pppeec__eeeeeeeeee pereeprrrrrrreeee
Question: What is the wording of the Specific Performance clause regarding the parties' entitlement in the event of a contractual breach?
Options: "entitled to seek" specific performance; "entitled specific performance
Example: Section 9.10 Specific Performance. The parties hereto hereby agree that irreparable damage would occur in the event that any provision of this Agreement were not performed in accordance with its specific terms were otherwise breached, and that money damages other legal remedies would not be an adequate remedy for any such damages. Accordingly, the parties acknowledge and agree that cach party shall entitled to, accordance with provisions performance or other equitable relief to prevent breaches of this Agreement and/or to enforce specifically the terms and provisions hereof in any court, in addition to any other remedy to which they are entitled at law equity (Page 73)
Agreement, an injunction or injunctions, specific
Answer:
Significance and value
Reading comprehension is a particularly challenging part of contract review, both human and machine. The MAUD tasks evaluate LLM's ability to understand and categorize wide spectrum of legal clauses the context of merger agreements.
88

---

G.17 New York State Judicial Ethics
In LEGAL the New York State Judicial Ethics task is denoted as nys_judiicill_ethie
Background The New York State Advisory Committee on Judicial Ethics posts rulings real ethical scenarios. The Committee, established in 1987, offers guidance roughly New York State judges and justices, as well as other judicial personnel and candidates in the state. By interpreting the Rules Governing Judicial Conduct and the Code of Judicial Conduct, the Committee assists these individuals in maintaining high ethical standards. Actions taken by judges in accordance with the Committee's 's formal opinions are deemed proper. which helps protect them during any future investigations by the New York State Commission Judicial Conduct.
Task 300 real-world scenarios and fact patterns have been reformulated into yes questions understand whether models understand ethical rules and how they might apply to different judicial situations.
example a 2022 decision the Committee noted that: "A judge who previously served as the District Attorney may not preside over a parole recognizance hearing concerning a parolee or releasee who had originally been convicted and sentenced during the judge's 's former tenure as the District Attorne.."
This is converted to Yes/No question:
Can a judge who previously served as the District Attomey preside over parole recognizance hearing concerning parolee or releasee who had originally been convicted and sentenced during the judge's 's former tenure the District Attorney?
Question:
Answer: No
Question
Answer
Can judge's ' law clerk assist the judge with No any clection-related matters during a general election when the judge is on-call?
Can a part-time town justice be employed part- No time as a community school liaison with the county sheriff's 's office simultaneously?
Can appellate judge who successfully sought to vacate a vexatious lien filed by a disgruntled litigant against their real property preside over appeals from other decisions or orders rendered by the lower court judge who granted the peti- tion to vacate?
Yes
Can part-time town justice serve as part-time assistant conflict defender in the same county as their court?
Yes
Table 37: Examples for nys_judiccilleehhccs
Construction process
We collect digest statements from the New York State Unified Court System Advisory
Committee on Judicial Ethics 22
We collect samples from 2010, 2021, 2022, and 2023 and then use ChaiGPT to reformulate the statements into yes no questions. To ensure that data is used for training OpenAl models, we opt out data use for accounts used task creation. We leave 2010 2021 data for understanding scope of data leakage from opinions being online. 2022 and 2023 data should not have been seen by most models that were trained prior to these years.
Significance and value An important part of legal practice is abiding by ethics rules. As agents become more involved in the legal process will he important to understand not only whether they can understand and reason about rules for the public, but also whether they can reason about ethical principles and rules governing judges and lawyers.
89

---

G.18 OPP-115 Tasks
In LEGALBENCH, the OPP-115 tasks denoted as opp115_*.
Background The OPP-115 Corpus, consisting of 115 online privacy policies, provides comprehensive collection privacy statements expressed in natural language [47.. Each of these policies has been meticulously read annotated by a team three law graduate students. The annotations present the text specifically outline various data practices.
These privacy policies are classified into ten distinct categories:
1. First Party Collection/Use:
This describes how and why service provider collects user information.
Third Party Sharing/Collection: This explains how user information may shared or collected by third parties.
3. User Choice/Control: This delineates the choices and control options available to users.
4. User Access, Edit, Deletion: This describes if and how users may access, edit, or delete their information.
Data Retention: This states how long user information stored.
6. Data Security: This communicates how user information is protected.
Policy Change: This explains if how users will be informed about changes to privacy policy,
8. Do Not Track: This discusses if and how Do Not Track signals for online tracking and advertising are honored.
9. International & Specific Audiences: This focuses on practices that pertain only to specific group of users (e.g., children)
10. Other: This encompasses additional sub-labels for introductory or general text, contact information, and practices not covered by the other categories.
Task and construction process A separate binary classification task has been created for cach category, with negative samples drawn from the rest of the text. To ensure consistency, any text with less than 10 words has been eliminated. The 'other' category not included in the categorization because broad and wouldn't provide much value terms specific classification.
deemed
Significance and value The classification task associated with OPP-1 15 Corpus serves as a significant measure of an LLM's logical reasoning ability. By assigning privacy policy segments to the right categories, LLMs demonstrate their understanding and interpretation of the language and nuances within these privacy policies. Although the task may be invaluable and objective gauge of L.IM's progress in logical reasoning and language comprehension.
"simple" from human legal practitioner's viewpoint, provides an
90

---

Task/category
Example of clause
opp115_ [aplla_data_________ retention
The name of the domain from which you access the Internet (for example, gmail.com, you are connecting from a Google account):
oppl15 dal__daa____a security
However, system be 100% secure and human errors occur, so there is the possibility that there could be unauthorized access your information. By using our services, you assume this risk.
oppl15_de _ot_noo___o__o__ee track
Do Not Track Signals Our websites not treat browsers that send a do not track signal differ- ently from browsers that do not send one.
oppl15__fisspprry_oolliiooononm
Send-a-friend: In the case of send-a-friend email card, only collect
oppl 15 httpe/aaatttoaaaaadeppccccaaaaaa
CalOPPA the first state law in the nation to require commercial websites and online services to post privacy policy. The law's reach stretches well beyond California to require a person or company in the United States (and conceivably the world) that operates websites collecting personally identifiable information from California consumers to post a conspicuous privacy policy on its website stating exactly the information being collected and those individuals with whom it is being shared, and to comply with this policy. See more at: prrp--cc-vctctttttionne caloppa/sthash.0uk www.OOFRRbSS..ppo
opp 1all1__poliyy_chagee
If we make significant or material change in the way we use your personal information,
hhppll__tircdpptty_ssriinnggg
We use third-party payment service providers such Amazon.com ( Privacy Policy), Stripe.com ( Privacy Policy), and PayPal (Pii- vacy Policy)
opp115 1115__uu 1111__________ seee_acccess_edit and deletion
you can access your personal information by contacting ABITA.COM described the bot- tom of this statement, or through alternative means of access described by the service.
do wish receive any additional marketing material, you can indicate your preference on our store partners order form.
Table 38: Examples for OPP-115 tasks.
91

---

G.19 Purpose of Oral Argument Questions
In LEGALBENCH, the Purpose of Oral Argument Questions task is denoted oral_argument araa.aarumenttguustiioo. purpose.
Background Before court decides a case, it typically calls before the attomeys for the parties the lawsuit to orally present their arguments for why the case should be resolved in favor of their clients and to answer any questions that the judge or judges have of them. In modern times, however, oral argument is not lawyers" primary avenue for explaining their positions. Instead, parties submit their arguments in written form ("briefs") and use their oral argument time primarily to supplement those submissions by reiterating their key positions, clarifying areas of ambiguity, and seeking persuade judges who are uncertain how the should be resolved.
Although there is no universally accepted listing, judges questions at oral argument tend to fall into few categories
Background: A question seeking factual or procedural information that is missing or not clear in the briefing
Clarification:
A question seeking to get an advocate to clarify her position or the scope of the rule
being advocated.
Implications: A question about limits a rule or its implications for future cases.
Support: A question offering support for the advocate's position.
Criticism: question criticizing an advocate's 's position.
Communicate:
A question designed primarily to communicate with one more other judges on the
court.
Humor: A question designed to interject humor into the argument and relieve tension.
A lawyer presenting her case before court oral argument must be able to quickly and accurately determine why the judge has asked particular question so answer it on behalf of her client in the most persuasive way possible. It also difficult task. Under the pressure persistent and difficult questioning it easy a lawyer to misread a question and offer an unresponsive or misguided answer. Skillful lawyers learn to quickly understand not only judges' questions but the reasons they are asked.
Task The Purpose of Oral Argument Questions task requires an LLM to determinc-gingg -iinnnngiiennnn question from an oral argument transcript-for for which of the seven purposes above the judge asked the question.
Construction process We created dataset of questions from U.S. Supreme Court oral argument transcripts,
classified into one of the seven functions above. Questions were taken from cases argued in the 2022-23 Supreme Court term, in reverse chronological order. A question was defined as the totality a judge's words prior to an advocate's 's response, regardless whether the words constituted a true interrogatory sentence. To include a sufficient number of questions of cach type in the datasct, questions were not drawn at random. Instead rarer question types (c.g. humor and communication) were targeted for inclusion, questions of more common types (e.g. clarification) were frequently omitted.
Significance and value Young attorneys, and even many experienced ones, struggle with oral advocacy. requires comfort in the courtroom, quick thinking, and careful demeanor to assess from the tone and content a question why the judge is asking it and how most effectively to respond. In one regard, LLMs will be superior. They do not suffer from human nervousness. However, given only text prompt rather than audible question from which to infer tone, and given only judge's question and not the full case context, this task would be challenge even for a seasoned lawyer. Whether LLMs can succeed be an extremely interesting measure of progress in legal analysis.
Thiis categorization scheme is from [48]. employed and discussed along with other possible classification schemes at [45).
92

---

Question
Purpose
May I ask you question about standing? So it's the case, isn't it, that if any party in either of these two cases has standing, then would be permissible for us to reach the merits of the issue?
Background
guess don't i understand that answer. In other words, is simply adding for religious reasons to the label that would change whether could be regulated not?
Clarification
And have amicus brief from different stake- holders, some saying it may not apply in parody, but could apply in movie titles, might ap- ply in something else and not this, in novels, et cetera. Why should we rule broadly? And if we rule narrowly, what basis? You heard earlier least three alliterations, one, the Justice Ka- gan's, one Justice Jackson, one me, limit this just to parodies, because parodies really do rely on is this joke that people are going to get.
Communicate
Mr. Martinez, I think one of the problems that you have, as evidenced by a lot of the questions that you' 've been getting, with the derivative works protection, you know, which, in, you know, 106(2), actually talks about transform- ing any other form in which a work may be recast, transformed, or adapted. And it seems to me like your test, this meaning or message test, risks stretching the concept of transformation so broadly that it kind of eviscerates Factor and puts all of the emphasis on Factor I mean, when you' ve been asked about book to movie and and and, you know, songs, you keep flip- ping to Factor 4. So, if work is derivative, like Lord of the Rings, you know, book to movie, your answer just like, well, sure, that's new meaning or message, it's transformative, so all that matters is 4?
Criticism
What are your two ones that you'te like killers?
Humor
So what's the limiting line of yours - of yours? Justice Kagan asked you about another web- site designer. But how about people who don't believe in interracial marriage about people who don't believe that What's where's the line? choose serve whom I want. If dis- agree with their personal characteristics, like race disability, can choose to sell to those peopledisabled people should get mar- ried?
Implications
There were several questions earlier about the justification for granting preference for foster adoptive parents who are members an entirely
Support
different tribe. Could you speak to that?
Table 39; Examples for oral_ oral_argument, wrgument_question.prrpee.
93

---

G.20 Overruling
In LEGALBENCIL, the Overruling task is denoted overruling.
Background A hallmark of the common-laww legal system is that courts will overrule previous judicial decisions. The act of overruling is significant, and indicates that the overruled decision either in accurate articclatoonnaapliiattoonn of particular law, that overruling court wishes to announce substantive change in law.
Task In this task, an LLM is required given an excerpt of judicial text to determine if the text overrules previous decision.
Construction process This task is taken from 155
which previously studied the capacity for finetuned BERT models perform this task. Please refer to 155 for more information the construction process.
Significance and value Identifying when judicial text overrules another case is basic but essential lawyering skill. From practical standpoint, the capacity for L.MSs correctly classify overruling sentences could have practical applications for the design and construction legal opinion databases. When using or citing a case in legal arguments, lawyers must ensure that the case hasn't been an overruled, and is still "good law." Tools which automatically parse legal databases and extract cases which have been overruled would thus be helpful for constructing legal arguments.
Sentence
Overruling sentence?
brockett v. spokane arcades, inc., 472 u.s. 491, 501 (1985) (citations omitted).
No
we overrule so much of kerwin as holds that Yes
criminal defendant is not entitled to inspect and make an analysis of the seized controlled substance.
Table 40: Examples for overruling.
94

---

G.21 Personal Jurisdiction
In LEGAL the Personal Jurisdiction task is denoted as personal..urrsstiction.
Background Personal jurisdiction refers to the ability of particular court (e-g. court in the Northem District of California) preside over dispute between specific plaintiff and defendant. A court (sitting in particular forum) has personal jurisdiction over defendant only when that defendant forum. We focus on a simplified version of the rule for federal personal jurisdiction, using the rule:
relationship with the
There personal jurisdiction over defendant in the state where the defendant domiciled, or when (1) the defendant has sufficient contacts with the state, such that they have availed themselves of the privileges of the state and (2) the claim arises out of the nexus of the defendant's 's contacts with the state.
Under this rule, there two paths for court have jurisdiction over defendant: through domicile through contacts.
Domicile: A defendant is domiciled in state if they citizen of the state (i.e. they live in the state). Changing residency affects change in eitizenship.
Contacts: Alternatively, court may exercise jurisdiction over a defendant when that defendant has sufficient contacts with the court's 's forum, and the legal claims asserted arise from the nexus of the defendant's contacts with the state. In evaluating whether look at the extent which
of contacts are sufficient, lawyers defendant interacted with the forum, and availed themselves of the benefits and privileges of the state's 's laws. Behavior which usually indicates sufficient contacts include: marketing in the forum or selling/Shipping products into the forum. In assessing nexus, lawyers ask if the claims brought against the defendant arise from their contacts with the forum. In short: is the conduct being litigated involve the forum its citizens in some capacity?
Task
The personal jurisdiction task requires an LLM to determine given fact pattern describing the events leading up a legal claim-whetler particular court has personal jurisdiction over the defendant.
Construction process We manually construct dataset test application of the personal jurisdiction rule, drawing inspiration from exercises found online and legal cascbooks. Each sample in our datasct describes a "fact pattern"" and asks court located particular state (A) can exercise personal jurisdiction over an individual (B) named in the fact pattern. In designing the dataset, we use base fact patterns, and create 4 slices, where each slice evaluates different aspect of the personal jurisdiction rule:
Domicile: Fact patterns where B domiciled in A. Henee, personal jurisdiction exists.
No contacts: Fact patterns where B has insufficient contacts with A. Hence there is no personal jurisdiction.
Yes contacts, no nexus: Fact patterns where B has sufficient contacts with A, but the claims against B do not arise from those contacts. Hence, personal jurisdiction does not exist.
Yes contacts, yes nexus: Fact patterns where has sufficient contacts with A, and the claims against B arise from those contacts. Henee, there is personal jurisdiction.
Caveat. Personal jurisdiction is rich and complex doctrine. Our dataset focuses on a narrow class of fact patterns, related jurisdiction over individuals. We don't 't consider, for instance, more complex questions related adjudicating citizenship (c.g. the Hertz test) or the classic oream-of-commeree problems. We leave this to future work.
Significance and value Identifying when personal jurisdiction exists is skill that law students learn in their first-year civil procedure course. The personal jurisdiction task interesting because applying even the simplified version of the rule requires reasoning over the degree of connection hetween a defendant and the forum state.
95

---

Facts
Personal jurisdiction?
Dustin is repairman who lives in Arizona and repairs computers in California, Oregon, and Washington. Dustin is avid skier, so his favorite place to go on vacation Colorado. While travelling repair computer in Wash- ington, Dustin involved a car crash in Ore- gon with Laura, citizen of Oregon. After the accident, Dustin returns to Arizona. Laura sues him in Colorado.
No
No David is citizen of California. He flies New York for vacation, where he meets Maggie, who is also visiting from Rhode Island. While they chat, Dave fraudulently tricks Maggie into giving him her savings, David then continues his vacation and visits Texas, Oregon, Florida, and New Mexico. After he returns home, Mag- gie sues David for fraud in Oregon.
Yes Ana is lawyer who resides in Texas. While visiting Louisiana, she meets David, who runs bike shop. David's bike shop is famous, and he frequently advertises his bikes in Texas newspa- pers. Ana buys bike from David and rides back home. Right after she crosses the border, the bike seat explodes, injuring Ana. Ana sues David in Texas.
Tony (from Texas) is regional manager for a cookbook company, Tasty Eats Books (in- corporated and principal place of business in Delaware). Tony's job requires him to travel from city city show cookbooks to chefs. In January 2022, he was scheduled to visit restaurants Illinois, Indiana, and Michi- gan. While in Michigan, Tony goes to Lake Erie to blow off some steam. He ends up get- ting into a fight with Arthur, a lawyer from Detroit, Michigan. Tony and Arthur each blame the other for starting the fight. Arthur sues Tony Texas.
Yes
Table 41: Examples for jersonal.jurisdicttonn
96

---

G.22 Privacy Policy Entailment
In LEGALBENCH, the Privacy Policy Entailment task is denoted as privacy.policy_entaal
Background The Privacy Policy Entailment task is created from the APP-350 corpus [599] which consists of 350 Android app privacy policies annotated with different privacy practices. In this corpus, individual clauses are annotated based on whether they do do not perform a certain practice (e.g.. "We access your location information").
Task Given a clause from a privacy policy and a description of the practice, the LLM must determine if the clause describes the performance of that practice. This is analagous to entailment task, where the premise is the clause, and the hypothesis is the practice description.
Construction process For each practice coded in the APP-350 corpus, we derive natural language "hypothesis." " Each instance of this task corresponds triple
deseription of that practice, which serves containing clause, practice description, and binary classification (Yes/No) based on whether the clause performs the practice. Across the dataset there are 57 unique policy descriptions.
Clause
Description
Performed?
We may collect and record information through the SN Service in accordance with the policies and terms of that SN collect
The policy describes receiving data from an unspecified single sign on service
Yes
Service. The information when you connect your user account to an SN Service may include: (1) your name, (2) your SN Service user identi- fication number and/or user name, (3) locale, city, state and country, (4) sex, (5) birth date, (6) email address, (7) pro- file picture or its URL, and (8) the SN Service user identification numbers for your friends that are also connected to Supercell's 's game(s).
Your e-mail address will not be stored.
The policy describes collection of the No
user's e-mail by party to the contract.
Table 42: Example clause-descriptioo-labll pairs for Privacy Policy Entailment task.
evaluate LLM's ss capacity to perform entailment-style reasoning over formal legal language. From a lawyerly perspective, understanding whether a policy performs certain functions empowers one of the parties to pursue practices is essential element of legal comprehension. From practical perspective, the ability for LLMs perform this task could empower researchers conduct broader studies of privacy agreements. As 159 observes, annotation cost limitations often restrict the scope of empirical studies privacy agreements.
Significance and value
privacy policy entailment task similar to ContractNLLI
97

---

G.23 Privacy Policy QA
In name, the Privacy Policy QA task is denoted as privacy-policy.pe
The Privacy Policy QA task derived from [100, which annotated clauses in mobile application privacy policies based whether they contain the
Background
question.
Task Given excerpt from privacy policy and question, the LLM must determine whether the excerpt relevant answering the question or not.
used the snippet annotations available in [10 to construct task, removing all saippets with fewer than 10 words. Examples excerpiquestion/ellaaan tuples are shown in the table below. 5449 instances correspond to relevant question-classe pair, and 5474 instances correspond to an irrelevant question-clause pair.
Construction process
Excerpt
Question
Class
We also use cookies, tags, web beacons, local shared ob- jects, files, tools and pro- grams to keep records, store your preferences, improve our advertising, and collectNon- Identifying Information, in- cludingDevice Dataand infor- mation about your interaction with the Site and ourBusiness Partners' web sites.
is my search and purchase his- Relevant
tory shared with advertisers?
We collect information about the value added services you are using over Viber and/or apps (such as games) you have downloaded through Viber.
does viber sell my information Irrelevant to advertisers and marketers?
Table 43: Examples for Privacy Policy QA
Significance and value Determining when a particular legal excerpt is relevant to answering question is essential to interpreting legal documents. This task allows to evaluate LLMs for this capability. From more practical standpoint, the Privacy Policy QA task is helpful evaluation task when developing LLM systems which involve decompositions over long documents. common approach order to account many long documents exceed ordinary context windows-is to chunk documents into smaller segments, and apply LLM independently to filter out irrelevant segments. For QA tasks involving long policies, this task allows practitioners to measure performance for the filtering step.
fact that
98

---

G.24 Private Right of Action (PROA)
In LEGALBENCH, the Private Right of Action task is denoted as proa.
Background A private right of action (PROA) exists when a statute empowers an ordinary individual (i.e. private person) to legally enforce their rights by bringing action in court. In short, PROA creates the ability for individual to ubiquitous antitrust law which individuals harmed by anti-competitive behavior can sue offending firms for compensation) and environmental law (in which individuals can sue entities which release hazardous substances for damages) 50
in order to recover damages halt offending conduct. PROAs
Task In the PROA task, an LLM must determine statutory clause contains a private right of action.
Construction process We construct a dataset of PROAs by hand, drawing inspiration from clauses found in different state codes. We construct 50 clauses which do contain PROA, and 50 clauses which do not. Clauses which do contain private right may either create cause of action, empower non-private individual (e-g., an attorney general) to bring a claim. randomly sampled clauses constitute the training set, and the remaining 95 form the test set.
Input
Answer
The attorne general or an attorney representing the state may bring action for an injunction to prohibit person from violating this chapter
No
or rule adopted under this chapter.
The administrator may bring action in court No of competent jurisdiction to enforce this chap- ter.
The sheriff the sheriff's designee shall main- No tain permanent personnel file oneach depart- ment employee.
If any laborer, without good cause, shall aban- don his her employer before the expiration of his or her contract, or she shall be liable to his her employer for the full amount of any account he or she may owe his or her employer.
Yes
No employer may discharge any employee by reason of the fact that earnings have been sub- execution. If an em-
Yes
jected to garnishment ployer discharges an employee in violation of this section, the employee may within ninety days of discharge bring a civil action for recov- ery of twice the wages lost result of the violation and for an order requiring reinstate- ment.
addition to all other penalties, rights, or reme- dies provided by law, an individual or entity that uses or attempts to use its official authority or influence for the purpose of interfering with the right of a legislative employee to make a pro- tected disclosure is liable in civil action for damages brought by a legislative employee.
Yes
Table 44: Examples for proa
Significance and value The PROA task evaluates an LLM''s ability to perform two-step reasoning test: (1) does the statute allow party to bring claim in court, and (2) is that party private? Law students and legal professionals should be capable of performing this task at near-perfect accuracy.
99

---

The PROA task derives additional significance from recent movement towards studying state statutory language [S3]. Legal scholars have long been unable conduct large scale empirical studies of state statutory language, given the sheer volume of state statutes. ability for LLMs to accurately classify or annotate statutes could thus empower new empirical studies of state statues.
100

---

G.25 Rule QA
In LEGALB the Rule QA task is denoted as rule_qa.
Background Lawyers are regularly required to recall specifical legal rules that are drawn from cases, statutes, other sources. Rules take many shapes and forms. For instance, the rule pertaining the federal requirements class (in a class action lawsuit) are codified in Rule 23(a) of the Federal Rules Civil Procedure and are simply known as the need for "numerosity, commonality, typicality, and adequacy."
Task The Rule QA task evaluates LLM's ability to answer questions on different legal rules. The rules are drawn from subjects typically studied in the first year of law school (e.g.. civil procedure, constitutional law, etc.). This is an open-generation task.
Construction process We manually wrote 50 question-anse answer pairs, focusing the types of rules which regularly tested in law school courses on civil procedure, evidence, and intellectual property. The questions the LLM to either (I) restate a rule. (2) identify where rule codified, or (3) list the factors employed particular rule. Several questions explicitly narrow their scope to a jurisdiction (e.g.. California state evidence law), in order to avoid bias towards merely federal law.
Question
Answer
Area of law
What are the four categories of patentable subject matter?
"process, machine, manufac- IP ture, or composition of mat- ter."
What are the requirements for diversity jurisdiction?
Diversity jurisdiction exists when the amount in contro- versy exceeds $75,000 and the plaintiffs and defendants are completely diverse (i.c. no plaintiff shares a state cit- izenship with any defendant)
Civil Procedure
Under which statute are patentable subject matter requirements codified?
35 USC 101
IP
What are the factors of the Mathews balancing test?
A three-part test that deter- mines whether individual has received due process un- der the Constitution. The test balances (1) l) the importance the interest stake; (2) the risk of an erroneous deprivation of the interest because of the pro- cedures used, and the probable value of additional procedural safeguards; and (3) the govern- ment's interest.
Constitutional law
Table 45: Examples for Rule QA
Significance and value The Rule QA task is an initial effort to evaluate the propensity for legal hallucination in LLMs. The questions asked are exceedingly basic, and law students taking the relevant course would be expected them nearly perfectly.
101

---

G.26 SARA Tasks
In LEGALBENCH, the SARA tasks are denoted as sara_*.
Background An important skill for lawyers is determine, given the facts of a case, whether given law applies and what preseribes. For example, does the payment received by the defendant August 2/st. 2017 qualify as wages under $3306(b) of the US Tax Code? This task has been introduced by [76 as statutory reasoning. (63) further introduce the Statutory Reasoning Assessment dataset (SARA). SARA contains (1) a set of 9 sections, taken from US federal tax law statutes, pruned and simplified; and (2) hand-crafiee cases that test the understanding of those 9 sections. In this context, Each case comes either with an entailment prompt- statement about the statutes and the case that may be true or false or a question- asking how much tax one the case's 's protagonists owes. The SARA dataset a simplified version of real-world cases, that retains many of the features of statutory reasoning for tax law. poses, however, a significant challenge to NLP models [12]
is paragraph of text stating facts in plain language.
Tasks There are two SARA tasks. The first, sara_ sara_entaill entailment. corresponds to the entailment cases. The entailment cases that given law applies to given case, and require the LLM to produce binary akin to Recognising Textual Entailment (39]. This is an approximation real-world statutory reasoning, where the answer is usually not strictly binary.
The second task, sara_ murericmm,i, consists of the numeric cases. Here, the goal is to compute the amount of tax owed. We frame this as a floating point number. To measure numerical accuracy, we use the metric introduced by 633, which includes tolerance for inaccurate predictions.
Construction process We framed the SARA dataset for the paradigm of language modeling. Due to dependencies across sections in the statutes, the entirety of the statutes are generally relevant determine the answer to any of cases. However, all 9 sections do not into the LLM's context window, and must be pruned. In entailment cases, the entailment prompt specifies which law from the statutes to apply. We automatically extract the text of that law, and use it as the language model prompt. For numerical cases, the entirety of the statutes relevant, and we use that as the language model prompt. Pruning left to the LLM's pre- processing.
Significance and value Statutory reasoning is important skill for lawyers, that is used within many other legal tasks. is fundamental task for legal AI, probing whether computational model can understand and reason with legal rules expressed natural language. The types reasoning involved in SARA are diverse defeasible, temporal, numerical reasoning, inter alia - and relevant beyond the legal domain. Statutory reasoning combines natural language understanding and logical reasoning, major goal for AL.
If statutory reasoning were solved, it could serve a basis for more complex legal tasks. For example, it could be used to automate the computation of taxes and benefits, without the need for coding the expert systems in many parts of the world [99]. A system that can statutory reasoning would also step towards machine reading models that can analyze legislation and anticipate its effects, coming up with possible application scenarios [I]] As a final example, a statutory reasoning agent could be used for basic legal advice, increasing access to justice.
102

---

Task name: sara, entaiimet
Statute: (2) an individual legally separated from his spouse under a decree of divorce or of separate maintenance shall not be considered married.
Description: Alice and Bob got married on April 5th, 2012. Alice and Bob were legally separated under decree of divorce on September 16th, 2017. Statement: Section 7703(a)(2) applies to Alice for the year 2018.
Answer: Entailment
Task name: @ara__nmerrs
Statute: $3301. Rate of tax <br> <br> There is hereby imposed on every employer (as de- fined in section 3306(a)) for each calendar year an excise tax, with respect to having individu- als...(Ommitted from clarity]...Thr section shall not apply to any taxable year beginning after December 31, 2017, and before January 1, 2026. <br> <br>
Description: Bob is Charlie and Dorothy's 's son, born on April 15th, 2015. Alice married Charlie on August 8th, 2018. Alice's and Charlie's 's gross incomes 2018 were $324311 and $414231 respectively. Alice, Bob and Charlie have the same principal place of abode in 2018. Alice and Charlie file jointly in 2018, and take the standard deduction.
Question: How much tax does Alice have to pay in 2018?
Answer: $259487
Table 46: Example of each SARA task. For numeric, ommit part of the statute for brevity.
103

---

G.27 SCALR
In LEGALBENCH, the SCALR task is denoted as scalr.
Background Each case decided by the Supreme Court addresses specific question presented for review. Both the questions and the Court's opinions are published the Supreme Court's website.
Many of the Court's opinions briefly descrihed by other judges who recount the holdings of the Court in their own writing. For example, consider the following passage from State of South Carolina v. Key, 27971 (S.C. 2020; emphasis added):
The United States Supreme Court has addressed the constitutionality of warrantless blood draws in several DUT cases. See Schmerber, 384 U.S. at 770-71 (holding the warrantless blood draw of DUT suspect was valid because the law enforcement officer, dealing with a car accident, could "reasonably have believed that be was confronted with emergency, in which the delay necessary obtain a warrant, under the cireumstances, threatened 'the destruction of evidence"")...
We refer to these brief descriptions 'holding statements' or 'holding parentheticals, since they often enclosed by parentheses. Identifying the holding parenthetical that corresponds a question presented for review requires a notion 'responsiveness' or relevance between questions and answers as well as an understanding of the kinds of legal issues that could be implicated by a specific question presented for review.
The SCALR benchmark is a collection of 571 multiple choice questions designed to assess the legal reasoning and reading comprehension ability of large language models. Each multiple-choice task gives the question presented for review in a particular Supreme Court case. The solver must determine which holding parenthetical describes the Court's ruling response to the question presented. Here example from &T&T Mobility LLC v. Concepcion, 563 U.S. 333 (2011) with the correct response emphasized:
Task
Question: Whether the Federal Arbitration Act preempts States from conditioning the enforcement of an arbitration agreement on the availability of particular procedures here, class -wide arbitration-when those procedures are not necessary to ensure that the parties to the arbitration agreement are able to vindicate their claims.
A: holding that when the parties in court proceedings include claims that are subject to an arbitration agreement, the FAA requires that agreement to be enforced even if a state statute common-lan law rule would otherwise exclude that claim from arbitration
B: holding that the Arbitration Act "leaves no place for the exercise of discretion by district court, but instead mandates that district courts shall direct the parties proceed to arbitration issues as which an arbitration agreement has been signed"
C: holding that class arbitration "changes the nature of arbitration such degree that it cannot be presumed the parties consented to it by simply agreeing submit their disputes to an arbitrator"
D: holding that California law requiring classwide arbitration was pre- empted by the FAA because it "stands an obstacle to the accomplishment and execution of the full purposes and objectives of Congress," which is to promote arbitration and enforce arbitration agreements according to their terms
E: holding that under the Federal Arbitration Act, a challenge to an arbitration provision is for the courts to decide, while a challenge to an entire contract which includes an arbitration provision is an issue for the arbitrator
Construction
The data used to create this task comes from two sources:
1. Questions presented were gathered from the Supreme Court of the United States' website, which hosts PDFs of questions granted for review in each case dating back to the 2001 Term.
2. Holding statements that comprise the "choices" for each question were compiled from (a) CourtLis- tener's collection of parenthetical descriptions and (b) extraction of parenthetical deseriptions from Courtlistener's and the Casclaw Access Project's collections of court decisions using Eyecite.
Because questions presented review in Supreme Court cases are not easily available prior to 2001, the benchmark is limited to questions from cases decided in the 2001 Term and later. To ensure that "holding" statements would address the particular question presented, we limited the set of cases to those in which exactly question was granted for review. We also perform some manual curation to exclude questions which not
104

---

answerable without specific knowledge of case. For example, eliminated a case that presented this question: "Whether this Court's decision in Harris United States, 536 U.S. 545 (2002), should be overruled."
To create choices for each question presented, we first filter set of parenthetical descriptions as follows:
We limited our parenthetical descriptions to only those that begin "holding that...". as these are most likely to describe the core holding of the case, rather than some peripheral issue.
2. We only parentheticals that describe Supreme Court cases. This avoids the creation of impossible questions that ask the solver distinguish between "holding" statements dealing with exactly same issue at different stages appellate review.
3. We then seleet for each case the longest parenthetical meeting the above eriteria. We use the longest parenthetical because is most likely
descriptive enough make the question answerable.
We then create a task for each case which has both a question presented and "holding" statement meeting the above requirements. (While creettioncorree holding pairs are only for cases decided after 2001, we allow the use of parentheticals describing any Supreme Court case alternative answer choices.) We then need to select the four alternative answer choices for each question in manner that makes the task challenging. To select choices that are at least facially plausible, we find the four "holding" statements from remaining pool that are most TF-IDF similar to the question presented. The inclusion of difficult alternative choices requires the solver to draw nuanced distinctions between legal issues that share overlap in terminology.
Significance and value This task is significant because it tracks the useful and challenging skill of identifying passage relevant or responsive given query. LLMs that are able to perform well at this task have the potential to be more useful for complex legal question- question-answering and retrieval. The poor performance of simpler models on this task demonstrates that challenging one that requires a level of understanding beyond the word/synonym level.
105

---

G.28 Securities Complaint Extraction
In LEGALBENCH, the Securities Complaint Extraction tasks are denoted ssla_*.
Background Securities Class Actions (SCAs) are lawsuits filed by, and behalf of, investors alleging economic injury as result of material misstatements or omissions in public disclosures made by corporate directors and officers. These actions allege violations of the Securities Act of 1933 and Securities Exchange Act of 1934 and are predominately filed federal court, though in 2018 the United States Supreme Court determined actions brought under the '33 Act were permitted in state court.
"Plaintiff(s)" is the legal term to describe the individual, company, or organization bringing forth a lawsuit. Under the class action system, one more plaintiff's are appointed "Lead Plaintiff"' by the court to represent the interests a larger group "similarly situated" parties. securities class actions, investors that suffered the greatest financial loss, often public pensions or unions, are appointed lead plaintiff.
"Defendant(s)" the legal term to deseribe the individual, company, or organization that must defend themself against the alleged violations or misconduct outlined in the lawsuit. There is always at least one defendant. The majority of securities class actions name the company. its CEO and its CFO. Many name additional C-suite level officers, members of the Board of Directors and additional third-parties such as the company's independent auditor the underwriters of public offerings.
Each designated lead plaintiff,, and named defendants, are explicitly identified under "Parties" section of the class action complaint.
Tasks
There are three extraction tasks.
The plaintiff' task requires an L.M to extract the named plaintiffss within a text.
The individual defendants tasks require an LLM to extract named defendants who are individuals from within a text.
The company defendants tasks require an LLM to extract named defendants who are corpora- tions/companies from within a text.
For certain samples, the complaint excerpt may not exactly contain the answer. For example, the correct may be "Strongbridge Biopharma PLC," while the complaint only mentions "Strongbridge." In order to maintain fidelity to the workflow used by SSLA, we evaluate an LLM's ability to generate the official name of the entity, as represented in the answer. This requires the LLM to possess some background knowledge regarding official corporation names. We find that larger models generally able account for this.
Sometimes, the provided text will not explicitly name the plaintiff. an individual defendant, corporate defendant. In these cases, the I.I is expected to return "Not named".
Lost Plavalf Juin Vaaleene sat faek lis anryy Bil
Preat dom in reall Bidedd sountirs MRggl beals. 40-2
Care os dlilk Dacuameet Aidd D411-00
Defrelads
Ddicatiz spocidte wtw.anacoooaaaaaaccaacoooooo
from you Aably co oe-bodgaarand Ficicc Caldicia uo 2475 Asconvad 3odioaad Denasa Caliiiia
Figure 2: Example of the typical SCA structure (Case 4:21-cv-05868-HSC
Construction process Stanford Securities Litigation Analytics (SSLA) identifies, tracks, and aggregates data on the several hundred private sharcholder lawsuits and public SEC/DOJ enforcements filed cach year. SSLA fellows manually extract and analyze information including plaintiffs, defendants, judges, mediators, plaintiff and defense firms, key litigation events, real-time case statuses, settlement timing, settlement dollar amounts, attorneys' fees and expenses, and imposed SEC/ / DOJ penalties. There no ambiguity regarding the answers for this task given its nature.
This dataset is an extract from the corpus of texts of securities class action complaints in the SSLA database. Given the typical structure and headings for these types of cases, this dataset represents text extracted from the complaint, between the sections titled "Parties" and "Substantive Allegations". For cases where the second
106

---

heading was not found. texts fragments were limited to 2,000 characters. Cases with both headings were then filtered to include only those with texts up 2,000 characters, which excluded cases with longer "Parties" sections. Thus, all observations this dataset are 2,000 characters or less.
Text was scraped from complaints using python's 's PyPDF2 library and left unformatted and uncleaned. This training set includes several observations where the text does not include all or some of the named entities due to the method of text collection and variation in case structure. In several of these cases, plaintiff names are not present in the selected text because the plaintiff' had been named earlier in the complaint.
Table 47: Examples from Securities Complaint Tasks
Task
Task name: sala_ wwtppacoaeeeeeeeaeeeeeee _defendants
Excerpt: 6. Plaintiff' Don L. Gross, as set forth in the accompanying certification and incorporated by reference herein, purchased the common stock of KCS during the Cla Period and has been damaged thereby. 7. Defendant KCS, headquartered in Kansas C ity. Missouri, operales railroads the Midwest and Mcxico that run north to south, unlike most other U.S. ra ilroads that run cast to west. The Company's 's stock traded on the NYSE, an ef ficient market, during th e Class Period under the ticker symbol "KSU." As of October 11, 2013, there were more than 110 million shares issued and outstanding. 8. Defendant David L. Starling ("Starlingg), ) all relevant times, served KCS's President and Chief Executive Officer ("CEO"). Case 4:14-cv-00345-BCY
Document Filed 04/15/14 Page 2 of 293 Defendant David R. Ebbrecht ("Ebbrecht"), at all relevant times, served as KCS's ' Executive Vice President and Chief Operating Officer ("CO0"). 10. Defendant Patrick J. Ottensmeyer ("Ottensmeye r"), at all relevant times, served as KCS's Executive Vice President Sales & Marketing. 11. Defendant Michael W. Upchurch ("Upchurch 7), at all relevant times, served as KCS's Executive Vice President and Chief Financial Officer ("CFO"). 12. Defendants Starling, Fbbrecct, Ottensmeyer an Upchurch are collee tively referred herein as the "Individual Defendants." 13. During the Class Period, the Individual Defe ndants, as senior executive officers and/or directors of KCS, were privy to confidential and propric tary information concerning KCS, its operations. finances. financial cond ition and present and future business prospects. The Individual Defendants also had access to material adverse non-public information concerning KCS, as discussed in detail below. Because of their positions with KCS, the Individual Defendants had access to non-public information about its busine ss, finances, products, markets and present and future business prospects via interna
Answer: Kansas City Southern
Task name: ssla_saaaalaaaaaaaaaaaaaaaaaaaas individual indiviidhal_defeendaatta _efeeddntts Excerpt: Plaintiff Don L. Gross, as set forth in the accompanying certification and incorporated by reference herein. purchased the common stock of KCS during the Cla ss Period and has been damaged thereby. 7. Defendant KCS, headquartered in Kansas C ity. Missouri, operates railroads in the Midwest and Mexico that run north to south, unlike most other U.S. ilroads that run east to west. The Company's stock traded on the NYSE, an ef ficient market, during th Class Period under the ticker symbol "KSU." As of October 11, 2013, there were more than 110 million shares issued and outstanding. 8. Defendant David I. Starling (" ("Starllng""), ",, at all relevant times, served as KCS's s President and Chicf Executive Officer ("CEO"). Case 4:14-cv-00345-BCC -0W Document Filed 04/15/14 Page 2 of 293 9. Defendant David R. Ebbrecht ("Ebbrecht"), at all relevant times, served as KCS's Executive Vice President and Chief Operating Officer ("CO0"). 10. Defendant Patrick Ottensmeyer ("Ottenmmeye r"), at all relevant times, served as KCS's Executive Vice President Sales & Marketing. 11. Defendant Michael W. Upchurch ("Upchurch "), at all relevant times, served as KCS's Executive Vice President and Chief Financial Officer ("CFO"). Upchurch are collee tively referred herein the
12. Defendants Starling, Ebbrecht, Ottensmeyer "Individual Defendants." " 13. During the Class Period, the Individual Defe ndants, as senior executive officers and/or directors KCS, were privy to confidential and proprie tary information concerning KCS, its operations, finances, financial cond ition and present and future business prospects. The Individual Defendants also had access to material adverse non- -ublic information concerning KCS, as discussed in detail below. Because of their positions with KCS, the Individual Defendants had access non-public information about its busine ss, finances, products, markets and present and future business prospects via intern
Answer: David L Starling, David R Ebbrecht, Patrick J. Ottensmeyer, Michael W. Upchurch
Task name: ssla plaintiff'
107

---

Table 47 continued from previous page
Task
Excerpt: 11. Plaintiff, as set forth in th attached certification, purchas ed Catalyst securities at artificially inflated prices dur ing the Class Period and has b cen damaged upon the revelation of the alleged corrective disclosures. 12. Defendant Catalyst is Coral Gates, Florida headquartered company located at 355 Alhambra Circle Suite 1500 Coral Gates, FL 33134. The common stock is traded the NASDAQ Stock Market ("NASDAQ") unde
ticker symbol "CPRX." 13. Defendant Patrick McEnany ("McEnn ny") is the Company's ' co-founder, CEO and President. 14. Defendant Dr. Hubert E. Huckel M.D. ("Huckel") is the Company's 's co-founder and one of its directors. 15. Defendant Steven R. Miller Ph. D. ("M iller") is the company's COO and CSO. 16. The defendants referenced above in 91 13- 15 are sometimes referred to herein the "Individual Defendants." DEFENDANTS' WRONGDOING Background Case 1113-vv-3878-UU Document Entered on FLSD Docket 10/25/2013 Page 4 of 20 5 17. Catalyst is a specialty pharmaceutical company which develops and commercializes drugs treating orphan (rare) neuromuscular d neurological discases. 18. Lambert-Faton Myasthenic Syndrome ("I.EM S") is extremely serious disase which also extremely rare, afflicting about 3.4 persons per million, and ahout two thousand patients in United States. 19. FDA rules permit so-called "compassionate use" use drug that has not been approved by the FDA outside of clinical trials. A patient may be given drugs under a compassionate use program if the patient may benefit from the treatment, the therapy can be given safely outside the clinical trial setting, other alternative therapy available, and the drug developer agrees provide access to the drug. 20, Jacobus a tiny privat pharmaccutical company in New Jersey, with only dozens of employecs, and only 35 as of 2009, Jacobus has b een manufacturing 3,4 DAP and providing it to patients through
Answer: Not named
Significance and value Extracting data from legal documents is an extraordinarily resource- and time- intensive effort prone human error. As result, there known databases of non-securities class action litigation, despite obvious public policy implications the class action system. Automation identification tasks coupled with human approval would improve efficiency and reduce collection costs and data errors. This task may he useful to other legal rescarchers and industry practitioners extracting structured data from complex texts. Identification is very simple task that can be done by those with an understanding and familiarity with the underlying legal documents and legal system, but LLM's ability to accurately and precisely identify entities a useful metric assess.
108

---

G.29 Successor Liability
In LEGAL, the Successor Liability task is denoted as puccessor_liabille
Background When one company sells its assets to another company, the purchaser is generally not liable for the seller's debts and liabilities. Successor liability is common law exception to this general rule. In order spot a successor liability issue, lawyers must understand how courts apply the doctrine.
The doctrine holds purchasers of all, or substantially all, of seller's assets liable for the debts and liabilities of the seller if:
1. the purchaser expressly agrees to he held liable;
2. the assets are fraudulently conveyed to the purchaser in order to avoid liability;
3. there de facto merger between the purchaser and seller;
4. the purchaser is a mere continuation of the seller.
Express agreement governed by standard contract law rules. In practice, a purchase agreement contains a provision assume liabilities, litigation will rarely arise. Courts, however, sometimes interpret an implied agreement in the absence of a written provision.
Assets are fraudulently conveyed when the seller intends to escape liability through a sale or knows that liability will be avoided through a sale.
De facto merger is multifactor test that consists of (I) continuity of ownership; (2) cessation of ordinary business and dissolution of the acquired corporation as possible; (3) assumption by the purchaser of the liabilities ordinarily necessary for the uninterrupted continuation of business of the acquired corporation: and (4) continuity of management, personnel, physical location, assets, and general business operation. Some jurisdictions require a showing of all four elements. Others do not, and simply emphasize that the substance of the asset sale is one of merger, regardless of its form.
Mere continuation typically requires showing that after the asset sale, only one corporation remains and there overlap of stack, stockholders, and directors between the two corporations. There are variations of the mere continuation exception. The first variation the "continuity of enterprise" exception. In order to find continuity of enterprise, and thus liability for the purchaser of assets, courts engage in multifactor analysis. Factors include: (1) retention of the same employees; (2) retention of the same supervisory personnel: (3)
retention of the same production facilities in the same physical location; (4) production of the same product; (5) retention of the same name; (6) continuity of assets: (7) continuity general business operations; and (8)
whether the successor holds itself out as the continuation of the previous enterprise. The second variation is the product line exception. This exception imposes liability on asset purchasers who continue manufacturing products of seller's 's product line. This exception generally requires that defendants show that the purchaser of assets is able assume the risk spreading role of the original manufacturer. and that imposing liability is fair because the purchaser enjoys the continued goodwill the original manufacturer.
Scholars have noted that fraud, de facto merger, and mere continuation (and its variants) overlap. They share the common thread of inadequate consideration, that is, the consideration given in exchange for the assets is unable to fund the liabilities that underwrite those assets. Because of the overlap. different courts might apply different doctrines to identical sets of facts, but arrive the same policy 49
Successor liability doctrine is commonly taught in course corporate law business associations in law school. Sometimes Students are expected to spot successor liability issues and understand how to determine if successor will be held liable.
reserved for upper level courses in corporate finance or mergers and acquisitions.
The Successor Liability task requires an L.M to spot successor liability issue and identify its relevant exception to no liability. If more than one exception is relevant. the LLM is required to state the additional exception(s). The task does not include identification of the two variations to the mere continuation exception (continuity of enterprise and product line).
Task
109

---

Facts
Issue
Relevant exception
Large Incarceration Services purchased substantial amount of Small Prison's ' assets last year. The asset purchase agree- ment expressly disclaimed Small Prison's potential liability for an employment discrimination claim arising out of its prison service activities. Small conveyed its assets Large because Small's owners were concemed that the liability arising from the lawsuit would lead to bankruptcy. Several months following the asset purchase, Small Prison lost the discrimination lawsuit. The plaintiffs now seek relief from Large Incarceration Services.
successor liability
fraudulent conveyance, mere continuation
Large Incarceration Services purchased substantial amount of Small Prison's 's assets last year. The asset purchase agree- ment expressly assumed any liability arising out of its prison service actívities. Several months following the asset pur- chase, Small Prison lost number discrimination lawsuits. The plaintiff's now seek relief from Large Incarceration Ser- vices.
successor liability
express agreement
Big Pharma purchases substantially of DW I's assets. The purchase agreement expressly provides for assumption of only those liabilities necessary for continuing operations of DW I. DW had developed successful drug that regulated oxygen levels in the blood. After the purchase of DW I's assets, DW I dissolves. DW I's shareholders maintain owner- ship in Big Pharma equivalent to their ownership in DW I. In addition, there is some overlap between the companies' management teams. Moreover, Big Pharma continues to em- ploy seventy percent of DW I's workforce. Past users of DW I's drug bring mass tort claim against Big Pharma alleging that DW I's drug incorrectly measured oxygen levels in the blood leading to harm.
successor
de facto merger, mere continu- ation
liability
Big Pharma purchases substantially all of DW I's assets. The purchase agreement expressly provides for assumption of only those liabilities necessary for continuing operations of DW DW had developed a successful drug that regulated oxygen levels in the blood. After the purchase of DW I's assets, DW dissolves. DW I's shareholders do not own any stock in Big Pharma. However, there some overlap between the two companies' management teams. In addition, Big Pharma continues to employ seventy percent of DW I's workforce. Past users of DW I's drug bring mass tort claim against Big Pharma alleging that DW I's drug incorrectly measured oxygen levels in the blood leading to harm.
successor liability
mere continuation
Table 48: Examples for successor liability.
110

---

G.30 Supply Chain Disclosure Tasks
In LEGALBENCH, the Supply Chain Disclosure Tasks denoted serply__chaii_ddislosurree
Background Corporations frequently legally required disclose information that may be relevant investors, regulators, or members of the public. One example of this kind disclosure requirement laws that require corporations doing business in particular jurisdictions to provide detailed information on their supply chains, which is intended to ensure that the company's 's business practices are not supporting things like human trafficking human rights violations. One example of these kind of disclosure requirements is the California Transparency in Supply Chains Act (CTSCA). The CTSCA applies to corporations that are a: "[I] retail seller and manufacturer [2] doing business in this state [of California] and [3] having annual worldwide gross receipts that exceed one hundred million dollars (S1 ($100,000,,00), If a corporation meets these criteria, they are required to post information on five topics:
Verification: "[A]t a minimum, disclose to what extent, if any, that the retail seller or manufacturer.. [elngages in verification of product supply chains to evaluate and address risks of human trafficking and slavery. The disclosure shall specify if the verification was not conducted by third party."
Audits: "[A]t a minimum, disclose to what extent, if any, that the retail seller or manufacturer.. [e]onducts audits of suppliers to evaluate supplier compliance with company standards for trafficking and slavery in supply chains. The disclosure shall specify if the verification was not an independent, unannounced audit, -2
Certification: "[A]t minimum, disclose to what extent, if any, that the retail seller or manufacturer ... [r]equires direct suppliers certify that materials incorporated into the product comply with the laws regarding slavery and human trafficking of the country or countries in which they are doing business."
Accountability: "[A]t a minimum, disclose to what extent, if any, that the retail seller or manufacturer [m]aintains internal accountability standards and procedures for employees contractors failing to meet company standards regarding slavery and trafficking."
Training: "[A]t a minimum, disclose to what extent, if any, that the retail seller or manufacturer. [p]rovides company employees and management, who have direct responsibility for supply chain management, training on human trafficking and slavery, particularly with respect to mitigating risks within the supply chains of products."
In addition to requiring corporations that meet the specified criteria to post disclosures that provide this information, the California Attorney General's office has also posted a guide informing firms of "Best Practices" for what specific information to provide on each of these five topics
However, prior research has suggested that companies do not always post disclosures that cover each of these topies; and, even when they do, the disclosures not always consistent with the recommended best.
practices 27
Construction process We constructed this task based on an existing dataset supply chain disclosures. In the summer of 2015, we, with the help of research assistants, we searched the websites of corporations that had previously been identified by organization called "KnowTheChain as being required to post supply chain disclosures to be compliant with the California Supply Chain Transparency Act. Through this process, found disclosures for roughly 400 firms out roughly 500 firms for which TnoThhhhain suggested were required to post disclosures.
For each of these roughly 400 firms, we saved copies of their supply chain disclosures. We then had research assistants read the disclosures and code whether they included each of the five required topics for disclosure and, if so, whether the disclosures on those five topies were consistent with the best practices outlined by the Californi Attorne General's 's office.
We convert each of these 10 coded variahles into distinct binary classificationt task, producing 10 tasks. Table lists each task, along with the precise question used code the disclosure.
49
See California Transparency in Supply Chains Act, CAL. CIV. CODE $ 1714.43(a)(1) (West 2012). CCAL. CIv. CODE $ 1714.43(c)()..
CLL. Civ. CODE & 17)4.
CAL.. CIv. CODE $ 1714.43(c)3)..
CAL CIV. CODE $ 171..43())
CCLL CIV. CODE $ (7)..43(0)55..
COLA DEP'T OF JUSTICE, THE CALIFORNIA TRANSPARENCY IN SUPPLY CHAINS ACT: A RESOURCE GUIDE (2015), httoss) hattps://aag http:///ww.g.g..e/o/////////////
guide. guf.e.pdf..
111

---

Significance and value Corporate disclosure requirements a commonly used regulatory tool, but evidence suggests that firms do not always fully comply with these disclosure requirements. The Supply Chain Disclosure task evaluates whether LLMs may able to determine whether corporations are complying with those disclosure requirements. Because these disclosures are often formatted very differently, written in complex language, and may be designed to obfuscate relevant information, this task provides a useful measure of whether LLMs parse the content covered in legal documents.
Task
Question
disclosed_ verification
Does the statement disclose to what extent, if any, that the retail seller or manu- facturer engages in verification of product supply chains to evaluate and address risks of human trafficking and slavery? If the company conducts verification), the disclosure shall specify if the verification was not conducted by a third party.
alaclosed_audits
Does the statement disclose to what extent, if any, that the retail seller or manufac- turer conducts audits of suppliers to evaluate supplier compliance with company standards for trafficking and slavery in supply chains? ? The disclosure shall spec- ify if the verification was not independent, unannounced audit.
disclosed_ certification
Does the statement disclose to what extent, any, that the retail seller or manu- facturer requires direct suppliers to certify that materials incorporated into the product comply with the laws regarding slavery and human trafficking of the country or countries in which they are doing business?
disclosed_
Does the statement disclose what extent, if any, that the retail seller or manufac- turer maintains internal accountability standards and procedures for employees or contractors failing to meet company standards regarding slavery and trafficking?
accountability
disclosed_
Does the statement disclose to what extent, any, that the retail seller or manu- facturer provides company employees and management, who have direct respon- sibility for supply chain management, training on human trafficking and slavery, particularly with respect mitigating risks within the supply chains of products?
training
best_ pratppractic_ verification
Does the statement disclose whether the retail seller manufacturer engages in verification and auditing as one practice, expresses that it may conduct an audit, or expressess that is assessing supplier risks through review of the US Dept. of Labor's List?
best.practice.
Does the statement disclose whether the retail seller or manufacturer performs any type of audit, or reserves the right to audit?
audits
best_ _est_practice_ certification
Does the statement disclose whether the retail seller or manufacturer requires direct suppliers certify that they comply with labor and anti-trafficking laws?
best.practice_
Does the statement disclose whether the retail seller or manufacturer maintains internal compliance procedures on company standards regarding human traffick- ing and slavery? This includes any type internal accountability mechanism. Requiring independently of the supply to comply with laws does not qualify or asking for documentary evidence of compliance does not count either.
accountability
best.practice_
Does the statement disclose whether the retail seller or manufacturer provides training to employees on human trafficking and slavery? Broad policies such as ongoing dialogue on mitigating risks of human trafficking and slavery or increasing managers and purchasers knowledge about health, safety and labor practices qualify as training. Providing training to contractors who failed to comply with human trafficking laws counts as training.
training
Table 49: Supply Chain Disclosure Tasks
112

---

G.31 Telemarketing Sales Rule
In LEGALBENCH, the Telemarketing Sales Rule task is denoted telemarketing_eales.
Background The Telemarketing Sales Rule (16 C.F.R. Part 310) is set of regulations promulgated by the Federal Trade Commission implement the Telemarketing and Consumer Fraud and Abuse Prevention Act. Its purpose is to protect consumers from specified deceptive and abusive telemarketing practices. This task focuses on 16 C.F.R. $ (11),((a) and 16 C.F.R. $ 310.3(a)2), which outline a series of specific telemarketing practices prohibited as "deceptive." 16 C.F.R. 3.13833000 (1)))(a)(I) lists information that must be disclosed to a consumer before sale is made, and 16 C.F.R. & 310.3(a)(2) lists categories of information that telemarketer is prohibited from misrepresenting. 16 C.E.R. $ 310.2 provides definitions relevant both these subsections.
The Telemarketing Sales Rule (TSR) not commonly taught in law school its own topic, but may he used as examples courses on consumer protection law, administrative law, telccommunications law, and the like. Because of its simplicity, it has also been used in beginner-level legal research exercises tasking students with finding the TSR in the Code of Federal Regulations and applying it to set of facts.
Applying the TSR would require an LLM to classify set of facts as either falling within or outside of the specific prohibitions outlined in the rule. For example, the TSR requires that telemarketers disclose certain material information before sale made, such as the total cost the goods or services, the quantities of goods or services being purchased, and exchange and return restrictions. It also forbids telemarketers from making material misrepresentations as to cost, quantity, quality, endorsement or sponsorship, and the like. In many real-life situations, it would be ambiguous whether certain telemarketer behavior would violate the TSR; for example, it could he contentious whether given misrepresentation fits the definition of "material." However, this task limited to clear, unambiguous violations or non-violations, such as a telemarketer told a consumer that they were selling four apples for four dollars, when in fact they were selling four apples for six dollars.
The following subsections 16 C.F.R. $ 310.3(a)(I) and 16 C.F.R. $ 310.3(a)(2) were ignored in the task, given their complexity or their reference to other statutes and regulations:
C.F.R $ 310. (2)
* 16 C.F.R. & (1vvi
Task The TSR task is meant to test whether an LLM can classify simple sets of facts as describing a violation of the TSR, or not describing a violation of the TSR.
Construction process We manually created 50 samples, such that examples of at least one violation and at least one non-violation of each relevant subsection of 16 C.F.R. $ 310.3(a)(1) and 16 C.F.R. 310.3(a)(2) were present.
Significance and value Determining whether simple and unambiguous set of facts falls within the ambit of 16 C.F.R. $ 310.3(a)(1) or C.F.R. $ 310.3(a)2) would be easy task for law students lawyers, as well as many non-lawyers. However, an LLM that was trained to recognize clear violations of consumer protection laws could help administrative agencies like the Federal Trade Commission inform normal citizens of their rights.
Input
Answer
Acme Toys is a telemarketer subject to the Tele- marketing Sales Rule. Acme Toys sold a cus- tomer frisbee. It disclosed the brand of the frishee, but did not tell the customer the frishee was manufactured in Portugal. Is this viola- tion the Telemarketing Sales Rule?
No
Acme Toys a telemarketer subject to the Tele- Yes marketing Sales Rule. Acme Toys told a cus- tomer that it would sell them a handful of fris- bees very reasonable price, and that ship- ping would be $5. Then, the customer agreed to the sale. Is this a violation of the Telemarketing Sales Rule?
Table 50: Examples for telemarketting.saees..rr
113

---

G.32 Textualism Tasks
In LEGALBENCH, the Textualism tasks are denoted as textualism_tooe
Background Courts regularly interpret statutes to determine the precise meaning of words contained in the statute. For instance, suppose statute specifies that "It shall be illegal park vehicle inside public parks for longer than thirty minutes." " court may be asked to determine whether the statute prohibits persons from parking bicycles inside public parks. This requires defining the term "vehicle" and determining if a bicycle is a type of vehicle.
To guide the interpretation of ambigous statutory terms, American jurisprudence has developed numerous principles of statutory construction interpretation. These principles-aie referred to as tools or canons are rules which dictate how terms in statutes should he interpreted. For instance, the principle of ejusdem generis states that where general words or phrases follow a number of specific words or phrases, the general words are specifically construed as limited and apply only to persons or things of the same kind or class as those expressly mentioned [130].
One approach to statutory interpretation-KKuae as textualism- states that only the text of the statute should be considered [3T] In contrast, other approaches to interpreting ambigous term might call for court to analyze the purpose the statute, the history of the statute, or the intent of the legislature.
Task The Textualism tasks ask a LLM to determine if an excerpt judicial text is applying a specific textual tool when performing statutory interpretation. There are two tasks: dictionaries (textualism tool dictionaries) and.
The first task is plain miiaimg (textualism tool poolin, and requires an LLM to determine a court is applying the "plain meaning" rule. The plain meaning rule says that statutory text should be interpreted according to its plain or ordinary meaning.
The second task is dictionaries (textualism_nnt hrt___ooooccooorrieeieeo and it requires LLM determine if a court is using dictionaries to define the statutory text.
Construction process For each task extracted paragraphs from Court of Appeals opinions and manually annotated whether the paragraphs showed the court as "using" the respective tool.
In order to count as using plain meaning, the paragraph must reference the plain or ordinary meaning of the text. This includes directly saying "plain meaning" or referencing the general logic of the plain meaning rule. There must also be evidence that the court used the tool its decision. This latter condition is notable because legal scholars often care about whether the court actually used the tool when defending its decision. Common examples of using include stating it as general rule of decision ("[O]ur obligation is to look to the plain language of the statute to effectuate the intent of congress") applying it the facts (heh statute's 's plain language indicates the 1.50% fee cap applies if (1) the plaintiff was "a prisoner" at the time brought the action and (2) he was awarded attorney's 's fees pursuant to 1988."). "Using" does not, for example, include paragraphs that criticize the use of the plain meaning rule.
In order to count as using dictionaries, the paragraph must reference a dictionary. There must be evidence that the court used a dictionary as part of its rationale. This latter condition is notable because legal scholars often care about whether the court actually used the tool when defending its decision. Common examples of using include stating it as general rule of decision ("[We use dictionary help determine the plain meaning of statutory text") or applying it the facts ("According to the Websters dictionary, a vehicle any means in or by which someone travels, or something is carried or conveyed"). "Using" does not, for example, include paragraphs that criticize the use of dictionaries.
Significance and value Recognizing when a court is applying a particular canon of interpretation is a classical skill law students are expected to learn. L.M performance on this task thus offers a heuristic for comparing L.M comprehension of judicial text to that of a law student's. More practically, the capacity for LLMss to detect when certain canons are being applied could make them valuable tool for empirical legal scholars.
114

---

Input
Answer
overcome our prior interpretation of statute depends, in turn, on whether we regarded the statute as unambiguously compelling our inter-
No
pretation.
the statutory waiver is express, and its range is defined in unmistakable language. to say that a private person, but the united states, liable under title vii for interest as an element of an attorneys fee would rob the unambiguous statu- tory language of its plain meaning, it would defeat the statutory imposition upon the united states of a liability for costs, and the statutory inclusion of reasonable attorneys fee as part of the costs, identical to that of private party in similar circumstances. the scope-setting statu- tory words the same private person mark out the united states liability for attorneys fees as well as costs in the traditional sense. our responsibility as judges enforce this provi- sion according to its terms.
Yes
Table 51: Examples for textualien aacuuallimmnee samllllmmm_tooo tool _plain.
115

---

Input
Answer
we pause to note that even if congress sought, through the csra, to regulate the nonuse of interstate channels, it would still be within its constitutional command to do so, the supreme court has often held, in several contexts, that the defendants nonuse of interstate channels alone does not shield him from federal purview under the commerce clause. in heart of atlanta motel, inc. v. united states, 379 u.s. 241, 250, 85 s.ct. 348, 353, 13 l.ed.2 2d 258 (1964), the court upheld commerce clause jurisdiction over a local motel that failed to engage in interstate commerce when it refused to rent rooms to black guests. the court held that by failing to rent the rooms, the hotel inhibited black travelers from crossing state lines and thus obstructed interstate commerce that otherwise would have occurred. id. at 253, 85 s.ct. at 356, in standard oil co. V. united states, 221 u.s. I, 68, 31 s.ct. 502, 518, 55 l.ed. 619 (1911), the court upheld the sherman act, 15 u.s.c. 1, 2, as permissible congressional action under the commerce clause. the sherman act prohibits restraints of trade and obstructions of interstate commerce in order to facilitate commerce that otherwise would occur absent the defendants monopolistic behavior. finally, in united states v. green, 350 u.s. 415, 420, 76 s.ct. 522, 525, 100 led. 494 (1956), the court found constitutional the hobbs act, 18 u.s.c. 1951, which punishes interference with interstate commerce by extortion, robbery or physical violence [by] ... [ntlaw[ing] such interference in any way or degree. to accept baileys nonuse argument would mean, as emphasized by the second circuit, that congress would have power to prohibit monopoly so complete as to thwart all other interstate commerce in a line of trade[;] or to punish under the hobbs act someone who successfully prevented interstate trade by extortion and murder. sage, 92 f.3d at 105.
No
Yes our primary area of concern with the district courts determination is its confident assertion that the language of 326(a) is unambiguous. see lan assocs., 237 b.r. at 56-57. in this day and age when we exchange by a keystroke or series of keystrokes what we used to handle only in cash, we do not think that the term moneys is so clear as the district court indicated. in fact, one of the definitions cited by the district court refers to money as a measure of value, see id. at 55-56 (citing websters third new intl dictionary 1458 (1986)), which surely is a concept that evolves along with and is dependent upon changing cultural, social, and economic practices and institutions. for example, in todays society the term money could easily encompass the concept of credit, which increasing numbers of people use method of payment. the term money might also encompass property, especially when property is used as a method of payment or a measure of wealth. see websters ii new college dictionary 707 (defining money as [a] medium that can be exchanged for goods and services and is used as a measure of their values the market and as [p]roperty and assets considered in terms of monetary value); supra note 5 (describing the nabts argument that an exchange of property involves an exchange of value). but see in brigantine beach hotel corp., 197 f.2d 296, 299 (3d cir. 1952) (referring to precode statute governing receiver compensation and stating that [i]t is clear that the word moneys in the clause upon all moneys disbursed or turned over is not the equivalent of property.). these reasonable interpretations of the term moneys render it ambiguous for purposes of our interpretation of 326(a). see taylor v. continental group change in control severance pay plan, 933 f.2d 1227, 1232 (3d cir. 1991) (a term is ambiguous if is subject to reasonable alternative interpretations.); accord united states v. gibbens, 25 f.3d 28, 34 (Ist cir. 1994) (a statute is ambiguous if reasonably can be read in more than one way.).
Table 52: Examples for
116

---

G.33 UCC vs Common Law
In LEGALBENCH, the UCC vs Common Law task is denoted as ucc_v_common_ __a.
Background In the United States, contracts are lypically governed by one of two different bodies of law depending on the subject matter of the contract. Contracts for the sale of goods (physical, moveable things) are governed by the Uniform Commercial Code (UCC), a uniform set of laws created by the Uniform Law Commission and adopted in all US jurisdictions. Contracts for services and real estate, on the other hand, are governed by state common law. For example, contract for Alice to sell Bob her bike would be governed by the UCC (sale of good), but contract for Bob repair Alice's bike would he governed by the common law (service).
This distinction is significant because the UCC and the common law diverge numerous important legal issues such as:
Offer and acceptance: The common law requires an offeree's acceptance to exactly match the terms of the offeror's s offer in order for contract to be formed (the "mirror image" rule). The UCC, on the other hand, allows for some variation in the terms under UCC Section 2-207.
Definiteness: For a common law contract to be enforceable, it must he reasonably definite with respect to all material terms. For example, a service contract would not be enforceable without price term or adequate description of the service be provided. The UCC only requires that goods contract include the good being sold and the quantity. any other term missing from the contract (such as price or delivery), it will filled in by UCC default rules.
To create option contract (by which the offeror provides the offeree with defined period of irrevocability), ), the common law requires that the offeree give the offeror separate consideration for the option. The UCC allows merchants to make "firm offers" (effectively option contracts) without the offeree providing separate consideration.
Options:
Modification: To modify an existing contract, the common law requires both parties to provide new consideration (the "preexisting duty rule") whereas the UCC only requires that modifications be made in good faith.
Task The UCC vs. Common Law task requires LLM to determine whether contract is governed by the UCC by the common law.
The dataset was manually created to test LLM's ability determine whether contract governed by the UCC or by the common law. The dataset is composed of 100 descriptions of simple contracts such as "Alice and Bob enter into a contract for Alice to sell her bike to Bob for $50" (UCC) and "Aria pays Owen $100 to mount television on the wall of her living room" (common law). Each description is followed the question, "Is this contract governed by the UCC or the common law?"
Construction process
The dataset does not include "mixed purpose" contracts which incorporate both the sale of good and service. For example, a contract in which Alice sells Bob her bike for S100 and agrees inflate the tires each week for the first month would he a mixed purpose contract. To determine whether a mixed purpose contract is governed by the UCC or the common law, most jurisdictions apply the "predominant purpose" test under which the predominant purpose of the contract (good or service) determines which body of law applies.
Significance and value The UCC vs. Common Law task is significant for a number of reasons. First, it provides measure of an LLM's legal reasoning ability relative to human lawyer (who would almost certainly score a 100% on the task). Sccond, it demonstrates an LLM's ability to determine the subject matter of legal text, which has implications for the use LLMs for legal tasks far beyond contract classification. Third, this task could prove useful in the context of contract lifecycle management (CLM) in which a CLM software product could automatically sort contracts by subject matter for review purposes. Fourth, while the sample contracts in the dataset were simple and easily identifiable as either UCC common law contracts, real-world mixed purpose contracts can be difficult to classify and sometimes generate costly litigation. This task could be used as a starting point for developing a more fine-tuned task that can classify mixed purpose contracts.
117

---

H Extended empirical study
We now present fuller discussion of LEGAL.BENCE experiments. Specifically, use LEGAI.BENCH to conduct a three-part study.
In the first part (Section H.2
conduct a sweeping evaluation of 20 LLMs from 11 different families, at four different size points. We this study to make initial observations performance differences across families, the role model size, and the gap between open-source and commercial LLMs.
In the second part (Section H.3)
show how LEGALBENCH can be used to conduct in-depth LEGALLBenCI highlight similarities and differences in
evaluations of models. To illustrate, the performance of three popular commercial models: GPT-4, GPT-3.5, and Claude-1.
In the final part (Section H.4). specific I.LM methods. We Tocus on prompting. and conduct series of experiments that hegin surface tradeoffs and challenges with regards to guiding LLMSs towards certain tasks.
show how LEGALBENCH
support the development of law-
Ultimately, our study here to illustrate the types of analyses that LEGAI enables, and highlight potential directions for future work.
H.1
Setup
H.1.1 Models
Commercial models We study three commercial API-access models. From the OpenAl GPT family, we study GPT-3.5 [14 (text-davinci-003)) and GPT-4 (96]. Results from these models were retrieved between May and August of 2023. From the Anthropic family, we study Claude-1 (v1.3) []. Results from this model were retrieved in July of 2023. These models are believed to be large (hundreds of billions of parameters), though details their architecture and training process unknown. It is thus possible that LEGALBENCH tasks leaked into pretraining data.
Open-source models We study 17 open models at three different size points: 3B parameters, 7B parameters, and 13B parameters. All inference was performed on two-GPU GCP 40GB A100s, using the Manifest library 197
From Together, we study three models: Mecite-nsttruct-85 Incite-Base and Incite- Instte-Instruct-BB 34 133].
From Meta's OPT family, study three models: OPT-2.7B, OPT-6.7B, and OPT-13B 154
From TIl's Falcon family, we study Falcon-7B-Ing 2 103]
From MosaicML's MPT family, we study MPT-7B-8 8k-Instruc 129
From LMSYS' Vicuna family, study Vicuna-7B- 16k and Vicuna- Hicuna-13B-16 133 26
From Google's FLAN-T5 family, we study Flan-T5-XL. (3B parameters) and Flan-T5 Monn-T-XXLL (1IB
parameters)
3
From Meta's LLama-2 family, we study LLaMA-2-7B, and LLaMA-2-13B B3
From the Wizard family, study WizardLM-13B 150
From the BigScience BLOOM family, we study BLOOM-3b and BLOOM-7B (16
HuggingFace links for the studied open-source models be found below.
Future work Our selected LLMs represent only sample of the models available. For instance, do not evaluate LLMss larger than 13B parameters, which have been observed perform well ( Studied L.MSs also "general domain," that we don't 't find evidence that any were specifically customized to perform well on logal text in future work we hope to expand our evaluation to broader set of LLMs.
H.1.2 Prompts
We designed prompt for each task by manually writing instructions for the task, and selecting between zero and cight samples from the available train split selected depended on the availability of data and the sequence length samples. For instance, inputs the Supply Chain Disclosure tasks are disclosure statements between 1-2 pages long, making the inclusion of
in-context demonstration. The number of samples
31 We note that as of July 2023, we were unable to identify public law-specific English large language models evaluate.
118

---

LLM
HuggingFace URL
3B
Incite-Base-
Incite-Instruct-7B
BLOOM-3B
bigscience/bloom--bb
BLOOM-7B
bigscience/bllomm-7bl
OPT-2.7B
facebook/opt-2.7b
OPT-6.7B
facebook/opt-6.7b
OPT-13B
facebook/opt-135
Palssnc7B-Instruct
dinue/fflconn----instu
MPT-77-6k--nntrucc
mosaicem//nnpt-7--instuuu
Vicuna-7B-166
Imsys/vicuna-7--v1555666
Vicuna- 13B-16k
Imsys/vicuna-13--v1.55566k
Flan-T5-XL
google/flan-t5-xl
Flan-T5-XXL
google/flan-t5-xxl
LLaMA-2-7B
metalllamallLaaMA-2-77bhhf
LLaMA-2-13B
metallanna/laaMMA2213hhhh
(izardI.M-13R
Wirandl./Wizaadd.M11133
Table 53: HuggingFace links for open-source models.
multiple demonstrations infcasible. For application evaluation, we augmented the prompt with an instruction for the LLM to explain its reasoning.
We used the same prompts across all LLMs with one exception. In contrast to the OpenAl and open-source LLMs, Anthropic recommends specific prompting formats when using Claude This includes surrounding in-context samples with <example>/<xxamp)) observed that failing to adhere to these guidelines led Claude to generate text which made extracting prediction challenging. Therefore, when prompting Claude, we added example-tage to the in-context demonstrations and instructions specifying the prediction space (c.g.. "Reply with either: generie, descriptive, suggestive, arbitrary, fanciful").
tags, and adding instructions specifying the output space.
LLM outputs were generated using next-token generation temperature of 0.0. For classifiatton//etraation tasks, we terminated new-line token. For rule_ge jurisdiction_6 we generated 150 tokens. For _a and all application tasks except diversity. we generated 300 tokens.
We believe there is significant scope for improving and refining prompts on LEGALBENCH. Hence, our results here provide a lower- bound on performance. as better prompts may elicit higher scores. Our prompts correspond what believe would be reasonable, based experience with prompt engineering in other settings, and the guidance provided by model developers. We make all prompts available starting point for future work on LEGGALBENCH.
Prompts for all LEGAL BENCH experiments are available on the Github repository. Table 54 provides the number of in-context demonstrations used.
H.1.3 Evaluation
Classificattin tasks are evaluated using "exact-match" (following HELM 78) Because some tasks contain significant label imbalances, use balanced balanced-accurace accuracy metric. For extraction tasks, we perform normaliza- few tasks (e.g.. successor. liability tity and sela_ iall__iidiiidal...m.. individdall_deeeeddaatta) requires the L.M produce multiple classes or extracted terms per instance. For these, we evaluate using Fl.
tion on generated outputs to account differences in tenss/casing/punetuation. delendantn)))))
119

---

Number of in-context demonstrations
Tasks
0
Canada Tax Court Outcomes, Consumer Contracts
QA, Corporate Lobbying, International Citizenship Questions, Rule QA, Supply Chain Disclosure Tasks,
SARA (Numeric)
MAUD Tasks, SCALR
2
Citation Prediction Tasks
3
Securities Complaint Extraction Tasks
4
Legal Reasoning Causality, Personal Jurisdiction, Sue- cessor Liability, SARA (Entailment) Telemarketing Sales Rule, Textualism Tools
5
Abercrombie, Hearsay, Insurance Policy Interpreta- tion, Private Right of Action
6
CUAD Tasks, Diversity Tasks, J.Crew Blocker, Learned Hands Tasks, Overruling, UCC v. Common Law
7
Function of Decision Section, Oral Argument Ques- tion Purpose
8
Contract NLJ Tasks, Contract QA, Definition Tasks, New York State Judicial Ethics, OPP-115 Tasks, Pri- vacy Policy Entailment, Privacy Policy QA
9
Unfair Terms of Service
Table 54: Number of in-context demonstrations used for each type.
Rulc-application tasks were evaluated manually by a law-trained individual, who analyzed LLM responses for both correctness and analysis This type of manual evaluation is consistent with previous works evaluating LLM generations in the legal domain [29] [69. As rule-application requires LLMs to generate "explanations" detailing legal reasoning- a capability primarily exhibited by larger models-we only evaluated GPT-4, GPT-3.5, and Claude-1. rule_qa was also manually evaluated by a law-trained individual. All manual evaluation was performed with reference to a grading guide, which we additionally make available.
H.2 Performance trends
Table 55 provides the average task performance for all 20 models in five reasoning categories fissue-spottnng The first block of rows correspouds
rule-recall, rule-conclusion, interpretation, and neetorical-udderstanddee large commercial models, the second block corresponds models the 1 1IB-13B range, the third block corresponds to models in the 6B-7B range, and the final block corresponds to models in the 2B-3B range. Table 56 provides the average task performance for the three large models on rulc-application.
Overall, we find significant variation in performance across tasks, suggesting that LEGALBBENC BENCH captures a diverse spectrum of difficulty. These results emphasize that assessments of LLM capabilities for legal applications must be made tasks appear beyond the scope of current-day LLMs, others seem more within reach. In this section, we offer preliminary observations on performance trends across model size, family, and reasoning categorics.
task-by-task basis, and informed by the nuances of specific tasks. While certain types of
Parameter count Within LLM familics, we observe that larger models usually outperform smaller models. For instance, XXL (1IB parameters) outperforms Flan-Ts T5-XL (3B parameters) average across all average across four reasoning
five reasoning categories. and LLaMA-22 1-2-11133 13B outperforms LLaMA-2--7 categories. Notably, the margin of the gap varies across LLM families and reasoning categories. For instance, on rule-recall, the 7B Instre-Instruct model outperforms the 3B meite-Insstrue model by almost 10pts, while the 6.7B OPT model outperforms the 2.7B OPT model by less than 1pt. We additionally note that the largest L.L (GPT-4) outperforms virtually all other models.
For the six diversity jurisdiction tasks, we sampled 30 instances from each task. For all other rule application tasks, manually evaluated the entirety of the dataset.
120

---

LLM
Issue
Rule
Conclusion
Interpretation
Rhetorical
GPT-4
82.9
59.2
89.9
75.2
79.4
GPT-3.5
60.9
46.3
78.0
72.6
66.7
Claude-1
58.1
57.7
79.5
67.4
68.9
Flan-TS Man-TS-XXLL
63.3
64.4
70.7
LLaMA-2-13B
50.2
37.7
59.3
50.9
54.9
OPT-13B
52.9
28.4
45.0
45.1
43.2
Vicuna Mocnn-13B-BBe 13B-16k
34.3
29.4
34,9
40.0
30.1
WizardL.M-133
24.1
38.0
62.6
50.9
59.8
BLOOM-7B
50.6
24.1
47.2
42.8
40.7
Falcon-T 7B-Instruct
25.0
$2.9
44.2
Incite-BB 1ee--B---Baa 7B-Base
50.1
36.2
47.0
46.6
40.9
Siciee-B-Innstuct
54.9
35.6
52,9
54.5
45.1
LLaMA-2-7B
50.2
33.7
55.9
47.7
47.7
MPPT-BB-k--Instrue
54.3
25.9
48.9
42.1
44.3
OPT-6.7B
52.4
23.1
48.9
42.2
Vicuna-7B-16k
3.9
14.0
35.6
28.1
14.0
BLOOM-3B
47.4
20.6
45.0
45.0
36.4
Flan-TS-XX XL
56.8
31.7
52.1
5144
67.4
Incite-3B- InnstttB
51.1
26.9
47.4
49.6
40.2
OPT-2.7B
53.7
22.2
46.0
44.4
39.8
Table 55: Average performance for each LLM over the different LEGALLBNCC categories. The first block of rows corresponds large commercial models, the second block corresponds models in the 11B-133 13B range, the third block corresponds to models the 6B-7B range, and the final block corresponds models in the 2B-3B range. The columns correspond to (in order): issue-spotting rule-recall, rule-conelusion, interpretation, and phetorical-undersanndie (large, 13B, 7B, and 3B), the best performing model in each category of reasoning is underlined.
For each class of models
LLM
Correctness
Analysis
GPT-4
82.2
79.7
GPT-3.5
58.5
44.2
Claude-v1
61.4
59.0
Table 56: Average performance for the large LLMs rule-application tasks.
Variation across families Even for LLMs of the same size, we find considerable differences in performance. For instance, we observe significant gaps in performance hetween Flan-T5-XXI. (1IB parameters) and Vicuna- 13B-16k (13B parameters), across all reasoning categories. This suggests, unsurprisingly, that the choice of pretraining data, regime of instruction-tuning and architecture play an important role in determining performance, and that certain configurations may be better aligned for LEGALBENT such choices may affect which types of reasoning categories LLMs appear to perform well at. For instance, we observe that WizardL 13B performs worse than all peers on issue-spotting tasks, best rule-recall tasks, and nearly matches the performance of the best-performing peer rule-conclusion tasks. Comparing Iheit--7--Instruuee
tasks. Interestingly, we observe that
Incite-7B-Bas also provides insight into the effect of Instruction-tuning across different categorics, at one size point (7B parameters). We observe that instruction tunning improves performance on four categories (issue-spotting, rule-conclusion, interpretation, and inetorical-understandde performance on rule-recall.
and worsens
We additionally find that family-specific trends appear to hold across different size points. For instance, the Flan-T5 models outperform all others both the 3B and 13B scale, while the Vieuna models appear underperform competitors at both their frequency tendency to generate poorly-formed outputs, which did not map to the expected verbalizer tokens (e.g.. blank spaces, random characters. etc.) di) This could possibly be attributed to the type of data used to fine the model (e.g., user-conversation), although more in-depth experimentation is necessary.
and scale. We attribute the Vicuna models"
performance
"' further experimentation, we found that writing prompts using the "### Human:" and "*### Assistant:"" templates did not appear help.
121

---

The gap between open-source and commercial models Finally, we find evidence that open-source models capable of performance that matches exceeds certain commercial models. For instance, Flan- TS-XXL outperforms GPT-3.5 and Claude-1 on two calegories (issue-spolting and rhetorical-undee -nderssaanndderssniinnnngg despite the relative gap in parameter count. Notably, the gap between closed and open-soure, models is largest for the rule-conclusion category. Amongst LEGALBENCH tasks, rule-conclusion tasks most like the other types of pettttisstp/commo--ssmmtte sense reasoning tasks where commercial LLMs have been found to perform well.
H.3 Comparing GPT models
This section provides more in-depth study of performance, focusing on the three commercial models (GPT-4, GPT-3.5, and Claude-1). The purpose of this section is to illustrate how LEGALBENCH enables fine-grained analysis of LLM performance. In particular we highlight how LEGALBENCH can provide more rigorous empirical support for anecdotal observations arising out the legal community's 's use of these models, and explain performance differences between models.
H.3.1 Issue-spotting
We first consider average model performance across all issue-spotting tasks. We observe that GPT-4 outperforms GPT-3.5 and Claude-1 (hoth p <, 0.001) [ In absolute terms, issue tasks present the largest gap in performance between GPT-4 and other closed-APl models, with an absolute margin of 20+ points. GPT-3.55 and Claude-1, I, contrast, appear to match each other performance, separated by an average gap only points. We additionally find that the open-source models perform poorly here. On 9 tasks, Incite-Base collapses to predicting a single class for all samples.
Wc note one limitation to our results: because 16/17 of our issuc-spotting tasks are drawn from one source (Learned Hands data), average issue performance is skewed by properties of the Learned Hands data distribution (i.c.,, user-generated questions). For instance, though GPT-3.5 outperforms Claude-1 on 12/16 Learned Hands tasks, Claude-1 outperforms GPT-3.5 on the one non-Learned Hands task (corporate (corporate_lobbyinng) iobbyinggb..... Despite the skew, we observe that these tasks appear to vary in difficulty. While GPT-4's balanced-accuracy on learned_ hands_ torts is only 70.6%, on three lasks-learne Saaeeed_aaaa_____e _____mmiggg weiiiimmmaiiii gration, learned haand_traffee and learned_hands
it scores > 95%.
H.3.2 Rule-recall
We first consider average model performance across all rule-recall tasks. While GPT-4 outperforms GPT-3.5 (p 0.05), we surprisingly find that Claude also outperforms GPT-3.5 (p < 0.05). and appears almost on par with GPT-4. Moreover, Claude-1 outperforms GPT-4 on three tasks: rule_qa, international. wttprrsssppuuussttiooo
and nys_judicial_ _eticss.h.t... This is the only task category where Claude-1 provides performance comparable to GPT-4. Because little is known regarding the architecture and training processes for these models however, it difficult to explain why this is the case.
Because rules/laws can be analogized to law-specifie "facts," rule-recall tasks are similar to general domain LLM tasks designed to measure "hallucination." There, an extensive literature has documented the propensity for LLMs to both generate factually incorrect information, and answer fact-based questions incorrectly [78) [104]. Our results align with the primary findings of that literature. For example, we observe that the small open source models perform considerably worse than the larger models, consistent with the observation that model size plays an important role retentinnn Overall, performance on the necelrecall tasks also lend additional empirical support to more anecdotal reports from the legal community regarding how LLMs often misstate the law or cases 112.
H.3.3 Rule-application
Application tasks evaluate whether LLMs can explain how a legal rule applies to a set of facts, and verbalize the necessary inferences. With respect to correctness, observe that GPT-4 outperforms both GPT-3.5 (p 0.01) and Claude-1 (p 0.01). Across L.Mss, we find that variation in performance across tasks is consistent with subjective impressions of task difficulty. For instance, performance on diversity wwwerssty.uuursiiieee... jwrrjddictiionn..oo (an casy task requiring a model to determine if an amount is greater than S75k and if the plaintiff' and defendant are from different states) is much higher than performance successor liability (a harder task requiring model identify multiple successor liability exceptions in fact pattern describing complex transaction).
We observe that LLM generations may be incorrect in many different ways. On the Diversity Tasks, LLMs sometimes perform incorrect arithmetic operations or mathematical comparisons (i.c., stating that $50,000 greater than $75,000). On telemarte tmmarketing... salessssl___.. rulleeee in contrast, LLMs will cite to an incorrect portion of the rule. For instance, a generation may explain that certain conduct by telemarketer runs afoul of the rule
sales_rule
Statistical significance is computed using a paired f-test the tasks in the category.
122

---

because the telemarketer failed to make mandatory disclosure (16 CFR 310.3(aXI)). but cite to the portion of the rule prohibiting misrepresentations (16 CFR 310.) Examples of other types of incorrect generations can found in Table
Fact Pattern (Task)
LLM generation
Why is it incorrect?
Sophia is from Ne- braska. Ava is from
"No. The parties are not diverse because Sophia and Ava are both from different states. The AiC is too small because it is less than 75k. Because the AiC requirement not met there is no jurisdiction." " (GPT-3.5)
While GPT-3.5 correctly identi- fies that there is no jurisdiction because the AiC is less than $75k (and thus too small), it fails recognize
Washington. Sophia sues Ava for legal mal- practice $34,000. (diversity_1)
the parties (Sophia
and Ava) are diverse because they are from different states.
The mark "Trim" for nail clippers. (abercrombie)
"The mark trim for nail clippers would be considered suggestive mark. It suggests a characteristic quality of the product which that the nail clippers are used to trim ones nails. It implies this purpose and function without directly describing it so trim for nail clippers would be classified as suggestive trademark." " (Claude-1)
Claude- 's assertion that the mark "implics this purpose and func- tion without directly describing it" misses fact that "Trim" rectly describes, rather than im- plies, what nail clippers are used for (i.e., the trim nails). Thus, the mark is actually descriptive, and not suggestive.
Ana is a lawyer who resides in Texas. While visiting Louisiana, she meets David, who runs a bike shop. She buys bike from him and uses it to bike back Texas. Right after she crosses the border, the bike seat explodes, in- juring Ana. Ana sues David in Louisiana. (personal_ jurisdiction)
"Yes. There personal jurisdiction over David in Louisiana. Explanation. 1. David is domiciled in Louisiana so there is gen- eral jurisdiction over him in Louisiana. 2. David has sufficient minimum contacts with Louisiana such that he has availed himself of the privileges of conducting business in I Louisiana. He operates bike shop in Louisiana so he purposefully avails himself doing business in the state.. 3. Ana's 's claim arises out of Davids contacts with Louisiana she bought the defective bike from David's bike shop in Louisiana the claim is di- rectly related David's business contacts the state. Requiring David to litigate Louisiana would not offend traditional no- tions of fair play." (Claude-1)
The LLM states that David domiciled in Louisiana, when the facts make no mention of this.
Table 57: Examples of generations for rule-application tasks which are incorrect. The last column provides analysis for why the generation is incorrect.
With respect to analysis, we observe that GPT-4 again outperforms both GPT-3.5 (p 0.01) and Claude-1 (p < 0.01). Explanations which failed to exhibit analysis can be grouped into several categorics. First, some generations will contain just prediction as to the legal outcome, without an explanation (even when the LLM has been prompted to generate one). The same prompt-applied to other samples in the dataset-will elicit explanations containing analysis. Second, we observe tendeney for L.LMs sometimes generate explanations which merely restate the facts and legal rule, without actually offering an explanation for how the outcome reached. Examples of such instances are provided in the table below.
H.3.4 Rule-conclusion
Rule-conclusion evaluates on the same tasks as rule-application, but only requires the LLM to generate a prediction as to the outcome, and not an explanation. We observe that GPT-4 once again outperforms GPT-3.5 (p < 0.01) and Claude-1 (p 0.01). Claude-1 and GPT-3.5 appear approximately level on performance.
The rule-conclusion tasks offer heuristic for characterizing the types of legal inferences LLMs are capable of struggle with. In particular, several of these tasks organize samples into slices, where the samples contained within slice all represent a similar type of fact pattern, and thus interact with the legal rule comparable way. For instance, the hearsay task contains a slice corresponding to "non-verbal hearsay." This slice contains fact patterns where an individual communicates something non- aeneerbllly (e.g., pointing). thus qualifying their conduct "statement" under the hearsay rule. In order make accurate predictions this slice, an LLM
123

---

Fact Pattern (task)
LLM generation
What analysis is missing?
Martha makes special pinwheel hats in her home in Salem, Ore- gon. She only markets her hats on the west coast (Califor- nia, Oregon, and Washington). Alex, citizen of Maine, pur- chases one the hats while vis- iting Salem. When Alex re- turns to Maine, the pinwheel comes off and seriously cuts Alex. Alex sues Martha for dam- ages in Washington. (personal_ jurisdiction)
"No. There is not personal jurisdiction." (Claude-I)
Claude-1 does not provide any explanation for why there is no personal jurisdic- tion, so the explanation does not contain analysis.
On the issue of whether Henry negligently hit Kelly with his car, Kelly's 's statement, "a witness told me after the crash that Henry was on his cell phone while driving. (hearsay)
"Yes, there is hearsay in this situation. The rcasoning is as follows: 1. The statement in question is an out-of-court statement, was made by a witeess who fying in court.. introduced to prove the truth of the matter asserted, which is that Henry was his cell phone while driving.. 3. Since the statement an out-of-court statement being introduced to prove the truth of the matter asserted, it is considered hearsay." (GPT-4)
GPT-4 does not explain how the truthfulness of the out-of-court statement (whether or not Henry was on his phone) relates to the purpose of its introduction (proving/disproving that Henry negligently drove).
testi- The statement is being
Table 58: Examples of generations for rule-application tasks which do not contain analysis. The last column explains why the generation is deficient.
must recognize that (1) the hearsay rule applics to non-verbal communicative conduct, and (2) the non-verbal conduct in these fact patterss is communicative.
Though slices are small- and thus not intended for rigorous statistical analysis- provide some intuition to the source of GPT-4's improvement over GPT-3.5, and the overall areas of strength and weakness for both models. On the hearsay task for instance (Table[5)). the difference between GPT-4 and GPT-3.5 appears primarily attributable improvements over the slices corresponding to non-verhal hearsay and statements made in court. In looking across slices moreover, it's clear that some are comfortably within the realm of model capabilities (e.g.. non-assertive conduct), while others (e.g., not introduced to prove the truth of the matter asserted) still pose considerable challenge.
Another example is provided by the abercrombie task, in which an LLM must determine the relationship between a product and a potential trademark name, by classifying the product-name pair into one five categories recognized by courts: generic, descriptive, suggestive, arbitrary, and fanciful. Loosely, these categorics measure how distinctive product name is for a product, with generic being the least distinctive, and fanciful being the most distinctive. Just as with hearsay, comparing LLM performance insight into the relative most attributable to performance on marks which are suggestive or arbitrary. However, GPT-4 still makes a number of errors for both categorics. Interestingly, performance on deseriptive marks is consistent between both models.
each of these categories provides of improvement (Table (60) Here, GPT-4's improved overall performance appears
H.3.5 Interpretation
On the interpretation tasks, we find that on average GPT-4 outperforms GPT-3.5 (p 0.01), ans GPT-3.5 outperforms Claude-1 (p < 0.01). Here, the larger API-models are highly performant on tasks which involve binary classification over short clauses. Averaged across the 38 CUAD tasks (contract clauses), for instance, GPT-4, GPT-3.5, and Claude-1 all have balanced-accuracy 2 88%. And on proa (statutory clauses), both GPT-4 and GPT-3.5 have balanced-accuracy 2 90%. Notably, performance degrades tasks which contain longer text sequences involve multi-class classification. On the Supply Chain Disclosure tasks for instance which LLMs must classify disclosures which are 1-2 pages in length- the average balanced -cccurccyy of the large commercial models ranges between 74%75% And on the MAUD tasks-which require answering multiple choice questions about merger deals the average ollacced-accuraal of GPT-4 drops to 47.8% accuracy.
124

---

Slice
Slice description
GPT-3355
GPT-4
Non-assertive conduct (n 19)
The fact pattern describes conduct which non-communicative and therefore hearsay.
100%
100%
Statement made in-court (r1 14)
The fact pattern describes a statement that was made in court and therefore not hearsay.
57%
93%
Standard hearsay (n = 29)
The fact pattern describes traditional hearsay (out-of-court statement introduced prove the truth of the matter asserted).
97%
97%
Non-verbal hearsay (n
12)
The fact pattern describes non-verbal communicative conduct that qualifies as hearsay.
33%
75%
Not introduced to prove 20)
fact pattern describes a statement not introduced to prove the truth of the matter asserted, which is therefore not hearsay.
25%
45%
Table 59; Comparison between GPT-3.5 and GPT-4 on hearsay slices. Accuracy reported each slice.
Mark
Mark description
GPT-3.5
GPTT4
Generic (n 19)
The name connotes the basic nature of the productt/ervvcc.
94%
100%
Descriptive (n 19)
The name identifies a characteristic quality of the prod- uct/service.
73%
72%
Suggestive (n 20)
The name suggests, rather than describes, characteristic of the product/service.
38%
70%
Arbitrary (n 18)
The name is real world but has relation the prod- uct/service.
41%
82%
Fanciful (n
The name is made-up word.
84%
100%
Table 60: Comparison between GPT-3.5 and GPT-4 on abercrombie categories. Accuracy is reported for each slice.
H.3.6 Rhetorical-analysis
intoricall-nderstandii tasks, we find that GPT-4 outperforms both GPT-3.5 (p S 0.05) and Claude-1 (p & 0.05). We note several results. First, on definition extraction which requires LLM to extract the term defined by sentence taken from a Supreme Court opinion Iirtiion-nnittt--ieee Incite- Base almost equals GPT-4 in performance (80.6% accuracy to 81.8%). Second, nearly all evaluated models struggle on two tasks requiring I.LMs label the legal "roles" played by either question excerpt from an opinion (function. of peecision_sesee classify text into one of six or more categories
On average across
and oral_ _all_aagueent___mm prgguent.qqusttion questiinnppuppssee Notably, both tasks require the I.I.M to
H.4 Prompt engineering strategies
Finally, we illustrate through series nicro-stuuiess how LEGALBENCH can be used to explore different aspects prompt-cngincering for LLMSs legal settings. We focus on three questions:
1. Can LLMs rely their latent knowledge of rule for rule-conclusion tasks?
2. Does simplifying task descriptions to plain language affect performance?
3. Are LLMs sensitive to the choice of in-context demonstrations?
Reliance on latent knowledge When prompting for general-doman domain tasks like sentiment or topic classifi- cation, prompt-engineen will often rely on the LLM's 's' latent knowledge of the task (5). In topic classification for instance, prompt may the instructions to label whether article about "sports," without offering
125

---

Description Reference
0.5
Ralanced accuracy
0.2
0.0
abercromble
hearsay
diversity
diversity 2
diversity .3
diveriity 4
diveriity
diversity
personal jurisiccioon
ucc common, law
Figure 3: We compare performance of prompts which describe the legal rule to be applied ("descrip- tion") against prompts which reference the legal rule to be applied ("reference"). Error bars measure standard error, computed using a bootstrap with 1000 resamples.
a detailed description of what "sports" refers to or encompasses. Such a description is not necessary, because general-domain terms like "sports" appear frequently in LLM training corpora, and LLMs can learn from these occurrences what general-domain terms mean. Prompting for legal tasks, however, may require different strategy. Because legal terms occur less frequently in general domain training corpora, legal prompting may require practitioners to provide additional background information. For example, general domain LLM may not know what the requirements for diversity jurisdiction are, because diversity jurisdiction is not as commonly discussed in pretraining corpora.
We explore this question through a study of cule-conclusion tasks. For selection of these tasks, we evaluate GPT-3.5 with two zero-shot prompts: reference- cerreneebbaedd prompt and deseription- -basodd prompt. In the reference prompt, the task instructions merely state the rule be applied, i.c., "Determine if the following fact patterns give rise to diversity jurisdiction." In the description-base based prompt, the instructions provide an explicit description of the rule, i..," "Diversity jurisdiction exists when there is (1) complete diversity between plaintifl's and defendants, and (2) the amouut-in-cootrooern and description prompt, can measure whether providing a description of the rule the prompt provides additional performance boost over the LLM's latent knowledge of the rule.
(AiC) is greater than $75k." By comparing performance between the reference
Figure BerreBBpros provides comparison for the different prompts. Interestingly, we find considerable variation across tasks. On tasks like abercrombie, ucc_v_ w_coomon__sse diversity_2 -2, and diversity 4. description prompts appear to offer significant increase in performance. On the other tasks, performance is approxi- mately the same (or even worse). We identify two possible explanations for diverging results across tasks. First, on certain tasks, subsets of fact-patterns are challenging for L.LMs like GPT-3.5, and description-based prompts do not provide sufficient guidance I.IMs to reason through those fact patterns. Second, legal rules may be described to varying extents within pretraining corpora. Hence, tasks where we observe performance im- provements from description-base based prompting may correspond to rules which occur less frequently in pretraining data.
Plain language descriptions of tasks Next, examine the extent which domain specialization in the language of the prompts affects performance. Like experts other specialized domains, lawyers have developed their own language (i.c., "legalese"), which forms the basis for most legal writing and communication. unelear whether-in interacting with large language models through comppinng-awyeee rely on formalistic legal language, instead use simpler plain language. While most large language models are "general domain" and thus less specialized to legalese, formalistic legal language is more precise, and may thus induce more accurate behavior from the model.
should continue to
We explore this question by comparing "plain language" and "technical language" prompts. For subset of LEGALBENCH tasks, we have access to the formal language provided to law-trained annotators when creating task data. By comparing the performance of prompt which uses this language to one which uses Main-languugee version can measure how the technicality of language affects results.
126

---

Rain
accuracy
0.4
cuad audi, rights
cuad insurance
contract nii permiss bie sgreemen__ossessionn
leamed hands
agreement, possession
Figure 4: We compare performance of prompts which describe the task in plain language to prompts which describe the task in technical legal language (for GPT-3.5). Error bars measure standard error, computed using bootstrap with 1000 resamples.
definition classificationn
GPI-3.5
cuad audit rights
cap____p_o__ jo_libliitty
learne_ hands, family
leamnd_ hamne__anndse hands, housing
app115_thas third palplpaaarrtiiinn wwwrriggahncccoe srr___aannncllllcttioon
choice control
averruling
0.5
0.9
Belsnced--ccuracyy
Figure 5: We evaluate GPT-3.5 and Incite-Instruct on five prompts constructed by randomly selecting different samples to use as in-context demonstrations (maintaining class balance the prompt). In the figure above, each point corresponds to different prompt.
conduct preliminary experiments on sample of five LEGALBENCH tasks (Figure On four five tasks, we find that the pangugla prompt significantly outperforms the technical language prompt, by up to 21 points (balanced baalance-acccuracy). Interestingly, catract_nli__ ppe.missibbbl. wwwptpggeeeeeeeeeeeo possession, find the opposite phenomenon holds; the plain language prompt is substantially worse than the technical prompt.
Sensitivity to in-context demonstrations Finally, we investigate the influence of the in-context demon- strations used prompts. Prior work in general domain L.MSs have abserved few-shot performance highly sensitive to the choice of demonstrations [56] 123 [41 We evaluate whether I.LMs are similarly sensitive for legal tasks, focusing on subset of binary classification tasks. For each task merge the train and evaluation split into single dataset, and randomly sample four in-context samples to include in the prompt (two from each with
class), five different times. We evaluated GPT-3.5 Incite-Instrun and plot the balanced-accuraacy each prompt in Figure
generated prompts,
Consistent with findings on general-domain tasks, we observe that LLMs on logal tasks are also highly sensitive to the choice of in-context samples. Notably, this appears to be the case for both GPT-3.5 and Incite-Instrrut.
Prompps made available the LEGALBENCH repository.
127

---

Under permutation test, we find significant differences (p < 0.01) between the best and worst performing prompt for mncite-Instruct (on all lasks), and for GPT-3.5 (on all tasks excepi opp115_ tthirdppatty.aaeee strry_shhiing... difference substantial. partrrd_arttssharnggggg overruling
collection and overruling) For many tasks, the magnitude for instance, the best Incite-Instruct prompt improves upon the worst prompt by over 20 points (balanced- accuracy). Overall, these results suggest that future work is needed to understand how different demonstrations influence performance.
I
All tables
We provide results for each LLM on each of the tasks. Models are divided into four groups based on type: commercial models, 13B models, 7B models, and 3B models. Model names are abbreviated to the family name ensure well-formed tables.
Task
GPT-4
GPTT-35
Claude-1
abercrombie
84.2 / 84.2
48.4 48.4
65.2 63.1
abercrombie
84.2 / 84.2
48.4 / 48.4
65.2 63.1
diversity_1
96.7 / 96.77
86.7 / 86.7
86.7 86.7
diversity_2 2
100.0 100.0
53.3 50.0
73.3 73.3
diversity_3
96.7 / 96.77
83.3 / 66.7
53.3 53.3
diversity_
93.3 / 93.3
70.0 / 66.7
43.3 /3.33
diversity_5
76.6 / 76.6
66.7 66.7
36.7 36.7
diversity_6 6
80.0 80.0
6.7 / 6.7
53.3 / 53.3
hearsay
75.5 / 47.9
55.3 / 23.4
68.1 /41.5
personal jurisdiction
94.0 / 94.0
68.0 10.0
70,0 70.0
successor Jicblssorllaabllity
19.1 / 19.1
15.2 15.2
38.3 38.3
lelemarketing_sakee Baapggg__lll__ellle rule
72.3 / 70.2
48.9 / 42.5
55.3 55.3
ucc_y soc_v_cove _ommonn_aww
97.8 / 97.8
100.0 / 47.8
93.6 93.6
Table 61: Performance on rule-application tasks for commercial models. ness/analysis.
We report correct-
Task
GPT-4
GPT-3.5
Claude-1
rule_qa
72.0
46.0
77.0
59.3
52.7
60.0
nys_judicial_ethies
78.3
74.3
79.1
citation preeiictrreeeeeeeee prediction_caasiffattioo classification
71.3
54.6
61.1
citation_ Betpttppppedddoppppeeeeeee ooeen
15.1
3.8
11.3
Table 62: Commercial models rule-recall tasks.
We conduct the permutation test with 1000 resamples.
128

---

Task
Flan
Llama-2
OPT
Vicuna
WizardLM
rule_qa
0.0
22.0
8.0
14.0
20.0
international httpa/ooonnnliiiieeeeiiiiieeeiio
52.4
50.0
18.0
21.5
49.8
nys_judicial_ethhe
69.1
67.6
63.3
61.3
63.8
citation prrddiciioncllssifeeeeeeo
56.5
47.2
52.8
50.0
52.8
ciation_prediction__pqee
1.9
0.0
0.0
3.8
Table 63: 13B models rule-recall tasks.
Task
Bloom
Falcon
Incite-Base
Incite-Inst.
Llama-2
MPT
OPT
Vicuna
rule_
6.0
8.0
18.0
28.5
22.0
20.0
6.0
0.0
qa
international0.0
10.2
49.5
47.0
27.2
2.4
2.4
3.0
citizenship_
questions
nys_ judicial ethics
62.5
55.8
62.7
54.9
67.5
57.2
59.7
52.3
citation prediction
50.0
50.9
50.9
47.2
50.0
50.0
47.2
14.8
classification
citation prediction
1.9
0.0
0.0
0.0
1.9
0.0
0.0
0.0
open
Table 64: 7B models on rule-recall tasks.
Task
Bloom
Flan
Incite
Opt
rule_qa
0.0
0.0
14.0
2.0
international international_ciizenshhpppp questionn
0.0
50.0
18.9
3,0
jys_judicial_ethin ethics
54.0
58.5
49.9
56.7
citation_ caaatoo_predictionncllattt ccasssiiiinnaassiiiattiiiioo
49.1
50.0
51.9
49.1
0.0
0.0
0.0
0.0
Table 65: 3B models on rule-recall tasks.
129

---

Task
GPT-4
GPT-3.55
Claude-1
seaporatelobbyinng
81.7
59.1
75.8
learned_ hww___aannd_ee_eeeeeeee
87.9
62.1
66.7
81.6
58.6
47.7
learne_hhands_coee
76.2
59.3
58.1
learned Sar____hand_____ee
52.6
54.2
46.9
learned hands_crime
81.0
62.4
59.0
learned_ Saadd__hhnd____d_e
84.0
59.3
53.3
learned Lannd_domestte violence
83,9
60.9
62.6
learned cunds_education
91.1
57.1
55.4
learned hands cepodddenmlyymettt
69.9
67.7
49.3
learned hhands_estates
96.6
59.0
74.7
learned hand__aanddllll
86.2
57.1
50.9
learned hands__aa____ _eaath
87.2
65.0
49.6
learned hands _oouinn
85.0
63.9
58.7
learned hands immigration
98.5
79.9
73.1
learned hands torts
70.6
60.0
53.2
learned_ Laadd__hann__ratae
95.3
49.8
52.3
Table 66: Commercial models on issue-spotting tasks.
Task
Flan
Llama-2
OPT
Vicuna
WizardLM
corporate, corporate_lobbying
55.9
55.9
50.9
50.3
50.2
learned _ands Lands_benefits
68.2
50.0
50.0
4.5
0.0
learned Landd__bssiness
61.5
50.6
48.9
1.1
46.6
72.8
50.0
45.0
0.0
44.6
58.9
49.5
50.0
50.0
0.0
learned har___aannddseeeeee
83.0
50.1
51.5
48.4
26.7
learned_ Baarmed_hand__ddde _dvoore
57.3
49.3
50.0
49.3
0.0
learned handsd_aand____ feeaddd_dmmstic_wvi violence
68.4
48.9
50.0
0.0
0.0
heamme__handseddiee hands_educatiio
89.3
50.0
46.4
50.0
26.8
learned Laiieed_hand__eeee hands_emploommnnt
74.2
49.2
49.9
23.1
0.0
learned hands_estates
67.4
50.0
68.5
49.4
38.8
learned_ saatdd_hans__aaiiiii
5.4
50.1
63.6
49.3
0.0
learned health
66.8
49.6
63.7
50.0
38.1
learned hands housingoungggg
68.7
49.8
49.5
47.8
0.0
learned hands immigration
79.9
51.5
53.0
50.0
53.0
learned hands ttrts
60.6
50.0
50.2
49.8
46.1
learned hands hands_traffic
84.2
49.8
58.6
10.8
38.7
Table 67: 13B models on issue-spotting tasks.
130

---

Task
Bloom
Falcon
Incite-Base
Incite-Inst. 49.7
Llama-2
MPT
OPT
Vicuna
corporate.
40.3
44.8
50.1
50.0
49.6
50.9
45.5
lobbying
learned_
48.5
51.5
50.0
51.5
51.5
56.1
50.0
0.0
hands_
benefits
learned_ hands_
55.7
60.3
50.0
55.7
50.0
50.0
59.8
0.0
business
learned_ hands
45.9
45.0
49.7
48.0
50.0
46.6
48.9
7.5
consumer
learned_ hands_ courts
53.6
50.5
50.5
48.4
49.5
57.8
49.5
0.0
learned_ hands_
50.3
51.6
50.0
54.1
51.0
53,9
51.7
0.0
crime
learned hands_
48.0
50.0
50.0
62.0
46.7
58.0
49.3
0.0
divorce
learned_ hands
48.9
48.9
51.7
59.8
51.1
49.4
50.6
0.0
domestic
violence
learned_ hands_
53.6
62.5
50.0
60.7
50.0
53.6
48.2
0.0
education
learned_ hands
50.6
49.7
49.9
54.8
49.4
51.1
49.7
0.0
employment
learned_ hands_
51.7
51.1
50.0
46.1
50.6
62.4
55.6
0.0
estates
learned hands_
55.0
57.3
49.6
60.8
50.2
56.0
53.3
0.0
family
learned hands_
49.1
52.7
50.0
59.3
52.2
57.1
53.5
0.0
health
learned_ hands_
49.2
49.6
49.8
47.1
50.5
49.3
50.1
0.0
housing
learned_ hands
53.7
47.0
50.0
64.9
50.0
67.9
62.7
0.0
immigration
learned hands
51.2
48.8
50.0
56.0
50.0
48.4
50.0
0.0
torts
learned_ hands_
54.7
50.0
49.8
54.3
50.2
56.5
57.2
13.3
traffic
Table 68: 7B modelslad issue-spotting tasks.

---

Task
Bloom
Flan
Incite
Opt
corporate_ lobbbygg
26.2
51.9
47.6
51.0
43.9
43.9
47.0
53.0
learned _andss__aa_____ _usinesss
51.1
49.4
47.7
44.8
learned maadd__cosumne
39.7
70.8
50.2
38.1
learned_haad saatrr__hnns__ottt
56.2
46.9
47.4
60.4
49.7
60.6
51.9
51.7
58.0
50.0
61.3
56.0
learned hands domestic violence
36.2
49.4
52.9
51.7
learned hands cunddseducation
50.0
67.9
51,8
46.4
learned hands_ empold_mmpoymmet
50.7
46.6
48.0
50.1
learned hands_ _etadds_stttsssssssssss_ss_s___t
42.7
66.3
52.8
62.4
learned Bhamee_handd_ffee famill
52.1
47.6
59.0
64.6
learned hands _ealth
46.9
58.4
53.1
52.2
learned _ands joousng
51.2
51.7
47.1
51.8
learned hands fands_immigration
59.0
73.1
53.7
56.7
learned hands hands_torts torts
44.4
66.2
51.2
56.7
46.8
65.3
46.4
65.3
Table 69: 3B models on issue-spotting tasks.
Task
GPT-4
GPT-3.5
Claude-1
abercrombie
85.3
63.2
66.3
diversity_I
100.0
88.4
83.4
diversity_2 _
99.8
87.3
92.9
diversity_3 3
97.0
89.4
84.3
diversity_4
100.0
90.1
97.9
diversity_5 5
93.2
92.6
81.0
diversity_6 6
90.5
77.3
56.4
hearsay
83.8
69.2
76.4
personal_jurisdicn
91.4
63.3
81.9
pricessor_liabiliin
57.1
52.3
72.7
delemarketing_sakee
rule rule
82.4
63.1
71.9
uec_ www.coomoonaaa
98.8
100.0
88.8
Table 70: Commercial models on rule-conclusion tasks.
132

---

Task
Flan
Llama-2
OPT
Vicuna
WizardLM
abercrombie
42.1
40.0
0.0
22.1
44.2
diversity__
76.0
50.0
55.0
25.5
60.5
diversity_ 2
59.4
62.1
53.9
48.4
57.1
diversity_3
78.6
65.8
50.0
52.6
77.8
diversity_
53.2
68.3
49.6
55.4
82.5
diversity_5
53.5
57.7
52.4
52.7
53.2
diversity_6 6
50.0
50.0
48.1
50.0
54.3
hearsay
64.0
56.5
52.3
48.7
66.3
personal_jurisdictn
62.6
57.9
51.3
0.0
64.0
Saccessor_liabilne
52.7
51.2
26.4
0.0
39.1
69.8
74.3
50.0
63.1
72.7
ucc common law
98.1
77.9
52.5
0.0
79.8
Table 71: 13B models on rule-conclusion tasks.
Task
Bloom
Falcon
Incite-Base
Incite-Inst.
Llama-2
MPT
OPT
Vicuna
abercrombid 7.9
24.2
27.4
34.7
32.6
34.7
17.9
2.1
diversity_ 47.6
51.7
56.8
61.5
73.4
54.9
58.0
55.3
diversity_ 50.0
56.3
50.5
65.7
50.0
50.0
49.8
49.1
2
diversity 51.9
53.6
50.0
50.8
62.8
50.0
49.9
49.3
3
diversity_ 58.3
68.7
50.0
67.9
72.9
50.0
49.7
49.7
4
diversity_ 57.9
50.0
50.4
49.4
51.3
50.0
50.0
46.8
5
diversity_ 50.0
50.0
50.0
44.7
50.0
50.0
49.7
50.0
hearsay 51.7
53.7
50.0
73.2
64.1
61.7
36.4
30.7
personal_
51.8
54.6
40.0
46.1
52.6
43.8
50.6
50.0
jurisdiction
successor_ 21.7
39.1
37.2
32.6
43.4
35.7
38.8
0.0
liability
telemarketing..4 sales
55.3
51.8
58.6
61.3
55.8
54.5
43.6
rule
ucc_v_ common_
50.0
50.0
50.0
50.0
56.6
50.0
50.0
0.0
law
Table 72: 7B models on rule-conclusion tasks.
133

---

Task
Bloom
Flan
Incite
Opt
abercrombie
20.0
31.6
25.3
26.3
diversity_
50.0
50.0
50.3
51.5
diversity_2
50.0
50.0
49.3
52.6
diversity_3
50.0
50.0
46.2
50.0
diversity_ _
50.0
50.0
62.5
51.4
diversity_5 5
51.8
50.0
52.8
50.0
diversity_6 6
50.5
50.0
51.5
50.0
hearsay
51.2
57.5
57.0
49.7
versonal_jurisdiction
50.0
51.1
46.1
50.0
peicessor_liabiliy
21.7
44.6
24.8
14.4
telemarketing saaee rule
44.5
51.4
53.5
55.9
ucc_ v common law
50.0
88.8
50.0
50.0
Table 73: 3B models on rule-conclusion tasks.
Task
GPT-4
GPT-3.5
Claude-1
canada_tax, tali_ttx_____a_aaa
98.9
80.0
76.9
defintiion_cassifcatiio
96.6
80.2
87.9
definition_extractiin
81.8
85.0
82.7
pnnctioo_o__deciisoonssee
43.3
35.2
37.6
legal_reasoning gegala_raasonng__caulllly
84.5
72.1
66.9
oraLLangument_ggusiiinouppppppp
37.4
28.4
35.1
overruling
95.2
88.9
95.4
scalr
77.9
58.8
64.7
textualism tool dictionaries
93,9
65.1
71.2
textualism tool polin
84.7
73.2
70.2
Table 74: Commercial models on rhetorical rhetorical-understen understanding tasks.
134

---

Task
Flan
Llama-2
OPT
Vicuna
WizardLM
canada_tax tax__tt__uuuaa tax_coout_ooucomee
71.7
34.4
1.8
2.9
75.1
sefinition_clasiffcaatte
80.8
51.2
57.4
50.0
81.5
definition_extractin
80.5
85.0
80.1
62.4
82.0
dunction_of_deciise
34.6
18.0
20.8
14.2
13.2
legalreeasoning__aassliy
78.6
52.8
52.3
48.4
46.2
oral oraLaagument_questiinnppupopg
24.5
20.0
15.0
16.1
29.3
overruling
94.2
92.3
78.3
52.7
89.5
scalr
66.5
56.7
20.2
5.0
45.6
textualism tool dictionaries
93.9
66.5
54.9
5.6
61.1
textualism tool_oopllll plain
81.5
72.6
51.7
43.6
74.6
Table 75: 13B models rhetoriiall tasks.
Task
Bloom
Falcon
Incite-Base
Incite-Inst.
Llama-2
MPT
OPT
Vicuna
canada
0.0
0.0
0.0
5.2
35.4
0.0
28.5
0.0
tax_
court_
outcomes
definition_ 54.4
80.2
50.1
65.4
50.0
57.5
58.3
42.5
classification
definition 77.4
77.7
80.6
73.4
84.1
80.9
73.5
4.2
extraction
function 14.2 of_ decision section
2.3
10.2
10.6
16.7
13.9
22.3
0.0
legal_ reasoning_
47.4
59.4
57.6
56.0
53.2
55.5
50.0
38.7
causality
oral_ argument_
13.7
15.5
20.2
27.0
14.5
17.2
14.4
1.2
question_
purpose
overruling
82.3
67.9
50.3
75.2
92.2
72.4
53.3
28.7
scalr
18.6
20.7
22.0
23.0
33.7
25.7
19.3
0.0
textualism_ 48.5 tool_
60.9
55.7
59.6
47.4
59.5
51.0
19.8
dictionaries
textualism_ 55.0 tool_
57.2
61.6
55.9
50.0
60.5
51.0
4.6
plain
Table 76: 7B models
tasks.
135

---

Task
Bloom
Flan
Incite
Opt
canada tax__ax__ _ux_court outcomes
16.0
62.3
11.7
0.3
definition definition_llassifeei classification
51.0
78.5
51.3
57.2
definition exttracioo
59.1
77.6
59.4
70.6
function of dif_deccsion_seee section
9.9
34.8
26.7
14.1
gggal_reasoning_caasllly
50.1
67.5
49.5
55.7
oral_argun pin_argument_questtonnpoop purpose
5.3
19.9
21.5
14.3
overruling
63.4
93.6
54.4
54.8
scalr
17.7
64.5
21.5
20.0
39.7
92.9
55.6
54.8
textualism_tool_pheee
51.5
82.2
50.7
55.9
Table 77: 3B models on tasks.
136

---

Table 78: Commercial models on interpretation tasks.
Task
GPT-4
GPT-3.5
Claude-1
consumer_ massumer_contracts.c
93.6
85.9
90.3
contract nli confidentialityy.o. _frllyo________e agreement
96.3
96.3
92.7
contract contract_nli_exppeeee aliieeppliitiideniiieiee identification
82.4
81.1
65.0
contract_nli wtw_ouluiiiinioonnneiooooo of htwp:/yooooeyeyooooonooee
90.7
83.0
83,4
contract _ii_limmte___eee use
86.6
85.4
80.8
contract nli no licensing
92.5
76.7
79.2
contract aalii_otice_onn_o
disclosure
97.2
97.2
96.5
contract wewippeemisibbeeeeee acquiremetttnt similar information
96.1
96.6
93.8
contract_nli_permisibbee.
80.4
77.7
72.8
contract_nli_ hroereeeniiippmmmisiiee..eeeeeee saloosssalarrrraeee similaar_information information
98.5
99.3
96.3
contract ali permissible post-ggreeetttltteeeeeetttt possession
94.6
89.3
92.2
95.6
92.5
89.4
contract nli Siar_shringg.. witp_wtthhhheeeeeeeeee employees
94.6
94.8
95.9
contract_nli www.iiiggihhrigtitttth.t thirdd-patiies
93.3
75.0
86.6
contract survisaviialloo of _ollliggaai
94.0
74.5
78.3
contract_ga
96.2
93.6
98.7
cuad affiliate license-licensee
90.9
90.9
85.9
cuad_ affiliate license-licensor
92.0
95.5
89.8
cuad antd_antiiaae antannntisstn assignment
91.4
89.1
92.4
cuad_ Lundd_audi__me audia_udihllttt rights
97.9
89.5
92.7
cuad cap_on liability
95.6
94.1
92.9
gunad_change_of cocttgeer____ooorollllo
88.9
89.7
89.2
cuad competitiee restriction exception
84.1
80.0
71.4
cund__oveeant_not_oo_suu
95.8
88.0
89.3
cuad_effice Luad__effective_datt
92.8
75.0
74.2
cuad exclusivity
92.9
89.0
87.3
quad_expiration__u
82.0
87.0
78.8
cuad governing law
99.3
98.3
98.7
cuad_insui jinad_issuranee
99.2
95.3
94.9
cuad ip_
91.7
91.0
89.2
cuad jrre_devvveabl_or wet_peppetul__icnee license
97.5
95.4
88.6
94.3
91.1
85.4
Lind__lienee_ee_ee grant
94.0
90.3
91.0
cuad_ lllglliiitedddamaaeeeeee
96.4
86.4
90.9
p_m__iiimum__mmmmmenttt commitment
89.1
86.1
88.6
cuad most favored nation
96.9
95.3
96.9
cuad no-ddint- of customers
100.0
98.8
92.9
cuad no-solicit of employees
100.0
97.9
96.5
cuad_non-compete
93.0
91.0
90.5
cuad_non-dixparagement
97.0
95.0
87.0
137

---

Table 78 - continued from previous page
Task
GPT-4
GPT-3.5
Claude-1
cuad_nome moad_nontransfcablle ciccr
90.2
82.1
87.6
cuad notice period to terminate renewal
95.9
97.7
96.4
94.6
89.0
77.8
cuad price restrictions
95.7
87.0
ruad_renewal_tem
96.1
95.9
95.6
cuad__reeeue--prffishhiee
95.3
91.2
88.9
cuad rofi-rofo-rofn
88.6
81.9
86.2
quad_ssuree__eee saatcceeeoddesseeeeee
96.6
91.5
94.1
teaedttttinnnion for convenience
96.7
94.2
95.8
cuad_third ttt_tttirr__pp parttd_ppp_____aaa beneficiary
89.7
83.8
82.4
cuad_ uncappeceppddpd liability
85.4
70.4
74.1
cuad_ wwwwumteeddd--ee you you-can caae-can-cattt eat-license
93.8
93.8
87.5
cuad_ wwwlvooolletriittiieeeeee
80.7
68.6
80.1
cuad_ warranty_ durraiion
77.8
81.2
insurance cesurance_policyy__ee pnlicy_interpretatiin
69.6
55.0
64.0
jerew_blocke blocker
100.0
88.9
55.6
maud ability consummate possuemate_concepptt is subject
carveouts
50.0
50.0
31.5
faaud_fnaaciillPr ciinanccallpoont_offoo _tt_:ot_oo_vie________________ ji____hh___soccccossderrrrttt
50.0
38.8
50.0
maud accuracy of fundamental harpeetttttttttteeeeee rws www.wwwwwwb.wwwa....e...oooo standard
29.3
33.3
10.4
63.6
51.0
46.9
aaaud_accuacyy..o http://wwwcciiiiiiioooooooooaaaa brrnggdww__stadddee -nnadderd__aswwrr
20.7
16.2
13.4
maud_
57.4
43.3
24.5
otwpgppiiiiiiiaaoooooooooooooooo covenant)
63.7
68.8
35.3
maud_ hww.pu/www..ooeeeeeeeiiieoeeeerr
50.0
60.8
28.9
maud_change_nn in ja_pee_____________ piippsiieee_oooiiirrrrrrttaeeeen modifier
53.0
48.3
65.2
maud_ gouud_changes_ing _at___nnnn__p____ cacapp_o__tlliii
51.7
47.4
53.8
disproportionate_ingee htcple_imaaaammmiimiei
maud_cor_permittee_innnno_ Baciil_innnnssse jtn__eesposs____jirrvennnng
50.0
52.5
50.1
maud_cor_peee hat____oo__ermteee_ieiii woith_hoard.. fiducciarydeetermmnatte
21.4
50.0
50.6
maud cor standard (intervening event)
0.5
36.5
0.0
mraud_cor_standane cet_oooossaaarrd__sieee (acaaaaddddd_urrrrrorreeeee
40.5
45.5
0.0
maud definition contains knowledge requirement - answer
25.0
33.7
20.7
maud_ wwt._lddefiiiionnnuuueeeeeeeeeee
33.3
30.5
0.5
maud definition dedinition_inclluus includes stock deals
33.3
37.5
27.4
maud ficpuuccarrrcccccppttooonnneonee board wwweeeiinttinnsttaaneeeee standard
maud http:pppptoooobboococccciitotato
40.1
27.5
0.0
(ntsgeer_no_shop)
50.0
48.8
50.0
maud fls_ (1__mmae__stanaa ssaandaddddd
25.0
44.6
22.5
maud wwwggeeeeeecoooooee aaapiiiaann_iiaania Lnw.anciia..ondiiiosssssiiccllcl
54.2
56.0
53.6
disproportionate_ingee saaplee_mpppppeeeee modifier impacc_modiffer
maud includes jit_iuuues_coosssstettttttt petteetttttttthpeeeeeee tetth_past_pacttiie
54.2
55.3
84.2
138

---

Table 78 - continued from previous page
Task
GPT-4
GPT-3.5
Claude-1
15.4
31.9
20.7
maud initial matching rights rightssperiod (ftr)
49.4
32.8
14.0
[etpuirrd_tt__cccur_afteee tn__occur__attrrsiiinnng. - _answer
51.9
51.4
12.5
maud knowledge definition
51.1
49.1
38.5
maud_ strr_sadddddddoooooooooooo cacpoop__reah__yyee representatives
44.2
51.9
49.4
maud ordinary_commee course efforts standard
91.1
70.3
53.6
maud_pandemic, cttttemmmmmooootteetteeeeeeee _rt___ootrr__utttteeee httprrttttttttiiiippeememiiiiiie
48.7
50.0
51.3
maud pouud__padmiic_.mmm Batpemmmccor_ooooe wwwooorrrrrrp_eeeee public_healthy eeeelleeeettteeeeeeeeeeeeeee _veentsspecffiiicccccceeeee speciffc_reference reference to
79.5
70.9
60.8
goveeggmettalllll..mmmmm reepaasees_________eee or_measures
maud rewationalla Laagggtoooooalaaagggeeeeeeeo ( (mae)_appliees_o
57.9
47.2
26.0
maud patpcific__erffrmaaee
51.5
90.6
73.7
maud tail tailtaallliiaaeee peripeeeooddollee length
68.1
39.5
60.4
maud_ _f__yy______oosierrtiiiooo
99.5
82.7
75.1
oppl 15_ 15_ddt5__reeentiiii
67.0
70.5
55.7
oppl15 data security
87.5
84.2
55.6
oppl 1oppin5__onotttraaee
99.1
93.6
90.0
05 first parrt____rt____ collection use
76.7
80.6
63.0
oppl jttpiieenattonalaaa__spiifeeeeee
92.3
82.6
79.4
oppl15_ policppllcc____ policccchanneee
91.9
89.3
83.8
oppl15_thhird_parm http:hiidd_ppttyyssshnnggggoeooo wwwrsshaarrrlllieeen collection
80.1
77.0
71.0
15 wswuuueeraae potr_acccesssee edit and deletion
90.2
87.7
79.3
15 uswr_ueeeueuuu sneer_hhoice_enee control
82.9
79.3
71.3
privacy_oolicy_cntilmeen
85.5
78.8
89.6
privacy_policy_ge
71.3
65.5
63.0
proa
99.0
90.6
88.5
ssla companyy_efendlee
65.0
65.3
16.5
ssla individual defendants
29.6
25.8
11.1
ssla ssla_plaintiff
92.0
86.5
86.7
sara_entailment
86.8
68.4
67.6
sara numeric
8.3
4.2
6.2
71.5
69.5
74.6
supply chain disclosure best, catl_ppratce_eaat audits
74.4
76.6
75.5
76.6
77.7
77.4
hupply_chaine discllouur_bbes__paatiiee training
83.3
87.1
85.3
68.3
59.4
64.3
77.0
80.4
75.5
audits
81.6
83.7
80.0
71.2
67.3
75.8
htaply_chainndisclosuee http:chiin.bcllluuur...ddioooood
89.1
83.0
75.6
139

---

Table 78 - continued from previous page
Task
GPT-4
GPT-3.5
Claude-1
56.6
62.0
67.6
unfair tos
9.1
13.7
5.5
Table 79: 13B models on interpretation tasks.
Task
Flan
LLaMA-2
OPT
Vicuna
WizardLM
consumer_contraee contracts_qp
92.6
68.1
24.8
37.9
67.8
contract nli, confidentiality. _ofllooo____ooooo of_agreement
85.4
50.0
54.9
48.8
contract ali all__expliit
81.4
49.4
51.6
50.0
67.0
contract_ nli wwwliinncnnnnneec of oeroo___verhlllyyoooveeee
56.2
50.0
53.0
50.0
58.2
information
contract nli limited use
71.1
44.5
60.6
49.5
57.1
contract_nli _ii__oni_______ _iniccciing
54.2
53.3
47.0
51.2
58.4
contract nli notice compelled disclosure
90.8
52.1
61.3
55.6
73.2
contract_nfe nli pew.iesiiiieeeeeeeeeeeeeenoo
89.3
50.0
45.5
52.2
61.2
dimilar_information
contract ali pwwmiseible copy
84.3
47.8
47.8
50.0
56.2
contract_nli nli wwtll/siemmmseeeeem ponnnaee_iiippmmmiiiieee_ddede
httpmissibbl_ddeellommee..oo of_
98.5
50.0
55.1
54.4
86.0
iinilar_information
contract nli hatmmiisibbleppotttggreeeeet possession
93.4
48.8
44.0
50.0
52.5
contract returntttmmo of confideeniiill information
89.4
50.0
44.3
51.3
66.2
contract nli wiiisshhahnngiiie hhipgg__tthhhhhhee wittriin_wittheepleeeee__
92.2
48.2
70.5
50.0
65.7
contract nli sharing with third-parties
86.7
49.5
50.9
50.5
64.4
contract_ nli o oollooggooooiiionsss
74.6
50.0
44.8
50.6
59.6
contract
96.3
82.7
56.8
73.1
35.9
cuad affiliate license-licensee
83.8
58.1
49.0
50.0
65.7
90.9
72.7
53.4
50.0
54.5
cuad anti-assigmment
85.0
52.2
48.1
50.2
76.8
boadd_audit_righh
87.1
51.6
71.7
50.7
58.6
cuad cap_acpppp__ on liability
81.5
74.0
0.6
50.0
$7.5
cuad_ cuad__chngee_f__eoe
75.5
56.2
52.9
49.0
74.8
cuad cooooppeiiveeeeestttttiooooooooo restriction exxception
81.8
$1.8
38.2
50.0
50.0
cuad covenant _wt_noot_____eeooeoe to sue
86.0
66.6
49.4
50.0
$6.5
ciindddeffcetve_ddee _ate
92.8
50.0
50.4
48.7
58.1
quadd_exluusing
84.0
86.1
61.7
49.6
58.7
cuad_ cond__expration_ddee date
60.5
51.6
53.1
51.0
73.2
cuad Lund__oovrming_larg
99.5
90.6
71.7
59.5
54.3
cuad insurance
90.2
55.5
66.3
55.0
55.7
cuad_ip_ htwppp/wwwesep_sssippppeeep assiggmment cuad irrevocable or perpetual license
89.8
74.0
49.8
49.7
59.9
95.7
85.4
67.9
50.0
73.9
cuad joint ip ownership
76.0
53.6
50.5
50.0
60.4
140

---

Table 79 - continued from previous page
Task
Flan
LLaMA-2
OPT
Vicuna
WizardLM
cuad_ Laa__llceeeeeeeee grant
92.8
62.2
50.8
49.9
68.7
cuad liquidated damages
73.6
50.9
48.6
49.5
71.8
62.3
54.5
51.6
49.9
47.0
cuad most favored nation
75.0
56.2
43.8
50.0
57.8
cuad_ mond__no-ollittcoo Bttpliitttt__u_sttteeeee
97.6
57.1
56.0
50.0
85.7
cuad__nosooicii_oo__mmpp
95.8
97.9
69.0
50.0
75.4
cuad non-compete
91.6
52.7
46.8
49.3
$5.2
cuad_ gundd_non-dispanee diseaaageent
83.0
73.0
67.0
49.0
74.0
cuad_ not--trseetaaeebbelee _
69.6
55.0
54.2
49.4
75.6
92.8
50.0
54.5
50.0
83.3
cuad quad__post-terminaioo services
88.7
50.7
51.4
49.6
62.5
cuad_ cuid_price____pppppppppc wwl_pprccceeeseeeeeee restrictions
76.1
71.7
50.0
47.8
50.0
cuad renewll term
89.1
50.5
50.3
57.5
83.9
cuad_ www.eee--proottt...... cuadd_reecuue-prfffftshinnnn sharrin
72.1
58.4
54.5
49.7
52.7
cuad roff-rofo-rofn
65.1
50.4
54.1
50.0
59.6
quad_souree
66.1
58.5
67.8
50.0
52.5
cuad termination for convenience
91.4
50.9
49.8
55.8
84.2
cuad_ taa___hhrrdd__y__beee benefccrenyy
85.3
54.4
63.2
47.1
69.1
cuad uncapped liability
50.3
55.4
46.6
51.0
74.5
83.3
52.1
64.6
47.9
54.2
cund_volume_rrstriciion
55.6
50.0
55.3
50.0
54.0
cuad warranty duration
74.7
57.5
55.9
50.6
63.1
insurance_poiie hhcp:epoooccccclllleleeeeeee polirpocccciciterrtttttooon
44.8
46.6
38.0
13.2
51.9
jerew_ blocker
86.7
66.7
51.1
50.0
www.cccmmmteecoomm iotppatpts.ccss__________ is_
50.0
47.3
51.8
50.0
subject_to to www_mmaaaeeeaeaaae car_ccurs
maud financial_ponn of view tt__te___tte_________ sole
53.6
51.0
50.0
53.1
consideration
maud accuracy_oor of fundamental Ias
12.5
33.3
33.3
31.7
33.3
bringdowo_standae sttandrrddddddddda
maud cuuuddaccuray__se _f____________ oof_ttaret_geeeee
sanget_general_wv.
46.6
50.0
49.1
49.8
50.0
bringdown _tminttiinngg.ee answer
maud_ louu_accuracy_ole sactenccyoo__trrgge wowooofatttoeeeeeee capptttaapaaaaiiii____oo
6.1
25.8
26.9
20.9
24.8
hhariii___shares__binodowan standard__ answer
maud additional matching rights fightt_pperiod___ for
0.0
25.0
19.8
18.4
12.9
modifications (cor)
maud_ applucdappiiationn..o buyer_consene
63.4
47.8
45.0
3.1
49.4
requirement (nnggraemen__negattiveeeeeee interim covenant)
maud_ beuld__uyer__oom wtwprrocoonettteeiiiieeennnnnooo course)
34.4
56.5
44.4
45.1
50.0
maud change_innnn.e in _i___an___________ haw__subectttoo to__
50.0
55.9
46.6
21.3
disproportionate_impee modifier jiipact_modiffier
141

---

Table 79 continued from previous page
Task
Flan
LLaMA-2
OPT
Vicuna
WizardLM
maud_ maud_changen chaplechahgs____ee in _r__aa__or_________
50.0
57.5
47.1
20.6
0.0
modifier
maud_cor_peetee cerooo_eeemttted____spssss intervening_event
50.0
57.6
58.8
26.8
47.0
maud permitted with board fiduciary determination oolliloooooooooooooooooo
50.0
50.0
51.2
46.7
42.0
maaud_co_sstaalee wtw.saaaaddmmtttreeeiiiii event)
0.0
24.0
16.7
10.0
23.3
maud_ cor standard (sspaliir_d__errr__orroeeeeeeeee offer)
11.9
16.8
3.0
0.0
26.9
maud www.eenittonnooniiiieie knowledge_______
0.0
28.9
25.0
24.5
24.1
requirement____nsverr
maud definition defiintiooninncuueeee includes__sneeeee _assss_ssssal_ss
2.8
35.4
32.7
0.9
34.2
maud definition_inccuues includes stock deals
4.6
31.8
24.7
30.1
17.3
maud fiwwciarryeeeeeeeeaccc _xccppticeeeeen_____ board_
1.2
6.5
12.5
0.5
14.1
Aetermination_standaae standard
maud -eteiiaryeeeeptionn board determination. trigger (no (no_shop)
50.0
59.2
44.7
8.8
48.8
maud_ [t_(mm))_ssaaddr
17.1
5.0
24.5
25.0
4.6
maud general economic and financial conditions subject to disproportionate impact modifier
50.0
53.6
52.4
6.0
0.0
ceaddd_nnudus__ssseee Latpee__tttth__ttt_______ _ast__paccttce
62.3
50.0
52.6
53.2
61.0
maud initial matching wwwptrgggghthrgeeeee ppriio___eidd___ee (cor)
0.0
8.3
20.8
0.9
14.2
maud_ initidinnialllliiiii caantcinng_rghht__sooo
0.0
13.3
23.2
19.1
18.8
maud_ fedddd_iieeenngg__n Seteeniiiineeeeeeeeeee eeeeeteeeeeeeeeeeeeeeee -eequireee to_rd_tttrucurrreeeee
39.2
46.2
48.9
47.1
47.1
after signing_-n_anne -_nswer
maud_ cerellddge__eeiiittioo
50.6
46.0
48.4
51.2
maud_ conudliabilityy. standarddddddd... for no-shop__heeaee breach_________
jiability_standard__
48.7
48.7
49.4
0.6
50.0
representaaivvss
maud wwwdinarr_couuuree course efforts standard
81.1
57.8
33.7
0.0
67.3
maud_ saauudd_andmiccoogg cacrscoo_____hhh cacrr__oorottrrttteeeeeee public_ health _eeett___et_____
48.1
52.2
46.2
7.6
31.0
_ubject_to totttto_oogg dioippropootiooatte cettttiooonttiipptttttete ltmpjictppooiiiiiiie modifier
maud pandemic other _ublicuuullleeeee health
50.0
50.0
51.9
50.7
37.4
event_specificc reference_to tat_tt___aaaaa to_pandemic-rclatee
petormnenta_rrspoosss_orr _or_mccsur_sssss
maud rew.aetoaaaaaaale Janguage_
to
51.0
44.2
1.4
51.7
5.1
maud_ coudd_spcific_pprrfrmaaac
94.9
52.1
50.0
0.0
56.7
maud lail pelppeerrrippppeeeee
25.8
51.5
52.5
13.5
34.6
maud hawpelleeeeeeeeeeeeeeeeeee connsiddeaationn
73.6
39.1
30.2
27.8
27.2
55.7
51.1
45.5
50.0
63.6
Soppl15_dtta_eecul
75.1
49.8
51.2
55.5
59.4
oppl15_de 1i15__do_ot__________ ho__not_racck
79.1
50.0
42.7
51.8
88.2
oppl
75.0
67.0
68.5
52.3
55.2
15 iitt_iiittrnatinnal.... and specific audiences
80.
59.5
18.2
50.4
66.2
oppl 1pp_1__icy__changee
70.5
64.9
60.9
52.8
56.5
142

---

Table 79 continued from previous page
Task
Flan
LLaMA-2
OPT
Vicuna
WizardLM
oppl paatiidd_attshhhhnnnntgo
71.4
54.9
58.8
52.4
60.9
oppl15 user access. and deletion
75.4
54.3
60.9
49.1
59.4
oppll1__ser_choice_coo
80.9
53.7
50.0
47.5
privacy privacy_policy. htttppoooyettlileeeeeee entailment
58.9
56.2
50.1
65.9
privacy_policy_aa
52.5
50.5
50.9
0.0
55.6
proa
94.8
76.0
52.1
50.0
80.1
ssla company wwwppaamneeeeeeeeieeooeee
defendants
34.4
63.4
56.6
3.1
2.8
ssla ssla_individual individuul_defenddats
21.9
21.6
16.2
0.0
0.0
ssla ail_ppaanttff
88.8
26.4
31.6
0.0
0.0
sara_entailment
35.3
58.1
48.9
15.4
50.0
sara_mume cur__nueric
1.0
0.0
0.0
0.0
0.0
supply_chiee chain desclosure_best__ee
73.6
49.4
48.8
67.2
52.1
accountability
69.2
49.2
66.6
46.5
65.7
supply_cie chain diwpldddusccbbeeeeeeee bestte__ee_____eeeeeepee practic...
69.9
51.6
56.1
67.2
64.8
certification
supply_chain_diseee httpr//iiiiilleeesssssrriiiiiiee
81.0
49.7
49.4
71.5
64.9
supply chain disclosure best_ practice. verification
70.5
49.1
50.8
51.6
52.5
supply_chain_disee httpchaiin_diccouurrrddsssssssdd accountability
80.7
49.4
45.8
59.1
58.8
disclosure_disccosseeeeeeoe supply supply_chain_eeee baclly_hhhnniniiiii discoosdddsssseeeeeeee disclosed audits
83.3
49.3
48.9
64.1
64.7
68.7
49.4
53.8
61.6
59.3
87.0
49.2
48.0
58.1
62.8
supply_chatee
64.3
49.3
48.6
58.0
50.4
unfair tos
10.0
12.8
10.0
8.3
Table 80: 7B models on interpretation tasks.
Task
BLOOM
Falcon
Incite-Rase
Incite-Inst.
LLaMA-2
MPT
OPT
Vicuna
consumer
57.9
42.4
40.5
14.7
34.0
contracts.
contract_ nli confidentiality_ of
53.7
64.6
65.9
59.8
50.0
52.4
57.3
45.1
agreement
contract_ nli explicit_
61.6
68.9
50.0
50.0
59.9
34.8
identification
143

---

Task
BLOOM
Falcon
Incite-Base
Incite-Inst.
LLaMA-2
MPT
OPT
Vicuna
contract_
50.0
67.5
66.7
60.9
50.0
49.3
55.5
33.1
nli
inclusion_ of verbally
conveyed information
contract_ nli_ limited
56.2
46.0
59.0
66.3
51.1
61.8
61.7
24.8
contract_
49.4
44.6
51.9
56.3
48.2
45.9
49.5
15.9
nli_no_ licensing
contract nli_
51.4
64.1
50.7
65.5
68.3
27.5
notice_
compelled_ disclosure
contract nli permissible acquirement of similar information
31.5
26.4
53.4
50.0
39.9
contract_ nli_ permissible_ copy contract_ nli
49.9
36.8
46.4
57.6
58.9
55.0
49.6
33.2
49.3
43.4
59.6
44.1
50.0
54.4
53.7
41.9
permissible development of_ similar_ information
contract nli_ permissible_ post-
43.1
53.2
55.3
44.5
2.4
agreement_
possession
contract_ nli
57.7
50.2
56.5
75.6
5.9
return of confidential information
contract nli sharing_ with_ employees
55.6
61.6
48.2
70.3
50.0
48.1
58.9
6.1
144

---

Task
BLOOM
Falcon
Incite-Base
Incite-Inst.
LLaMA-2
MPT
OPT
Vicuna
contract_ nli_
49.5
48.1
39.3
53.7
50.0
48.4
48.7
11.9
sharing_
parties
contract_ nli
49.5
55.1
41.2
43.2
50.0
41.7
49.3
1.8
survival_ of obligations
contract
14.8
7.7
11.4
87.7
31.9
9.0
39.0
50.3
qa
cuad affiliate_
53.0
50.5
68.7
71.7
50.0
69.2
66.2
34.3
license- licensee
cuad_
45.5
70.5
64.8
85.2
50.0
76.1
52.3
38.6
affiliate_
license-
licensor
cuad anti- assignment
47.7
33.2
60.4
76.2
58.8
56.5
55.1
37.8
cuad_ audit..
67.4
59.7
80.0
71.5
50.7
52.5
72.8
28.2
rights
cuad_
41.8
42.1
40.4
57.4
50.2
70.3
37.0
16.8
cap_on_ liability
cuad_ change_
55.0
49.8
57.0
67.8
50.0
56.0
55.5
39.2
of_conto
cuad_
37.3
30.9
50.0
47.7
48.6
46.8
36.4
16.4
competitive_
restriction
exception
cuad_
47.7
48.1
67.9
78.2
57.5
71.8
50.0
0.0
covenant_ not to sue
cuad_ effective_ date
56.8
60.2
53.0
46.6
50.0
44.5
65.7
6.4
cuad_..
62.9
57.1
66.4
65.4
63.4
56.7
60.0
14.4
exclusivity
cuad
75.9
62.7
55.5
67.6
50.1
52.9
85,0
10.5
expiration date
cuad
79.3
77.3
28.5
68.2
55.8
66.3
73.1
17.6
governing
law
cuad_
83.2
66.8
65.4
72.0
50.
72.7
77.2
39.1
insurance
145

---

Task
BLOOM
Falcon
Incite-Base
Incite-Inst.
LLaMA-2
MPT
OPT
Vicuna
cuad_ip_
53.0
65.8
61.1
73.8
51.6
60.4
70.8
14.4
ownership_
assignment
cuad_
57.5
58.6
72.1
83.2
52.9
59.6
80.0
46.1
irrevocable_
perpetual_ license
cuad
58.3
60.9
55.2
72.4
30.2
joint ip ownership
cuad_
62.7
60.3
65.5
77.0
63.0
39.0
67.9
20.8
license_
grant
cuad
54.1
65.0
65.0
70.5
50.0
51.4
57.7
36.4
liquidated_
damages
cuad_
58.4
55.3
49.9
55.3
59.2
31.3
minimum
commitment
cuad_ most_
51.6
48.4
60.9
59.4
51.6
57.8
48.4
23.4
favored_ nation
cuad_no- solicit_of_ customers
47.6
38.1
61.9
77.4
50.0
51.2
27.4
22.6
cuad no- solicit_of_
39.4
35.9
54.9
80.3
69.0
70.4
42.3
19.7
employees
cuad_ _on- compete
32.8
42.5
55.2
67.4
63.3
52.3
29.0
42.3
cuad disparagement
41.0
50.0
64.0
70.0
64.0
51.0
47.0
34.0
cuad non- transferable_
67.2
65.9
56.5
78.0
50.0
48.5
68.5
27.9
license
cuad_ notice
50.0
58.6
50.5
76.6
50.0
50.9
81.5
35.6
period_to__
terminate.
renewal
cuad_post-
48.1
45.7
60.0
57.2
50.0
64.7
57.7
20.0
termination
services
cuad_ price_
58.7
43.5
45.7
58.7
50.0
54.3
52.2
41.3
restrictions
cuad_
50.3
57.3
45.1
75.6
50.0
53.1
82.1
35.2
renewal_
term
146

---

Task
BLOOM
Falcon
Incite-Base
Incite-Inst.
LLaMA-2
MPT
OPT
Vicuna
cuad_
54.3
57.9
50.6
$2.3
50.0
62.4
11.1
revenue-
profit_
sharing
cuad rofr-
54.6
43.3
55.9
57.0
50.0
59.9
53.0
20.6
rofo-rofn
cuad_
75.4
59.3
62.7
65.3
52.5
51.7
79.7
24.6
source_ code
escrow
cuad_
70.5
67.7
47.9
83.7
50.0
49.8
64.4
42.8
termination_ for convemience
cuad_ third_
70.6
50.0
69.1
79.4
50.0
67.6
70.6
29.4
party
beneficiary
cuad
46.9
53.4
53.4
61.2
51.0
77.9
36.4
33.0
uncapped
liability
cuad_
72.9
62.5
75.0
79.2
62.5
60.4
72.9
41.7
unlimited-
all-you-
can-eat-
license
cuad_
52.8
47.5
55.0
54.0
50.9
52.5
63.0
42.2
volume
restriction
cuad_
67.5
59.4
57.8
64.1
51.9
54.4
65.3
5.3
warranty_
duration
insurance_
34.5
42.8
35.4
36.9
43.2
34.9
33.7
29.1
policy
interpretation
jerew_ blocker
45.6
46.7
47.8
51.1
58.9
57.8
56.7
11.1
maud ability
42.3
48.2
30.9
50.0
50.0
0.9
50.5
45.5
consummate concept is_ subject_
to_ m__ carveouts
maud_
48.5
56.6
45,9
43.9
48.5
4.1
63.3
50.0
financial point_of_ view_is the sole. consideration
147

---

Task
BLOOM
Falcon
Incite-Base
Incite-Inst.
LLaMA-2
MPT
OPT
Vicuna
maud_
33.3
34.5
31.7
35.9
34.4
31.6
37.5
33.3
accuracy_ of fundamental target_ rws bringdown.
standard
maud_
50.0
45.8
53.7
48.3
51.7
50.8
53.9
50.0
accuracy of target general
rw
bringdown_ timing answer
maud_ accuracy_
30.7
30.2
25.1
20.2
22.1
22.3
24.7
18.1
of' target_ capitalization
(outstanding shares) bringdown. standard
maud_ additional_ matching. rights_ period_ for modifications.. (cor) maud application of buyer consent requirement_ (negative_ interim_ covenant)
19.5
18.3
21.1
16.3
22.0
8.4
20.3
20.0
47.8
40.0
50.0
50.9
50.0
47.5
43.1
maud_ buyer_
50.6
54.1
49.7
50.0
50.0
7.4
57.5
50.0
consent
requirement.
(ordinary_ course)
maud change law subject_ to disproportionate. impact_
36.4
45.8
11.2
49.4
50.6
12.2
50.0
50.0
modifier
148

---

Task maud_ changes_ in in_gaap_ or or_other_ accounting principles. subject__
BLOOM
Falcon
Incite-Base
Incite-Inst.
LLaMA-2
MPT
OPT
Vicuna
35.9
48.4
10.0
53.3
50.6
50.0
50.0
disproportionate_ impact_ modifier maud cor permitted_ in_ response_ to intervening
47.6
62.5
47.5
7.3
65.6
50.7
event
maud permitted_ with board fiduciary
45.8
42.1
37.2
48.8
50.0
13.5
49.4
49.4
determination. only maud_ cor
26.7
9.3
10.9
16.7
21.6
13.8
16.7
16.7
standard (intervening...
event)
maud_ standard (superior offer)
16.8
7.8
7.0
15.8
17.8
9.7
11.5
6.4
maud_ definition_ contains_
25.3
26.9
20.8
23.1
22.8
28.0
25.3
25.0
knowledge
requirement
__answer
maud_
23.9
29.4
18.2
28.9
32.8
6.8
35.1
28.6
definition_
includes_ asset_
deals
maud definition_ includes_ stock_
27.7
16.4
5.0
24.5
18.4
7.6
17.2
22.6
deals
149

---

Task
BLOOM
Falcon
Incite-Base
Incite-Inst.
LLaMA-2
MPT
OPT
Vicuna
maud_
4.3
9.6
6.9
14.6
9.9
13.7
1.0
fiduciary_
exception_
hoard
determination
standard
maud_
48.8
44.7
43.3
49.3
55.8
20.1
57.0
42.1
fiduciary_
exception_ board
determination trigger
(no_ (no_shop)
maud fls_ (mae)
38.3
16.0
13.0
28.6
1.8
1.8
24.6
0.0
standard
maud general economic and financial_ conditions...
54.8
62.5
38.1
48.8
50.6
14.3
50.0
50.0
subject
to disproportionate impact_ modifier
maud includes...
50.0
48.2
61.6
49.6
52.5
5.4
52.8
50.0
consistent
with_past past_
practice
maud_ initial
20.7
11.2
21.5
23.4
27.2
10.8
21.0
18.9
matching.
rights
period (cor)
maud initial matching.. rights_ period_ (ftr)
20.6
14.5
16.4
11.1
22.8
7.7
20.0
16.9
maud_ intervening event_-_ required_ to occur after_ signing_-_
50.2
51.5
43.4
50.0
42.6
43.5
47.5
50.0
maud knowledge__ definition
47.1
46.8
40.0
46.7
51.8
49.3
50.7
45.5
150

---

Task maud_
BLOOM
Falcon
Incite-Base
Incite-Inst.
LLaMA-2
MPT
OPT
Vicuna
59.6
50.0
50.0
53.8
50.0
46.8
58.3
50.0
liability
standard for_no- shop_
breach
by target non-do
representatives
maud
34.2
32.8
41.9
32.9
77.4
32.7
34.2
33.8
ordinary.
course_
efforts_
standard
maud pandemic otherr_ public_
48.2
10.8
49.4
48.9
51.3
8.5
47.5
49.3
health
event
subject_ to_
disproportionate. impact_ modifier
maud_ pandemic_ or other_ public_ health_ event
41.7
51.9
46.0
48.1
50.0
4.2
50.8
50.0
specific_
reference
to pandemic- related_
governmental_ responses_ or measures
maud_
37.0
47.1
18.8
50.0
7.1
16.5
28.3
46.4
relational_ language_ (mac)_
applies_to to
maud
60.4
50.0
49.2
43.5
58.5
0.6
39.4
50.0
specific performance
maud
6.9
32.0
25.3
31.4
63.9
14.2
8.6
3.1
tail.
period_
length
maud
25.0
28.5
27.1
25.3
39.3
24.3
25.0
type type_of_ of consideration
15 data
48.9
37.5
46.6
50.0
50.0
50.0
51.1
42.0
retention
151

---

Task
BLOOM
Falcon
Incite-Base
Incite-Inst.
LLaMA-2
MPT
OPT
Vicuna
oppl15_
53.4
60.3
53.8
63.6
50.2
49.8
63.7
45.9
data
security
opp115_ do not track
43.6
34.5
47.3
69.1
50.0
49.1
31.8
45.5
opp115_ first_ party
63.9
59.3
69.5
69.9
61.3
59.2
70.4
46.4
collection
use
opp115_
58.1
60.3
50.1
64.9
51.1
52.7
57.5
37.4
international
and
specific_
audiences
opp115_ policy_
68.2
66.2
55.6
55.9
50.0
50.0
74.5
44.0
change
opp115_ third
53.8
$7.7
50.9
68.4
50.4
50.3
54.1
36.5
party_
sharing_
collection
oppl15 user
60.2
51.5
56.1
64.5
54.7
50.8
59.0
41.7
access,
edit_and_
deletion
opp115_
40.0
46.5
48.6
47.4
44.2
user
choice_
control
privacy- policy
51.4
49.3
56.8
58.1
57.7
64.1
52.8
26.6
entailment
privacy_
50.0
49.7
52.0
56.3
50.2
50.0
50.4
0.1
policy_qa
proa
$2.1
56.2
50.0
52.1
58.3
51.0
47.9
ssla..
35.5
44.9
54.8
54.1
60.7
51.0
50.9
0.0
company_
defendants
ssla
14.3
17.4
20.8
19.6
23.1
20.8
13.9
0.0
individual
defendants
ssla plaintiff
26.1
9.3
52.3
64.4
60.5
76.0
59.8
0.0
50.4
50.0
50.0
51.1
50.0
50.0
50.0
0.0
entailment
sara
2.1
1.0
2.1
0.0
0.0
2.1
1.0
0.0
numeric
152

---

Task
BLOOM
Falcon
Incite-Base
Incite-Inst.
LLaMA-2
MPT
OPT
Vicuna
supply-
55.8
50.0
58.4
50.4
56.8
32.7
31.5
chain
disclosure
best
practice
accountability
supply_
1.5
71.6
55.9
63.0
55.8
42.2
31.1
8.7
chain_
disclosure_
best
practice
audits
supply_
2.8
67.1
52.1
57.6
52.3
40.8
51.5
11.3
chain
disclosure best_ practice
certification
supply_ chain_
6.6
52.3
50.9
55.4
49.3
56.2
50.5
30.7
disclosure best
practice
training
supply_ chain_
4.5
55.8
49.8
54.3
49.1
36.0
38.1
13.9
disclosure
best practice
verification
supply_ chain
2.1
48.2
49.7
48.7
49.0
24.6
20.4
15.7
disclosure
disclosed
accountability
supply_ chain_.
3.6
50.2
48.7
49.6
51.5
52.0
49.7
16.3
disclosure_
disclosed
audits
supply_ chain disclosure
3.2
50.5
52.0
52.2
54,9
45.8
32.3
20.1
disclosed
certification
supply chain
5.6
48.6
49.1
49.2
49.3
44.0
39.5
10.1
disclosure
disclosed
training
supply chain_.
48.9
49.6
48.6
48.5
49.3
23.1
disclosure_
disclosed_
verification
unfair_tor
8.3
12.3
12.7
8.7
10.6
11.0
10.5
0.0
153

---

Table 81: 3B models on interpretation tasks.
Task
BLOOM
Flan
Incite
Opt
consumer_ ceasumer_contrattssog
0.8
93.2
46.1
31.7
contract nli confidentiality..o. of agreement
57.3
86.6
56.1
46.3
contract_nli: _li_eeppiiiit___c_pppp
p_texppiciiiidenffcttioonn identification
57.2
81.2
63.6
53.3
contract_nli_ wtw.oaiiiiiinnoonnn_oooooo of wttp:/oooveeee__oooonooee
50.1
82.2
67.8
60.9
contract _ii_limmte___eee use
63.1
72.8
60.1
63.0
contract nli no licensing
50.0
65.1
43.5
38.5
contract pall_notice__eee caceiiccooncoompeee disclosure
57.0
66.2
65.5
71.8
contract pewippeemisibbeeeeee acquiiemetttnt similar information
46.6
54.5
contract_lli_pemmisibbbee
51.3
83.7
55.3
56.2
contract_nli hoomreeeniiieemmissiieeeeeooooed salossssmmarrraaeee infoommatono
60.3
75.0
49.3
59.6
contract ali permissible postalgreeetttttaeeeeentt possession
43.8
66.3
43.4
42.4
51.6
82.3
61.3
51.9
contract ali siar_shaingg.. with_wtthhhheeee employees
53.3
69.7
47.1
52.8
contract_nli www.:iiiiishrrngittttttt wiittttttirr-ppee third--prtiies
50.0
80.5
51.4
49.6
contract survisaviialloo of _ololiggaaions
50.0
72.9
49.4
44.1
contract_ga
35.9
0.0
82,9
44.6
cuad affiliate license-licensee
66.7
60.1
68.7
cuad_ affiliate license-licensor
48.9
77.3
71.6
50.0
cuad intd_antiiaae antannniisstn assignment
50.0
74.7
51.5
33.7
cuad_ Landd__udi__mg audit_udiilllll rights
50.0
56.3
56.2
64.6
cuad cap_on, liability
49.2
50.9
49.4
26.8
cuad_ehange__ae _octtggrr____ooooolll
51.4
59.6
60.8
47.4
cuad competitive restriction exception
46.4
73.2
45.5
40.5
cund__oveennt_not_oo_uuu
50.0
59.1
49.4
44.8
cuad_effic Luad__effective_datt
51.7
92.4
47.0
65.7
cuad exclusivity
64.6
55.2
53.9
60.9
quad_expiration__u
50.2
67.7
52.1
69.3
cuad governing law
56.8
84.4
50.5
74.8
cuad_insee jinau_issuranee
$3.5
53.9
57.5
70.3
cuad ip_
49.0
57.6
$7.3
65.1
cuad irreddevvvecbleor wwt_ppppptal_llceeeeee license
51.8
76.1
53.6
71.1
48.4
65.1
64.6
68.8
Lind__lienee_ee_ee grant
50.1
69.1
51.4
69.2
cuad_ liqgiddatedddmaaaeeeeee
50.0
52.3
57.3
34.5
p_m_miiimuu__mmmmmenttt commitment
52.7
54.1
50.6
57.0
cuad most favored nation
50.0
57.8
56.2
43.8
quad_no-solice of customers cuad no-solicit of employees
47.6
59.5
32.1
48.6
63.4
66.9
42.3
cuad_non-compete
48.0
63.1
56.1
30.3
cuad_non-dixparagement
49.0
57.0
60.0
44.0
154

---

Table 81 continued from previous page
Task
BLOOM
Flan
Incite
Opt
cuad_nee mond_nontrannfcablle cinetrannfcrahll__ccccee
license
56.8
58.7
50.9
62.0
cuad notice noticeeperioo to terminate renewal
59.5
61.3
54.5
68.5
quad__pos--cmmintiinnsssss
64.0
50.7
55.9
cuad price restrictions
50.0
58.7
47.8
ruad_renewal_tem
50.3
70.7
59.8
73.3
cuad__reeeue--rrffishhiee
54.1
60.9
52.5
64.1
cuad rofi-rofo-rofn
50.3
49.4
51.6
55.4
quad_ssuree__eee saatcceeeoddeeseeeeeeeee
55.9
42.4
70.3
56.8
teaedttttiinnion wtr_rforrr_rreeee convenience
50.0
64.4
58.1
38.4
cuad_thine ttt_ttthr___pp parttllpp_t___aaaa beneficiary
58.8
79.4
50.0
64.7
cuad_ uncappeceppedpd liability
50.0
53.4
49.7
28.9
cuad_ wwwwumteeddd--ee you you-can caae-can-cattt eat-license
66.7
81.2
54.2
70.8
cuad_ wwwlwoeeleetriitiiieeeee
50.3
49.4
57.5
59.6
cuad_ wwwrawtaaddurrnaeee durratoo
53.1
61.9
51.6
54.4
insurance mnsurance_policyy..eceee sagicy_interpretation
32.2
38.8
37.1
32.9
jerew_blocks blocker
55.6
56.7
48.9
53.3
maud ability consummate cww.eeeeccccpppcceceeec is subject mae
47.3
13.4
50.0
49.1
carveouts
53.1
49.0
50.0
43.4
maud accuracy of fundamental target rws bringdown standard
32.6
0.0
32.6
34.8
maud_accuracy..o wtwgggeegenorlllllbbbbnrnooooonn
50.0
0.0
48.2
53.2
maud accuracy of tatr_tttrraappae cangttt_aataaiiatinn. catpiattoooon__otsttteee tw_(ooustandingg_hhl shares)
26.1
0.0
23.4
28.2
bringdown standard answer
maud additional hawpiiiioona_mtthinnnnnngoo righths righggg__rhttttiroodddo peeiodd___ood_for___o_ooo horod__oommficctiiooooooooo
20.0
0.0
18.8
20.3
souud_appicaation of catpreoooonenettiiiiee (negative_
49.1
62.5
50.3
50.0
interim covenant)
maud buyer wtweeecoonentttuuuneeeeeeen
course)
49.4
50.0
50.0
49.7
maud maud_ change in hw__eaaww_sje_ttttoo_ wttpssseettuussppppppppeeeeeeeee ttt__/dipppoppotiontteippppppa
44.3
6.3
44.0
51.1
modifier
maud changes in gaap other accounting princcplesriceeeeeppee subject
41.9
5.9
51.0
47.9
to disproportionate coopactmmodifie
maud_cor Beruud_cor_perrie caad___oo__eemtt_______ in ietpiittdddd___rpsossse__eee httprressssee____eeee_ee___eeeee tow.otterccningeeeeee
$1.2
48.8
71.7
50.6
maud cor permitted with board fiduciary determination only
49.4
36.3
50.0
45.9
16.7
1.0
16.7
16.7
maud cor cor_ wwt.staaddrd_suuupiooorrrr _
10.8
4.1
8.8
5.4
24.7
0.0
26.2
25.0
maud www.ddeeiinniinniieeeee asset deals
34.4
0.0
22.5
33.3
maud ieciiittionnincuudee sttck_toocata
40.4
0.0
9.9
0.0
maud wtt.f/uciaay__cccppttioooo httpr//ooooooooddddccccccintttoo
sandddddmmnnatttosssaddddddd standard
6.3
0.5
12.9
13.3
maud fiduuciay board determination deeeerrmnnatiooorriirrr_...
$6.7
50.0
50.5
50.0
maud_ fis_ fls__(mae_standarr
23.9
10.5
24.2
1.2
155

---

Table 81 continued from previous page
Task
BLOOM
Flan
Incite
Opt
maud_general__ _et__eeeeeeccccomiicc.. Laapamieeann_aieeeeee ilaprpprtionnte_impaat, modifier
50.0
3.0
66.1
53.0
maud includes consiistttttttttnncneecee with past practice
50.4
89.7
50.0
43.3
maud_initial haggaiiii_mmmciinngggiisoooooooo
13.5
0.0
18.0
17.0
maud initial matching rights period (ftr)
rights_perrid
18.9
0.0
19.6
19.4
maud_ htwparvwninngeetttttooppoooooooo
48.5
3.3
50.0
50.0
answer
maud inowledge_definitionn definition
46.0
50.0
47.5
49.4
maud_ Jiabildiaallttytio iiaaility_sttndard__tttt sotpiarsd_fooooooopp__o_p representatives
50.0
54.5
50.0
50.0
@oaud_oddinary. ctlcuuueeuefootttsssessss _ _standadd
33.3
58.3
32.1
33.3
maud_ baclud__anemic___mm catpaniccc____orrrooeee cwrooootr__uurttteeee wttppppuubbllbheeeeeeeeeeeee htttgaaataethtttttteee subject______
2.6
62.3
60.4
47.4
htnppoottionate__ppptt__ppp m_odifier
htt_ghtthheatt_speetteeeee event_specific reference_
50.0
59.9
50.0
50.0
to maud relational language (mae)_applic)
rescsuress_o_mmcssure measures
32.6
0.0
50.0
49.3
craud_seccfficppeffreee
50.0
89.3
48.0
50.0
maud tail period length
1.5
0.0
35.6
25.0
mmaud_type__e co
24.4
27.2
25.2
15 data retention
52.3
64.8
39.8
60.7
69.7
55.9
53.6
oppl15_do_ coloooo____a
lto_oo__rrre track
45.5
55.5
42.7
37.3
appl15_first_ppp collection panllp_pttttcoollctiionnnoo
60.9
52.3
69.5
56.1
opplll5_internationalaadddd specispeeeediieeeeeeeee
59.9
70.5
51.8
57.0
oppl15_policy_ciee _ccaage__heeeeeee
52.1
60.4
61.2
57.4
oppl15_third__pp paaplparrrtrrnngeeeeeee collection challccttarnnooollltto
55.6
63.3
64.8
47.0
and deletion
49.2
82.3
66.4
53.5
Suplll__seer_choic.e control
48.5
58.5
59.0
47.5
privacy.policy_cntaimmee
53.6
66.1
54.1
privacy_policy_ga
50.3
61.0
55.1
50.5
proa
54.1
64.5
53.1
ssla_com wtwppooommmpeeeeeeee defendants
36.2
13.1
51.1
47.0
ssla individual defendants
11.9
20.0
20.5
17.3
ssla_plaintiff
40.1
86.4
36.8
42.8
sara ent_eentamment
48.5
0.0
50.7
48.9
sara_numeric
3.1
1.0
0.0
1.0
coplyycchiin.o hoaplyychainddicclssieee disclosure_ best hrrpiieeesssstppteeeeeeeeeee lttp__rrrcicccccccaaalllll accountability
43.0
74.5
55.1
37.4
supply_chain_dine dwwcllosueeebest/ www.b/eetpaccee_auuuueeeeee audits
42.7
74.3
61.0
54.6
46.8
77.4
52.0
46.6
44.9
83.4
64.2
41.0
47.7
54.9
53.1
21.8
46.9
77.7
60.6
14.6
156

---

Table 81 continued from previous page
Task
BLOOM
Flan
Incite
Opt
46.5
73.0
57.3
2.6
supply wttppccaaaaaalaaaaeeeee disclosureeeeeeeeeeooee disclosed certification
45.3
77.3
52,0
12.6
47.9
87.2
58.8
16.9
supply chain disclosure disclosed verification
44.1
55.3
0.2
unfair_tos
9.0
8.2
4.0
15.3
157


Reviews:
[{"review_id": "5Ho6FYA0sb", "reviewer": "Reviewer_ZLU8", "summary": "", "questions": "", "limitations": "The limitations described in the previous section under a. and d. are very briefly mentioned in the paper, and perhaps would benefit from further elaboration, while the limitation described in b. is not explained.\n", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Recent progress in LLMs has not previously led to public datasets and benchmarks providing broad evaluation of LLMs' legal reasoning capabilities. The present work makes progress towards categorizing legal reasoning tasks and benchmarking LLMs on such tasks.  This work is done as a collaboration between ML and legal experts, and appears to be amongst the first broad-based collaborative efforts of its kind.  This is a novel work towards an important research direction. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "R3AWhvxbBb", "reviewer": "Reviewer_cUyJ", "summary": "", "questions": "", "limitations": "* Several limitations are acknowledged in a section dedicated to this topic. One particularly pertinent and constraining limitation is that the benchmark is exclusively in English - despite other legal benchmarks like LEXTREME (Niklaus et al., 2023) offering multilingual features (though they are less comprehensive) - and tends to focus on American law, particularly contract law. These factors must be borne in mind for future work before drawing general conclusions about the performance of LLMs \"in the legal field.\"\n* An additional limitation that I would suggest is the inherent difficulty in accurately evaluating some of the tasks presented in this benchmark, particularly the seven open-generation tasks. In their assessment, the authors enlist legal experts to evaluate rule-application tasks, but this might not be feasible or affordable for others wishing to work with LegalBench. The research community would likely benefit from a more rigorous study of suitable metrics for each type of task or from guidelines on manual evaluation.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The proposed benchmark undoubtedly serves as a unique and highly valuable resource for the research community.\n* The collaboration with legal experts is a significant advantage. It enabled precise analysis of the types of legal reasoning required in the real world, hence assessing models on concrete tasks that will add substantial value in practice.\n* The level of detail included in the documentation is remarkable.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "gQWlW5UN1b", "reviewer": "Reviewer_BYVU", "summary": "", "questions": "", "limitations": ".", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Timely contribution in the era of LLM.\n- Neat organization of tasks from the view point of lawyers.\n- GPT-4 and GPT-3.5 experimental results that gives the information of current status of AI on legal reasoning tasks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "mezEeuSuyC", "reviewer": "Reviewer_XJRt", "summary": "", "questions": "", "limitations": "The limitations and and potential negative societal impact of their work are well addressed.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The benchmark consists of 162 tasks, which is comprehensive and professional.\n- The framework for legal reasoning is very clear, reasonable, and motivative. \n- The supplementary material provides lots of details for this benchmark, which is very solid.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "58BtLHIiYQ", "reviewer": "Reviewer_tmL2", "summary": "", "questions": "", "limitations": "1. The benchmark is created with the sole focus of testing LLMs, however, if in the future a new class of models come into being then these tasks may not serve a good purpose. The benchmark should be generic and there should be a clear separation between the tasks and systems that are used to solve those. The focus of the tasks should be promoting development of legal-tech and this can be independent of what techniques/models are used to achieve this goal. \n2. The benchmark is mainly restricted to US legal system and it is not clear if a system/LLM is performing well on LegalBench, would it generalize well to legal systems of other countries as well? Authors should discuss about this and possibly evaluate on benchmarks for other legal domains (e.g., EU, India, China, etc.). Moreover, the benchmark is restricted to English language only. Authors briefly mention about this in the paper.\n3. Currently, LegalBench involves tasks that are not subjective and result in an unambiguous outcome. However, in the practice of law, such cases are straight forward to solve. For the benchmark to be reflective of real life conditions it should also involve tasks which may involve subjective outcome. It would be nice to evaluate how LLMs perform over these. Moreover, it will serve a good benchmark for future LLMs. \n4. Authors are proposing various tasks and some of these can have negative societal impact and some may not have much impact. Possibly, authors should discuss in detail which of the tasks are more prone to ethical considerations and what could be done to mitigate the impact.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. LegalBench benchmark covers various types of reasoning that are required by legal professionals in day to day working. And the benchmark is created with the help of legal professionals so it covers the practical aspects of the legal profession. \n2. The tasks are well organized (there is a typology of tasks) into various types of reasoning, this can be useful from the perspective of future research.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper reports on a collaboratively assembled benchmark for assessing legal reasoning. The authors include both CS/AI folks and the stake holders from legal profession--making it more likely that the benchmark may find acceptance in the legal community.

The reviews are mixed--with three reviewers pretty strongly advocating it, and two viewing the paper and the contribution as borderline. 

Given the sensitivity of the benchmark, an ethics review was also done (and the authors responded to the concerns raised). 

Speaking for myself (AC), one question that I had was this: The benchmark seems to focus mostly on reasoning by pattern matching (rule application, rule recall, rule conclusion etc). To what extent is legal reasoning really glorified pattern matching, as against what logical reasoning involving long inference chains? I ask because as the authors may have noticed, there is considerable divergence of opinion as to whether LLMs are actually able to do the latter type of deep reasoning. I think the paper and the benchmark would be strengthened if it discusses the issue.
Decision: Accept (Poster)