SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Background Courts regularly interpret statutes to determine the precise meaning of words contained in the statute. For instance, suppose a statute specifies that "It shall be illegal to park a vehicle inside public parks for longer than thirty minutes." A court may be asked to determine whether the statute prohibits persons from parking bicycles inside public parks. This requires defining the term "vehicle" and determining if a bicycle is a type of vehicle.
To guide the interpretation of ambiguous statutory terms, American jurisprudence has developed numerous principles of statutory construction or interpretation. These principles—also referred to as tools or canons—are rules which dictate how terms in statutes should be interpreted. For instance, the principle of ejusdem generis states that where general words or phrases follow a number of specific words or phrases, the general words are specifically construed as limited and apply only to persons or things of the same kind or class as those expressly mentioned [130].
One approach to statutory interpretation—known as textualism—states that only the text of the statute should be considered [131]. In contrast, other approaches to interpreting an ambiguous term might call for a court to analyze the purpose of the statute, the history of the statute, or the intent of the legislature.
Task The Textualism tasks ask a LLM to determine if an excerpt of judicial text is applying a specific textual tool when performing statutory interpretation. There are two tasks: dictionaries (textualism_tool_dictionaries) and .
- The first task is plain-meaning (textualism_tool plainly), and it requires an LLM to determine if a court is applying the "plain meaning" rule. The plain meaning rule says that statutory text should be interpreted according to its plain or ordinary meaning.
- The second task is dictionaries (textualism_tool_dictionaries), and it requires an LLM to determine if a court is using dictionaries to define the statutory text.
Construction process For each task we extracted paragraphs from Court of Appeals opinions and manually annotated whether the paragraphs showed the court as "using" the respective tool.
- In order to count as using plain meaning, the paragraph must reference the plain or ordinary meaning of the text. This includes directly saying "plain meaning" or referencing the general logic of the plain meaning rule. There must also be evidence that the court used the tool in its decision. This latter condition is notable because legal scholars often care about whether the court actually used the tool when defending its decision. Common examples of using include stating it as a general rule of decision ("[0]ur obligation is to look to the plain language of the statute to effectuate the intent of congress") or applying it to the facts ("The statute's plain language indicates the  $150\%$  fee cap applies if (1) the plaintiff was "a prisoner" at the time he brought the action and (2) he was awarded attorney's fees pursuant to  $\S 1988$ "). "Using" does not, for example, include paragraphs that criticize the use of the plain meaning rule.
- In order to count as using dictionaries, the paragraph must reference a dictionary. There must also be evidence that the court used a dictionary as part of its rationale. This latter condition is notable because legal scholars often care about whether the court actually used the tool when defending its decision. Common examples of using include stating it as a general rule of decision ("[We use a dictionary to help determine the plain meaning of the statutory text") or applying it to the facts ("According to the Websters dictionary, a vehicle is any means in or by which someone travels, or something is carried or conveyed"). "Using" does not, for example, include paragraphs that criticize the use of dictionaries.
Significance and value Recognizing when a court is applying a particular canon of interpretation is a classical skill law students are expected to learn. LLM performance on this task thus offers a heuristic for comparing LLM comprehension of judicial text to that of a law student's. More practically, the capacity for LLMs to detect when certain canons are being applied could make them a valuable tool for empirical legal scholars.
Input	Answer
overcome our prior interpretation of a statute depends, in turn, on whether we regarded the statute as unambiguously compelling our interpretation.	No
the statutory waiver is express, and its range is defined in unmistakable language. to say that a private person, but not the united states, is liable under title vii for interest as an element of an attorneys fee would rob the unambiguous statutory language of its plain meaning. it would defeat the statutory imposition upon the united states of a liability for costs, and the statutory inclusion of a reasonable attorneys fee as part of the costs, identical to that of a private party in similar circumstances. the scope-setting statutory words the same as a private person mark out the united states liability for attorneys fees as well as costs in the traditional sense. our responsibility as judges is to enforce this provision according to its terms.	Yes
Table 51: Examples for textualism_tool plainly.
Table 52: Examples for textualism_tool_dictionaries
G.33 UCC vs Common Law
In LEGALBENCH, the UCC vs Common Law task is denoted as ucc_v_common_law.
Background In the United States, contracts are typically governed by one of two different bodies of law depending on the subject matter of the contract. Contracts for the sale of goods (physical, moveable things) are governed by the Uniform Commercial Code (UCC), a uniform set of laws created by the Uniform Law Commission and adopted in all US jurisdictions. Contracts for services and real estate, on the other hand, are governed by state common law. For example, a contract for Alice to sell Bob her bike would be governed by the UCC (sale of a good), but a contract for Bob to repair Alice's bike would be governed by the common law (service).
This distinction is significant because the UCC and the common law diverge on numerous important legal issues such as:
- Offer and acceptance: The common law requires an offeree's acceptance to exactly match the terms of the offeror's offer in order for a contract to be formed (the "mirror image" rule). The UCC, on the other hand, allows for some variation in the terms under UCC Section 2-207.
- Definiteness: For a common law contract to be enforceable, it must be reasonably definite with respect to all material terms. For example, a service contract would not be enforceable without a price term or an adequate description of the service to be provided. The UCC only requires that a goods contract include the good being sold and the quantity. If any other term is missing from the contract (such as price or delivery), it will be filled in by UCC default rules.
- Options: To create an option contract (by which the offeror provides the offeree with a defined period of irrevocability), the common law requires that the offeree give the offeror separate consideration for the option. The UCC allows merchants to make "firm offers" (effectively option contracts) without the offeree providing separate consideration.
- Modification: To modify an existing contract, the common law requires both parties to provide new consideration (the "preexisting duty rule") whereas the UCC only requires that modifications be made in good faith.
Task The UCC vs. Common Law task requires an LLM to determine whether a contract is governed by the UCC or by the common law.
Construction process The dataset was manually created to test an LLM's ability to determine whether a contract is governed by the UCC or by the common law. The dataset is composed of 100 descriptions of simple contracts such as "Alice and Bob enter into a contract for Alice to sell her bike to Bob for  $50" (UCC) and "Aria pays Owen$ 100 to mount a television on the wall of her living room" (common law). Each description is followed the question, "Is this contract governed by the UCC or the common law?"
The dataset does not include "mixed purpose" contracts which incorporate both the sale of a good and a service. For example, a contract in which Alice sells Bob her bike for $100 and agrees to inflate the tires each week for the first month would be a mixed purpose contract. To determine whether a mixed purpose contract is governed by the UCC or the common law, most jurisdictions apply the "predominant purpose" test under which the predominant purpose of the contract (good or service) determines which body of law applies.
Significance and value The UCC vs. Common Law task is significant for a number of reasons. First, it provides a measure of an LLM's legal reasoning ability relative to a human lawyer (who would almost certainly score a  $100\%$  on the task). Second, it demonstrates an LLM's ability to determine the subject matter of a legal text, which has implications for the use of LLMs for legal tasks far beyond contract classification. Third, this task could prove useful in the context of contract lifecycle management (CLM) in which a CLM software product could automatically sort contracts by subject matter for review purposes. Fourth, while the sample contracts in the dataset were simple and easily identifiable as either UCC or common law contracts, real-world mixed purpose contracts can be difficult to classify and sometimes generate costly litigation. This task could be used as a starting point for developing a more fine-tuned task that can classify mixed purpose contracts.
H Extended empirical study
We now present a fuller discussion of LEGALBENCH experiments. Specifically, we use LEGALBENCH to conduct a three-part study.
- In the first part (Section H.2), we conduct a sweeping evaluation of 20 LLMs from 11 different families, at four different size points. We use this study to make initial observations on performance differences across families, the role of model size, and the gap between open-source and commercial LLMs.
- In the second part (Section H.3), we show how LEGALBENCH can be used to conduct in-depth evaluations of models. To illustrate, we use LEGALBENCH to highlight similarities and differences in the performance of three popular commercial models: GPT-4, GPT-3.5, and Claude-1.
- In the final part (Section H.4), we show how LEGALBENCH can support the development of lawspecific LLM methods. We focus on prompting, and conduct a series of experiments that begin to surface tradeoffs and challenges with regards to guiding LLMs towards certain tasks.
Ultimately, our study here serves to illustrate the types of analyses that LEGALBENCH enables, and highlight potential directions for future work.
H.1.1 Models
Commercial models We study three commercial API-access models. From the OpenAI GPT family, we study GPT-3.5 [14] (text-davinci-003) and GPT-4 [96]. Results from these models were retrieved between May and August of 2023. From the Anthropic family, we study Claude-1 (v1.3) [3]. Results from this model were retrieved in July of 2023. These models are believed to be large (hundreds of billions of parameters), though exact details on their architecture and training process are unknown. It is thus possible that some LEGALBENCH tasks leaked into pretraining data.
Open-source models We study 17 open-source models at three different size points: 3B parameters, 7B parameters, and 13B parameters. All inference was performed on two-GPU GCP 40GB A100s, using the Manifest library [97].
- From Together, we study three models: Incite-Instruct-7B, Incite-Base-7B, and Incite-Instruct-3B [34, 133].
- From Meta's OPT family, we study three models: OPT-2.7B, OPT-6.7B, and OPT-13B [154].
- From TII's Falcon family, we study Falcon-7B-Instruct [2, 103].
- From MosaicML's MPT family, we study MPT-7B-8k-Instruct [129].
- From LMSYS' Vicuna family, we study Vicuna-7B-16k and Vicuna-13B-16k [26].
- From Google's FLAN-T5 family, we study Flan-T5-XL (3B parameters) and Flan-T5-XXL (11B parameters) [31].
- From Meta's LLama-2 family, we study LLaMA-2-7B, and LLaMA-2-13B [134].
From the Wizard family, we study WizardLM-13B [150].
- From the BigScience BLOOM family, we study BLOOM-3b and BLOOM-7B [116].
HuggingFace links for the studied open-source models can be found below.
Future work Our selected LLMs represent only a sample of the models available. For instance, we do not evaluate LLMs larger than 13B parameters, which have been observed to perform well [9]. Studied LLMs are also "general domain," in that we don't find evidence that any were specifically customized to perform well on legal text.[31] In future work we hope to expand our evaluation to a broader set of LLMs.
H.1.2 Prompts
We designed a prompt for each task by manually writing instructions for the task, and selecting between zero and eight samples from the available train split to use as in-context demonstration. The number of samples selected depended on the availability of data and the sequence length of samples. For instance, the inputs to the Supply Chain Disclosure tasks are disclosure statements between 1-2 pages long, making the inclusion of
31We note that as of July 2023, we were unable to identify public law-specific English large language models to evaluate.
LLM	HuggingFace URL
Incite-Instruct-3B	togethercomputer/RedPajama-INCITE-Instruct-3B-v1
Incite-Base-7B	togethercomputer/RedPajama-INCITE-Base-7B-v0.1
Incite-Instruct-7B	togethercomputer/RedPajama-INCITE-Instruct-7B-v0.1
BLOOM-3B	bigscience/bloom-3b
BLOOM-7B	bigscience/bloom-7b1
OPT-2.7B	facebook/opt-2.7b
OPT-6.7B	facebook/opt-6.7b
OPT-13B	facebook/opt-13b
Falcon-7B-Instruct	tiuiae/falcon-7b-instruct
MPT-7B-8k-Instruct	mosaicml/mpt-7b-instruct
Vicuna-7B-16k	lmsys/vicuna-7b-v1.5-16k
Vicuna-13B-16k	lmsys/vicuna-13b-v1.5-16k
Flan-T5-XL	google/flan-t5-x1
Flan-T5-XXL	google/flan-t5-xxl
LLaMA-2-7B	meta-llama/LLaMA-2-7b-hf
LLaMA-2-13B	meta-llama/LLaMA-2-13b-hf
WizardLM-13B	WizardLM/WizardLM-13B-V1.2
Table 53: HuggingFace links for open-source models.
multiple demonstrations infeasible. For application evaluation, we augmented the prompt with an instruction for the LLM to explain its reasoning.
We used the same prompts across all LLMs with one exception. In contrast to the OpenAI and open-source LLMs, Anthropic recommends specific prompting formats when using Claude.[32] This includes surrounding in-context samples with `<example>>/<example>` tags, and adding instructions specifying the output space. We observed that failing to adhere to these guidelines led Claude to generate text which made extracting a prediction challenging. Therefore, when prompting Claude, we added example-tags to the in-context demonstrations and instructions specifying the prediction space (e.g., "Reply with either: generic, descriptive, suggestive, arbitrary, fanciful").
LLM outputs were generated using next-token generation at a temperature of 0.0. For classification/extraction tasks, we terminated at a new-line token. For rule_qa and all application tasks except diversity Jurisdiction_6 we generated 150 tokens. For diversityJurisdiction_6 we generated 300 tokens.
We believe there is significant scope for improving and refining prompts on LEGALBENCH. Hence, our results here provide a lower-bound on performance, as better prompts may elicit higher scores. Our prompts correspond to what we believe would be reasonable, based on experience with prompt engineering in other settings, and the guidance provided by model developers. We make all prompts available as a starting point for future work on LEGALBENCH.
Prompts for all LEGALBENCH experiments are available on the Github repository. Table 54 provides the number of in-context demonstrations used.
H.1.3 Evaluation
Classification tasks are evaluated using "exact-match" (following HELM [78]). Because some tasks contain significant label imbalances, we use balanced-accuracy as a metric. For extraction tasks, we perform normalization on generated outputs to account for differences in tense/casing/punctuation. A few tasks (e.g., successor_liability and ssla_individual_defendants) require the LLM to produce multiple classes or extracted terms per instance. For these, we evaluate using F1.
<sup>32</sup>https://docs.anthropic.com/claude/docs/introduction-to-prompt-design
Number of in-context demonstrations Tasks
0	Canada Tax Court Outcomes, Consumer Contracts QA, Corporate Lobbying, International Citizenship Questions, Rule QA, Supply Chain Disclosure Tasks, SARA (Numeric)
1	MAUD Tasks, SCALAR
2	Citation Prediction Tasks
3	Securities Complaint Extraction Tasks
4	Legal Reasoning Causality, Personal Jurisdiction, Successor Liability, SARA (Entailment) Telemarketing Sales Rule, Textualism Tools
5	Abercrombie, Hearsay, Insurance Policy Interpretation, Private Right of Action
6	CUAD Tasks, Diversity Tasks, J.Crew Blocker, Learned Hands Tasks, Overruling, UCC v. Common Law
7	Function of Decision Section, Oral Argument Question Purpose
8	Contract NLI Tasks, Contract QA, Definition Tasks, New York State Judicial Ethics, OPP-115 Tasks, Privacy Policy Entailment, Privacy Policy QA
9	Unfair Terms of Service
Table 54: Number of in-context demonstrations used for each type.
Rule-application tasks were evaluated manually by a law-trained individual, who analyzed LLM responses for both correctness and analysis.33 This type of manual evaluation is consistent with previous works evaluating LLM generations in the legal domain [29, 69]. As rule-application requires LLMs to generate "explanations" detailing legal reasoning—a capability primarily exhibited by larger models—we only evaluated GPT-4, GPT-3.5, and Claude-1. rule_qa was also manually evaluated by a law-trained individual. All manual evaluation was performed with reference to a grading guide, which we additionally make available.
H.2 Performance trends
Table 55 provides the average task performance for all 20 models in five reasoning categories (issue-spotting, rule-recall, rule-conclusion, interpretation, and rhetorical-understanding). The first block of rows corresponds to large commercial models, the second block corresponds to models in the 11B-13B range, the third block corresponds to models in the 6B-7B range, and the final block corresponds to models in the 2B-3B range. Table 56 provides the average task performance for the three large models on rule-application.
Overall, we find significant variation in performance across tasks, suggesting that LEGALBENCH captures a diverse spectrum of difficulty. These results emphasize that assessments of LLM capabilities for legal applications must be made on a task-by-task basis, and informed by the nuances of specific tasks. While certain types of tasks appear beyond the scope of current-day LLMs, others seem more within reach. In this section, we offer preliminary observations on performance trends across model size, family, and reasoning categories.
Parameter count Within LLM families, we observe that larger models usually outperform smaller models. For instance, Flan-T5-XXL (11B parameters) outperforms Flan-T5-XL (3B parameters) on average across all five reasoning categories, and LLaMA-2-13B outperforms LLaMA-2-7B on average across four reasoning categories. Notably, the margin of the gap varies across LLM families and reasoning categories. For instance, on rule-recall, the 7B Incite-Instruct model outperforms the 3B Incite-Instruct model by almost 10pts, while the 6.7B OPT model outperforms the 2.7B OPT model by less than 1pt. We additionally note that the largest LLM (GPT-4) outperforms virtually all other models.
33For the six diversity jurisdiction tasks, we sampled 30 instances from each task. For all other rule-application tasks, we manually evaluated the entirety of the dataset.
Table 55: Average performance for each LLM over the different LEGALBENCH categories. The first block of rows corresponds to large commercial models, the second block corresponds to models in the 11B-13B range, the third block corresponds to models in the 6B-7B range, and the final block corresponds to models in the 2B-3B range. The columns correspond to (in order): issue-spotting, rule-recall, rule-conclusion, interpretation, and rhetorical-understanding. For each class of models (large, 13B, 7B, and 3B), the best performing model in each category of reasoning is underlined.
LLM	Correctness	Analysis
GPT-4	82.2	79.7
GPT-3.5	58.5	44.2
Claude-v1	61.4	59.0
Table 56: Average performance for the large LLMs on rule-application tasks.
**Variation across families** Even for LLMs of the same size, we find considerable differences in performance. For instance, we observe significant gaps in performance between Flan-T5-XXL (11B parameters) and Vicuna-13B-16k (13B parameters), across all reasoning categories. This suggests, unsurprisingly, that the choice of pretraining data, regime of instruction-tuning, and architecture play an important role in determining performance, and that certain configurations may be better aligned for LEGALBENCH tasks. Interestingly, we observe that such choices may affect which types of reasoning categories LLMs appear to perform well at. For instance, we observe that WizardLM-13B performs worse than all peers on issue-spotting tasks, best on rule-recall tasks, and nearly matches the performance of the best-performing peer on rule-conclusion tasks. Comparing Incite-7B-Instruct to Incite-7B-Base also provides insight into the effect of instruction-tuning across different categories, at one size point (7B parameters). We observe that instruction-tuning improves performance on four categories (issue-spotting, rule-conclusion, interpretation, and rhetorical-understanding), and worsens performance on rule-recall.
We additionally find that family-specific trends appear to hold across different size points. For instance, the Flan-T5 models outperform all others at both the 3B and 13B scale, while the Vicuna models appear to underperform competitors at both the 7B and 13B scale. We attribute the Vicuna models' low performance to their frequency tendency to generate poorly-formed outputs, which did not map to the expected verbalizer tokens (e.g., blank spaces, random characters, etc.).<sup>34</sup> This could possibly be attributed to the type of data used to fine the model (e.g., user-conversation), although more in-depth experimentation is necessary.
34In further experimentation, we found that writing prompts using the "######## Human:" and "######## Assistant:" templates did not appear to help.
The gap between open-source and commercial models Finally, we find evidence that open-source models are capable of performance that matches or exceeds certain commercial models. For instance, Flant-T5-XXL outperforms GPT-3.5 and Claude-1 on two categories (issue-spotting and rhetorical-understanding), despite the relative gap in parameter count. Notably, the gap between closed and open-source models is largest for the rule-conclusion category. Amongst LEGALBENCH tasks, rule-conclusion tasks most like the other types of multi-step/common-sense reasoning tasks where commercial LLMs have been found to perform well.
H.3 Comparing GPT models
This section provides a more in-depth study of performance, focusing on the three commercial models (GPT-4, GPT-3.5, and Claude-1). The purpose of this section is to illustrate how LEGALBENCH enables fine-grained analysis of LLM performance. In particular we highlight how LEGALBENCH can provide more rigorous empirical support for anecdotal observations arising out of the legal community's use of these models, and explain performance differences between models.
H.3.1 Issue-spotting
We first consider average model performance across all issue-spotting tasks. We observe that GPT-4 outperforms GPT-3.5 and Claude-1 (both at  $p < 0.001$ ).<sup>35</sup> In absolute terms, issue tasks present the largest gap in performance between GPT-4 and other closed-API models, with an absolute margin of  $20+$  points. GPT-3.5 and Claude-1, in contrast, appear to match each other in performance, separated by an average gap of only 2 points. We additionally find that the open-source models perform poorly here. On 9 tasks, Incite-Base collapses to predicting a single class for all samples.