SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We note one limitation to our results: because 16/17 of our issue-spotting tasks are drawn from one source (Learned Hands data), average issue performance is skewed by properties of the Learned Hands data distribution (i.e., user-generated questions). For instance, though GPT-3.5 outperforms Claude-1 on 12/16 Learned Hands tasks, Claude-1 outperforms GPT-3.5 on the one non-Learned Hands task (corporate_lobbying). Despite the skew, we observe that these tasks appear to vary in difficulty. While GPT-4's balanced-accuracy on learnedHands_torts is only  $70.6\%$  , on three tasks—learnedHands-immigration, learnedHandsTraffic, and learnedHands_estate—it scores  $>95\%$
H.3.2 Rule-recall
We first consider average model performance across all rule-recall tasks. While GPT-4 outperforms GPT-3.5  $(p < 0.05)$ , we surprisingly find that Claude-1 also outperforms GPT-3.5  $(p < 0.05)$ , and appears almost on par with GPT-4. Moreover, Claude-1 outperforms GPT-4 on three tasks: rule_qa, international_citizenshipquestions, and nys_judicial_ethics. This is the only task category where Claude-1 provides performance comparable to GPT-4. Because little is known regarding the architecture and training processes for these models however, it is difficult to explain why this is the case.
Because rules/laws can be analogized to law-specific "facts," rule-recall tasks are similar to general domain LLM tasks designed to measure "hallucination." There, an extensive literature has documented the propensity for LLMs to both generate factually incorrect information, and answer fact-based questions incorrectly [78, 104]. Our results align with the primary findings of that literature. For example, we observe that the small open source models perform considerably worse than the larger models, consistent with the observation that model size plays an important role in fact-retention. Overall, performance on the rule-recall tasks also lend additional empirical support to more anecdotal reports—from the legal community—regarding how LLMs often misstate the law or cases [112].
H.3.3 Rule-application
Application tasks evaluate whether LLMs can explain how a legal rule applies to a set of facts, and verbalize the necessary inferences. With respect to correctness, we observe that GPT-4 outperforms both GPT-3.5 ( $p < 0.01$ ) and Claude-1 ( $p < 0.01$ ). Across LLMs, we find that variation in performance across tasks is consistent with subjective impressions of task difficulty. For instance, performance on diversity_jurisdiction_1 (an easy task requiring a model to determine if an amount is greater than  $\$75$ k and if the plaintiff and defendant are from different states) is much higher than performance on successor_liability (a harder task requiring a model to identify multiple successor liability exceptions in a fact pattern describing a complex transaction).
We observe that LLM generations may be incorrect in many different ways. On the Diversity Tasks, LLMs sometimes perform incorrect arithmetic operations or mathematical comparisons (i.e., stating that  $50,000 is greater than$ 75,000). On telemarketing_sales_rule in contrast, LLMs will cite to an incorrect portion of the rule. For instance, a generation may explain that certain conduct by a telemarketer runs afoul of the rule
35Statistical significance is computed using a paired  $t$  -test over the tasks in the category.
because the telemarketer failed to make a mandatory disclosure (16 CFR § 310.3(a)(1)), but cite to the portion of the rule prohibiting misrepresentations (16 CFR § 310.3(a)(2)). Examples of other types of incorrect generations can be found in Table 57.
Fact Pattern (Task)	LLM generation	Why is it incorrect?
Sophia is from Nebraska. Ava is from Washington. Sophia sues Ava for legal malpractice for $34,000. (diversity_1)	“No. The parties are not diverse because Sophia and Ava are both from different states. The AiC is too small because it is less than 75k. Because the AiC requirement is not met there is no jurisdiction.” (GPT-3.5)	While GPT-3.5 correctly identifies that there is no jurisdiction because the AiC is less than $75k (and thus too small), it fails to recognize that the parties (Sophia and Ava) are diverse because they are from different states.
The mark “Trim” for nail clippers. (abercrombie)	“The mark trim for nail clippers would be considered a suggestive mark. It suggests a characteristic or quality of the product which is that the nail clippers are used to trim ones nails. It implies this purpose and function without directly describing it so trim for nail clippers would be classified as a suggestive trademark.” (Claude-1)	Claude-1’s assertion that the mark “implies this purpose and function without directly describing it” misses the fact that “Trim” directly describes, rather than implies, what nail clippers are used for (i.e., the trim nails). Thus, the mark is actually descriptive, and not suggestive.
Ana is a lawyer who resides in Texas. While visiting Louisiana, she meets David, who runs a bike shop. She buys a bike from him and uses it to bike back to Texas. Right after she crosses the border, the bike seat explodes, in-juring Ana. Ana sues David in Louisiana. (personal_jurisdiction)	“Yes. There is personal jurisdiction over David in Louisiana. Explanation. 1. David is domiciled in Louisiana so there is general jurisdiction over him in Louisiana. 2. David has sufficient minimum contacts with Louisiana such that he has availed himself of the privileges of conducting business in Louisiana. He operates a bike shop in Louisiana so he purposefully avails himself of doing business in the state.. 3. Ana’s claim arises out of Davids contacts with Louisiana she bought the defective bike from David’s bike shop in Louisiana so the claim is directly related to David’s business contacts in the state. 4. Requiring David to litigate in Louisiana would not offend traditional notions of fair play.” (Claude-1)	The LLM states that David is domiciled in Louisiana, when the facts make no mention of this.
Table 57: Examples of generations for rule-application tasks which are incorrect. The last column provides analysis for why the generation is incorrect.
With respect to analysis, we observe that GPT-4 again outperforms both GPT-3.5 ( $p < 0.01$ ) and Claude-1 ( $p < 0.01$ ). Explanations which failed to exhibit analysis can be grouped into several categories. First, some generations will contain just a prediction as to the legal outcome, without an explanation (even when the LLM has been prompted to generate one). The same prompt–applied to other samples in the dataset–will elicit explanations containing analysis. Second, we observe a tendency for LLMs to sometimes generate explanations which merely restate the facts and legal rule, without actually offering an explanation for how the outcome is reached. Examples of such instances are provided in the table below.
H.3.4 Rule-conclusion
Rule-conclusion evaluates on the same tasks as rule-application, but only requires the LLM to generate a prediction as to the outcome, and not an explanation. We observe that GPT-4 once again outperforms GPT-3.5  $(p < 0.01)$  and Claude-1  $(p < 0.01)$ . Claude-1 and GPT-3.5 appear approximately level on performance.
The rule-conclusion tasks offer a heuristic for characterizing the types of legal inferences LLMs are capable of or struggle with. In particular, several of these tasks organize samples into slices, where the samples contained within a slice all represent a similar type of fact pattern, and thus interact with the legal rule in a comparable way. For instance, the hearsay task contains a slice corresponding to "non-verbal hearsay." This slice contains fact patterns where an individual communicates something non-verbally (e.g., pointing), thus qualifying their conduct as a "statement" under the hearsay rule. In order to make accurate predictions on this slice, an LLM
Fact Pattern (task)	LLM generation	What analysis is missing?
Martha makes special pinwheel hats in her home in Salem, Oregon. She only markets her hats on the west coast (California, Oregon, and Washington). Alex, a citizen of Maine, purchases one of the hats while visiting Salem. When Alex returns to Maine, the pinwheel comes off and seriously cuts Alex. Alex sues Martha for damages in Washington. (personal_jurisdiction)	“No. There is not personal jurisdiction.” (Claude-1)	Claude-1 does not provide any explanation for why there is no personal juris dication, so the explanation does not contain analysis.
On the issue of whether Henry negligently hit Kelly with his car, Kelly’s statement, “a witness told me after the crash that Henry was on his cell phone while driving. (hearsay)	“Yes, there is hearsay in this situation. The reasoning is as follows: . 1. The statement in question is an out-of-court statement, as it was made by a witness who is not testi fying in court.. 2. The statement is being introduced to prove the truth of the matter asserted, which is that Henry was on his cell phone while driving.. 3. Since the statement is an out-of-court statement being introduced to prove the truth of the matter asserted, it is considered hearsay.” (GPT-4)	GPT-4 does not explain how the truthfulness of the out-of-court statement (whether or not Henry was on his phone) relates to the purpose of its introduction (proving/disproving that Henry negligently drove).
Table 58: Examples of generations for rule-application tasks which do not contain analysis. The last column explains why the generation is deficient.
must recognize that (1) the hearsay rule applies to non-verbal communicative conduct, and (2) the non-verbal conduct in these fact patterns is communicative.
Though slices are small—and thus not intended for rigorous statistical analysis—they provide some intuition as to the source of GPT-4's improvement over GPT-3.5, and the overall areas of strength and weakness for both models. On the hearsay task for instance (Table 59), the difference between GPT-4 and GPT-3.5 appears primarily attributable to improvements over the slices corresponding to non-verbal hearsay and statements made in court. In looking across slices moreover, it's clear that some are comfortably within the realm of model capabilities (e.g., non-assertive conduct), while others (e.g., not introduced to prove the truth of the matter asserted) still pose a considerable challenge.
Another example is provided by the abercrombie task, in which an LLM must determine the relationship between a product and a potential trademark name, by classifying the product-name pair into one of five categories recognized by courts: generic, descriptive, suggestive, arbitrary, and fanciful. Loosely, these categories measure how distinctive a product name is for a product, with generic being the least distinctive, and fanciful being the most distinctive. Just as with hearsay, comparing LLM performance on each of these categories provides insight into the relative areas of improvement (Table 60). Here, GPT-4's improved overall performance appears most attributable to performance on marks which are suggestive or arbitrary. However, GPT-4 still makes a number of errors for both categories. Interestingly, performance on descriptive marks is consistent between both models.
H.3.5 Interpretation
On the interpretation tasks, we find that on average GPT-4 outperforms GPT-3.5 ( $p < 0.01$ ), ans GPT-3.5 outperforms Claude-1 ( $p < 0.01$ ). Here, the larger API-models are highly performant on tasks which involve binary classification over short clauses. Averaged across the 38 CUAD tasks (contract clauses), for instance, GPT-4, GPT-3.5, and Claude-1 all have a balanced-accuracy  $\geq 88\%$ . And on proa (statutory clauses), both GPT-4 and GPT-3.5 have a balanced-accuracy  $\geq 90\%$ . Notably, performance degrades on tasks which contain longer text sequences or involve multi-class classification. On the Supply Chain Disclosure tasks for instance—in which LLMs must classify disclosures which are 1-2 pages in length—the average balanced-accuracy of the large commercial models ranges between  $74 - 75\%$ . And on the MAUD tasks—which require answering multiple choice questions about merger deals—the average balanced-accuracy of GPT-4 drops to  $47.8\%$  accuracy.
Slice	Slice description	GPT-3.5	GPT-4
Non-assertive conduct (n = 19)	The fact pattern describes conduct which is non-communicative and therefore not hearsay.	100%	100%
Statement made in-court (n = 14)	The fact pattern describes a statement that was made in court and therefore not hearsay.	57%	93%
Standard hearsay (n = 29)	The fact pattern describes traditional hearsay (out-of-court statement introduced to prove the truth of the matter asserted).	97%	97%
Non-verbal hearsay (n = 12)	The fact pattern describes non-verbal communicative conduct that qualifies as hearsay.	33%	75%
Not introduced to prove truth (n = 20)	The fact pattern describes a statement not introduced to prove the truth of the matter asserted, which is therefore not hearsay.	25%	45%
Table 59: Comparison between GPT-3.5 and GPT-4 on hearsay slices. Accuracy is reported for each slice.
Mark	Mark description	GPT-3.5	GPT-4
Generic (n = 19)	The name connotes the basic nature of the product/service.	94%	100%
Descriptive (n = 19)	The name identifies a characteristic or quality of the product/service.	73%	72%
Suggestive (n = 20)	The name suggests, rather than describes, a characteristic of the product/service.	38%	70%
Arbitrary (n = 18)	The name is a real world but has no relation to the product/service.	41%	82%
Fanciful (n = 19)	The name is a made-up word.	84%	100%
Table 60: Comparison between GPT-3.5 and GPT-4 on abercrombie categories. Accuracy is reported for each slice.
H.3.6 Rhetorical-analysis
On average across all rhetorical-understanding tasks, we find that GPT-4 outperforms both GPT-3.5 ( $p \leq 0.05$ ) and Claude-1 ( $p \leq 0.05$ ). We note several results. First, on definition Extraction—which requires a LLM to extract the term defined by a sentence taken from a Supreme Court opinion—Incite-Base almost equals GPT-4 in performance (80.6% accuracy to 81.8%). Second, nearly all evaluated models struggle on two tasks requiring LLMs to label the legal "roles" played by either a question or excerpt from an opinion (function_of_decision_section and oral ArgumentQuestionpurpose). Notably, both tasks require the LLM to classify text into one of six or more categories
H.4 Prompt engineering strategies
Finally, we illustrate—through a series of micro-studies—how LEGALBENCH can be used to explore different aspects of prompt-engineering for LLMs in legal settings. We focus on three questions:
1. Can LLMs rely on their latent knowledge of a rule for rule-conclusion tasks?
2. Does simplifying task descriptions to plain language affect performance?
3. Are LLMs sensitive to the choice of in-context demonstrations?
Reliance on latent knowledge When prompting for general-domain tasks like sentiment or topic classification, prompt-engineers will often rely on the LLM's latent knowledge of the task [5]. In topic classification for instance, a prompt may use the instructions to label whether a news article is about "sports," without offering
Figure 3: We compare performance of prompts which describe the legal rule to be applied ("description") against prompts which reference the legal rule to be applied ("reference"). Error bars measure standard error, computed using a bootstrap with 1000 resamples.
a detailed description of what "sports" refers to or encompasses. Such a description is not necessary, because general-domain terms like "sports" appear frequently in LLM training corpora, and LLMs can learn from these occurrences what general-domain terms mean. Prompting for legal tasks, however, may require a different strategy. Because legal terms occur less frequently in general domain training corpora, legal prompting may require practitioners to provide additional background information. For example, a general domain LLM may not know what the requirements for diversity jurisdiction are, because diversity jurisdiction is not as commonly discussed in pretraining corpora.
We explore this question through a study of rule-conclusion tasks. For a selection of these tasks, we evaluate GPT-3.5 with two zero-shot prompts: a reference-based prompt and a description-based prompt. In the reference prompt, the task instructions merely state the rule to be applied, i.e., "Determine if the following fact patterns give rise to diversity jurisdiction." In the description-based prompt, the instructions provide an explicit description of the rule, i.e., "Diversity jurisdiction exists when there is (1) complete diversity between plaintiffs and defendants, and (2) the amount-in-controversy (AiC) is greater than $75k." By comparing performance between the reference and description prompt, we can measure whether providing a description of the rule in the prompt provides additional performance boost over the LLM's latent knowledge of the rule.
Figure 3 provides a comparison for the different prompts. Interestingly, we find considerable variation across tasks. On tasks like aberrcrombie, ucc_v_common_law, diversity_2, and diversity_4, description prompts appear to offer significant increase in performance. On the other tasks, performance is approximately the same (or even worse). We identify two possible explanations for diverging results across tasks. First, on certain tasks, subsets of fact-patterns are too challenging for LLMs like GPT-3.5, and description-based prompts do not provide sufficient guidance for LLMs to reason through those fact patterns. Second, legal rules may be described to varying extents within pretraining corpora. Hence, tasks where we observe performance improvements from description-based prompting may correspond to rules which occur less frequently in pretraining data.
Plain language descriptions of tasks Next, we examine the extent to which domain specialization in the language of the prompts affects performance. Like experts in other specialized domains, lawyers have developed their own language (i.e., "legalese"), which forms the basis for most legal writing and communication. It is unclear whether—in interacting with large language models through prompting—lawyers should continue to rely on formalistic legal language, or instead use simpler plain language. While most large language models are "general domain" and thus less specialized to legalese, formalistic legal language is more precise, and may thus induce more accurate behavior from the model.
We explore this question by comparing "plain language" and "technical language" prompts. For a subset of LEGALBENCH tasks, we have access to the formal language provided to law-trained annotators when creating task data. By comparing the performance of a prompt which uses this language—to one which uses a plain-language version—we can measure how the technicality of language affects results.
Figure 4: We compare performance of prompts which describe the task in plain language to prompts which describe the task in technical legal language (for GPT-3.5). Error bars measure standard error, computed using a bootstrap with 1000 resamples.
Figure 5: We evaluate GPT-3.5 and Incite-Instruct on five prompts constructed by randomly selecting different samples to use as in-context demonstrations (maintaining class balance in the prompt). In the figure above, each point corresponds to a different prompt.
We conduct preliminary experiments on a sample of five LEGALBENCH tasks (Figure 4).<sup>36</sup> On four of the five tasks, we find that the plain-language prompt significantly outperforms the technical language prompt, by up to 21 points (balanced-accuracy). Interestingly, on contract_nliisable_post-agreement possession, we find the opposite phenomenon holds: the plain language prompt is substantially worse than the technical prompt.
Sensitivity to in-context demonstrations Finally, we investigate the influence of the in-context demonstrations used in prompts. Prior work in general domain LLMs have observed that few-shot performance is highly sensitive to the choice of demonstrations [56, 123, 141]. We evaluate whether LLMs are similarly sensitive for legal tasks, focusing on a subset of 8 binary classification tasks. For each task we merge the train and evaluation split into a single dataset, and randomly sample four in-context samples to include in the prompt (two from each class), five different times. We evaluated GPT-3.5 and Incite-Instruct-7B with each of the five generated prompts, and plot the the balanced-accuracy of each prompt in Figure 5.
Consistent with findings on general-domain tasks, we observe that LLMs on legal tasks are also highly sensitive to the choice of in-context samples. Notably, this appears to be the case for both GPT-3.5 and Incite-Instruct.
36Prompts are made available in the LEGAL BENCH repository.
Under a permutation test, we find significant differences  $(p < 0.01)$  between the best and worst performing prompt for Incite-Instruct (on all tasks), and for GPT-3.5 (on all tasks except opp115_threeParty_sharingcollection and overruling). For many tasks, the magnitude of difference is substantial. On overruling for instance, the best Incite-Instruct prompt improves upon the worst prompt by over 20 points (balanced-accuracy). Overall, these results suggest that future work is needed to understand how different demonstrations influence performance.
I All tables
We provide results for each LLM on each of the tasks. Models are divided into four groups based on type: commercial models, 13B models, 7B models, and 3B models. Model names are abbreviated to the family name to ensure well-formed tables.
Task	GPT-4	GPT-3.5	Claude-1
abercrombie	84.2 / 84.2	48.4 / 48.4	65.2 / 63.1
abercrombie	84.2 / 84.2	48.4 / 48.4	65.2 / 63.1
diversity_1	96.7 / 96.7	86.7 / 86.7	86.7 / 86.7
diversity_2	100.0 / 100.0	53.3 / 50.0	73.3 / 73.3
diversity_3	96.7 / 96.7	83.3 / 66.7	53.3 / 53.3
diversity_4	93.3 / 93.3	70.0 / 66.7	43.3 / 43.3
diversity_5	76.6 / 76.6	66.7 / 66.7	36.7 / 36.7
diversity_6	80.0 / 80.0	6.7 / 6.7	53.3 / 53.3
hearsay	75.5 / 47.9	55.3 / 23.4	68.1 / 41.5
personal_jurisdiction	94.0 / 94.0	68.0 / 10.0	70.0 / 70.0
successor_liability	19.1 / 19.1	15.2 / 15.2	38.3 / 38.3
telemarketing_sales_rule	72.3 / 70.2	48.9 / 42.5	55.3 / 55.3
ucc_v_common_law	97.8 / 97.8	100.0 / 47.8	93.6 / 93.6
Table 61: Performance on rule-application tasks for commercial models. We report correctness/analysis.
Task	GPT-4	GPT-3.5	Claude-1
rule_qa	72.0	46.0	77.0
international_citizenshipquestions	59.3	52.7	60.0
nys_judicial_ethics	78.3	74.3	79.1
citation_prediction_classification	71.3	54.6	61.1
citation_prediction_open	15.1	3.8	11.3
Table 62: Commercial models on rule-recall tasks.
37We conduct the permutation test with 1000 resamples.
Task	Flan	Llama-2	OPT	Vicuna	WizardLM
rule_qa	0.0	22.0	8.0	14.0	20.0
international_citizenshipquestions	52.4	50.0	18.0	21.5	49.8
nys_judicial_ethics	69.1	67.6	63.3	61.3	63.8
citation_prediction_classification	56.5	47.2	52.8	50.0	52.8
citation_prediction_open	1.9	1.9	0.0	0.0	3.8
Table 63: 13B models on rule-recall tasks.
Task	Bloom	Falcon	Incite-Base	Incite-Inst.	Llama-2	MPT	OPT	Vicuna
rule_qa	6.0	8.0	18.0	28.5	22.0	20.0	6.0	0.0
international0.0citizenship_ questions	10.2	49.5	47.0	27.2	2.4	2.4	3.0
nys_ judicial_ ethics	62.5	55.8	62.7	54.9	67.5	57.2	59.7	52.3
citation_ prediction_ classification	50.0	50.9	50.9	47.2	50.0	50.0	47.2	14.8
citation_ prediction_ open	1.9	0.0	0.0	0.0	1.9	0.0	0.0	0.0
Table 64: 7B models on rule-recall tasks.
Task	Bloom	Flan	Incite	Opt
rule_qa	0.0	0.0	14.0	2.0
international_citizenshipquestions	0.0	50.0	18.9	3.0
nys_judicial_ethics	54.0	58.5	49.9	56.7
citation_prediction_classification	49.1	50.0	51.9	49.1
citation_prediction_open	0.0	0.0	0.0	0.0
Table 65: 3B models on rule-recall tasks.