SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 9 - continued from previous page 1
maud_cor_permitted_with_board_fiduciary_determination_only	101	303.1
maud_cor_STANDARD_(intervening_event)	85	326.1
maud_cor_STANDARD_(superior Offer)	101	308.0
maud-definition.contains knowledge Requirement__answer	148	243.4
maud_defined INCLUDES_asset Deals	147	314.7
maud_defined INCLUDES stock Deals	149	313.6
maud_fiduciary_exception: _board_determination_STANDARD	180	246.8
maud_fiduciary_exception: _board_determination_trigger_(no.shop)	180	245.1
maud_fls_(mae)_standard	78	705.1
maud_general_economic_and_financial_conditions: subject_to_ "disproportionate_impact" _modifier	99	704.6
maud Includes "consistent_withpast practice"	182	122.7
maud_initializer_MATCHing_rights_period_(cor)	159	313.4
maud_initializer Matching Rights period (ftr)	133	336.4
maud_intervening_event__required_to_ occur_after_signing__answer	148	242.2
maud_knowledge-definition	168	334.8
maud_liability_STANDARD_for_no-shop_breach_by_target_non-d&oRepresentatives	157	38.1
maud_ordinary_course_efforts_STANDARD	182	122.7
maud_pandemic_or_other_public_health_event: _subject_to_ "disproportionate_impact" _modifier	99	707.5
maud_pandemic_or_other_public_health_event: _specific_reference_to_pandemic-related_governmentalresponses_or_MEasures	99	707.5
maud_Relational_language_(mae)_applies_to	91	705.5
maudSpecific_performance	179	96.8
maud_tail_period_length	180	95.1
maud_type_of consideration	173	128.2
nys_judicial_etherics	300	25.7
opp115_data_retention	96	31.5
opp115_data_security	1342	38.6
opp115_do_not_track	118	37.1
opp115_firstPartyCollection_use	2094	32.6
opp115_international_andSpecific_audiences	988	52.1
opp115_policy_change	439	33.2
opp115ThirdParty_sharing.collectionition	1598	35.3
opp115_user_access,_edit_and_deletion	470	35.1
Table 9 - continued from previous page 1
opp115_user_choice_control	1554	33.5
oral Argument question purpose	319	50.2
overruling	2400	27.5
personal_jurisdiction	54	67.8
privacy_policy_entailment	4343	111.9
privacy_policy_qa	10931	41.1
proa	100	42.6
rule_qa	50	11.7
scalr	571	275.4
sslacompany_defendants	1231	310.0
sslindividual_defendants	1015	313.7
ssl Plaintiff	1036	308.4
sara_entailment	276	148.0
sara_numeric	100	12222.1
successor_liability	50	71.5
supply_chain_disclosure_bestPracticeaccountability	387	510.0
supply Chain Disclosure best practice audits	387	508.3
supply Chain Disclosure best practice certification	386	508.4
supply Chain Disclosure best practicetraining	387	508.2
supply Chain Disclosure best practiceverification	387	507.0
supply Chain Disclosure disclosedaccountability	386	510.4
supply Chain Disclosure disclosed audits	387	508.0
supply Chain Disclosure disclosedcertification	386	509.9
supply Chain Disclosure disclosed_training	387	506.9
supply Chain Disclosure disclosedverification	387	507.6
telemarketing_sales_rule	51	58.4
textualism_tool_dictionaries	111	151.3
textualism_toolplain	169	160.9
ucc_v_common_law	100	20.9
unfair_tos	3822	34.3
This section describes metrics and evaluation protocols.
Rule-application To evaluate an LLM's performance on a rule-application task, a law-trained expert manually validated each generation. We computed two metrics. The first metric—correctness—corresponds to the proportion of generations which do not misstate the fact pattern, the legal outcome, the rule, or contain a logical error. Logical errors include arithmetic mistakes, or assertions which are plainly wrong (e.g., that an apple is not a tangible object).
- The LLM would incorrectly assert the legal outcome. For instance, an LLM would assert diversity jurisdiction existed, when it actually did not.
- The LLM would incorrectly assert an intermediate conclusion. For instance, an LLM would assert that a rental agreement for a boat was a contract for a service, rather than a moveable and tangible good. In this case, the LLM would fail to realize that a boat is a moveable or tangible good.
- The LLM would hallucinate a piece of information not explicitly stated in the fact pattern.
- The LLM would misstate the content of a rule. For instance, the LLM would assert that subprovision of a statute barred one type of conduct, when in fact, that conduct was barred by a different subprovision
The second metric—anlaysis—corresponds to the proportion of generations which contain the necessary inferences to reach the correct legal conclusion from the provided fact-pattern. Thus, it is insufficient for a LLM explanation to merely state that a piece of evidence is hearsay is insufficient: an explanation must reference the qualities of the evidence which make it hearsay. We introduced this measurement after discovering that for some tasks, LLMs often generate explanations which—though correct—largely restate the rule being applied, without any reference to the underlying facts. Explanations which are incorrect are automatically deemed to be insufficient on analysis. We compute an overall analysis score for a LLM on a task by measuring the proportion of samples for which the explanation contains sufficient analysis.
Generation tasks LEGALBENCH contains the following generation-tasks, which are evaluated as follows:
- rule_qa is a question-answer task in which a LLM must generate a response to an open-ended question. A law-trained individual evaluated the generations against an answer-key for the task, which is available for download from the website.
- The Securities Complaint Extraction tasks require an LLM to extract the names of different parties from excerpts of securities class-action complaints. Because some samples require the extraction of multiple entities, we evaluate using F1 score.
- definition Extraction requires the LLM to identify the term that is being defined in a sentence from a Supreme Court opinion. For a small number of sentences, any one of multiple terms may constitute the correct answer. We therefore evaluate performance on this task using accuracy, and by counting the fraction of sentences for which the LLM identified a permissible term. To account for edge-cases involving word tense, we compare stemmed versions of the answers to stemmed versions of the generation.
- sara_numeric requires an LLM to generate an estimate of the amount of tax that is owed. We compute performance here using an accuracy metric, which treats a prediction as accurate if it is within  $10\%$  of the true amount.
- citation_open requires an LLM to predict the name of the case that should be cited for a particular sentence of judicial text. We evaluate by checking if the LLM generation contains the correct case name.
- successor_liability requires an LLM to identify the multiple possible successor liability exceptions to a fact pattern. We evaluate using F1.
Classification tasks We evaluate all classification tasks in LEGALBENCH using exact-match on class-balanced-accuracy. We do this because a number of LEGALBENCH tasks are class-imbalanced.
G Task descriptions
This section provides a detailed description for each family of tasks.
G.1 Abercrombie
In LEGALBENCH, the Abercrombie task is denoted as abercrombie.
Background A particular mark (e.g., a name for a product or service) is only eligible for trademark protection if it is considered to be distinctive. In assessing whether a mark is distinctive, lawyers and judges follow the framework set out in the case Abercrombie & Fitch Co. v. Hunting World, Inc.,<sup>7</sup> which enumerates five categories of distinctiveness. These categories characterize the relationship between the dictionary definition of the term used in the mark, and the service or product it is being attached to. They are:
- Generic: A name is generic with respect to a product or service if it connotes the basic nature of the product/service, rather than more individualized characteristics of the product. For example, the mark "Salt" for packaged sodium chloride would be generic under Abercrombie because "salt" is the common name for sodium chloride. It is also common to think of generic marks as merely referring to the class of goods for which a particular product is a species.
- Descriptive: A name is descriptive if it identifies a characteristic or quality of an article or service, such as color, odor, function, dimensions, or ingredients. For example, the name "Sharp" for a television would be descriptive, because it describes a plausible characteristic of television (i.e., their sharp image quality).
- Suggestive: A name is suggestive if it suggests, rather than describes, some particular characteristic of the goods or services to which it applies. An essential aspect of suggestive names is that it requires the consumer to exercise the imagination in order to draw a conclusion as to the nature of the goods and services. For example, the name "Greyhound" would be suggestive for a bus service, because greyhounds are considered to be fast, and "fast" is an adjective that could be used to describe a bus service.
- Arbitrary: A name is arbitrary if it is a "real" word but seemingly "arbitrary" with respect to the product or service. For example, the mark "Apple" for a software company is arbitrary, because apples are unrelated to software.
- Fanciful: A name is fanciful if it is entirely made up, and not found in the English dictionary. For example, "Lanmbe" is a fanciful mark, because it is a made-up word.
The Abercrombie spectrum is commonly taught as part of Intellectual Property courses in law school, and students are expected to understand how to determine the Abercrombie classification for a particular product/mark combination.
Performing the Abercrombie task requires reasoning about the literal meaning of a word and the degree of its connection to a particular product/service. It requires having some understanding of the types of words that could plausibly be used to describe a particular good/service, and the extent to which those words relate to a particular mark. It also requires reasoning as to whether a particular word is a real English word.
Task The Abercrombie task requires an LLM to determine—given a candidate mark and a description of a product/service—which of the five Abercrombie categories above apply.
Facts	Abercrombie Classification
The mark "Whirlpool" for an oven.	arbitrary
The mark "Compact" for wallets.	descriptive
The mark "Imprion" for a line of sports drinks.	fanciful
The mark "Car" for a line of automobiles.	generic
The mark "Quick Green" for grass seed.	suggestive
Table 10: Task examples.
7Abercrombie & Fitch Co. v. Hunting World, 537 F.2d 4 (2nd Cir. 1976).
Construction process We manually create a dataset to evaluate a model's ability to classify a mark's distinctiveness (into one of the above 5 categories) with respect to a product. In writing samples, we draw inspiration from similar exercises available in legal textbooks and practice study guides. Hence, the samples provided have a definite answer, and are not subject to ambiguity. There is an expectation that a law student learning intellectual property would be able to answer these questions to a high degree of accuracy.
We create approximately 20 samples for each category of distinctiveness, and randomly select a single sample from each category to constitute the train set. The remaining 19 samples (for each category) are assigned to the test set (for a total of 95 samples).
Class	Number of samples
generic	19
descriptive	19
suggestive	20
arbitrary	18
fanciful	19
Table 11: Test set class distribution.
Significance and value Given how easy this task is for lawyers with a basic training in intellectual property law, it is unlikely that LLMs will be called on to perform this task in the actual practice of law, or that the ability for LLMs to perform this task would alter the way in which lawyers approach IP practice. Instead, the Abercrombie task is significant as a measurement of reasoning ability. Because it is "simplistic" by the standards of human lawyers, it provides a useful objective measure of reasoning progress for LLMs.
G.2 Canada Tax Court Outcomes
In LEGALBENCH, the Canada Tax Court Outcomes task is also denoted as canada_tax court_outcomes.
Background The Tax Court of Canada hears appeals of government decisions related to taxation.<sup>8</sup> The Court's decisions, which are written in natural language, are published on the Court's website, in both French and English.<sup>9</sup> Decisions typically include a section at the beginning summarizing the outcome of the appeal, followed by sections describing the factual background and various procedural steps, a section identifying the issues under consideration, sections with legal analysis, and a concluding section. While this is the standard format, judges are free to use other formats if they prefer. Decision length varies depending on the complexity of the litigation, with some decisions being only a few hundred words, and others being many thousands of words.
Appeals in Tax Court of Canada cases are brought by individuals or organizations who ask the Court to overturn a government taxation decision. Outcomes of appeals are generally binary: appeals are either granted, in which case the government taxation decision is overturned in whole or in part, or appeals are denied in which case the government taxation decision is upheld. Occasionally published decisions will not involve the outcome of an appeal, including where the decision is about a procedural step (e.g. the admissibility of particular evidence).
The canada_tax court\_outcomes task involves identifying whether an excerpt from a Tax Court of Canada decision includes the outcome of the appeal and, if so, what the outcome is. While the task is straightforward, one challenge is that the model must distinguish between outcomes of the appeal as a whole and outcomes of particular aspects of the appeal. Another challenge is that where the excerpt does not include the outcome, the model must avoid predicting the outcome - even if the model might plausibly correctly infer the likely outcome from the excerpt provided.
Task The Canada Tax Court Outcomes task requires an LLM to classify whether an excerpt from a given decision includes the outcome of the appeal, and if so whether the appeal was allowed or dismissed. Some excerpts do not include an outcome of the appeal, in which case the model should return 'other'. Where the excerpt includes the outcome and the appeal is allowed in whole or in part, the model should return 'allowed'. Where the excerpt includes an outcome, and the appeal is dismissed the model should return 'dismissed'. The model should disregard outcomes that are not about the ultimate outcome of the appeal, such as costs awards (i.e. orders requiring a party to pay the other party's legal costs).
Construction process We obtained the full text of English-language versions of decisions from 2001 to 2022 by scraping the Tax Court of Canada website. We then cleaned and parsed the text to extract excerpts that are most likely to contain the outcome of the appeal. For example, many decisions contain a brief introductory section describing the outcome of the appeal using a specific header, and if the decision contained a section with such a header, we excerpted only that section. Where our parsing code could not identify such a section, we excerpted the first and last 2,500 characters, because outcomes are generally described at either the beginning or end of decisions. After initially attempting outcome classification on these excerpts using OpenAI's ChatGPT, we selected a quasi-random sample of 250 excerpts (quasi random because we selected these manually, we oversampled excerpts where the outcome is 'other', and we chose some excerpts that were challenging due to factors such as length or unusual format). We manually reviewed outcomes for these excerpts, correcting some that had been miscategorized.
Two random cases from each class are selected for the training split, while the remainder are used as the test set.
Significance and value Legal scholars frequently gather data about outcomes in large numbers of legal decisions in order to examine patterns in judicial decision-making. For example, a legal scholar may be interested in comparing outcomes in similar processes across jurisdictions or they might examine whether a legislative change resulted in different outcomes over time. Lawyers and legal information technology companies may also be interested in gathering data on outcomes for the purposes of judicial analytics or to predict future outcomes.
Gathering such data is typically straightforward. It is, for example, a common task assigned to first year law student research assistants who can frequently achieve close to  $100\%$  accuracy on such tasks with only minimal training. However, because the data is often useful only when gathered on large numbers of decisions, this type of data gathering using human research assistants can be cost prohibitive. If LLMs can obtain high accuracy on these tasks, substantial savings could be achieved – which would increase the ability of researchers to pursue new projects.
$^{8}$ Tax Court of Canada Act, RSC, 1985, c T-2, online: https://laws-lois.justice.gc.ca/eng/acts/t-2/index.html, s 12.
${}^{9}$  Tax Court of Canada, "Find a Decision", online: https://decision.tcc-cci.gc.ca/tcc-cci/en/ nav.do.
<sup>10</sup>Ibid. As per the terms of service of the website, we are required to note that the text of the scraped decisions are not the official versions (official versions can be obtained from the website), and that the reproduction of these cases has not been produced in affiliation with or with the endorsement of the Government of Canada.
Excerpt	Outcome
The appeal is allowed in part and the assessment is referred back to the Minister of National Revenue for reconsideration and reassessment to reflect a 25% reduction of the tax owed by the appellant and adjustments to the interest and penalties, as agreed to by the respondent. Costs are to be determined after hearing both parties. In all other respects, the assessment is confirmed. Signed this 23rd day of February 2012. "Franois Angers" Angers J.	allowed
IN ACCORDANCE with the Reasons for Judgment attached, the appeal from the decision of the Respondent in relation to the income of the Appellant for the purposes of determining his entitlement to the Guaranteed Income Supplement under the Old Age Security Act for the payment period from July 1, 2014 to June 30, 2015 is dismissed, without costs. Signed at Ottawa, Canada, this 24th day of October 2017. R.S. Bocock Bocock J.	dismissed
(These Reasons for Judgment are issued in sub-stitution for the Reasons for Judgment signed on January 22, 2002) Lamarre, J. [1] These are appeals under the informal procedure against assessments made by the Minister of National Revenue ("Minister") under the Income Tax Act ("Act") for the 1995, 1996, 1997, 1998 and 1999 taxation years. [2] In filing her 1995 income tax return, the appellant claimed a business investment loss of $268,897 with respect to investments in eight mortgages held "in trust" for the appellant and her father Henry Sokolowski by Kiminco Acceptance Co. Ltd. ("Kiminco"), a member of the Glen Coulter group of companies. The eight mortgage investments were made in 1987 and 1988 and are identified as follows in paragraph 13 of the Reply to the Notice of Appeal: Account/Mortgage Number Ultimate Borrower ... For the Appellant: Name: Firm: For the Respondent: Morris Rosenberg Deputy Attorney General of Canada Ottawa, Canada	other
Table 12: Task examples.
Outcome	Number of samples
allowed	101
dismissed	131
other	12
Table 13: Test set class distribution.
G.3 Citation Prediction Tasks
In the LEGALBENCH, the Citation Prediction tasks are also denoted as citation_prediction_*.
Background The importance of locating relevant legal materials, or "law search" has long been recognized as an essential aspect of legal practice. This process involves uncovering case law, statutes, and other materials pertinent to legal questions or arguments. As a fundamental aspect of legal reasoning, law search plays a crucial role in bridging the gap between the initial translation of behaviors into legal questions and the subsequent interpretation and application of the relevant law.
Legal professionals are often valued for their ability to find and apply the appropriate law to their clients' situations. Given the intricate nature of the contemporary legal domain, the process of law search has evolved into a complex and nuanced task that demands a comprehensive understanding of the law.
A core component of law search is legal relevance. From a sociological perspective, the relevance of legal documents to a specific legal question is a social fact. This fact is determined by the judgments made by members of the legal community, who must determine which legal materials are applicable to a given question. Relevance relates legal questions to sources of legal authority.
In functional legal communities, law search leads to some degree of convergence over legal materials. Convergence occurs when competent members of a legal community, faced with the same legal question, identify the same sources of relevant legal authority. This process is essential to ensuring that the legal system operates consistently, predictably, and coherently.
As a critical process that connects the translation of behaviors into legal questions and the subsequent interpretation and application of the relevant law, law search is indispensable to legal reasoning.
The Citation Prediction task requires reasoning concerning the relationship between the text of judicial opinions and legal propositions. Successful prediction would entail encoding a notion of legal relevance and would allow a system to determine whether a legal proposition was or was not supported by the extant body of law.
Task The citation task is based on a version of the evaluation approaches used in [38]. There are two Citation Prediction tasks. The first (citation_prediction_classification) requires an LLM to predict whether a given sentence (i.e. legal proposition) is or is not supported by a given case. The second (citation_prediction_open) requires an LLM to predict a case (by name) that supports a provided sentence.
Construction process We collected a sample of circuit court opinions published after January 1, 2023. To the best of our knowledge, most existing LLMs haven't been trained on any data generated in 2023. For each opinion, we manually collected sentences which were supported by a citation to a judicial opinion, where (1) the sentence contained some quotation from the original case, and (2) the sentence was supported by a single cite. We chose sentences which included quotation fragments and were only supported by a single cite to avoid sentences which could be supported by a broad set of cases. When a sentence is supported by a much larger universe of cases, verifying that an LLM answer is incorrect is difficult. We also recorded the circuit for each opinion that we pulled language from. As a result, we can include the circuit information in the prompt, since circuits prefer citing their previous decisions. We collected 55 sentences using this process. For the citation generation task (citation_prediction_open), we ask the LLM to predict the citation given the sentence.
The citation_prediction_classification task is then constructed as follows. We use each sentence-citation pair to create two task samples. The first sample corresponds to the sentence and the correct citation (positive label). The second sample corresponds to the sentence and a randomly selected citation from the remainder of the data (negative label). This generates a dataset of 110 sentence-citation pairs, two of which are assigned to the training split.
Significance and value Law search is a core function of legal thinking. In addition, the difficulty of identifying relevant law is a core barrier in the public's ability usefully access the law. The ability of an LLM to accurately engage in citation prediction would have important practical value in providing access to law, and would also allow the LLM to more reliably support legal statements with relevant authority.