SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Construction process We construct a dataset of PROAs by hand, drawing inspiration from clauses found in different state codes. We construct 50 clauses which do contain a PROA, and 50 clauses which do not. Clauses which do not contain a private right may either create no cause of action, or empower a non-private individual (e.g., an attorney general) to bring a claim. 5 randomly sampled clauses constitute the training set, and the remaining 95 form the test set.
Input	Answer
The attorney general or an attorney representing the state may bring an action for an injunction to prohibit a person from violating this chapter or a rule adopted under this chapter.	No
The administrator may bring an action in a court of competent jurisdiction to enforce this chapter.	No
The sheriff or the sheriff's designee shall maintain a permanent personnel file oneach department employee.	No
If any laborer, without good cause, shall abandon his or her employer before the expiration of his or her contract, he or she shall be liable to his or her employer for the full amount of any account he or she may owe his or her employer.	Yes
No employer may discharge any employee by reason of the fact that earnings have been subjected to garnishment or execution. If an employer discharges an employee in violation of this section, the employee may within ninety days of discharge bring a civil action for recovery of twice the wages lost as a result of the violation and for an order requiring reinstatement.	Yes
In addition to all other penalties, rights, or remedies provided by law, an individual or entity that uses or attempts to use its official authority or influence for the purpose of interfering with the right of a legislative employee to make a protected disclosure is liable in a civil action for damages brought by a legislative employee.	Yes
Table 44: Examples for proa
Significance and value The PROA task evaluates an LLM's ability to perform a two-step reasoning test: (1) does the statute allow a party to bring a claim in court, and (2) is that party private? Law students and legal professionals should be capable of performing this task at near-perfect accuracy.
The PROA task derives additional significance from a recent movement towards studying state statutory language [153]. Legal scholars have long been unable to conduct large scale empirical studies of state statutory language, given the sheer volume of state statutes. The ability for LLMs to accurately classify or annotate statutes could thus empower new empirical studies of state statutes.
G.25 Rule QA
In LEGALBENCH, the Rule QA task is denoted as rule_qqa.
Background Lawyers are regularly required to recall specific legal rules that are drawn from cases, statutes, or other sources. Rules can take many shapes and forms. For instance, the rule pertaining to the federal requirements for a class (in a class action lawsuit) are codified in Rule 23(a) of the Federal Rules of Civil Procedure and are simply known as the need for "numerosity, commonality, typicality, and adequacy."
Task The Rule QA task evaluates a LLM's ability to answer questions on different legal rules. The rules are drawn from subjects typically studied in the first year of law school (e.g., civil procedure, constitutional law, etc.). This is an open-generation task.
Construction process We manually wrote 50 question-answer pairs, focusing on the types of rules which are regularly tested in law school courses on civil procedure, evidence, and intellectual property. The questions ask the LLM to either (1) restate a rule, (2) identify where a rule is codified, or (3) list the factors employed in a particular rule. Several questions explicitly narrow their scope to a jurisdiction (e.g., California state evidence law), in order to avoid bias towards merely federal law.
Question	Answer	Area of law
What are the four categories of patentable subject matter?	“process, machine, manufacture, or composition of matter.”	IP
What are the requirements for diversity jurisdiction?	Diversity jurisdiction exists when the amount in controversy exceeds $75,000 and the plaintiffs and defendants are completely diverse (i.e. no plaintiff shares a state of citizenship with any defendant)	Civil Procedure
Under which statute are patentable subject matter requirements codified?	35 USC 101	IP
What are the factors of the Mathews balancing test?	A three-part test that determines whether an individual has received due process under the Constitution. The test balances (1) the importance of the interest at stake; (2) the risk of an erroneous deprivation of the interest because of the procedures used, and the probable value of additional procedural safeguards; and (3) the government’s interest.	Constitutional law
Table 45: Examples for Rule QA
Significance and value The Rule QA task is an initial effort to evaluate the propensity for legal hallucination in LLMs. The questions asked are exceedingly basic, and law students taking the relevant course would be expected to answer them nearly perfectly.
G.26 SARA Tasks
In LEGALBENCH, the SARA tasks are denoted as sara_*.
Background An important skill for lawyers is to determine, given the facts of a case, whether a given law applies and what it prescribes. For example, does the payment received by the defendant on August 21st, 2017 qualify as wages under §3306(b) of the US Tax Code? This task has been introduced by [76] as statutory reasoning. [63] further introduce the Statutory Reasoning Assessment dataset (SARA). SARA contains (1) a set of 9 sections, taken from US federal tax law statutes, pruned and simplified; and (2) hand-crafted cases that test the understanding of those 9 sections. In this context, a case is a paragraph of text stating facts in plain language. Each case comes either with an entailment prompt — a statement about the statutes and the case that may be true or false — or a question — asking how much tax one of the case's protagonists owes. The SARA dataset is a simplified version of real-world cases, that retains many of the features of statutory reasoning for tax law. It poses, however, a significant challenge to NLP models [12].
Tasks There are two SARA tasks. The first, sara_entailment, corresponds to the entailment cases. The entailment cases state that a given law applies to a given case, and require the LLM to produce a binary answer — akin to Recognising Textual Entailment [39]. This is an approximation of real-world statutory reasoning, where the answer is usually not strictly binary.
The second task, Sara_numeric, consists of the numeric cases. Here, the goal is to compute the amount of tax owed. We frame this as a floating point number. To measure numerical accuracy, we use the metric introduced by [63], which includes a tolerance for inaccurate predictions.
Construction process We framed the SARA dataset for the paradigm of language modeling. Due to dependencies across sections in the statutes, the entirety of the statutes are generally relevant to determine the answer to any of the cases. However, all 9 sections do not fit into the LLM's context window, and must be pruned. In entailment cases, the entailment prompt specifies which law from the statutes to apply. We automatically extract the text of that law, and use it as the language model prompt. For numerical cases, the entirety of the statutes are relevant, and we use that as the language model prompt. Pruning is left to the LLM's pre-processing.
Significance and value Statutory reasoning is an important skill for lawyers, that is used within many other legal tasks. It is a fundamental task for legal AI, probing whether a computational model can understand and reason with legal rules expressed in natural language. The types of reasoning involved in SARA are diverse — defeasible, temporal, numerical reasoning, inter alia — and relevant beyond the legal domain. Statutory reasoning combines natural language understanding and logical reasoning, a major goal for AI.
If statutory reasoning were solved, it could serve as a basis for more complex legal tasks. For example, it could be used to automate the computation of taxes and benefits, without the need for coding the expert systems in use in many parts of the world [99]. A system that can do statutory reasoning would also be a step towards machine reading models that can analyze legislation and anticipate its effects, coming up with possible application scenarios [11]. As a final example, a statutory reasoning agent could be used for basic legal advice, increasing access to justice.
Task name: sara_entailment
Statute: (2) an individual legally separated from his spouse under a decree of divorce or of separate maintenance shall not be considered as married.
Description: Alice and Bob got married on April 5th, 2012. Alice and Bob were legally separated under a decree of divorce on September 16th, 2017.
Statement: Section 7703(a)(2) applies to Alice for the year 2018.
Answer: Entailment
Task name: sara_numeric
Statute: §3301. Rate of tax <br> <br> There is hereby imposed on every employer (as defined in section 3306(a)) for each calendar year an excise tax, with respect to having individuals....[Omitted from clarity]...This section shall not apply to any taxable year beginning after December 31, 2017, and before January 1, 2026. <br> <br>
Description: Bob is Charlie and Dorothy's son, born on April 15th, 2015. Alice married Charlie on August 8th, 2018. Alice's and Charlie's gross incomes in 2018 were  $324311 and$ 414231 respectively. Alice, Bob and Charlie have the same principal place of abode in 2018. Alice and Charlie file jointly in 2018, and take the standard deduction.
Question: How much tax does Alice have to pay in 2018?
Answer: $259487
Table 46: Example of each SARA task. For sara_numeric, we omit part of the statute for brevity.
G.27 SCALR
In LEGALBENCH, the SCALAR task is denoted as scalar.
Background Each case decided by the Supreme Court addresses a specific question presented for review. Both the questions and the Court's opinions are published on the Supreme Court's website.
Many of the Court's opinions are briefly described by other judges who recount the holdings of the Court in their own writing. For example, consider the following passage from State of South Carolina v. Key, 27971 (S.C. 2020; emphasis added):
The United States Supreme Court has addressed the constitutionality of warrantless blood draws in several DUI cases. See Schmerber, 384 U.S. at 770-71 (holding the warrantless blood draw of a DUI suspect was valid because the law enforcement officer, dealing with a car accident, could "reasonably have believed that he was confronted with an emergency, in which the delay necessary to obtain a warrant, under the circumstances, threatened 'the destruction of evidence'")....
We refer to these brief descriptions as 'holding statements' or 'holding parentheticals,' since they are often enclosed by parentheses. Identifying the holding parenthetical that corresponds to a question presented for review requires a notion of 'responsiveness' or relevance between questions and answers as well as an understanding of the kinds of legal issues that could be implicated by a specific question presented for review.
Task The SCALR benchmark is a collection of 571 multiple choice questions designed to assess the legal reasoning and reading comprehension ability of large language models. Each multiple-choice task gives the question presented for review in a particular Supreme Court case. The solver must determine which holding parenthetical describes the Court's ruling in response to the question presented. Here is an example from  $AT&T$  Mobility LLC v. Concepcion, 563 U.S. 333 (2011) with the correct response emphasized:
Question: Whether the Federal Arbitration Act preempts States from conditioning the enforcement of an arbitration agreement on the availability of particular procedures—here, class-wide arbitration—when those procedures are not necessary to ensure that the parties to the arbitration agreement are able to vindicate their claims.
A: holding that when the parties in court proceedings include claims that are subject to an arbitration agreement, the FAA requires that agreement to be enforced even if a state statute or common-law rule would otherwise exclude that claim from arbitration
B: holding that the Arbitration Act "leaves no place for the exercise of discretion by a district court, but instead mandates that district courts shall direct the parties to proceed to arbitration on issues as to which an arbitration agreement has been signed"
C: holding that class arbitration "changes the nature of arbitration to such a degree that it cannot be presumed the parties consented to it by simply agreeing to submit their disputes to an arbitrator"
D: holding that a California law requiring classwide arbitration was preempted by the FAA because it "stands as an obstacle to the accomplishment and execution of the full purposes and objectives of Congress," which is to promote arbitration and enforce arbitration agreements according to their terms
E: holding that under the Federal Arbitration Act, a challenge to an arbitration provision is for the courts to decide, while a challenge to an entire contract which includes an arbitration provision is an issue for the arbitrator
Construction The data used to create this task comes from two sources:
1. Questions presented were gathered from the Supreme Court of the United States' website, which hosts PDFs of questions granted for review in each case dating back to the 2001 Term.
2. Holding statements that comprise the "choices" for each question were compiled from (a) CourtListener's collection of parenthetical descriptions and (b) extraction of parenthetical descriptions from CourtListener's and the Caselaw Access Project's collections of court decisions using Eyecite.
Because questions presented for review in Supreme Court cases are not easily available prior to 2001, the benchmark is limited to questions from cases decided in the 2001 Term and later. To ensure that "holding" statements would address the particular question presented, we limited the set of cases to those in which exactly one question was granted for review. We also perform some manual curation to exclude questions which are not
answerable without specific knowledge of a case. For example, we eliminated a case that presented this question: "Whether this Court's decision in Harris v. United States, 536 U.S. 545 (2002), should be overruled."
To create choices for each question presented, we first filter our set of parenthetical descriptions as follows:
1. We limited our parenthetical descriptions to only those that begin with "holding that...", as these are most likely to describe the core holding of the case, rather than some peripheral issue.
2. We use only parentheticals that describe Supreme Court cases. This avoids the creation of impossible questions that ask the solver to distinguish between "holding" statements dealing with exactly the same issue at different stages of appellate review.
3. We then select for each case the longest parenthetical meeting the above criteria. We use the longest parenthetical because it is most likely to be descriptive enough to make the question answerable.
We then create a task for each case which has both a question presented and a "holding" statement meeting the above requirements. (While question-correct holding pairs are only for cases decided after 2001, we allow the use of parentheticals describing any Supreme Court case as alternative answer choices.) We then need to select the four alternative answer choices for each question in a manner that makes the task challenging. To select choices that are at least facially plausible, we find the four "holding" statements from the remaining pool that are most TF-IDF similar to the question presented. The inclusion of difficult alternative choices requires the solver to draw nuanced distinctions between legal issues that share overlap in terminology.
Significance and value This task is significant because it tracks the useful and challenging skill of identifying a passage as relevant or responsive to a given query. LLMs that are able to perform well at this task have the potential to be more useful for complex legal question-answering and retrieval. The poor performance of simpler models on this task demonstrates that it is a challenging one that requires a level of understanding beyond the word/synonym level.
G.28 Securities Complaint Extraction
In LEGALBENCH, the Securities Complaint Extraction tasks are denoted as ssla_**.
Background Securities Class Actions (SCAs) are lawsuits filed by, and on behalf of, investors alleging economic injury as a result of material misstatements or omissions in public disclosures made by corporate directors and officers. These actions allege violations of the Securities Act of 1933 and Securities Exchange Act of 1934 and are predominately filed in federal court, though in 2018 the United States Supreme Court determined actions brought under the '33 Act were permitted in state court.
"Plaintiff(s)" is the legal term to describe the individual, company, or organization bringing forth a lawsuit. Under the class action system, one or more plaintiffs are appointed "Lead Plaintiff" by the court to represent the interests of a larger group of "similarly situated" parties. In securities class actions, investors that suffered the greatest financial loss, often public pensions or unions, are appointed lead plaintiff.
"Defendant(s)" is the legal term to describe the individual, company, or organization that must defend himself against the alleged violations or misconduct outlined in the lawsuit. There is always at least one defendant. The majority of securities class actions name the company, its CEO and its CFO. Many name additional C-suite level officers, members of the Board of Directors and additional third-parties such as the company's independent auditor and the underwriters of public offerings.
Each designated lead plaintiff, and all named defendants, are explicitly identified under the "Parties" section of the class action complaint.
Tasks There are three extraction tasks.
- The plaintiff task requires an LLM to extract the named plaintiffs within a text.
- The individual defendants tasks require an LLM to extract named defendants who are individuals from within a text.
- The company defendants tasks require an LLM to extract named defendants who are corporations/companies from within a text.
For certain samples, the complaint excerpt may not exactly contain the answer. For example, the correct answer may be "Strongbridge Biopharma PLC," while the complaint only mentions "Strongbridge." In order to maintain fidelity to the workflow used by SSLA, we evaluate an LLM's ability to generate the official name of the entity, as represented in the answer. This requires the LLM to possess some background knowledge regarding official corporation names. We find that larger models are generally able to account for this.
Sometimes, the provided text will not explicitly name the plaintiff, an individual defendant, or a corporate defendant. In these cases, the LLM is expected to return "Not named".
Figure 2: Example of the typical SCA structure (Case 4:21-cv-05868-HSG)
Construction process Stanford Securities Litigation Analytics (SSLA) identifies, tracks, and aggregates data on the several hundred private shareholder lawsuits and public SEC/DOJ enforcements filed each year. SSLA fellows manually extract and analyze information including plaintiffs, defendants, judges, mediators, plaintiff and defense firms, key litigation events, real-time case statuses, settlement timing, settlement dollar amounts, attorneys' fees and expenses, and imposed SEC / DOJ penalties. There is no ambiguity regarding the answers for this task given its nature.
This dataset is an extract from the corpus of texts of securities class action complaints in the SSLA database. Given the typical structure and headings for these types of cases, this dataset represents text extracted from the complaint, between the sections titled "Parties" and "Substantive Allegations". For cases where the second
heading was not found, texts fragments were limited to 2,000 characters. Cases with both headings were then filtered to include only those with texts up to 2,000 characters, which excluded cases with longer "Parties" sections. Thus, all observations in this dataset are 2,000 characters or less.
Text was scraped from complaints using python's PyPDF2 library and left unformatted and uncleared. This training set includes several observations where the text does not include all or some of the named entities due to the method of text collection and variation in case structure. In several of these cases, plaintiff names are not present in the selected text because the plaintiff had been named earlier in the complaint.
Table 47: Examples from Securities Complaint Tasks
Task name: ssla_company_defendants
Excerpt: 6. Plaintiff Don L. Gross, as set forth in the accompanying certification and incorporated by reference herein, purchased the common stock of KCS during the Cla ss Period and has been damaged thereby. 7. Defendant KCS, headquartered in Kansas City, Missouri, operates railroads in the Midwest and Mexico that run north to south, unlike most other U.S. ra ilroads that run east to west. The Company's stock traded on the NYSE, an efcient market, during th e Class Period under the ticker symbol "KSU." As of October 11, 2013, there were more than 110 million shares issued and outstanding. 8. Defendant David L. Starling ("Starling"), at all relevant times, served as KCS's President and Chief Executive Officer ("CEO"). Case 4:14-cv-00345-BCW Document 1 Filed 04/15/14 Page 2 of 293 9. Defendant David R. Ebbrecht ("Ebbrecht"), at all relevant times, served as KCS's Executive Vice President and Chief Operating Officer ("COO"). 10. Defendant Patrick J. Ottensmeyer ("Ottensmeye r"), at all relevant times, served as KCS's Executive Vice President Sales & Marketing. 11. Defendant Michael W. Upchurch ("Upchurch"), at all relevant times, served as KCS's Executive Vice President and Chief Financial Officer ("CFO"). 12. Defendants Starling, Ebbrecht, Ottensmeyer and d Upchurch are colloc tively referred to herein as the "Individual Defendants." 13. During the Class Period, the Individual Defendants, as senior executive officers and/or directors of KCS, were privy to confidential and proprietary tary information concerning KCS, its operations, finances, financial condition and present and future business prospects. The Individual Defendants also had access to material adverse non-public information concerning KCS, as discussed in detail below. Because of their positions with KCS, the Individual Defendants had access to non-public information about its busine ss, finances, products, markets and present and future business prospects via interna
Answer: Kansas City Southern
Task name: ssla_individual_defendants