SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 19: Test set class distribution.
Clause	Question	Answer
This Agreement shall be gov-erned by and construed in ac-cordance with the laws of the State of New York, without giving effect to any choice of law or conflict of law provi-sions.	Does the clause discuss BIPA consent?	No
If a dispute arises between the parties under this Agreement that cannot be resolved through good faith negotia-tions within a reasonable pe-riod of time, such dispute shall be escalated to an executive of-ficer of each party for resolu-tion. If such executive officers are unable to resolve such dis-pute within a reasonable pe-riod of time after escalation, either party may pursue any available legal remedies.	Does the clause discuss how disputes may be escalated?	Yes
Table 20: Examples for contract_qa.
G.5 Consumer Contracts QA
In LEGALBENCH, the Consumer Contracts QA task is denoted as consumer_contracts_qa.
Background Consumer contracts govern many economic and social activities, ranging from retail purchases and online search to social media and entertainment. These contracts can affect consumers' access to services, control terms of payment, and determine the remedies available when consumers' rights are violated. Despite the importance of these legal agreements, consumers typically lack the time, expertise, and incentive to properly examine how consumer contracts impact their rights and interests. This issue is known as the "no-reading" problem [8, 7]. LLMs may offer a solution. By reading consumer contracts and explaining their legal ramifications, LLMs could enable consumers to better understand and exercise their legal rights in many everyday contexts.
Task The Consumer Contracts QA task, first introduced in [71], aims to examine the degree to which an LLM can understand certain consumer contracts. Specifically, the task is comprised of 200 yes/no legal questions relating to the terms of service of popular websites. Examples of questions are provided in the table below.
In addition to the original 200 questions, the task includes an alternatively worded version of all 200 questions. While each question's content is substantially the same across both versions of the question, the alternatively worded questions are, by design, less readable, that is, more difficult for a human to read. Comparing performance across the original questions and the alternatively worded questions can help assess an LLM's brittleness in performing the task at hand. An example is provided in the table below:
Construction process The task was introduced in [71]. To construct the dataset, an attorney drafted 200 yes/no questions relating to the terms of service of the 20 most-visited U.S. websites (10 questions per document), as well as an alternatively worded version of all 200 questions. The questions relate to a wide range of legal issues arising in the terms of service, including eligibility to access services, payment for services, limitations of liability, intellectual property rights, and dispute resolution procedures. Answers to all questions can be obtained from the applicable terms of service.
Significance and value Given the ubiquity of consumer contracts, LLMs capable of reading these documents and communicating their contents to consumers might offer significant benefits. These benefits, however, are contingent on a model's accuracy and reliability. LLMs that misinterpret the provisions of consumer contracts may hinder consumers' ability to understand and exercise their contractual rights. The Consumer Contracts QA task is a preliminary attempt at evaluating the ability of LLMs to read certain consumer contracts.
Contract: Content Removal and Disabling or Terminating Your Account
We can remove any content or information you share on the Service if we believe that it violates these Terms of Use, our policies (including our Instagram Community Guidelines), or we are permitted or required to do so by law. We can refuse to provide or stop providing all or part of the Service to you (including terminating or disabling your access to the Facebook Products and Facebook Company Products) immediately to protect our community or services, or if you create risk or legal exposure for us, violate these Terms of Use or our policies (including our Instagram Community Guidelines), if you repeatedly infringe other people's intellectual property rights, or where we are permitted or required to do so by law. We can also terminate or change the Service, remove or block content or information shared on our Service, or stop providing all or part of the Service if we determine that doing so is reasonably necessary to avoid or mitigate adverse legal or regulatory impacts on us. If you believe your account has been terminated in error, or you want to disable or permanently delete your account, consult our Help Center. When you request to delete content or your account, the deletion process will automatically begin no more than 30 days after your request. It may take up to 90 days to delete content after the deletion process begins. While the deletion process for such content is being undertaken, the content is no longer visible to other users, but remains subject to these Terms of Use and our Data Policy. After the content is deleted, it may take us up to another 90 days to remove it from backups and disaster recovery systems.
Content will not be deleted within 90 days of the account deletion or content deletion process beginning in the following situations:
where your content has been used by others in accordance with this license and they have not deleted it (in which case this license will continue to apply until that content is deleted); or
where deletion within 90 days is not possible due to technical limitations of our systems, in which case, we will complete the deletion as soon as technically feasible; or
where deletion would restrict our ability to: investigate or identify illegal activity or violations of our terms and policies (for example, to identify or investigate misuse of our products or systems); protect the safety and security of our products, systems, and users; comply with a legal obligation, such as the preservation of evidence; or comply with a request of a judicial or administrative authority, law enforcement or a government agency; in which case, the content will be retained for no longer than is necessary for the purposes for which it has been retained (the exact duration will vary on a case-by-case basis).
If you delete or we disable your account, these Terms shall terminate as an agreement between you and us, but this section and the section below called "Our Agreement and What Happens if We Disagree" will still apply even after your account is terminated, disabled, or deleted.
Question: According to the terms, 30 days after I've asked to delete content, can other users see that content?
Contract: 16. Termination You may terminate these Terms at any time and for any reason by deleting your Account and discontinuing use of all Services. If you stop using the Services without deactivating your Account, your Account may be deactivated due to prolonged inactivity.
We may suspend or terminate your Account, moderator status, or ability to access or use the Services at any time for any or no reason, including for violating these Terms or our Content Policy.
The following sections will survive any termination of these Terms or of your Account: 4 (Your Content), 6 (Things You Cannot Do), 10 (Indemnity), 11 (Disclaimers), 12 (Limitation of Liability), 13 (Governing Law and Venue), 16 (Termination), and 17 (Miscellaneous).
17. Miscellaneous These Terms constitute the entire agreement between you and us regarding your access to and use of the Services. Our failure to exercise or enforce any right or provision of these Terms will not operate as a waiver of such right or provision. If any provision of these Terms is, for any reason, held to be illegal, invalid, or unenforceable, the rest of the Terms will remain in effect. You may not assign or transfer any of your rights or obligations under these Terms without our consent. We may freely assign any of our rights and obligations under these Terms.
Question: Will certain terms remain in force notwithstanding a users termination of the service?
Table 21: Task examples.
Original wording	Alternative wording
Am I allowed to be paid for writing a Wikipedia article, assuming I disclose who’s paying me?	Are Wikipedia contributors permitted to receive payment in respect of their contributions, provided they disclose the identity of the person or institution providing such payment?
Table 22: Example of reworded question.
G.6 Contract NLI Tasks
In LEGALBENCH, the Contract NLI tasks are denoted as contract_nli_*
Task The Contract NLI tasks require a LLM—given an excerpt of a contract and an assertion about the legal effect of that excerpt—to determine whether the assertion is supported or unsupported by the excerpt.
Construction process These tasks are constructed by transforming data released by [72]. The original dataset consists of 607 contracts and 17 assertions (e.g., "Receiving Party shall not disclose the fact that Agreement was agreed or negotiated"). Each contract is labeled for each assertion as supporting, negating, or not mentioning the assertion. Please refer to the original paper for details on annotation.
We restructure this dataset for a short-context LLM setting. Specifically, we treat each assertion as a separate task, where the objective is to determine whether a contract excerpt is supportive (or not) of the assertion. For each instance where a contract is supportive of an assertion, [72] has annotated the excerpt of the contract that is supportive. When creating a task, we use the supportive excerpts for the assertion from the test set as positive instances. To generate negative instances, we combine excerpts where the assertion is contradicted with a random sample of excerpts associated with other assertions. We treat both groups of excerpts as instances which are "unsupportive" of the assertion. We transform the assertion into a Yes/No question, where the LLM is asked to determine if a clause satisfies the assertion.
Table 23 lists each task, the assertion associated with the task, and an example of an excerpt which supports the assertion.
Table 23: ContractNLI Tasks
Task name: contract_nli_return_of_confidential_information
Question: Identify if the clause provides that the Receiving Party shall destroy or return some Confidential Information upon the termination of Agreement.
Example: Upon receipt by the Recipient of a written demand from the Disclosers: 8.1.1 the Recipient must return or procure the return to the Disclosers or, as the Disclosers may require, destroy or procure the destruction of any and all materials containing the Confidential Information together with all copies; 8.1.2 if the Disclosers requires, the Recipient must provide the Disclosers with a certificate or such other evidence as the Disclosers may reasonably require duly signed or executed by an officer of the Recipient confirming that the Recipient has complied with all of its obligations under this Agreement including about return, destruction and deletion of Confidential Information and media; 8.1.3 the Recipient must delete or procure the deletion of all electronic copies of Confidential Information; and 8.1.4 the Recipient must make, and procure that the Authorised Persons shall make, no further Use of the Confidential Information.
Task name: contract_nli_no_licensing
Question: Identify if the clause provides that the Agreement shall not grant Receiving Party any right to Confidential Information.
Example: No license to the receiving party under any trade secrets or patents or otherwise with respect to any of the Proprietary Information is granted or implied by conveying proprietary Information or other information to such party, and none of the information transmitted or exchanged shall constitute any representation, warranty, assurance, guaranty or inducement with respect to the infringement of patents or other rights of others.
Task name: contract_nli_confidentiality_of_agreement
Question: Identify if the clause provides that the Receiving Party shall not disclose the fact that Agreement was agreed or negotiated.
Example: In addition, except as permitted herein, Recipient shall not disclose the fact that the parties are exchanging Confidential Information and having discussions. In connection therewith, it is agreed that no public release or disclosure of any contemplated transaction shall be made except by a mutually agreed disclosure except that each party may make such disclosure if advised by its outside securities counsel in writing that such disclosure is required; PROVIDED, HOWEVER, that in such event such party will notify the other party that it intends, as a preliminary matter, to take such action and the outside securities counsel of such party shall first discuss the mater with the outside securities counsel of the other party before any definitive decision is made on the disclosure.
Task name: contract_nli_explicit_identification
Question: Identify if the clause provides that all Confidential Information shall be expressly identified by the Disclosing Party.
Significance and value The Contract NLI tasks evaluate an LLM's capacity to reason over the rights and obligations created by a contract. The ability to perform this skill is essential to many types of legal work.
G.7 Corporate Lobbying
In LEGALBENCH, the Corporate Lobbying task is denoted as corporate_lobbying.
Background A significant amount of effort is devoted to identifying developing sources of law which implicate client or issue interests. Examples of such sources include: legislative bills, proposed regulations, or in-progress litigation. Identifying these sources serves multiple purposes. From a scholarly standpoint, researchers often aggregate sources into issue-focused databases, enabling them to identify emerging trends or patterns across different sources [36]. From an advocacy standpoint, identifying sources allows affected groups to better understand how their rights or obligations may be affected, and how to focus efforts on interacting with courts, legislatures, and other governmental bodies [15, 81, 43].
Task The Corporate Lobbying task requires an LLM to determine whether a proposed Congressional bill may be relevant to a company based on a company's self-description in its SEC 10K filing. The following information about a bill and a company are available:
- The title of the bill.
- A summary of the bill.
- The name of the company.
- A description of the company.
We expect higher accuracy of LLM predictions if we were to provide the model with more data about a bill, and especially if we provide it with more data about a company. Proprietary applications of this approach could leverage significant internal company data. More expensive deployments could leverage the full text of the bill
Construction process This data was manually labeled. This work was an extension of the research described in [90].
Significance and value Determining whether a particular bill is relevant for a company requires (1) identifying the legal consequences of the bill, and (2) whether those consequences are relevant to a company's business model, structure, or activities. As discuss above, this type of prognostication is a common legal practice. For instance, law firms regularly publish "client alerts" which seek to keep clients updated on new legal developments [120].
Class	Number of samples
No	345
Yes	145
Table 24: Test set class distribution
Table 25: An example of a relevant bill for corporate_lobbying.
G.8 Definition Tasks
In LEGALBENCH, the Definition Tasks are denoted as definition_classification and definition Extraction.
Background Judicial opinions regularly involve definition, assigning a particular meaning to words or phrases (Let us define words and phrases as "terms"). Definition of terms can occur when judges introduce or discuss legal concepts (e.g. parol evidence), and it frequently occurs when judges interpret terms in legal texts. This can include language from past judicial opinions and language appearing in legal texts like contracts, statutes, and the Constitution. Historically, interpreters have often evaluated the definition(s) of individual words. For example, in interpreting the meaning of "keep and bear arms" in the Second Amendment, courts consider the definition(s) of individual words (like "bear"). This approach—focusing on terms' definitions—has only increased in recent decades with the rise of textualist approaches to constitutional and statutory interpretation.
Judicial opinions define a wide range of terms, including ordinary terms, legal terms, and scientific terms. They also appeal to a wide range of defining sources, including ordinary dictionaries, legal dictionaries, and legal texts. For an example of the last, consider statutory definitions: 1 U.S.C. 1 offers generally applicable definitions of many frequent statutory terms.
It is useful for lawyers to identify when definition occurs (definition classification), as well as which terms have been defined (definition extraction). These tasks might seem simple at first. There are some intuitively plausible indicators of definition classification and extraction. For example, defined terms often (but not always) appear in quotation marks or near a citation to a dictionary.
However, these tasks are not entirely straightforward. Indicators like quotation will not lead to perfect definition classification and extraction. Consider for example, this sentence from the dataset related to the definition of "confidential": The term "confidential" meant then, as it does now, "private" or "secret." Webster's Seventh New Collegiate Dictionary 174 (1963).<sup>11</sup> As another example from the dataset, consider this definition of "brought": But a natural reading of § 27's text does not extend so far. "Brought" in this context means "commenced," Black's Law Dictionary 254 (3d ed. 1933).<sup>12</sup> Other examples exclusively quote the definition, rather than defined terms: Stare decisis ("to stand by things decided") is the legal term for fidelity to precedent. Black's Law Dictionary 1696 (11th ed. 2019).<sup>13</sup> In all of these examples, the presence of a dictionary would not indicate which term is extracted. In other examples, there is no dictionary cited; there is not a perfect correlation between dictionary citation and classification of a sentence as a defining one.<sup>14</sup>
Tasks The Definition Classification task requires an LLM to determine-given an excerpt from a Supreme Court opinion-whether the excerpt is defining any term (Yes/No). The Definition Extraction task requires an LLM to determine-given an excerpt from a Supreme Court opinion-which term the excerpt is defining (Open-ended response).
Construction process An original hand-coded dataset was constructed to study how the Supreme Court relies on dictionaries over time. Any case citing a dictionary was included in the dataset, and human coders identified relevant excerpts that defined terms and which terms were defined.
That dataset has been repurposed for the task here. For the definition extraction task, the original dataset includes the relevant information (excerpts, with the defined term coded separately).
For the definition classification task, the original dataset includes examples of language defining terms. To create a set of non-defining language, Neel Guha randomly selected similarly long excerpts of text from the same Supreme Court opinions. Kevin Tobia analyzed those randomly selected excerpts, identifying any that include definitions (for removal). The resulting dataset has 691 sentences which define sentences, and 646 sentences which do not.
Significance and value This is not a particularly difficult task for human lawyers, and it is unlikely that LLMs would replace lawyers as experts in this process. However, it is possible that LLMs successful in these tasks could provide beneficial legal research roles (e.g. quickly identifying all prior definitions of a specific term in a particular jurisdiction).
Moreover, the definition extraction task serves as a useful test of LLMs abilities, given the task's open-ended nature. The task is not limited to a small set of possible answers (e.g. Yes, No). Rather, it requires identifying which term of all terms in an excerpt is defined. Most of these choices will admit of over ten possible answers (i.e. excerpts of over ten words). Moreover, there is great variety in the language used across the examples. There are hundreds of possible answers, across all items.
<sup>11</sup>Food Mktg. Inst. v. Argus Leader Media, 139 S. Ct. 2356, 2363 (2019).
$^{12}$ Merrill Lynch, Pierce, Fenner & Smith Inc. v. Manning, 136 S. Ct. 1562, 1568 (2016).
13June Medical Services L.L.C. v. Russo, 140 S. Ct. 2103, 2134 (2020).
<sup>14</sup>E.g. "And "remuneration" means "a quid pro quo," "recompense" or "reward" for such services. Id., at 1528." BNSF Ry. Co. v. Loos, 139 S. Ct. 893, 905 (2019).
Sentence	Definition sentence?
The risk of that consequence ought to tell us that something is very wrong with the Court's analysis.	No
This term has long referred to a class of expenses commonly recovered in litigation to which attorney's fees did not traditionally be long. See Black's Law Dictionary 461 (1891) (defining “expensae litis” to mean “generally allowed” costs); 1 J. Bouvier, Law Dictionary 392 (1839) (defining the term to mean the “costs which are generally allowed to the successful party”); id., at 244 (excluding from the definition of “costs” the “extraordinary fees [a party] may have paid counsel”).	Yes
Table 26: Examples for definition_classification.
Sentence	Defined term
The term “plaintiff” is among the most commonly understood of legal terms of art: It means a “party who brings a civil suit in a court of law.” Black’s Law Dictionary 1267 (9th ed. 2009) see also Webster’s Third New International Dictionary 1729 (1961)”	plaintiff
The ordinary understanding of law enforcement includes not just the investigation and prosecution of offenses that have already been committed, but also proactive steps designed to prevent criminal activity and to maintain security.	law enforcement
Table 27: Examples for definition Extraction.
G.9 Diversity Jurisdiction
In LEGALBENCH, the Diversity Jurisdiction tasks are denoted as diversity_*
Background Diversity jurisdiction is one of two ways in which a federal court may have jurisdiction over a lawsuit pertaining to state law. Diversity jurisdiction exists when there is (1) complete diversity between plaintiffs and defendants, and (2) the amount-in-controversy (AiC) is greater than $75,000.
"Complete diversity" requires that there is no pair of plaintiff and defendant that are citizens of the same state. However, it is acceptable for multiple plaintiffs to be from the same state, or for multiple defendants to be from the same state.
The AiC requirement allows for certain forms of aggregation. Specifically, if plaintiff A asserts two independent claims against defendant B, the value of the claims may be added together when considering if the AiC requirement is met. However, a plaintiff may not aggregate the value of claims against two separate defendants, and two plaintiffs may not aggregate claims against the same defendant.
Tasks We define six different tasks, each of which tests the diversity jurisdiction rule under a different pattern of facts. The diversity jurisdiction tasks are:
- diversity_1: The fact patterns consists of one plaintiff, one defendant, and one claim per plaintiff-defendant pair.
- diversity_2: The fact patterns consists of one plaintiff, two defendants, and one claim per plaintiff-defendant pair.
- diversity_3: The fact patterns consists of one plaintiff, one defendant, and two claims per plaintiff-defendant pair.
- diversity_4: The fact patterns consists of two plaintiffs, one defendant, and one claim per plaintiff-defendant pair.
- diversity_5: The fact patterns consists of two plaintiffs, one defendant, and two claims per plaintiff-defendant pair.
- diversity_6: The fact patterns consists of two plaintiffs, two defendants, and two claims per plaintiff-defendant pair.