SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Input	Citation	Supported?
Exclusions are always strictly construed against the insurer and in favor of the insured.	Nationwide Mut. Ins. Co. v. Cosenza	Yes
The Supreme Court and this court have repeatedly "held that environmental plaintiffs adequately allege injury in fact when they aver that they use the affected area and are persons for whom the aesthetic and recreational values of the area will be lessened by the challenged activity."	United States v. Pearce	No
Table 14: Examples for citation_prediction_classification.
Input	Citation
In other words, the DJA "creates a means by which rights and obligations may be adjudicated in cases involving an actual controversy that has not reached the stage at which either party may seek a coercive remedy."	United States v. Doherty
To be “equivalent to a demotion,” the action need not “result in a decrease in pay, title, or grade; it can be a demotion if the new position proves objectively worse—such as being less prestigious or less interesting or providing less room for advancement.”	Alvarado v. Tex. Rangers
Table 15: Examples for citation_prediction_open.
G.4 Clause Classification Tasks
LEGALBENCH includes a number of tasks in which the LLM must determine the "type" or "category" of a provision/clause in a legal document. Specifically:
- The Contract QA task (Section G.4.4), in which the LLM is provided with the name of a common type of contractual clause and a clause, and must determine if the clause is an example of the example type.
- 38 tasks derived from the CUAD dataset (Section G.4.1), where each task is a binary-classification task requiring the LLM to identify whether a clause (from an EDGAR contract) belongs to a certain category (e.g., audit rights clauses) [60].
- The J.Crew blocker task (Section G.4.2), in which the LLM must classify whether a clause (from a loan agreement) is a J-Crew blocker provision.
- The Unfair Terms of Service task (Section G.4.3), in which the LLM must classify a clause (from a terms of service agreement) to one of eight types, where seven of the types denote clauses that would potentially be considered "unfair" under European law [80].
Significance and value Lawyers spend significant time and energy reviewing legal documents (e.g., contracts, leases, etc.). Manual review serves an important purpose, allowing parties to identify potentially problematic terms [117]. Parties will sometimes review agreements that have already been signed, in response to changing world events. For instance, the COVID-19 pandemic led many firms to inspect agreements for the existence of force majeure clauses, which ordinarily specify how contractual expectations should be handled in the event of major world crises [48]. Because legal documents are long and require legal training to understand, the process of reviewing is often extremely expensive [60]. This, in turn, presents significant access-to-justice concerns. Because most individuals do not have the financial capacity to consult lawyers prior to entering legal agreements, they are oblivious to when those agreements contain predatory, oppressive, or unconscionable terms. A rich legal scholarship has noted, for instance, the frequency at which legal agreements contain terms that would be invalidated by a court [25].
The clause classification tasks in LEGALBENCH are thus amongst the most practically useful tasks in LEGAL-BENCH, as they capture an actual current-day use case for LLMs. As the complexity of clause classification depends both on the clause category and document type, LEGALBENCH tasks span a range of clause categories and source documents.
G.4.1 CUAD Tasks
We adapt the CUAD dataset for LEGALBENCH [60]. The original dataset consists of 500 contracts, each annotated with up to 41 different clause types. These contracts varied significantly in length, ranging from a few pages to over one-hundred pages. In the original word, [60] studied the ability for BERT-base language models to identify the text spans corresponding to different types of clauses. The principal difficulties were (1) the length of the contract, and (2) the lack of significant training data.
We adapt the CUAD dataset as follows. We select 38 of the 41 clause categories. For each selected category, we construct a dataset consisting of (1) clauses in the CUAD contracts which are assigned to that category, and (2) an equal number of clauses randomly sampled from other categories. This produces a balanced binary classification task for clause category, where the purpose is to identify which clauses belong to the respective category. A table with the selected categories, and their descriptions is found below.
Table 16 lists each task, a "description" of the category corresponding to the task, and an example of a clause which meets the category criteria. In accordance with [60], the description is presented as the question posed to the annotators during data labeling. If a clause yields an affirmative answer with regards to the question, then the label is "Yes". Otherwise the label is "No".
In LEGALBENCH, the CUAD tasks are denoted as cuad_*.
Table 16: CUAD Tasks
Task name: cuad_affiliatelicense-licensee
Description: Does the clause describe a license grant to a licensee (incl. sublicense) and the affiliates of such licensee/sub licensor?
Example:  $[***]$ , Valeant hereby grants to Dova a fully paid-up, royalty free, non-transferrable, non-exclusive license (with a limited right to sub-license to its Affiliates) to any Valeant Property that appears on, embodied on or contained in the Product materials or Product Labeling solely for use in connection with Dova's promotion or other commercialization of the Product in the Territory.
Task name: cuadclosest_favored_nation
Description: Does the clause state that if a third party gets better terms on the licensing or sale of technology/goods/services described in the contract, the buyer of such technology/goods/services under the contract shall be entitled to those better terms?
Example: Eutectix agrees that in the event any Licensed Products shall be sold (1) to any Affiliate (as defined herein), or (2) to a corporation, firm, or association with which, or individual with whom Eutectix or its stockholders or Affiliates shall have any agreement, understanding, or arrangement (such as, among other things, an option to purchase stock, or an arrangement involving a division of profits or special rebates or allowances) without which agreement, understanding, or arrangement, prices paid by such a corporation, firm, association or individual for the Licensed Products would be higher than the Net Sales Price reported by Eutectix, or if such agreement, understanding, or arrangement results in extending to such corporation, firm, association, or individual lower prices for Licensed Products than those charged to outside concerns buying similar products in similar amounts and under similar conditions, then, and in any such events, the royalties to be paid hereunder in respect of such Licensed Products shall be computed based on an assumed or deemed Net Sales Price equal to those charged to such outside concerns.
Task name: cuad_no-solicit_ofCustomers
Description: Does the clause restrict a party from contracting or soliciting customers or partners of the counterparty, whether during the contract or after the contract ends (or both)?
Example: During the Term of this Agreement, and for a period of one year thereafter, except as expressly provided in this Agreement, PlanetCAD shall not market any services to Customers without the prior written approval of Dassault Systemes.
Task name: cuad_no-solicit_of_employees
Description: Does the clause restrict a party's soliciting or hiring employees and/or contractors from the counterparty, whether during the contract or after the contract ends (or both)?
Example: You covenant that during the term of this Agreement, except as otherwise approved in writing by us, you will not, either directly or indirectly, for yourself, or through, on behalf of, or in conjunction with any person, persons, partnership, corporation or company:<omitted>2. Employ or seek to employ any person who is at that time employed by us, our affiliates, or by any other franchisee of ours, or otherwise directly or indirectly induce or seek to induce such person to leave his or her employment thereat.
Task name: cuad_non-compete
Description: Does the clause restrict the ability of a party to compete with the counterparty or operate in a certain geography or business or technology sector?
Example: Agent may not offer or promote competitive products without the consent of Kallo.
Task name: cuad_non-disparagement
Description: Does the clause require a party not to disparage the counterparty?
Example: The Company shall not tarnish or bring into disrepute the reputation of or goodwill associated with the Seller Licensed Trademarks or Arizona.
Task name: cuad_non-transferable LICENSE
Description: Does the clause limit the ability of a party to transfer the license being granted to a third party?
Example: Subject to the terms and conditions of this Agreement, Licensor hereby grants to Licensee, and Licensee hereby accepts from Licensor, a personal, non-exclusive, royalty-free right and license to use the Licensed Mark solely and exclusively as a component of Licensee's own corporate name and in connection with marketing the investment management, investment consultation and investment advisory services that Investment Advisor may provide to Licensee.
Task name: cuad_post-termination_services
Description: Does the clause subject a party to obligations after the termination or expiration of a contract, including any post-termination transition, payment, transfer of IP, wind-down, last-buy, or similar commitments?
Example: Cisco agrees to repurchase all Product in Distributor's inventory within  $[****]$  days following the effective date of termination or expiration.
Task name: cuad_pricerestrictions
Description: Does the clause place a restriction on the ability of a party to raise or reduce prices of technology, goods, or services provided?
Example: The prices set forth in Section 2.4(a) shall be subject to adjustment annually on the first day of each Product Year beginning in the calendar year 2000 and on the first day of each succeeding Product Year for the remainder of the Term and all renewals of this Agreement in proportion to the increase or decrease in the Consumer Price Index (CPI) as compared to the CPI as it existed on the first day of the Term of this Agreement.
Task name: cuad_revenue-profit_sharing
Description: Does the clause require a party to share revenue or profit with the counterparty for any technology, goods, or services?
Table 16 - continued from previous page
Task name: cuad_governing_law
Description: Does the clause specify which state/country's law governs the contract?
Example: This Agreement shall be governed by and construed under the laws of the State of California, excluding conflict of laws provisions and excluding the 1980 United Nations Convention on Contracts for the International Sale of Goods.
Task name: cuad Notice_period_toTerminate_renewal
Description: Does the clause specify a notice period required to terminate renewal?
Example: Unless either party gives written notice to terminate this Agreement at least six (6) months prior to the end of said Initial Term, this Agreement shall continue on a year to year basis ("Extended Term(s)") until terminated by either party by giving written notice of termination thereof to the other party at least six (6) months prior to the end of the then current Extended Term.
Task name: cuad_renewal_term
Description: Does the clause specify a renewal term?
Example: This Agreement shall commence on the Effective Date and, unless sooner terminated in accordance with its terms, including by Ginkgo pursuant to Section 7.3 (Buy-Down Election) or extended by the mutual written agreement of the Parties, shall continue until the Intended End of Term (such time period, as may be extended pursuant to this Section 13.3.1 (Term - General), the "Term"); provided that, if,<omitted>at the expiration of the Intended End of Term, Ginkgo has paid the Minimum Cumulative Purchase Commitment, but will not have paid to BLI the Full Purchase Target, then the Term of this Agreement shall automatically extend for an additional  $[^{***}]$  ([\*\*\*]) year period from the date of the expiration of the then-Intended End of Term so that, among other things, BLI may potentially receive the benefit of the Full Purchase Target and Ginkgo may receive the continuing benefit of royalty-free licenses.
Task name: cuad_warranty_duration
Description: Does the clause specify a duration of any warranty against defects or errors in technology, products, or services provided under the contract?
Example: Airspan warrants that, following repair or replacement, the repaired or replaced Equipment or Software by Airspan shall be free from defects in materials and faulty workmanship and that the Software will conform in all material respects to Airspan's published specifications therefor ninety (90) days from date of shipment from Airspan to Distributor or until the end of the Initial Warranty Period, whichever is longer.
G.4.2 J.Crew Blocker
In LEGALBENCH, the J.Crew Blocker task is denoted as jcrew-blocker.
Background Loan agreements often contain restrictive covenants that place limits on a borrower's activities to protect the lender's interests. One such restrictive covenant that has become popular in recent years is the "J.Crew blocker" provision. This provision was created in response to actions taken by the retailer J.Crew in 2016. J.Crew transferred valuable intellectual property assets out of the collateral pool for its existing loans by moving them into a new unrestricted subsidiary. This subsidiary was then able to use the IP assets as collateral to obtain new financing.
The J.Crew blocker provision aims to prevent this type of activity by prohibiting borrowers from transferring IP assets out of the reach of existing lenders. There are two key components to a J.Crew blocker:
1. A prohibition on transferring IP assets to unrestricted subsidiaries. This prevents the borrower from moving assets outside the scope of lender restrictions.
2. A requirement to obtain lender consent for any IP transfers to subsidiaries. This gives lenders oversight and control over how IP assets are distributed within the corporate group.
The presence of a robust J.Crew blocker in a loan agreement is designed to keep material assets within the collateral pool, and thereby protect lenders from borrowers' attempts to secure additional debt through unexpected transfers of IP. For this reason, J.Crew blocker provisions have been widely adopted in leveraged loan agreements.
Task The J.Crew blocker task requires determining whether a given provision in a loan agreement qualifies as a J.Crew blocker. To make this determination, the provision must be analyzed to assess whether it contains:
1. A prohibition on transferring IP assets to unrestricted subsidiaries
AND/OR
2. A requirement to obtain lender consent for IP transfers to any subsidiary.
If the provision includes one or both of these components, it can be classified as a J.Crew blocker. If not, the provision does not meet the criteria.
Construction process The dataset for this task was constructed by legal experts extracting real examples of provisions from public loan agreements. Each example was labeled as either meeting the criteria for a J.Crew blocker or not. The dataset contains 60 total examples, organized into two columns: "Text" (containing the clause in question) and "Label" (indicating whether the clause is a J.Crew Blocker provision). Each clause was analyzed and classified as a J.Crew Blocker provision ("Yes") or not ("No"). The construction process involved manually reviewing and annotating these samples, ensuring that each clause was accurately categorized. This process, carried out by legal experts, provides definitive answers to each sample, eliminating ambiguity.
Significance and value The ability to identify J.Crew blocker provisions is important for both lenders and borrowers in leveraged finance. For lenders, it helps ensure key protections are included in loan agreements. For borrowers, it provides insight into restrictions being placed on their activities. Given the widespread adoption of J.Crew blockers, this is a task that requires proficiency to actively participate in the leveraged loan market. The task serves as an important measure of an LLM's ability to understand and apply legal concepts, particularly those related to secured lending and intellectual property law. It also tests the LLM's capacity to analyze and interpret legal provisions. Given the increasing complexity and sophistication of financial transactions, the ability to accurately identify and understand such provisions is a valuable skill for any LLM. This task, therefore, provides a useful measure of progress for LLMs in their understanding and interpretation of complex legal clauses.
Table 17: Examples from jcrew_blocker.
G.4.3 Unfair Terms of Service
In LEGALBENCH, the Unfair Terms of Service task is denoted as unfair_tos.
Background An array of recent work has found that consumers rarely read terms of service agreements [95, 62]. As a result, consumers regularly sign agreements or contracts containing provisions that (1) they lack awareness of, and/or (2) would consider as "unfair" or "predatory." Reasons for this phenomenon include the sheer amount of time it would take to read every terms of service agreement, the obtuse language of these agreements, and the lack of actual recourse on an individual basis.
With reference to European consumer law, [80] identify eight categories of clauses in terms-of-service agreements which could be considered "potentially unfair":
- Arbitration: clauses which mandated that all disputes between the parties would be resolved through arbitration.
- Unilateral change: clauses which allow the provider to modify the terms of service and/or the service itself.
- Content removal: clauses which give the provider a right to modify/delete a user's content
- Jurisdiction: clauses which specify a jurisdiction in which claims must be brought, regardless of where the user lives.
- Choice of law: clauses which specify the country's law which governs disputes arising under the contract, regardless of where the user lives.
- Limitation of liability: clauses which limit the liability of the service provider.
- Unilateral termination: clauses which empower the service provider to terminate/suspend the service at their discretion.
- Contract by using: clauses which stipulate that a consumer is bound by the terms of service simply by using the service.
A more detailed description of these categories can be found in [80].
Task The Unfair Terms of Service task requires an LLM to determine—given a clause from a terms of service agreement—whether it belongs to one of the above eight categories, and if so, which one.
Construction process We use the version of data available in [21], which takes a subset from [80]. Unlike [21]—which frames the task as distinguishing "fair" from "unfair" clauses—we cast the task as 8-way multiclassification task across the original categories identified in [80].
Significance and value Unlike the CUAD and J.Crew Blocker task, the Unfair TOS task evaluates a LLM's ability to perform multiclass clause classification across a highly imbalanced dataset.
Clause	Clause type
you also acknowledge that a variety of evernote actions may impair or prevent you from accessing your content or using the service at certain times and/or in the same way, for limited periods or permanently, and agree that evernote has no responsibility or liability as a result of any such actions or results, including, without limitation, for the deletion of, or failure to make available to you, any content.	Arbitration
if you do not terminate your agreement before the date the revised terms become effective, your continued access to or use of the airborne platform will constitute acceptance of the revised terms.	Choice of law
we may at any time and from time to time, in our sole discretion, change the fees and charges, or add new fees and charges, in relation to any of the products.	Content removal
you and academia.edu agree that any dispute, claim or controversy arising out of or relating to these terms or the breach, termination, enforcement, interpretation or validity thereof or the use of the site or services (collectively, “disputes”) will be settled by binding arbitration, except that each party retains the right: (i) to bring an individual action in small claims court and (ii) to seek injunctive or other equitable relief in a court of competent jurisdiction to prevent the actual or threatened infringement, misappropriation or violation of a party’s copyrights, trademarks, trade secrets, patents or other intellectual property rights (the action described in the foregoing clause (ii), an “ip protection action”).	Contract by using
if we find that any shared content in your account violates our terms of service (including by violating another person’s intellectual property or privacy rights), we reserve the right to un-share or take down such content.	Jurisdiction
if you live in the european union: you agree that the laws of ireland, excluding conflict of laws rules, shall exclusively govern any dispute relating to this contract and/or the services.	Limitation of liability
ocular does not endorse or guarantee the opinions, views, advice or recommendations posted or sent by users.	Other
if you object to the changes, nintendo reserves the right to terminate this agreement or any portion of it upon reasonable notice and you will have to register again if you wish to continue using the nintendo account service under the new terms and conditions.	Unilateral change
unless you and we agree otherwise, in the event that the agreement to arbitrate above is found not to apply to you or to a particular claim or dispute, either as a result of your decision to opt out of the agreement to arbitrate or as a result of a decision by the arbitrator or a court order, you agree that any claim or dispute that has arisen or may arise between you and ebay must be resolved exclusively by a state or federal court located in salt lake county, utah.	Unilateral termination
Table 18: Examples from unfair_tos.
G.4.4 Contract QA
In LEGALBENCH, the Contract QA task is denoted as contract_qa.
Background Each of the above tasks evaluates the capacity for LLMs to learn to recognize a single type of clause, given a description of that clause and/or examples of it. The Contract QA task generalizes this across
Class	Number of samples
Other	3454
Contract by using	15
Choice of law	38
Content removal	53
Unilateral change	70
Arbitration	98
Limitation of liability	28
Unilateral termination	32
Jurisdiction	25
multiple clause types, evaluating an LLM's ability to recognize legal provisions that are not described in the prompt.
Task Each sample in the dataset consists of (1) a contract clause, and (2) a question asking if the clause is an example of a provision type (e.g., "Is this a severability clause?"). Across the dataset, the questions correspond to 22 different legal provisions. Questions and provisions are paired such that for each provision type, the LLM is presented with two clauses that are an example of the type, and two clauses which are not.
**Construction Process** The data was manually extracted from a set of sample agreements contributed by a LegalTech vendor and from public sources. It represents a variety of contracts, such as:
- Vendor or Partner Data Protection Agreements (DPA)
- Master Services Agreements (MSA)
- Licensing Terms
- BIPA consents