SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- LEGALBENCH does not include tasks over long documents. Long documents are significant for legal practice, as writings like contracts, corporate filings, statutory codes, and judicial opinions can be hundreds of pages long [77].
- The legal reasoning dimensions identified in LEGALBENCH constitute a subset of the possible legal reasoning abilities for which we wish to evaluate LLMs. An example of a reasoning ability which is not currently evaluated in LEGALBENCH would be analogical reasoning grounded in case law.
- LEGALBENCH tasks are skewed towards certain legal domains (e.g., contracts and civil procedure) and others are unrepresented.
- LEGALBENCH tasks skew towards US Federal law, and thus may not be representative for studies of other jurisdictions, or tasks involving international law.
- LEGALBENCH does not enable evaluation for multilingual, or non-English, legal tasks.
- LEGALBENCH does not evaluate more subjective legal tasks, or tasks which contain more ambiguity. These tasks are common to the legal field.
We hope to work on these limitations as part of future work. In particular, we would like to expand LEGALBENCH to include other jurisdictions and a broader cross-section of legal domains.
Nothing in LEGALBENCH should be construed as legal advice.
Social impact A potential negative social impact of our work would be if others either (1) construed our work as unequivocally endorsing automation in the legal industry, or (2) used performance on LEGALBENCH as the sole justification for AI deployments. We therefore take efforts to mitigate these impacts, noting the following.
As we state in Section 1, the purpose of our work is not to determine whether large language models are capable of replacing legal professionals, the types of legal work that should/can be automated, or the broader implications of new technology on the practice of law. Rather, our focus is on developing technical artifacts which better enable stakeholders and affected parties to answer these questions themselves. Rigorous evaluation is essential to the safe and ethical usage of AI. LEGALBENCH, as a benchmark, is intended to improve the ability for
stakeholders to conduct evaluations. We additionally note that LEGALBENCH, as a tool for research, is not a substitute for more in-depth and context-specific evaluation efforts. The deployment of any AI application in the law must be accompanied by evaluation on in-domain data, and assessments for ethical and legal compliance.
We finally note that potential negative impact will depend significantly on the task studied and the broader social context. The consequences of mistakes in using LLMs to annotate datasets, for instance, has significantly different consequences from the cost of mistakes when LLMs are used to answer legal aid questions.
D Datasheet
Following recent work, we provide a datasheet [54] below. The datasheet below provides general answers to each of the questions, while Appendix G provides more in-depth details for each individual task. In addition, a number of LEGALBENCH tasks have been adapted from previously released datasets, and the datasheets accompanying their publication provide further details.
D.1 Motivation
For what purpose was the data set created? Was there a specific task in mind? If so, please specify the result type (e.g. unit) to be expected.
LEGALBENCH was created to evaluate LLMs on legal tasks and better understand their legal reasoning capabilities. Recent advances in language modeling techniques have led to the emergence of "large" language models, and spurred interest within the legal community. This has led to two questions:
- What technical adaptations are necessary to enable LLMs to perform legal tasks? Legal tasks often involve longer text sequences, jargon, and multi-step reasoning, making them more difficult than traditional NLP tasks.
- For which legal tasks can current LLMs be trusted to perform safely and reliably?
LEGALBENCH encompasses many different tasks. The specification for each task and the expected output can be found in the full task descriptions (Section G).
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g. company, institution, organization)?
LEGALBENCH consists of novel datasets (which were created by the authors of this paper), and transformed/adapted datasets (which were originally released as part of prior research). In Section G we discuss the origins of each dataset.
LEGALBENCH and its contributors have been generously funded by a range of entities that include the institutional affiliations provided for each author, governmental grants, and other sources.
D.2 Composition
All LEGALBENCH tasks consist of instances which are text. These include: sentences, paragraphs, and documents. Some instances are drawn from real world sources of text (e.g., actual contracts, corporate disclosures, judicial opinions, or complaints). Other instances were synthetically crafted. Section G provides details for each task.
Section E provides details for each task.
Nearly every LEGALBENCH task corresponds to a sample of a population, or entirely synthetic data. Section G contains a more detailed description for each dataset. We highlight several broader explanations for the difficulty in acquiring complete or representative data which generalizes across tasks:
- As prior work on legal benchmarks has noted [58, 118], not all legal documents are published or reported. Hence, many are only accessible through special request, or only available in paper. The lack of easily available representative data is a noted challenge in many justice systems [58, 100].
- Acquiring legal annotations is exceedingly expensive. The CUAD project, for instance, estimated that a modestly sized dataset of 500 contracts (relative to the standards of NLP) had an estimated cost of $2 million US dollars [60]. As a result, it is often possible to only annotate a small sample of data, even when a larger population is available.
Instances in LEGALBENCH largely correspond to unprocessed text. Section G contains a more detailed description for each dataset.
Yes. Labels correspond to: classes, extracted entities, and open-ended generation. Section G contains a more detailed description of the labels/targets for each dataset.
Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.
For reused/adapted datasets, we refer readers to the original data sheets which document redactions/missing data. Newly contributed tasks should not be missing information.
Yes. Tasks are split into train and test splits. Train splits consist of a small random sample of the original dataset (i.e., between 2-8 instances). We select small training splits in order to capture the true few-shot setting [105], in which a practitioner only has access to a handful of labeled instances. This design choice is also reflected in the structure of the RAFT benchmark [1].
A significant amount of legal data is the product of scanning and OCR. Hence, this data often contains artifacts of these processes, which appear as errant or missing characters.
LEGALBENCH is self-contained.
No. All LEGALBENCH data is derived from public sources or was generated by authors. There is no confidential information in our dataset.
LEGALBENCH data relates to people to the extent that LEGALBENCH contains tasks which contain language drawn from judicial cases involving individuals, or posts by individuals to legal forums (i.e., the Learned Hands Tasks).
Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset. No.
As LEGALBENCH is drawn entirely from public datasets—which themselves may contain additional information—it is possible to identify the original documents that LEGALBENCH data was drawn from.
The Learned Hands tasks correspond to posts on public forums. In these posts individuals discuss legal questions, and sometimes disclose information that would meet the above definition of "sensitive."
We note that the data distributions from which some LEGALBENCH tasks were drawn—like judicial cases or legal forums—have been used by prior work published in the NeurIPS Datasets and Benchmarks Track [58, 118]. These works offer additional information.
D.3 Collection process
Data underlying LEGALBENCH tasks were collected using different processes, and Section G contains a detailed discussion for each task.
Please refer to Section G for background on each task.
Please see the discussion in the Composition section above.
Section G contains a detailed discussion for each task.
Where applicable, Section G provides information relevant to each task.
The dataset relates to people insofar as it draws text from documents which relate to people, or people created.
No. Following other works which incorporate data from public judicial sources [58, 118], we note that judicial filings are public, and the individuals implicated in those proceedings are aware of the public nature.
Individuals whose names and circumstances appear in the original datasets did not separately consent to be a part of LEGALBENCH. Again, we note that these documents are generally public, and already accessible to a wide range of parties, through many different judicial data services.
D.4 Preprocessing, cleaning, labeling
D.5 Use
We have used the constructed datasets to evaluate several LLMs.
LEGALBENCH is available at https://github.com/HazyResearch/legalbench/.
We envision this dataset could be used for the following:
Evaluation of LLMs.
- Finetuning LLMs, either on task data directly, or self-instruct style generations derived from task data.
We emphasize that LEGALBENCH—like all generalized benchmarks—can offer only a preliminary understanding of LLM performance. LEGALBENCH tasks do not generalize to all legal reasoning tasks or all types of legal documents. We thus emphasize that practitioners seeking to deploy LLMs within their own applications should perform their own data collection and validation specific to their use case.
These datasets should not be used to predict the legality of real world events, the outcome of lawsuits, or as legal advice.
D.6 Distribution
Table 4 provides the license that applies to each individual LEGALBENCH task.
Yes. Tasks which consist of adapted/transformed data are released under the same license as the original dataset. Table 4 provides these licenses, and Section G provides a reference to the original dataset for transformed tasks.
D.7 Maintenance
Neel Guha will be supporting this dataset.
Neel Guha can be reached at nguha@cs.stanford.edu. He will be available to answer any questions.
We have currently not found any, but will make them available on the website.
Willthe dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?
Yes. There will be two types of updates to LEGAL BENCH:
- First, we will update LEGALBENCH to reflect new contributions from the legal community.
- Second, we will update LEGALBENCH to reflect identified errors in the data.
We will strive to make and publicize updates as soon as errors are identified and new tasks are contributed. Neel Guha will be in charge of managing these updates.
Yes. We will make older versions available on request by email.
Yes. We encourage members of the legal community to contribute new tasks. We are in the process of formalizing procedures for reviewing, validating, and incorporating submissions.
We additionally note that many of the LEGALBENCH tasks are available under permissive licenses, and other researchers may thus modify them.
E Task overview
E.1 Licenses
LEGALBENCH tasks are subject to different licenses, due to the choices of dataset contributors, or the license under which the original data was released. Table 4 summarizes the licenses. The authors bear all responsibility in case of violation of rights, and confirm the dataset licenses.
License	Tasks
Creative Commons Attribution 4.0	Abercrombie, CUAD Tasks, Citation Prediction Tasks, Contract NLI Tasks, Contract QA, Corporate Lobbying, Diversity Tasks, Function of Decision Section, Hearsay, Insurance Policy Interpretation, International Citizenship Questions, J.Crew Blocker, Legal Reasoning Causality, MAUD Tasks, Oral Argument Question Purpose, Overruling, Personal Jurisdiction, Private Right of Action, Rule QA, SCALAR, Securities Complaint Extraction Tasks, Successor Liability, Supply Chain Disclosure Tasks, Telemarketing Sales Rule, UCC v. Common Law, Unfair Terms of Service
Attribution-NonCommercial 4.0 International	Canada Tax Court Outcomes, Consumer Contracts QA, Textualism Tools
Attribution-ShareAlike 4.0 International	Definition Tasks
Attribution-NonCommercial-ShareAlike 4.0 International	Learned Hands Tasks
MIT	New York State Judicial Ethics, Privacy Policy QA, SARA Tasks
Creative Commons Attribution-NonCommercial License	OPP-115 Tasks
Attribution-NonCommercial 3.0 Unported	Privacy Policy Entailment
Table 4: Licenses
E.2 Public availability status
Given that many commercially available LLMs are trained on the "entirety of the web"—and little is known as to how they are trained—there are concerns that many benchmarks have inadvertently become part of the training data for these models. Therefore, this section identifies and organizes LEGALBENCH tasks into three categories:
- Previously published tasks, which were derived from datasets that were initially published as part of other works and available on the web for download.
- Original but available tasks, which are original creations of the LEGALBENCH project but previously made available online.
- Original and unavailable tasks, which are original creations of the LEGAL BENCH project but have not been released online.
Table 5 summarizes the availability status of each of the tasks.
E.3 Reasoning type
Table 6 organizes tasks by the LEGALBENCH reasoning type they can be used to assess. Table 7 similarly organizes tasks according to reasoning-types recognized in the NLP literature. For each reasoning type, we provide examples of general-domain NLP benchmarks which are similar. For more information on the types of reasoning required for each task, please see the individual task descriptions provided in Appendix G.
Publication status	Tasks
Previously published tasks	CUAD Tasks, Contract NLI Tasks, MAUD Tasks, OPP-115 Tasks, Overruling, Privacy Policy Entailment, Privacy Policy QA, SARA Tasks, Unfair Terms of Service
Original but available tasks	Abercrombie, Diversity Tasks, Hearsay, International Citizenship Questions, Learned Hands Tasks, New York State Judicial Ethics, Personal Jurisdiction, Private Right of Action, Rule QA
Original and unavailable tasks	Canada Tax Court Outcomes, Citation Prediction Tasks, Consumer Contracts QA, Contract QA, Corporate Lobbying, Definition Tasks, Function of Decision Section, Insurance Policy Interpretation, J.Crew Blocker, Legal Reasoning Causality, Oral Argument Question Purpose, SCALR, Securities Complaint Extraction Tasks, Successor Liability, Supply Chain Disclosure Tasks, Telemarketing Sales Rule, Textualism Tools, UCC v. Common Law
Table 5: Task publication status
LEGALBENCH reasoning type	Tasks
Issue-spotting	Corporate Lobbying, Learned Hands Tasks
Rule-recall	Citation Prediction Tasks, International Citizenship Questions, New York State Judicial Ethics, Rule QA
Rule-application	Abercrombie, Diversity Tasks, Hearsay, Personal Jurisdiction, Successor Liability, Telemarketing Sales Rule, UCC v. Common Law
Rule-conclusion	Abercrombie, Diversity Tasks, Hearsay, Personal Jurisdiction, Successor Liability, Telemarketing Sales Rule, UCC v. Common Law
Interpretation	CUAD Tasks, Consumer Contracts QA, Contract NLI Tasks, Contract QA, Insurance Policy Interpretation, J.Crew Blocker, MAUD Tasks, OPP-115 Tasks, Privacy Policy Entailment, Privacy Policy QA, Private Right of Action, SARA Tasks, Securities Complaint Extraction Tasks, Supply Chain Disclosure Tasks, Unfair Terms of Service
Rhetorical-understanding	Canada Tax Court Outcomes, Definition Tasks, Function of Decision Section, Legal Reasoning Causality, Oral Argument Question Purpose, Overruling, SCALR, Textualism Tools
Table 6: Tasks by LEGALBENCH reasoning type.
E.4 Task statistics
Table 9 provides statistics for the LEGALBENCH tasks. For each task, we list the number of samples and the average length (in words) of each input. LEGALBENCH encompasses tasks ranging from short (a single sentence)
NLP reasoning type	Tasks
Knowledge (e.g., MMLU [59], WikiFact in HELM [78, 106])	Citation Prediction Tasks, International Citizen-ship Questions, New York State Judicial Ethics, Rule QA
Linguistic inference (e.g., CoLA [142])	Canada Tax Court Outcomes, Definition Tasks, Function of Decision Section, Legal Reasoning Causality, Oral Argument Question Purpose, Overruling, Textualism Tools
Topic classification (e.g., RAFT [1])	Corporate Lobbying, Learned Hands Tasks, CUAD Tasks, J.Crew Blocker, OPP-115 Tasks, Private Right of Action, Unfair Terms of Service
Entailment (e.g., RTE [138])	Contract NLI Tasks, Privacy Policy Entailment, Privacy Policy QA, Insurance Policy Interpretation, SARA Tasks
Arithmetic (e.g., GSM8K [33])	Diversity Tasks, SARA Tasks
Multi-step reasoning (e.g., STREET [111])	Abercrombie, Hearsay, Personal Jurisdiction, Successor Liability, Telemarketing Sales Rule, UCC v. Common Law
Document-based QA (e.g., BoolQ [32])	Consumer Contracts QA, Contract QA, MAUD Tasks, Supply Chain Disclosure Tasks
Named entity recognition (e.g., CoNLL-2003 [113])	Securities Complaint Extraction Tasks
Casual reasoning (e.g., CoPA [138])	SCALAR
Table 7: Tasks by NLP reasoning type. For each reasoning type, we provide examples of general-domain NLP benchmarks which are similar. For more information on the types of reasoning required for each task, please see the individual task descriptions provided in Appendix G.
to longer inputs (two pages of text) (Figure 1). The average LEGALBENCH task contains between 500-600 instances. All tasks consist of at least 50 instances. A more detailed breakdown is available in Table 8.
Figure 1: LEGALBENCH task sizes and input text lengths.
Size range (samples)	Number of tasks
50-100	28
100-500	97
500-2000	29
2000+	8
Table 8: Number of LEGALBENCH tasks at different dataset sizes.
Table 9: Task Statistics
Task	Number of Samples	Mean Sample Length (Words)
abercrombie	100	7.1
canada_tax-court_outcomes	250	99.2
citation_prediction_classification	110	35.9
citation_prediction_open	55	32.4
consumer Contracts_qa	400	486.8
contract_nli_confidentiality_of_agreement	90	73.9
contract_nli_explicit_identification	117	75.7
contract_nli_inclusion_of verbally_conveyed_information	147	78.1
contract_nli_limited_use	216	63.8
contract_nli_no_licensing	170	65.6
contract_nli Notice_on_compelled_disclosure	150	77.9
contract_nliisable_acquirement_of_similar_information	186	66.6
contract_nliisable_copy	95	60.2
contract_nliisable_development_of_similar_information	144	61.0
contract_nliisable_post-agreement Possession	119	81.7
contract_nli_return_of_confidentialinformation	74	74.2
contract_nli_sharing_with_employees	178	83.9
contract_nli_sharing_with THIRD-parties	188	78.9
contract_nli_survival_of_obligations	165	64.6
contract_qa	88	48.3
corporate_lobbying	500	878.1
cuad_affiliate Licence-licensed	204	75.9
cuad_affiliate Licence-licensor	94	99.7
cuad_anti-assignment	1178	54.6
cuad_audit_rights	1222	53.6
cuad_cap_on_liability	1252	59.8
cuad_change_of_control	422	62.8
cuad_competitive_restriction_exception	226	67.1
Table 9 - continued from previous page 1
cuad_covenant_not_to_sue	314	64.6
cuad-effective_date	242	44.5
cuad_exclusivity	768	58.0
cuadexpiration_date	882	49.9
cuad_governing_law	882	46.9
cuad_insurance	1036	56.8
cuad_ip Ownership Assignment	582	65.3
cuad_irrevocable_or_perpetuallicense	286	72.1
cuad_joint_ip Ownership	198	59.1
cuad-License Granted	1402	63.5
cuad_liquidated_damages	226	57.2
cuad_minimum_commitment	778	57.9
cuad-most_favored_nation	70	68.0
cuad_no-solicit_ofCustomers	90	61.2
cuad_no-solicit_of_employees	148	66.6
cuad_non-compete	448	60.6
cuad_non-disparagement	106	63.8
cuad_non-transferable LICENSE	548	61.5
cuad Notice_period_toTerminate_renewal	228	57.0
cuad_post-termination_services	814	67.3
cuad_price restricions	52	53.6
cuad_renewal_term	392	55.2
cuad_revenue-profit_sharing	780	59.8
cuad_rofr-rofo-rofn	696	64.0
cuad_source_code_escrow	124	64.4
cuadtermination_for_convenience	436	52.0
cuad THIRDParty_beneficiary	74	42.4
cuad_uncaptured_liability	300	70.1
cuad_unlimited-all-you-can-eat-license	54	56.8
cuad_volume_restriction	328	49.0
cuad_warranty_duration	326	55.9
definition_classification	1345	41.0
definition.extraction	695	53.9
diversity_1	306	16.2
diversity_2	306	23.4
diversity_3	306	21.1
diversity_4	306	23.5
diversity_5	306	28.3
diversity_6	306	47.9
function_of_decision_section	374	87.4
hearsay	100	25.3
Table 9 - continued from previous page 1
insurance_policy_interpretation	138	87.2
international_citizenshipquestions	9310	33.2
intra_rule_distinguishing	60	33.8
jcrew_blocker	60	167.2
learnedHands_benefits	72	253.2
learnedHands.business	180	225.3
learnedHands_consumer	620	246.3
learnedHands_courts	198	225.6
learnedHands_crime	694	233.7
learnedHands_divorce	156	240.2
learnedHands/domestic_violence	180	262.3
learnedHands_education	62	265.7
learnedHands_employment	716	242.0
learnedHands_estrates	184	230.9
learnedHands_family	2271	258.4
learnedHands_health	232	286.8
learnedHands_housing	4500	254.8
learnedHands-immigration	140	232.8
learnedHands_torts	438	272.2
learnedHands_traffic	562	229.6
legal_reasoning_causality	59	245.7
maud "_ability_to_consummate"__concept_isSubject_to_mae_carveouts	70	688.6
maud "_financial_point_of_view"__is_the_sole considération	113	307.5
maudaccuracy_of_fundamental_target_r&ws:_bringdown_STANDARD	176	143.7
maud accuracies_of_target_"general"__r&w:_bringdown_timing_answer	182	142.5
maud accuracies_of_target_capitalization_r&w_(outstanding_shares):_bringdown_STANDARD_answer	182	142.0
maud(additional_MATCHing-rights_period_for_modifications)(cor)	159	314.3
maud.application_of_buyer_consentrequirement_(negative_interim_covenant)	181	85.6
maud_buyer_consent Requirement_(ordinary_course)	182	121.3
maud_change_in_law:__subject_to"disproportionate_impact"modifier	100	702.6
maud Changes_in_gaap_or_otheraccounting_principles:__subject_to"disproportionate_impact"modifier	99	703.1
maud cor_permitted_in_response_to_intervening_event	101	305.6