SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The Cambridge Law Corpus: A Dataset for Legal AI Research
Andreas Östling $^{1}$  Holli Sargeant $^{2}$  Huiyuan Xie $^{2}$  Ludwig Bull $^{3}$  Alexander Terenin $^{2}$  Leif Jonsson $^{4}$  Måns Magnusson $^{1}$  Felix Steffek $^{2}$ $^{1}$ Uppsala University  $^{2}$ University of Cambridge  $^{3}$ CourtCorrect  $^{4}$ Sudden Impact AB
We introduce the Cambridge Law Corpus (CLC), a corpus for legal AI research. It consists of over 250 000 court cases from the UK. Most cases are from the 21st century, but the corpus includes cases as old as the 16th century. This paper presents the first release of the corpus, containing the raw text and meta-data. Together with the corpus, we provide annotations on case outcomes for 638 cases, done by legal experts. Using our annotated data, we have trained and evaluated case outcome extraction with GPT-3, GPT-4 and RoBERTa models to provide benchmarks. We include an extensive legal and ethical discussion to address the potentially sensitive nature of this material. As a consequence, the corpus will only be released for research purposes under certain restrictions.
In recent years, transformer-based neural networks (Vaswani et al., 2017) have transformed the field of textual data analysis. These models have reached or surpassed human performance on many classical natural language processing tasks (Devlin et al., 2019; Liu et al., 2019). These recent developments have made it possible to train token prediction models at a larger scale than ever before, using extensive textual corpora gathered from sources such as social media, books, newspapers, web links from Reddit, and Wikipedia (Brown et al., 2020; Zhang et al., 2022). In late 2022, OpenAI released ChatGPT, a chatbot based on the GPT-3.5 language model, to the public. Just a few months later, OpenAI made the more powerful GPT-4 model available, both via ChatGPT and through an API (OpenAI, 2023). GPT-4 has shown promising results on a large variety of tests designed for humans, such as AP tests, the LSAT and the Uniform Bar Exam (Martínez, 2023).
Legal artificial intelligence is emerging as a rapidly-developing area of machine learning (Zhong et al., 2020), which focuses on problems such as case outcome prediction (O'Sullivan and Beel, 2019; Chalkidis et al., 2019), legal entity recognition (Leitner et al., 2019; Dozier et al., 2010), prediction of relevant previous cases or relevant statutes (Liu et al., 2015), contract reviews (Hendrycks et al., 2021), and more recently, passing legal exams (Choi et al., 2023; Katz et al., 2023). Using machine learning models to solve such tasks automatically presents a new way for citizens and businesses to interact with many aspects of the legal system.
The release of sizeable, specialised datasets has been pivotal to machine learning research progress. Most famously, Deng et al. (2009) introduced the ImageNet dataset, containing over 1 million images spanning 1000 classes, crucial for the early development of deep convolutional neural networks (LeCun et al., 2015). Similarly, large datasets such as the Google Book corpus (Davis, 2011) have played an important role in the development of transformers and the popularisation of bread-and-butter techniques like byte-pair encoding (Gage, 1994). Just as critically, in parallel with these large-scale
The Cambridge Law Corpus's project page, which includes example data and terms of use, can be found at: HTTPS://WWW.CST.CAM.AC.UK/RESEARCH/SRG/PROJECTS/LAW. DOI: 10.17863/CAM.100221.
general-interest resources, many other specialised datasets have proven fruitful for research (Rasmy et al., 2021; Deng et al., 2009; Deng, 2012; Lin et al., 2015). Therefore, high-quality datasets for legal artificial intelligence research are needed to facilitate research progress.
Legal language tends to be more specialised than other kinds of natural language. Legal terms and idioms, which are often difficult for a lay person to understand, have strong semantics and are often the keystone of legal reasoning (Nazarenko and Wyner, 2017; Dale, 2017). For example, the term stare decisis is a legal "term of art" that has a specific legal meaning. The translation from Latin is "to stand by things decided": this term refers to the legal principle of precedent, that is, that courts must adhere to previous judgments of the same court (or judgments of higher courts) while resolving a case with comparable facts. We discuss this example and related points in Section 2.1.
Specialised large language models, such as recent legal versions of the BERT model (Chalkidis et al., 2020; Masala et al., 2021; Zheng et al., 2021), have been shown to perform better when fine-tuned on legal corpora. Suitable training data is needed in order to make the development of such models possible. At present, large research-quality legal corpora are not as widely available as analogous resources in other areas of machine learning. There is a lack of general large legal corpora focused on the United Kingdom, especially in machine-readable formats and with relevant annotations to facilitate the development of methods for solving key machine-learning tasks.
Legal data involves unique challenges, including the need to comply with regulations such as the GDPR legislation and to appropriately address privacy and other ethical matters. These challenges also include the need to digitise cases whose presentation is not in any manner standardised, such as those that are hundreds of years old but which constitute a legal precedent and whose content defines the law in use today. Handling details such as these creates a need for maintaining an appropriate level of dataset quality.
This work introduces a corpus containing, in its present form, 258,146 legal cases from the UK, available for research. We release the corpus and annotations of case outcomes for 638 cases.
Prior Work and Current State of Affairs
Corpora containing legal decisions in English-language jurisdictions other than the UK are becoming increasingly available, with examples such as MultiLegalPile (Niklaus et al., 2023) and LEXTREME (Chalkidis et al., 2023) providing court decisions in the United States, Canada, and Europe. Notably, LeXFiles (Chalkidis et al., 2023) contains legislation and case law from six countries—including the UK, which is our focus—albeit at a limited scale. Complementing case judgments, corpora such as CUAD (Hendrycks et al., 2021) provide expert-annotated legal contracts.
The availability of legal judgment data varies by jurisdiction and is influenced by (i) the degree to which legal judgements are anonymised, (ii) the degree to which they are standardised, and (iii) the degree of privacy protection offered by the jurisdiction. For example, Hwang et al. (2022) have recently released a corpus of Korean legal judgments. These judgments are anonymised when they are written, in accordance with the Korean legal system, making such a release more straightforward compared to other jurisdictions which do not anonymise. Other examples of court decisions include, Xiao et al. (2018) who released a large corpus of Chinese criminal cases, and Poudyal et al. (2020), who published a corpus from the European Court of Human Rights.
Compared to these examples, the availability of UK data is more limited. In terms of the aforementioned factors, the UK (i) does not anonymise its legal decisions or (ii) present them in a standardised format while simultaneously offering (iii) comparatively strong privacy protections. Recently, the UK's National Archives have started a service making case law available. At present, however, this service is limited to cases decided after 2003, prohibits bulk download and does not allow computational analysis by its applicable license (The National Archives, 2023).
2 The Cambridge Law Corpus
The Cambridge Law Corpus (CLC) is a corpus containing legal cases, also called judgments or decisions, from the combined United Kingdom jurisdiction, and England and Wales. Courts from Scotland and Northern Ireland are currently excluded. Further details on content are available in Appendix A, and in this work's companion datasheet (Gebru et al., 2021). Before summarising the
Figure 1: A simplified view of the UK court and tribunal structure (Judiciary UK, 2023).
corpus's content, we begin by reviewing some of the legal background needed to understand the material. Legal and ethical implications are addressed in Section 3.
2.1 The United Kingdom's Legal System
The UK does not have one single unified legal system--instead, it has three distinct jurisdictions: England and Wales, Scotland and Northern Ireland. There are exceptions to this rule, where some courts or tribunals hear cases from across the three jurisdictions. The common law of England and Wales is one of the major global legal traditions. It has been adopted or adapted in numerous countries worldwide.
Each jurisdiction is a common law jurisdiction—that is, authoritative law includes both legislation and court judgments. Legislation comprises Acts of Parliament (under the authority of the Crown) and regulations, that is, rules that govern how we must or must not behave and contain the consequences of our actions. Judges apply legislation and case law principles decided in previous cases to explain their decisions. Judges in different courts may decide cases with different principles across these three jurisdictions. Court judgments may be appealed to the highest court—the UK Supreme Court.
Court decisions can operate as an independent primary source of substantive law (case law), meaning that courts can legally justify their decisions by applying case law in the same way they apply legislation (Raz, 2009). Lawyers often call past cases precedents, and in many contexts, precedents at least influence the decisions of courts when relevantly similar disputes arise. In Anglo-American legal systems, this legal principle is known as stare decisis. Courts in the UK apply stare decisis relatively strictly.
The authority of the court will depend on the judgment court's place in the court hierarchy (see Figure 1). Precedent may have binding authority if the judgment has been upheld by a higher court, or persuasive authority if the judgment is from a lower court (Cross and Harris, 1991; Lewis, 2021). For example, the Supreme Court (formerly the House of Lords) is the highest court in the United Kingdom, and its decisions bind all courts in the jurisdiction (Lewis, 2021). This makes common law jurisdictions different from civil law jurisdictions. In the civil law jurisdiction, prior judicial decisions are not considered "law" (Merryman and Pérez-Perdomo, 2019).
A person or party may only be judged by the law and punished for a breach of the law (Dicey, 1915). Therefore, judges are not able to refuse to apply legislation; only Parliament can change the substantive rules in legislation. However, judges are able to interpret the appropriateness of law as it has evolved in light of the aims of previous lawmakers and may acknowledge that they make a policy-influenced decision.
Different courts deal with criminal and civil law (civil law cases are different to civil jurisdictions). Civil cases consider remedies to a party who has an enforceable legal right arising from tortious claims, contract disputes, family matters, company articles and similar. Criminal cases concern the law of crime and punishment where the Crown prosecutes an accused whom the police have investigated.
2.2 Corpus Content
The corpus consists of 258 146 cases, spanning 53 courts, with cases as old as the 16th century such as for instance Walsingham's Case (1573). Most cases are from the 20th and 21st centuries. Detailed figures on the development of the number of cases can be found in Appendix A. In total, these cases include approximately 5GB of legal text, consisting of roughly 0.8B tokens.
In the Cambridge Law Corpus, each case is stored as a single XML file by court and year. We chose XML in order to be able to annotate both whole cases and parts of cases in a structured, easy-to-use fashion, and in order to support many character encodings, comments, user-defined tags and namespaces. Appendix B shows an example of what a stored file might look like.
In addition to the case files, we also store additional relevant information in separate tables. For example, the categorical outcome of an individual case. In this way, the XML files mainly store the textual content of the case and the CSV files store additional features separately. This simplifies adding new information and features to the corpus. Further, the corpus comes with a Python library for quickly converting the XML files into formats commonly used in machine learning settings such as the Hugging Face DATASETS class.
The corpus contains both legal information and additional technical information. Each case contains a case header that contains general information on the case, such as the judge, the claimant and defendant, the date of the judgment, and similar. The case header also includes information on the court and the date of either the final hearing or judgment delivery.
In addition to the case header, the corpus also contains the case body, which contains the body text of the case. The case body contains complete sentences of legal text. Usually, it starts with a summary of the facts of the case, followed by arguments and a decision, although there is not necessarily any formal structure, and therefore the content of the body varies between cases and over time.
The case outcomes are stored separately from the cases and follow a hierarchical structure with aggregate outcomes, such as claimant wins or claimant loses, and more detailed case outcomes, such as damages awarded or permission to appeal granted. For details on definitions of case outcomes and a detailed list, see Appendix D.
Each case in the corpus is assigned a unique identifier (Cambridge Law Corpus identifier, CLCID). For this, we have created universally unique identifiers (UUID), which are identifiers that can guarantee uniqueness across space and time (Leach et al., 2005). In addition, a case's metadata can contain legal identifiers, which are used by the legal community to identify cases. These allow cases to be found in a manner mirroring their indexing in law reports. We also include neutral case citations, which were introduced in the United Kingdom in January 2001 to facilitate identifying a case in a manner independent from citations assigned by the commercial providers of the law reports. For example, the cases decided by the Civil Division of the England and Wales Court of Appeal are neutrally cited as follows: [year] EWCA Civ [number].
2.3 Corpus Creation and Curation Process
The original cases of the Cambridge Law Corpus were supplied by the legal technology company CourtCorrect<sup>1</sup> in raw form, including Microsoft Word and PDF files. The Word files were cleaned and transformed into an XML format. PDF files were converted to textual form via optical character recognition (OCR) using the Tesseract engine (Kay, 2007, v4.1.1)—an estimate of the OCR error rate is given in Appendix A. The resulting text files were then converted to the XML standard format of the corpus. The original documents are stored separately for quality control purposes.
Due to the size of the Cambridge Law Corpus, it is not feasible to annotate or curate the whole corpus manually. Instead, we use a process inspired by Voormann and Gut (2008), whose principles can be summarised as follows:
1. Replace sequential corpus creation phases with an encompassing, cyclic process model and employ a query-driven approach.
2. Recognise general error potentials and take measurements for immediate modifications in each part of the corpus creation process.
3. Minimal effort principle: slim annotation schema and little upfront annotation.
These principles have inspired our corpus creation and curation process in two ways. First, our process focused on improving the corpus in many small iterative steps. These include adding new annotations, new metadata, new cases and correcting identified errors, such as OCR errors. Second, we adopt a release model consisting of many small corpus releases, following the general ideas of
1 HTTPS://COURTCORRECT.COM.
semantic versioning (Preston-Werner, 2013). We treat both the actual content of the corpus—the XML and CSV files—and the accompanying Python library as two complementary parts of the corpus's application programming interface (API).
These two principles have multiple benefits both for us and for the users of the corpus. First, rapid releases keep the corpus up to date in that new releases are made as soon as corrections or additions have been made to the corpus. This also encourages users to report errors in the data back to us in a structured way. Second, semantic versioning helps communicate the effect of the release to the users of the corpus and when to expect a change in the corpus API. Changes to the corpus are quality controlled through a random sample of the edits made.
2.4 Case Outcome Annotations
To enable researchers to study case outcomes, we added manual annotations tagging the tokens which contain the outcomes for a subset of cases. A case's outcome describes which party or parties are (partly) successful with their application(s) and which parties are not (partly) successful. In addition, the outcome contains the legal consequence decided by the court or tribunal. For example, a party may be ordered to pay a certain sum of money to the other party, or a person may be sentenced to a certain number of months in prison. Legal research and legal practice are particularly interested in the outcome of court cases. For instance, before going to court, it is helpful for parties to understand the expected outcome. After courts have decided a case, parties, advisers and researchers may be interested in knowing whether the decision is correct.
While it is easy to identify the outcome in the text of some cases, other cases are more difficult to understand. Some judgments report the speeches of multiple judges and it may be difficult to identify the final decision of the collective. As there are no formal rules or conventions on the writing of court decisions, the words and sentences defining the outcome vary considerably. In some cases, readers may even struggle to understand which legal outcome the judge aimed to lay down.
We started the annotation process by annotating a stratified random sample (by court) of legal cases. The annotations were made by legal experts at the Faculty of Law of the University of Cambridge following annotation instructions that can be found in Appendix D. The annotation instructions distinguish between the general outcome (in particular, whether a party is successful in court) and the detailed case outcome (for example, what remedy one party owes the other party). Following the ideas of Voormann and Gut (2008), we continuously updated the schema and instructions during the annotation process as problems arose.
The annotation process was undertaken by four lawyers (all of whom have a law degree, and three of whom are qualified solicitors and have graduate degrees in law) and one further legal expert (a Professor at the University of Cambridge's Faculty of Law, who also has higher qualifications in law) overseeing the annotation process. The four annotators and the legal expert met after a certain number of cases had been annotated to discuss problems that had arisen in the meantime. The main challenge was to align the annotation practice of the four annotators and to deal with new kinds of detailed case outcomes arising from the cases. As the corpus contains cases from various courts and tribunals as well as various areas of law, very specific detailed outcomes needed to be integrated into the annotation guidelines to capture such variability. While there is some research into formalising court outcomes, there is currently no accepted standard taxonomy of court outcomes the annotations could have referred to. Against this background, another contribution of this research is the first set of standardised annotation guidelines for UK case outcomes.
3 Legal and Ethical Considerations
The legality and ethics of collecting, processing and releasing the corpus is of paramount importance. Therefore, we have considered relevant safeguards to ensure legal compliance and the ethical design of this project. We now discuss these matters in more detail.
UK legislation and court decisions are subject to Crown copyright and licensed for use under the Open Government Licence. The Open Government Licence grants a worldwide, royalty-free, perpetual and non-exclusive licence (The National Archives, 2014).
The Data Protection Act 2018 (DPA) complements the UK implementation of the European Union's General Data Protection Regulation (GDPR), here referred to as the UK GDPR. Compliance with the DPA and UK GDPR has been the basis of the legality of this project as affirmed by the ethics approval from the University of Cambridge.
There are limitations on when personal data can be collected, processed and stored. The personal data in this corpus was not collected directly from data subjects and was only undertaken for research purposes in the public interest. Both these circumstances offer exemptions from obligations in the UK GDPR (DPA sch 2, pt 6, paras 26, 27; UK Information Commissioner's Office, 2022).
Given the practically impossible and disproportionate task of informing all individuals mentioned in this corpus and that these cases are publicly available and being processed exclusively for research purposes, makes this corpus exempt from notification requirements (DPA sch 2, pt 6, paras 26, 27; GDPR art 14(5)). Further, research in the public interest is privileged as regards secondary processing and processing of sensitive categories of data restrictions (GDPR art 6, rec 47, 157, 159). In particular, this aids the protection of the integrity of research datasets.
There are still important safeguards and considerations relevant to processing data of this sensitive nature. We apply safeguards in compliance with legal and ethical requirements and ensure that:
- Appropriate technical and organisational safeguards exist to protect personal data (DPA s 19; GDPR art 89)
- Processing will not result in measures being taken in respect of individuals and no automated decision-making takes place (DPA s 14; GDPR art 22).
- There is no likelihood of substantial damage or distress to individuals from the processing.
- Users who access the corpus must agree to comply with the DPA and the UK GDPR in addition to any local jurisdiction.
- Any individual may request the removal of a case or certain information which will be immediately removed (GDPR art 17; DPA s 47).
- The corpus will not pose any risks to people's rights, freedoms or legitimate interests (DPA s 19; GDPR art 89).
- Access to the corpus will be restricted to researchers based at a university or other research institution whose Faculty Dean (or equivalent authority) confirms, inter alia, that ethical clearance for the research envisaged has been received.
- Researchers using the corpus must agree to not undertake research that identifies natural persons, legal persons or similar entities. They must also guarantee they will remove any data if requested to do so.
We have considered the potential impact of processing such data. Given the public availability of all cases in the dataset in other repositories, the principle of open justice, the prohibition of research identifying individuals, the requirement of ethical clearance and our mechanisms for the erasure of data, we believe there is unlikely to be any substantial damage to individuals.