SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Courts included in the Cambridge Law Corpus
Figure 3: Number of cases per year.
Figure 4: Number of cases per year, 1900 forwards.
Figure 5: Number of cases per court.
Figure 6: Number of cases per court in descending order.
Figure 7: Number of tokens per case per year. Tokens are defined as separated by white space.
Figure 8: Number of tokens per case per year, from 1900 onward. Tokens are defined as separated by white space.
B Example XML case
<?xml version="1.0" encoding="UTF-8"?>
<basic(case_docmt
 $\rightarrow$  xmlns="https://github.com/cambridge-ai-and-law-project/CambridgeLawCorpus"
 $\rightarrow$  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi: schemaLocation="http"
 $\rightarrow$  s://github.com/cambridge-ai-and-law-project/CambridgeLawCorpus
 $\rightarrow$  https://github.com/cambridge-ai-and-law-project/CambridgeLawCorpus/schemas/cas
 $\rightarrow$  e_schema.xsd">
    <CLC-ID>5f916ee5-b3f9-4fb4-b044-1575765da226</CLC-ID>
    <court>England and Wales Court of Appeal (Civil Division)</court>
    <lower court />
    <last_date_hearingdelivery>2019-02-19</last_date_hearingdelivery>
    <neutral_citations>
        <case-neutral_citation>[2019] ICR 1155</case-neutral_citation>
        <case-neutral_citation>[2019] EWCA Civ 125</case-neutral_citation>
        <case-neutral_citation>[2019] IRLR 545</case-neutral_citation>
    </neutral_citations>
    <law_reportreferences>
        </law_report references>
    <case_NR_of_year />
    <judges>
        </judges>
    <claimants>
        </claimants>
    <defendants>
        </defendants>
    <solicitors>
        </solicitors>
    <barristers>
        </barristers>
    <parties>
        </parties>
    </case_text></! [CDATA[ England and Wales Court of Appeal (Civil Division) Decisions
<NAME> v <NAME> 8 <NAME> [2019] EWCA Civ 125 (19 February 2019)  
[2019] ICR 1155,  
[2019] EWCA Civ 125,  
[2019] IRLR 545  
Neutral Citation Number: [2019] EWCA Civ 125  
Case No: A2/2017/0040  
IN THE COURT OF APPEAL (CIVIL DIVISION) ON APPEAL FROM the Employment Appeal  
→ Tribunal Mitting J  
Royal Courts of Justice Strand, London, WC2A 2LL  
19/02/2019  
B e f o r e :  
LORD JUSTICE <NAME> (Vice-President of the Court of Appeal (Civil Division)) LORD  
→ <NAME> and LORD JUSTICE <NAME>
Between:
Appellant
- and -
Ms <NAME> (instructed by <NAME>) for the Appellant  
Mr <NAME> QC and Mr <NAME> (instructed by Waring & Co) for the Respondents  
Hearing date: 3 October 2018
Lord Justice <NAME>:
The Claimants in these proceedings were employed, during the period with which we  $\hookrightarrow$  are concerned, by a company which has since been dissolved called
DISPOSAL I would dismiss the appeal
Lord  $\langle \text{NAME}\rangle$  : I agree.
Lord Justice: I also agree.
```html
```html
```html
```html
```html
```html
```html
```html
```html
</basic(case_doc>
C Case Outcome Task Description
Below we provide additional details on the models used for the case outcome extraction task. Statistics for the data are given in Table 5, and an example annotated case outcome is given in Figure 9.
- End-to-end RoBERTa. In this setup, the RoBERTa (Liu et al., 2019) model is used to directly predict tokens that may contain case outcome information.
- Two-step RoBERTa pipeline. In this setup, we first extract sentences that explicitly contain case outcome information from those that do not. On sentences that contain case outcome information, we further perform token classification using RoBERTa, which is done similarly to the previous RoBERTa token classification setup. By dividing the problem into two levels of granularity—sentence-level and token-level—we can see whether explicitly enforcing an intermediate step helps achieve better end results for token classification.
- GPT-3.5-turbo in a zero-shot setting. Here we evaluate the performance of GPT-based models, which have been reported to achieve state-of-the-art results on multiple language-related benchmarks. In a zero-shot setting, we asked GPT-3.5-turbo to extract the exact text containing the case outcome when provided with the full case text. See Appendix F for further GPT-specific details.
- GPT-4 in a zero-shot setting. Here, we test the more-powerful GPT-4 model along similar lines as the GPT-3.5-turbo. GPT-4 is a multi-modal language model that allows multiple forms of input sources—including text and image—and accepts longer-form text as input compared to GPT-3.5-turbo. Note that supporting the latter format makes this model particularly suitable for the legal setting, which can include long cases.
Ablation study on sentence classification step in the two-step RoBERTa pipeline In this ablation study, we first report evaluation results obtained from the first component—that is, the sentence-level classifier—of the RoBERTa pipeline. After 20 epochs of training, the sentence-level RoBERTa classifier achieves an extraction accuracy of  $82.3\%$ .
We also explored an ablated specification where we sliced examples sequentially into train/validation/test splits rather than randomly shuffling all examples across courts. When cases are sequentially sliced from their original order in the CLC, cases in a particular split tend to belong to specific courts. For example, all test cases in this ablated setting are from the England and Wales High Court (EWHC) and the UK Supreme Court (UKSC), whilst cases in the other two splits are from different courts. As a result, cases in different splits are principally from different courts, as illustrated in Table 6. With this setting, studying extrapolation, we trained multiple sentence-level classifiers—BERT (Devlin et al., 2019) and T5 (Raffel et al., 2019)—to extract sentences that potentially contain case outcome
Num. cases	Num. sentences	Num. sent. with outcome	Prevalence
Training	388	99565	536	0.54%
Validation	100	29208	126	0.43%
Testing	150	44881	208	0.46%
Table 5: Data statistics for the annotations used in the case outcome extraction task.
Split	Courts
Training	EWHC (Comm, Ch, Patents, Fam, QB, Admlty), EWCA (Civ)
Validation	EWHC (TCC), UKHL
Testing	UKHL, UKSC
Table 6: Courts used in the sequentially sliced train/val/test splits.
information. Considering that in this ablated study, cases from each data split tend to belong to different courts compared to when the cases are randomly shuffled, extracting sentences that potentially indicate general case outcome in this case requires a stronger ability for the classifier models to generalise to unseen cases from other courts. The BERT model yields an extraction accuracy of  $50.8\%$ , whilst the T5 model yields an accuracy of  $53.9\%$ . These accuracy results are significantly lower than the extraction accuracy we obtained under the experiment setting where we shuffled all cases. This indicates that judge ruling statements vary significantly across English courts.
D Case Outcome Annotation Instructions
Below are the annotator guidelines for the annotation of general and detailed case outcomes.
D.1 Case Outcome General
The case outcome is annotated in two steps. First the part of the case that contains the information of the case outcome is marked. Then this case outcome is classified both with a general and a detailed label describing the outcome of the case.
How is the label used? The tag case outcome general (label: CaseOutcomeGeneral) is a document class with the categories:
[CLAIMANT_WINS] Claimant wins
[CLAIMANT_PARTLY_WINS] Claimant partly wins
[CLAIMANT_LOSES] Claimant loses
[OTHER_OUTCOME] Other outcome
[NONE] Outcome cannot be identified
Please distinguish this general outcome tag from the detailed outcome tag below.
Note: Sentences such as where the judge says 'I agree' are not tagged as [CaseOutcomeGeneral] and not as [CaseOutcomeDetailed]. The reason for not tagging such sentences is that words such as 'I agree' do not indicate a specific legal consequence and only reference other text containing the legal consequence.
If there are multiple judges then [CaseOutcomeGeneral] is only tagged for the lead judge and not for other majority judges and not for minority judges.
In the circumstances of his forbearance and not getting £ < amount > besides the costs he has incurred all these years, it would not be appropriate further to pursue the periodical payments application. I would therefore dismiss this application for permission to appeal. CASE OUTCOME LORD JUSTICE < NAME > : I agree .
Figure 9: An example of the case outcome annotations in the CLC.
Definition The result of the case expressed as winning or losing from the claimant's perspective. There is also a category in case the result cannot be determined or if it is a case where the outcome cannot be described in terms of winning or losing (e.g., an evidence collection).
Example If the claimant requires damages of 100 GPB and the court awards 100 GBP then it is: Claimant wins.
What parts of the text do you want annotated, and what should be left alone? Please tag both general and detailed outcomes if possible.
D.2 Case Outcome Detailed
What is the tag called and how is it used? The tag case outcome detailed (label: CaseOutcomeDetailed) is a document class with the following categories (designed as drop-down menu). Note that some of the tags may be overlapping. To solve this, priorities are given where necessary:
[DAMAGES_AWARDED] Damages awarded. This includes the award of equitable compensation.
[DAMAGES_NOT_AwarDED] Damages not awarded. This includes the rejection to award equitable compensation.
[EQUITABLE_REMEDY_AwarDED] Equitable remedy(s) awarded. Note to annotators: equitable remedies include specific performance, restitution, recission, rectification, estoppel, account of profits. There is a separate tag for declaration and injunction.
[EQUITABLE_REMEDY_NOT_AwarDED] Equitable remedy(s) not awarded
[FAMILY COURT REMEDY GRANTED] Family Court remedy granted
[FAMILY COURT REMEDY_NOT_GRANTED] Family Court remedy not granted
[SUMMARY_JUDGEMENT_GRANTED] Summary judgement granted
[SUMMARY_JUDGEMENT_NOT_GRANTED] Summary judgement not granted
[INJUNCTION_AWARDED] Injunction awarded
[INJUNCTION_NOT_AwarDED] Injunction not awarded
[DECLARATION MADE] Declaration made (includes both doctrinal and functional declarations). The declaration tag is understood in both a doctrinal and a functional way. It takes priority over BREACH_OF_STATUTORY_OBLIGATION. For example, it applies when a court confirms the validity of an appointment or the breach of an IP right (while the damages flowing from this would be annotated as DAMAGES_awARDED). It also applies where the court decides on a certain interpretation of the law or a contract. It also applies where court approval is needed to achieve a legal consequence (e.g., that the requirements for convening a meeting are present).
[DECLARATION_NOT_MADE] Declaration not made (includes both doctrinal and functional declarations). Note to annotators: for example, a breach of obligation under the Human Rights Act 1998.
[STAY_ORDERED] Stay ordered
[STAY_NOT_ORDERED] Stay not ordered
[STRUCK_OUT] Application struck out
[NOT_STRUCK_OUT] Application not struck out
[BREACH_OF_STATUTORY_OBLIGATION] Breach of statutory obligation
[NO_BREACH_OF_STATUTORY_OBLIGATION] No breach of statutory obligation
[APPEAL_DISMISSED] Appeal dismissed
[APPEAL_UPHELD] Appeal upheld
[GUILTY_VERDICT] Guilty verdict. This includes contempt of court.
[NOT_GUILTY_VERDICT] Not-guilty verdict
[CONVICTION_QUASHED] Conviction quashed
[CONVICTION_NOT_QUASHED] Conviction not quashed
[ORIG_decision_UPHELD UNDER JUDICIAL REVIEW_REMEODY] Original decision upheld under judicial review remedy(s)
[ORIGdecision QUASHED_JUDICIAL_REVIEW_REMEDY] Original decision quashed and/or judicial review remedy(s). Note to annotators: where a decision is quashed on judicial review, remedies may include quashing order, prohibiting order, mandatory order, declaration, injunction, damages.
E Topic Model Top Words
Below we list the top words found in each topic given by the topic model.
Immigration (Topic 32): leave, application, rules, immigration, remain, decision, paragraph, appeal, secretary, state, united, kingdom, requirements, made, granted, refusal, uk, enter, appendix, policy.
Financial Services (Topic 35): investment, transaction, financial, market, bank, client, risk, rbs, transactions, investors, million, fsa, business, trading, clients, investments, barclays, banks, management, fund.
Employment (Topic 55): employment, work, contract, employer, employee, employees, employed, working, pay, time, hours, worker, transfer, job, terms, service, business, paid, worked, workers.
Contract (Topic 95): agreement, contract, clause, parties, terms, party, agreed, term, obligation, contractual, breach, construction, obligations, effect, agreements, words, rights, contracts, termination, provision.
F Evaluation of GPT Models
Below are instructions regarding how we tested the two GPT-based models. GPT-3.5-turbo has a 4k token limit, while GPT-4 has an 8k token limit. We chose to input the same text to all GPT models for a fair comparison, not utilising the larger token limit of GPT-4. Since very few cases fit into the above token limit, we chose to simply segment the cases without overlap into approximately 3k token segments, which were processed independently via the OpenAI chat completion API.Chunks were created such that sentences were added one sentence at a time, until the total size of the chunk exceeded 3k tokens, to ensure we had enough tokens left for the prompt and the response. Tokens were calculated using the TIKTOKEN python library.
The prompts used for case outcome extraction are as follows:
{"role": "system", "content": "Please be a legal research assistant to assist legal professionals in analyzing and understanding legal cases"} {"role": "user", "content": ""Please carefully read the provided text and identify the sentence(s) that describe the outcome(s) of the case. Copy and paste the exact quotes from the text. If there are multiple outcomes, separate the quotes with "[SEP]". Provide the quotes in chronological order, being concise and avoiding unnecessary information. If the case outcome is not provided in the text, reply with 'NA'. Case outcome definition: The result of the case expressed as winning or losing from the claimants perspective. Examples of case outcome: For the reasons set out in this Ruling, we have decided to grant permission on all of the grounds proposed. I would dismiss the appeal. For these reasons the appeal is denied."""+ legal(case}
where LEGAL_CASE is the chunk of case text described above.
We set the temperature to zero, since the goal is to quote text. Both the system message and the prompt were created with the help of GPT, asking ChatGPT to improve an initial prompt which
was created based on our annotation guidelines. This prompt was then improved upon iteratively, by stating to ChatGPT what error the previous prompt caused and asking for an improved version, until satisfactory performance was reached on validation data. For the instances where the annotators annotated multiple parts of text within a chunk as case outcome, those parts were concatenated with the text [SEP] separating the outcomes, in order to account for multiple outcomes. This was done both to avoid performing extra parsing of output from the GPT models, and because it makes it straightforward to calculate the word error rate for an entire case, and not just each individual sentence.
Cambridge Law Corpus: Datasheet
Andreas Östling<sup>1</sup>
Holli Sargeant2
Leif Jonsson
Huiyuan Xie
Ludwig Bull3
Måns Magnusson
Felix Steffek2
<sup>1</sup> Uppsala University
$^{2}$ University of Cambridge
3CourtCorrect
Sudden Impact AB
The Cambridge Law Corpus (CLC) was created to enhance and facilitate research in law, natural language processing and machine learning. The corpus consists of 258,146 cases from UK courts, spanning from 1595 to 2020.
CourtCorrect Ltd represented by one of the authors, Ludwig Bull, as director has provided the original dataset (containing the raw data) to the research project. The CLC was then created by the authors as part of a research project based at the Faculty of Law of the University of Cambridge. Associated copyrights, such as Crown copyright, are acknowledged.
If there is an associated grant, please provide the name of the grantor and the grant name and number.
The dataset was created within the UKRI-JST funded research project Legal Systems and Artificial Intelligence at the University of Cambridge. The grant reference is ES/T006315/1.
The corpus consists of a collection of 258,146 court cases from the United Kingdom. The data contains information from the judgements, including the court, judges, parties and legal representation in the case. The main information is the decision of the judge(s). The corpus contains a wide variety of cases from different jurisdictions within the UK, different courts across different levels of court hierarchy (for example, county courts, High Court, Court of Appeal and Supreme Court) and time periods.
In addition to the original court cases, the data also contains annotations of case outcomes for 638 cases annotated by legal experts at the University of Cambridge. This was added since, in the UK, there is no strict formal way in which court decisions are written, and it is therefore, non-trivial to identify the case outcome within the text.
There are 258,146 cases in total. We release 638 annotated cases, split into 150 test cases, 100 validation cases and 388 cases for training.
If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
The Cambridge Law Corpus (CLC) is a legal corpus with 258,146 court cases from the combined jurisdictions of the United Kingdom. The United Kingdom does not have one single unified legal system. Instead, it has what are called three dis
tinct jurisdictions, which are England and Wales, Scotland and Northern Ireland. This rule has some exceptions: certain courts or tribunals hear cases from across these three jurisdictions. At present, we provide a sample corpus including 258,146 judgements from the combined United Kingdom jurisdiction and England and Wales.
We chose to exclude certain courts from the dataset, including courts from Scotland and Northern Ireland, and some European judgements published in the United Kingdom. The choice to exclude these cases is for two primary reasons. First, these legal jurisdictions are distinct and this corpus seeks to represent the laws of England and Wales and the combined courts of the United Kingdom. In particular, given the United Kingdom's exit from the EU, we have excluded European judgements. Second, according to the doctrine of precedent, a court is bound by the decisions of a court above it, so we have included the superior courts of the United Kingdom as their judgements are very relevant for the courts in England and Wales.
In terms of completeness and representativeness, we make available all court decisions that are legally and ethically available to us. As courts in the UK do not report all decisions they make, it is currently not possible to know how many other court decisions there are that are not in our dataset. Against this background, this dataset cannot claim to be comprehensive or representative. However, this dataset is the largest dataset currently available on UK court cases and, therefore, making it available can be seen as a major step for the research community.
The dataset consists of legal cases with the raw text from the case as well as metadata on most cases. Certain cases are OCRed from PDF and/or converted from Microsoft Word format. Metadata includes information on the name of the court, jurisdiction, names of judges and date of the hearing.
In addition to metadata described above, 638 cases have annotations at the token level for the outcome of the specific case. Some cases have more than one instance of case outcome. Legal experts have annotated information within the case text for meta-data and juridical information.
In particular, the annotations focused on annotating the case outcome at a general and detailed level. See the main publication's appendix for the annotation guidelines.
Case body text, year and court are available for all court cases with some exceptions: (i) the courts might anonymise certain cases due to their sensitive nature, for example those containing graphic content or cases related to child abuse, (ii) cases not available in raw text were OCRed from PDF versions, (iii) non-UTF8 characters were removed to comply with XML format.
Currently, there are no links between different cases. This also applies to the annotated cases. However, the content of many cases will refer to other cases using legal references.
The annotated data contains a pre-specified test set of size 150, a validation set of size 100 and a training set of 388, all of which were selected uniformly at random.
Some part of the material has been OCRed by us from original PDF files using Tesseract (v4.1.1). Hence, for some of the cases, there might be OCR errors. There might also be errors in the facts and decisions reported by the courts, as well as possible errors arising from transcription and digitalisation of printed material, particularly in older cases. However, in terms of the dataset, we do not technically view all such instances as errors, since our aim is to make court decisions available as they have been made.
Is the dataset self-contained, or does it link to or otherwise rely on external
resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
The corpus is self-contained.
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals nonpublic communications)? If so, please provide a description.
No. The dataset contains court judgments that are already publicly available and free of charge published by the UK Courts, UK Government and various commercial and charitable data providers. Therefore, the dataset only contains data that has already been made available to the public elsewhere.
The dataset contains all types of court judgments. There may be judgments that may be difficult for some users to read (e.g., court decisions describing criminal offences). However, the courts are sensitive in their description of challenging cases, and no information should be offensive, insulting or threatening to the usual reader. We cannot, of course, predict the reaction of every reader as the sensitivity of individuals differs. Also, since there are very old cases, these might contain language or words used that might be considered offensive today.
Subpopulations (e.g., grouped by age or gender) are not identified. However, individuals are iden
tified in certain cases: please see the answer to the next question.