SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the UK, court cases are not anonymised. Parties should, therefore, expect to be named in judgments because courts uphold the principle of open justice, promote the rule of law and ensure public confidence in the legal system (Judiciary UK, 2022). However, the court will anonymise a party if the non-disclosure is necessary to secure the proper administration of justice and to protect the interests of that party (CPR 39.2(4)). This decision includes weighing up the interests and rights of a party against the need for open justice. For example, individual names in asylum or other international protection claims are often anonymised by the court (Judiciary UK, 2022). Also, legislation sometimes prohibits naming certain individuals, such as victims of a sexual offence or trafficking (SOAA s 1, 2), or children subject to family law proceedings (CA s 97(2)).
While considering the ethics of this project, we examined the approaches of other jurisdictions to evaluate any relevant ethical and comparative legal considerations. Drawing from a jurisdiction with the opposite approach to the UK, in 2019, France amended its Justice Reform Act to prohibit the publication of judge analytics (Law 2019-22). The French restriction is based on express rules and a legal culture that varies considerably from the English common law system. France is a civil law system with different rules on how judges engage in cases, the binding nature of decisions and anonymisation (Cornu, 2014). For example, in the French legal system, individuals' names are
Model	Acc	F1	WER	BERTScore	BLEU	ROUGE_L
End-to-end RoBERTa	0.997	0.185	0.536	0.207	0.010	0.185
RoBERTa pipeline	0.739	0.012	0.776	0.030	0.015	0.028
GPT-3.5	-	-	4.281	0.788	0.217	0.300
GPT-4	-	-	3.396	0.840	0.282	0.517
Table 1: Evaluation results for the case outcome extraction task. The RoBERTa-based models are fine-tuned on the CLC annotations, while the GPT-based models are zero-shot—that is, they are evaluated without fine-tuning. We calculate accuracy and F1 scores for the RoBERTa-based models, and word error rate (WER), BERTScore, BLEU and  $\mathrm{ROUGE}_L$  for all baseline models.
usually anonymised. The French Justice Reform Act extends this restriction such that the "identity data of judges and court clerks may not be reused for the purpose or effect of evaluating, analysing, comparing or predicting their real or supposed professional practices" (Tashea, 2019). We have not identified any similar rules in the UK.
At present, the UK GDPR protects against autonomous decision-making that produces a legal or substantive effect on any individual. Researchers have to agree to comply with these and all GDPR rules to access the corpus, as well as any relevant obligations in their local jurisdiction. Taking a cautious approach, access to our corpus requires the researchers' promise not to use it for research that identifies natural persons, legal persons and similar entities.
We have considered the importance of using this corpus in a way that is in the public interest. Ethics approval has helped shape our safeguards and consider relevant risks. The emerging field of AI ethics evidences a growing interest in ensuring fair and just outcomes, in addition to legal compliance, as more personal data is used in algorithmic processes. Based on our legal and ethical evaluation of the project, it does not raise material risks to individuals' rights, freedoms or interests. Our safeguards also act to protect against any risks in further use of the corpus. The purpose, design and method of the corpus avoids such risks and prioritises positive outcomes that may be achieved through improving access to and research of case law. We will evaluate the requirements for access to the dataset on an ongoing basis to potentially further widen access, in particular, if the national and international legal framework becomes more permissive.
As part of the curation of the Cambridge Law Corpus, building upon the corpus creation process and the additional annotations, we have undertaken a thorough exploration of two research tasks using English judgments, namely case outcome extraction and topic model analysis. These tasks offer a comprehensive range of possible use cases for leveraging the CLC in legal analysis.
4.1 Case Outcome Extraction
Court rulings can be complex and exceedingly lengthy, making them less accessible to non-legal professionals. To explore how the corpus can potentially improve such accessibility, we formulated a text classification task to identify text segments within a court decision that explicitly indicate the judges' decision regarding the final case outcome. This is done in two stages: we first classify whether or not each token in the case concerns the outcome, and then, given the result of this classification, we extract what the actual outcome is. A full, detailed description of this task is given in Appendix C.
Quality annotations on case outcomes are essential for this task in order to produce and evaluate output that is useful for legal analysis. We further provide manually annotated labels regarding case outcomes, including general case outcomes and detailed outcomes. The fine-grained annotations enable the extraction of tokens that enclose case outcomes from variable lengths of legal ruling texts. An example of the case outcome annotations is illustrated in Appendix D.
We sampled a total number of 638 cases (173,654 sentences) within the CLC to be manually annotated by legal experts, which are split into train/validation/test sets consisting of 388/100/150 cases, respectively. The number of cases and segments for each data split is summarised in Table 5.
GPT-3.5-turbo	GPT-4	Annotator Reference
Lord Justice: I also agree.	I would dismiss these appeals.	I would dismiss these appeals.
I would answer the issue: The Claimant did comply with the service requirements of the Share Purchase Agreement. I will hear submissions as to the next steps to be taken in relation to this claim, and as to whether it is appropriate to embody the answers to the preliminary issues (disputed and undisputed) in formal declarations.	The Claimant did comply with the service requirements of the Share Purchase Agreement.	The Claimant did comply with the service requirements of the Share Purchase Agreement.
For the reasons I have given on each of the questions charterers are unsuccessful in their appeal. Here too it is unnecessary to address procedural questions and I consider it undesirable to seek to do so. It follows from my analysis above that charterers' appeal on questions 2 and 3 fails on the merits.	For the reasons I have given on each of the questions charterers are unsuccessful in their appeal.	For the reasons I have given on each of the questions charterers are unsuccessful in their appeal.
Table 2: Examples of outputs from GPT-3.5-turbo and GPT-4 and annotated references. Text highlighted in green italics denotes model output content that correctly correspond to annotations.
To extract case outcomes, we explored a number of approaches: (i) end-to-end RoBERTa, which uses the RoBERTa model to directly predict tokens containing case outcomes, (ii) a two-step RoBERTa pipeline, where we first extract the true sentences containing case outcome, then apply token-level RoBERTa classification on only those sentences, (iii) zero-shot GPT-3.5-turbo, which asks OpenAI's GPT-3.5-turbo language model to extract the exact text containing the case outcome from its full text, and (iv) zero-shot GPT-4, similar to (iii) except that we use the larger, more-powerful GPT-4 model. Note that the RoBERTa models are fine-tuned using our annotations, whereas the GPT models are not. Further details on our setup can be found in Appendix F.
Table 1 presents the evaluation results of the four baselines. Both RoBERTa-based models obtain relatively high accuracies, but comparatively low F1 scores, with end-to-end RoBERTa favouring a tradeoff more in the direction of higher accuracy. Both GPT models have a word error rate higher than 3, indicating that they produce too much text. Based on annotator feedback, this can be preferable, since it can be easier to work with output which contains the correct case outcome together with irrelevant sentences, compared to output that does not capture the outcome in enough detail.
Since, in real-world court ruling statements, the case outcome is described with very few words in relation to the total number of words in a case, the case outcome extraction task involves a large class imbalance. Moreover, the variability in how judgments are presented is very high: an ablation study, which uses different train-test splits to understand some of the variability, is given in Appendix C. Given these properties, models can simply learn to classify every sentence as not containing case outcome information, thus resulting in a very high accuracy but a very low F1 score. This may explain why, compared to the GPT models, the RoBERTa-based models exhibit lower scores in metrics such as BERTScore, BLEU, and ROUGE $_L$ . This imbalance is an inherent challenge in the case outcome extraction task.
GPT-based models do not directly perform classification for individual tokens, but instead, given a prompt, produce a text in natural language. We constructed a prompt for case outcome extraction as follows. First, we set the system message so that it asks the model to be a legal research assistant. Then, in the main prompt, we asked the model to carefully read and then quote back exact text from the case text given, with certain extra formatting details. Full details on prompts can be found in Appendix F.
To evaluate the natural-language output produced by the GPT-based models, we use the word error rate (Morris et al., 2004) evaluation metric to measure how close the generated text is to the text segmented by annotators. Examples of the models' outputs and our gold standard annotations are demonstrated in Table 2, showing strong performance, especially for the larger GPT-4 model. Our results therefore suggest that GPT-based models can produce output which mirrors human annotators.
Figure 2: Proportion of words in documents belonging to the listed topics. A word can belong to more than one topic. Left: Aggregated to a one-year period spanning 1950-2020. Centre: Aggregated to a one-year period spanning 2000-2020. Right: Aggregated to a ten-year period spanning 1573-2020.
These examples illustrate that CLC can be a useful resource for benchmarking, fine-tuning and potentially pre-training large language models for legal tasks.
4.2 Topic Model Analysis
The areas of law courts deal with change over time, reflecting social, technological, economic and other societal changes. Understanding the areas of law courts are dealing with can enable one to understand, for instance, where conflicts in society arise and what resources and competencies are needed in the administration of justice. Practically, understanding the areas and numbers of cases going to court supports court managers in creating the necessary infrastructure. At the same time, a better understanding of the types of cases decided by the judges helps to identify areas where legal remedies offered in the court system are not effective. This would be the case, for example, if legal conflicts relevant to a large number of citizens and businesses are not reflected in court cases.
To illustrate a possible use case for how the CLC can be used to follow and analyse changes in topics of law over time, we ran a latent Dirichlet allocation (Blei et al., 2003) topic model using the parallel partially-collapsed Gibbs sampling algorithm (Magnusson et al., 2017; Terenin et al., 2019). We trained for 1200 iterations with hyper-parameters  $\alpha = 0.1$ ,  $\beta = 0.1$ ,  $K = 100$  topics, a rare word threshold of 20 occurrences, and the default stop word list provided by Mallet (McCallum, 2002). The model reached a log-likelihood of  $-3.0832 \times 10^{9}$  after 1200 iterations, at which point we obtained a sample of topic indicators from its output.
To analyse the topics over time, we aggregated the topic indicators produced on a per-year basis, which were then labelled by legal scholars of the Faculty of Law at the University of Cambridge. Figure 2 shows the development of four areas of law distinguishing three time periods: 1573-2020, 1950-2020, and 2000-2020. The top word tokens for these topics can be found in Appendix E.
From these results, we see that the contract topic is generally relevant at all times. This is expected, as a contract is a fundamental legal mechanism which allows natural and legal persons to privately order their cooperation. Interestingly, the relevance of contractual disputes in the courts drops somewhat from around 1990. This may reflect that businesses and consumers increasingly turn to other mechanisms than court litigation, such as alternative dispute resolution and in-house solutions to settle contractual conflicts.
The financial services topic, on the other hand, was hardly present in the courts until the 1990s. Its growing relevance in the courts may be caused by both increased public financial investment and the introduction of legal rules protecting consumers in financial services. The more recent drop in cases in the courts may also be a result of the rise of alternative dispute resolution, in particular, the dispute resolution services offered by the Financial Ombudsman Service.
One can similarly map rises and falls in the immigration and employment topics to historical events in the respective time periods. In total, this short analysis shows the potential of the CLC's use not only for legal AI, but also for legal research using computational methods.
As a final note, the cases available within the CLC are those cases that are available from the courts. In the UK, not all judgments are consistently published or otherwise made available. Courts have some discretion as to what to publish, may decide whether or not to publish in a manner that is not
fully transparent and may change their practice over time. This introduces a selection bias into the corpus: we provide additional details and discussion on this in Appendix A. From this viewpoint, the topic model analysis presented exemplifies a general way to determine what cases are available within the corpus.
We present the Cambridge Law Corpus, a collection of over 250 000 cases. We provide the corpus in an easy-to-work-with format, collected and converted from multiple sources and release the corpus using semantic versioning to simplify continuous improvement of the corpus. Due to the sensitive nature of the material, great care has been and will be taken in the ethical management and distribution of the corpus. We also provide a discussion on how to legally and ethically treat the data. Finally, we give two examples of how to use the corpus for research both in legal AI and legal research, case outcome extraction and topic model analysis, and provide a first benchmark on case outcome extraction.
The work on the corpus is part of the UK Economic and Social Research Council (ESRC) and JST (Japan Science and Technology Agency) funded project on Legal Systems and Artificial Intelligence. The support of the ESRC and JST is gratefully acknowledged. We are grateful to Narine Lalafaryan, Joana Ribeiro de Faria and Lucy Thomas for excellent research assistance. We thank the participants of the Faculty of Law Research Seminar at the University of Cambridge for their helpful comments.
D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent Dirichlet Allocation. Journal of Machine Learning Research, 2003. Cited on page 9.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language Models are Few-shot Learners. In Advances in Neural Information Processing Systems, 2020. Cited on page 1.
N. Byrom. AI Risks Deepening Unequal Access to Legal Information. Financial Times, July 2023. URL: HTTPS://WWW.FT.COM/CONTENT/2ABA82C0-A24B-4B5F-82D9-EED72D2B1011. Cited on page 14.
I. Chalkidis, I. Androutsopoulos, and N. Aletras. Neural Legal Judgment Prediction in English. In Association for Computational Linguistics, 2019. Cited on page 1.
I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and I. Androutsopoulos. LEGAL-BERT: The Muppets Straight out of Law School. In Conference on Empirical Methods in Natural Language Processing, 2020. Cited on page 2.
I. Chalkidis, N. Garneau, C. Goanta, D. M. Katz, and A. Søgaard. LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development. In Association for Computational Linguistics, 2023. Cited on page 2.
J. Choi, K. Hickman, A. Monahan, and D. Schwarcz. ChatGPT Goes to Law School. Journal of Legal Education (Forthcoming), 2023. Cited on page 1.
G. Cornu. *Vocabulaire juridique*. Presses universitaires de France, 2014. Cited on page 6.
R. Cross and J. W. Harris. *Precedent in English Law*. Clarendon Press, 1991. Cited on page 3.
K. Dale. Legal Corpus Linguistics: Gambling to Gaming Language Powers and Probabilities Notes. UNLV Gaming Law Journal, 8(2):233-252, 2017. Cited on page 2.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is All You Need. In Advances in Neural Information Processing Systems, 2017. Cited on page 1.
H. Voormann and U. Gut. Agile Corpus Creation. Corpus Linguistics and Linguistic Theory, 2008. Cited on pages 4, 5.
C. Xiao, H. Zhong, Z. Guo, C. Tu, Z. Liu, M. Sun, Y. Feng, X. Han, Z. Hu, H. Wang, and J. Xu. CAIL2018: A Large-scale Legal Dataset for Judgment Prediction. arXiv Preprint 1807.02478v1, 2018. Cited on page 2.
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. OPT: Open Pre-trained Transformer Language Models. Technical report, Meta AI, 2022. Cited on page 1.
L. Zheng, N. Guha, B. R. Anderson, P. Henderson, and D. E. Ho. When Does Pretraining Help? Assessing Self-supervised Learning for Law and the Casehold Dataset of 53,000+ Legal Holdings. In International Conference on Artificial Intelligence and Law, 2021. Cited on page 2.
H. Zhong, C. Xiao, C. Tu, T. Zhang, Z. Liu, and M. Sun. How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence. In Association for Computational Linguistics, 2020. Cited on page 1.
Legal References
Children Act 1989 (UK).
Civil Procedure Rules (UK).
Data Protection Act 2018 (UK).
Loi 2019-22 du 23 mars 2019 de programmation 2018-2022 et de réforme pour la justice [Law 2019-22 of 23 March 2019 on the Programming 2018-2022 and the reform for justice] 2019, Journal Officiel de la République Française (FR).
Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data and repealing Directive 95/46/EC 2016, OJ L 119/1 (EU).
Sexual Offences Amendment Act 1992 (UK).
Walsingham's Case [1573] EWHC KB J99, [1573] 75 ER 805.
Mean	SE
WER	0.0283	0.0097
CER	0.0417	0.01
Table 3: Word error rate (WER) and character error rate (CER) for pages on which optical character recognition was performed.
A Detailed Information on Corpus Content
Table 4 gives the total number of cases for each court included in the Cambridge Law Corpus (CLC) and the abbreviations used in the following figures.
The corpus does not contain all UK cases. Not all judgments are consistently published or made otherwise available by the courts, even though access to court judgments is fundamental to the principle of open justice (discussed in Section 3) under the UK common law system. The Supreme Court and Privy Council generally publish their decisions comprehensively. However, lower court decisions may not always be captured by various sources and some discretion is held by judges and the judicial press office. Courts have discretion as to what judgments are published. For example, a study found that between 2015 and 2020 only  $55\%$  of 5,408 administrative court judgments identified by a commercial subscription service appeared on the website of the BAILII (Somers-Joce et al., 2022; Byrom, 2023). We did not explicitly select any cases in or out, except to exclude Scotland and Northern Ireland by jurisdiction, and include all cases that we were able to obtain.
We performed an analysis of the optical character recognition (OCR) error rate, by sampling 40 pages randomly from the pages OCRed by Tesseract, and then sampling 3 rows from each page for manual assessment—a two-stage cluster sampling design. We then compared the manually written sentences with the output from Tesseract. We calculate the mean word error rate (WER) and character error rate (CER) or the Levenshtein distance. Results can be found in Table 3.
Court	Abbreviation	Cases
United Kingdom Competition Appeals Tribunal	CAT	478
England and Wales Court of Appeal (Civil Division)	EWCA-Civ	18461
England and Wales Court of Appeal (Criminal Division)	EWCA-Crim	5787
England and Wales County Court (Family)	EWCC-Fam	111
England and Wales Court of Protection	EWCOP	531
England and Wales Care Standards Tribunal	EWCST	745
England and Wales Family Court (High Court Judges)	EWFC-HCJ	386
England and Wales Family Court (other Judges)	EWFC-OJ	873
England and Wales High Court (Administrative Court)	EWHC-Admin	11555
England and Wales High Court (Admiralty Division)	EWHC-Admlty	101
England and Wales High Court (Chancery Division)	EWHC-Ch	6623
England and Wales High Court (Commercial Court)	EWHC-Comm	2926
Court of Common Pleas	EWHC-CP	6
England and Wales High Court (Exchequer Court)	EWHC-Exch	11
England and Wales High Court (Family Division)	EWHC-Fam	2239
Intellectual Property Enterprise Court	EWHC-IPEC	170
England and Wales High Court (King's Bench Division)	EWHC-KB	34
Mercantile Court	EWHC-Mercantile	22
England and Wales High Court (Patents Court)	EWHC-Patents	642
England and Wales High Court (Queen's Bench Division)	EWHC-QB	4645
England and Wales High Court (Technology and Construction Court)	EWHC-TCC	1689
England and Wales Land Registry Adjudicator	EWLandRA	482
England and Wales Lands Tribunal	EWLands	570
England and Wales Leasehold Valuation Tribunal	EWLVVT	18445
England and Wales Patents County Court	EWPCC	135
English and Welsh Courts - Miscellaneous	Misc	451
Special Immigrations Appeals Commission	SIAC	174
United Kingdom Immigration and Asylum (AIT/IAC) Unreported Judgments	UKAITUR	65034
United Kingdom Employment Appeal Tribunal	ukeAT	16002
United Kingdom Employment Tribunal	UKET	36772
United Kingdom Financial Services and Markets Tribunals	UKFSM	69
First-tier Tribunal (General Regulatory Chamber)	UKFTT-GRC	2311
First-tier Tribunal (Health Education and Social Care Chamber)	UKFTT-HESC	437
First-tier Tribunal (Property Chamber)	UKFTT-PC	7860
First-tier Tribunal (Tax)	UKFTT-TC	7684
United Kingdom House of Lords	UKHL	4929
United Kingdom Asylum and Immigration Tribunal	UKIAT	1352
Information Commissioner's Office	UKICO	13659
United Kingdom Investigatory Powers Tribunal	UKIPTrib	51
United Kingdom Information Tribunal incl. National Security Appeals Panel	UKIT	250
The Judicial Committee of the Privy Council	UKPC	10334
United Kingdom Supreme Court	UKSC	743
United Kingdom Special Commissioners of Income Tax	UKSPC	439
UK Social Security and Child Support Commissioners'	UKSSCSC	2467
Upper Tribunal (Administrative Appeals Chamber)	UKUT-AAC	3048
Upper Tribunal (Immigration and Asylum Chamber)	UKUT-IAC	811
United Kingdom Upper Tribunal (Lands Chamber)	UKUT-LC	1009
United Kingdom Upper Tribunal (Tax and Chancery Chamber)	UKUT-TCC	918
United Kingdom VAT & Duties Tribunals	UKVAT	2767
United Kingdom VAT & Duties Tribunals (Customs)	UKVAT-Customs	104
United Kingdom VAT & Duties Tribunals (Excise)	UKVAT-Excise	785
United Kingdom VAT & Duties Tribunals (Insurance Premium Tax)	UKVAT-IPT	7
United Kingdom VAT & Duties Tribunals (Landfill Tax)	UKVAT-Landfill	12
Total	258146